Terrance Samson

Greenhorn
+ Follow
since Mar 11, 2020
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Terrance Samson

Stephan van Hulst wrote:
This is the worst form of security. People tend to think that just layering stuff on top of each other makes it safer, but in many instances it actually compromises security. Why are you performing all these transformations? It's not only pointless, it's unsafe.



Yes, I'm aware that in some cases it can be unsafe to layer algorithms, like for example if you use the same or a similar algorithm and the same key, because sometimes doing it twice can just undo the first time or change it in such a way as to make it less secure.  However, if each time you use completely different algorithms, with different keys, and even incompatible block sizes, then it really can't possibly be less secure because they have nothing to do with each other.  In any case, if it did make them less secure then that would make a cryptanalyst's job a whole lot easier, because all they'd have to do is apply various algorithms over top the already encrypted data, and that would weaken it significantly, which means that it wouldn't have been very strong to begin with if it's susceptible to such a ridiculous attack.

Stephan van Hulst wrote:
Only if the encrypted files are a multiple of 16 bytes. The example you quoted had an odd number of bytes, and that's will only be the case for stream ciphers or CTS mode.



The example was just a number that I made up to use as an example; I don't have any files of exactly that size.  In any case, I remember for certain that I was using blocks, not streams.

By the way, if I remember from what I read on the Wikipedia page about AES (at the time I was trying to see the inner workings, and then I realized that it's easier to use an algorithm that's already implemented than to write one manually), it seems to cut each block into smaller blocks, and then reorder them and manipulate their data in other ways, but without resizing anything, other than to perhaps add padding, so I'm not sure why you all seem to think that it's supposed to expand by some percentage or fraction like 1/3.
9 hours ago

Henry Wong wrote:There have been many extended ASCII formats over the years. Many of these are now replaced with Unicode. If this is one of the formats that you are using, there isn't much that Java can do to help. You will definitely need to use raw bytes -- and deal with all format specific stuff yourself.



Well do you happen to know whether RTF stores data as regular ASCII or extended ASCII?  I know it's not Unicode, because whenever it needs a character which is Unicode only, it puts it like "\character-name".

Stephan van Hulst wrote:The output will also be raw bytes. I don't know what you do with the raw bytes after encryption, but if you're converting it to a string, you must also use an encoding. Using an encoding like ASCII is pretty pointless here, because the data is certain to contain non-printable characters. If you're storing the encrypted data to disk, you might as well store the binary data directly.



Well, pointless or not, I MUST use ASCII, because once again, that's what C# is using and the algorithm must be IDENTICAL.

Stephan van Hulst wrote:The fact that your encrypted data is exactly the same size as your plaintext worries me for another reason. It means that you're not prepending the initialization vector (IV) to the ciphertext. That implies that you're either not using an IV, or you're using a hard-coded IV. This is extremely insecure and it's just begging to get hacked.



In C# I'm using the System.Security.Cryptography.Aes class.  I might possibly be using Reijndall but I don't remember.  However, I'm using a block, not a stream.  I don't think I actually did us an IV or if I did, I set it to some constant.  However, understand that I'm also XORing blocks together, reordering them, etc., and this isn't the only algorithm that I'm using, but rather I'm layering a bunch of completely different and unrelated algorithms together, so it would still be very secure in other ways.  In fact, the AES was kind of an afterthought that it just seemed like, "well, why not throw this in there too?"

Now that I think of it, the encrypted files from C# sometimes are a few bytes larger, which may be accounted for by the padding, but they certainly don't expand by 1/3 the size!

Hopefully I can get together an example for C# and get it actually working on Tuesday, so that we have something to which we can compare my Java.
9 hours ago

Henry Wong wrote:
Encrypted data are generally larger than their original data counterpart. This is because encrypted data also hold stuff needed to hide the original data. Additionally, many encryption algorithm purposely increases the size in a way that can't be calculated. This is because the algorithm is protecting the details of the original data -- which includes the size of the original data.

Henry  



Well I know for a fact that the C# program that I'm using does not expand the data at all.  If I start with a 94,820,763 byte unencrypted file then when I encrypt it, I get a 94,820,763 byte encrypted file.  And it's using AES.

Henry Wong wrote:To start, ASCII codes are *not* a single byte... To be exact, it is 7 bits long. And historically, most systems (that store and transfer the data) just use 8 bits (a byte), with the last bit being zero. This is because it is a lot easier to repurpose hardware and code for it, than to redevelop for 7 bits.



Are you absolutely certain that ASCII never uses all 8 bits?  I was pretty sure that it does, but that half of the values are stuff like the Greek alphabet and other characters which aren't used as often.

Paul Clapham wrote:In particular, Base-64 expands the original data by a factor of one-third.



That doesn't make sense, because I specifically disabled the part that converts between base 64, so I'm just using regular byte data, like Zachary suggested with his code samples (which I'm using), but it's still one third larger!

EDIT: By the way, I'm trying to get an example C# program to work, but so far I'm getting an exception that says "Padding is invalid and cannot be removed" when I try to decrypt it, so I'll have to get back to you about that when I get it working, and I'll get the code posted here.
1 day ago
Zachary:

Seriously, unsigned doesn't exist in Java?  Well that's hardly compatible with any data from other languages!

Thanks for the code, and I figured it might involve something like that, except that wouldn't translating to ints cause it to use a larger data type, therefore taking extra space and padding all bytes with like 3 extra bytes of 0 before them?

Though I see from your output that somehow it worked even despite that!  EDIT: I think I see why - you're converting back and forth between chars and ints, but you're encrypting while they're ints, right?  So then it will increase the size MUCH larger (and sometimes I'm using files that could be gigabytes in size, so I really don't want to increase them), and also, since the C# version doesn't do that it wouldn't be compatible, anyway.  Please correct me if I'm wrong though, and it's not actually increasing the size.  Actually though, judging by your encrypted string, it looks even smaller!  How did you manage that?

I also looked at my code again and it seems like I actually am converting from Unicode to ASCII by using UTF-8, which supposedly works up to character 127, and as far as I know I'm not using any characters after that (at least in the test data, though I will occasionally in the real data, so it will be necessary, but it wouldn't necessarily explain why testing isn't working as well as I'd hoped).

I also just now tried changing the getBytes() in the decrypt function to getBytes("UTF-8"), and unfortunately that didn't fix it but it did change from a BadPaddingException to an IllegalBlockSizeException, so maybe that's progress.  By the way, I don't know why it takes a string rather than an enumeration to set the mode to UTF-8 - that seems very silly to me.

I think I'll make a simple experimental C# program now, which will encrypt/decrypt and save files, and then I'll put the code here, so that we'll have something to compare to the Java one that I'm trying to make.

EDIT: Looking at it some more, I noticed that a char seems like it can be treated as a positive number which can exceed 127 (and may even be unsigned), so what do you think of the idea of using a char array instead of a byte or int array?  Though I suspect a char will take two bytes, so that might not quite fix the problem.  Also, I suspect that your negation of negative number or numbers greater than 127 is causing them to reverse their order, because if you take negative numbers and negate them to make them positive, the ones that were higher will become lower and vice-versa.

EDIT: As I was afraid would happen, I printed the length of the original unencrypted string and it's 48 characters, but then when I printed the length of the toDecrypt array (which is of course encrypted) it's 64 bytes, so it's expanding by one third (I don't know why it's that amount).
1 day ago

Henry Wong wrote:
Neither Unicode or ASCII can hold binary data. You need to go through all the code that uses either to hold the encrypted data -- convert them to use something that can hold the data (such as a byte array). You also need to modify all the code that works with the data type to use it correctly too.



You may have slightly misinterpreted me, or maybe I misspoke, but I didn't mean that it needs to be a string and can't be a byte array.  All I meant is that ultimately a string is essentially a byte array, or an array of numbers, each of which is interpreted as a character, either in ASCII or in Unicode, but the difference is that in ASCII each character is one single byte, so if I convert that to a byte array, I'd think that I'd get a different result than if I convert a Unicode string to a byte array, because then each character would become two bytes in the array, right?  So the resulting array would be different even with the same readable text.  So what I need is to make an ASCII string in Java, or otherwise take a Unicode string (which should only have ASCII compatible characters, theoretically, considering what I'm doing) and convert that into either an ASCII string or a byte array equivalent of an ASCII string, but not a Unicode string.  But why should this be so difficult to do in Java?

Henry Wong wrote:
Binary data can't really be held using ASCII. So, there is either some sort of lack of checking on the C# side, or the C# code is using some sort of encoding that is not base64. I would advise against guessing, as you can't replicate it on the Java side, if you are not sure what the C# side is doing.

EDIT: BTW, forgot to mention. If the file is in RTF format, then, your Java side needs to process that too.

Henry



For one thing, I'm absolutely certain that the C# version isn't using base 64, because I never converted to that, and because if I did then it would be taking more bytes, which it isn't.

And I don't think that RTF will make a difference, because it's still just ASCII text.  I know this because if I open a RTF file using Notepad it prints like regular text with a bunch of extra stuff in it for formatting, but that also prints as regular text, so all I'm doing is using that as a text string in C# and I want to do the same thing in Java.

Zachary Griggs wrote:
When you create your cipher, it returns bytes. These are the values of the first few bytes:
byte[0] = 80
byte[1] = -22
byte[2] = -118
byte[3] = 83
So, how are you going to represent -22 as an ASCII character? You can't. It's out of the spec.
I promise that the C# application isn't taking these bytes and converting them directly to ASCII either. Since it's not possible.
There isn't really a simple way you can make this work in ASCII without an encoding, which is why all examples online posted of this snippet use an encoding to show the value to you.

Can you post the C# source for the section where it encrypts?



Well first of all, that tells me that it's using -22 as a signed byte, but I really need unsigned bytes ranging from 0 to 255 because that's what ASCII is, after all.  Unicode has an even greater range, but neither use negative numbers, so I don't know why it's doing that, unless for some reason it just wants to interpret it as a signed byte even though given the context of text, only unsigned bytes are reasonable.  Can't I make an array of unsigned bytes and use that instead?

As for posting the C# code, that's really going to be tricky, because for security purposes, I really can't put it online.  The only reason why I can put the Java version online is because at this stage it's just a simple experimental thing, whereas the C# one has a ton of other stuff, and I can't just pull out a little piece of the code, because at the moment it's inaccessible to me (to make an incredibly long story short, it was on an air-gapped computer which needed temporary Internet access, so it had to be thoroughly reformatted first, and that code can't be put back onto it until it no longer needs Internet access and can once again become air-gapped, but at the moment it still need the Internet, and ideally it would be very convenient if I could figure this out before I re-air-gap the computer).
2 days ago

Zachary Griggs wrote:Can you save it as a binary file which then feeds in to your other application?
I recommend unit tests to test functionality, which would not necessarily require strings.



Well, I suspect one of my problems is going to also be that Java uses Unicode and C# uses ASCII, so it might also be necessary for me to have functions in Java which can translate back and forth between ASCII and Unicode, but I'm not sure what functions those would be.
2 days ago
Sorry for posting so much, but people keep posting while I do, and then I notice something new to which I want to reply:

Zachary Griggs wrote:The code needs to either directly use bytes, or to base64 encode it (or some similar encoding that can represent all the bytes AES uses)



I can't encode it in base 64 or anything like that!  I keep telling you all that it needs to be in a format that is compatible with the C# version of the program, and that one uses:

- Regular strings/bytes/whatever with NO base 64 conversion
- ASCII strings 1-byte-per-character for storing RTF, which sometimes uses non-ASCII-compatible Unicode characters specified as RTF commands within the string (like \rquote for a closing quotation mark, for example), but I don't think that should affect this because it could just be treated as a regular ASCII string

Basically, the C# program takes a text string in ASCII format, does NOT convert it to base 64 or anything else, and just straight encrypts it and saves it in a file.  It can also read from the file and decrypt the text.  It doesn't even have to be text, because it also can encrypt/decrypt any data, but whenever it is text, it's ASCII default byte-sized characters, with NO 64 bit conversion.

What I need the Java program to do is exactly the same thing as that, and whatever is encrypted with either program must be decryptable with the other program, so that it turns out identical to the original, and that if it happens to be text then it must still be readable in both programs.

I'm sorry to sound harsh, but it's just frustrating, and it seems like I have to keep saying that I can't use base 64, and people keep suggesting, "Well why don't you just use base 64 then?"

Also, I do appreciate the suggestions but it would be really nice to get a bit of source code, even if it's just to modify a couple of my lines to get it to do what I want, because if you just explain, "do it like this..." but don't provide any code then I may or may not be able to get it to do what you're saying.  I mean the whole point is that if I could get it to do anything just from a description of what I want then I wouldn't even be having this problem in the first place, because I already know what I want it to do, but I can't seem to figure out how to get it to do it.  Keep in mind that I'm very rusty with Java and haven't used it in about 12 years until I just made this program, so bear with me please.
2 days ago

Zachary Griggs wrote:

Stephan van Hulst wrote:

Zachary Griggs wrote:How are you testing your code without base64? Are you copy/pasting the output from the encrypt into the decrypt? If so, it may be charset issues causing the problem, not an actual issue with the algorithm.


You can easily pass around byte arrays without transporting them as Base64. Again, Base64 is unrelated to encryption/decryption.


Yes - but as shown, he is attempting to use Strings, not Bytes. This is why Base64 is used. It's not directly related to encryption/decryption but it is a character encoding that is used to make the result of encrypting/decrypting printable. Which is the problem that the poster is running in to.

I would recommend either converting to Base64 or storing the result as bytes rather than a String.



Well like I said, I don't want to use base 64, because that's the whole thing that I'm trying to avoid.  As for storing in bytes, I guess that's alright, except that ultimately I'll need to be able to read it as a string, because:

1: I need to test to make sure that the data is alright, so if I can read a string that makes it a lot easier.
2: Ultimately this will be used for encrypting/decrypting text, among other things, but remember, the files that it reads/writes must be compatible with a C# program which already exists, does not use base 64, and uses ASCII strings (as far as I can tell, but ultimately it stores RTF, but I'm pretty sure it's using ASCII to store the RTF, but with formatting that can make Unicode characters as necessary).
2 days ago

Henry Wong wrote:Without the encoding, the result is binary data. It can't be held in a string. Doing so, will likely mess up the binary data, and hence, can no longer be decrypt correctly.



Alright, so then how would I fix those lines of code?  I'm not sure of a better alternative way which would achieve what I want.
2 days ago
Alright, I said I'd supply the code, and here it is.  I pretty much got this from examples on the Internet:





That works fine, but with base 64.  When I try without it I change the return line in encrypt to:

return Base64.getEncoder().encodeToString(cipher.doFinal(strToEncrypt.getBytes("UTF-8")));

And I change the return line in decrypt to:

return new String(cipher.doFinal(Base64.getDecoder().decode(strToDecrypt)));

But doing so causes it to print this to the console:

PêŠSò)ú/{]HK%Ç76bYp片ɨq=

© ï†û8¼­;²ÃPrc‘œLóEYɾ6?³ 7𪾯E
Error while decrypting: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
null

So it seems to encrypt, though possibly not correctly, then there's an error when decrypting, which seems to have to do with padding.  Honestly I didn't remember what it was doing until I saw it just now, but I don't see why there would be an error, since the only difference is whether or not I use base 64.  In any case, I can try using the ToString function instead of the String constructor, but I don't think it makes a difference.
2 days ago

Zachary Griggs wrote:Why do you not want to use Base64?
It's an encoding which can be very easily undone.



As I said, it increases the size of the data, and it's also not compatible with the already existing C# version of the application which must be compatible with the Java one that I'm making, so that files written using either application can be read using the other one.

As for example code, I might be able to get you some when I have time but all I really need is something like public String encrypt(String plainText, String key) and public String decrypt(String cipherText, String Key) or byte arrays in place of strings, or whatever.  And I don't really care about the padding mode and all that, because I can adjust those settings as necessary.

Really, the only thing that's not working is that when I use base 64 it encrypts and decrypts just fine, but when I remove the base 64 conversion, it doesn't.
3 days ago
Oh, sorry, I didn't realize there was a section for security or I would have put it there.

I know all about different algorithms for different purposes and all of that, but it's not really the issue here.

The problem is that I just don't seem to be able to encrypt without first converting to base 64, which I do not want to do, but all examples seem to do it, so I was wondering if anyone could paste or point me to some source code that does it without the conversion.

I think something might possibly be going wrong when I convert between a string and byte array but I'm not sure.  I just know that the result of the whole thing is wrong, and when I use base 64 it seems to be able to encrypt and then decrypt back to the original data, but when I don't use base 64 it can't.  Also it seems like the string might have been somehow cut short when converting to a byte array, but there weren't any null characters or anything that might trick it into thinking that it's the end of the string (though I can't promise that would always be the case - I was just using some test data).
3 days ago
Does anyone have any source code example for how to use AES in JAVA, but without transforming strings into base 64?  For some reason all of the examples that I've found do that, and when I try not to I get errors, like sometimes I think it seems to cut strings short or mess them up when converting into byte arrays and so on.

Anyway, I really need to avoid using base 64, because for one thing it's less efficient for memory, but more importantly, I need to make my files compatible with the equivalent algorithm already implemented in C#, and it does not use base 64, but rather it just takes plain strings (they're ASCII, and I guess Java uses Unicode, but regardless, I really just need a way to pull the string/array in as is, no matter what the data contains, and encrypt it straight from that).

Help would be much appreciated, thank you!
4 days ago

Tim Moores wrote:While Wordpad can open RTFs and DOCs in general, it's a really limited app. I wouldn't take its behavior to signify anything about how something is working correctly or not. In general, RTF is a good format for exchanging styled/formatted text. Certainly much better than DOC or DOCX if you don’t know which word processors might be used to handle it, anyway.



Well I definitely need to use RTF anyway, but I was just comparing it to DOC as an experiment.

Tim Moores wrote:All standard word processors can do this, and all of them let you turn it off. In LibreOffice: Tools -> AutoCorrect -> AutoCorrect Options -> Localized Options. That's another area where Wordpad shows its limitations.



Hmm, I'll look into that, but in this case, my point is that Wordpad is actually working better than LibreOffice for what I'm trying to do.

Tim Moores wrote:Indeed. It handles only a subset of RTF, and is buggy at that, too. See https://bugs.openjdk.java.net/browse/JDK-4723383 to get an idea how long people have been complaining about that, and how long nothing has been done about it.



As a side note, I also noticed that when using JavaFX, it seems to handle HTML formatted text just fine, but doesn't even have anything compatible with RTF at all, as far as I can tell.



Anyway, thanks a lot; you're much better than the jerks over at Stack Overflow.  I'll tell you if I have any more problems but I think that solves this one (fingers crossed).
1 week ago
Well I figured out something weird.  But it's very good information to know for anyone who may wish to load/save/convert RTF files between various programs.  First I made my function that reads lines append a "\r\n" after each line to compensate for removing them automatically, but before I looked into the link that you gave me, I thought I'd try something simple.

First of all, I tried making two identical test RTF documents - one using Wordpad, and the other using LibreOffice (sort of a substitute/competitor to MS Office, and it can save in RTF).

I found that the LibreOffice version puts a whole bunch of extra and seemingly superfluous junk in it that the Wordpad version doesn't.  It also seems that the documents saved from LibreOffice are the only ones that have that problem with apostrophes, or at least Wordpad doesn't seem to (so when I tried it before, I must have made the document using LibreOffice I guess).

Anyway, I was still curious why the LibreOffice version wouldn't be compatible, so when I examine both documents in Notepad, I find that the one generated using Wordpad just uses a regular apostrophe character, so when I typed "Here's some text" it appeared exactly as I typed it.  But the LibreOffice version does this to it: "Here\u8217\'92s some text".  It seems that "\u8217" means to print Unicode character 8217, so I looked it up, and evidently it seems that code corresponds to the right single quote, which I guess is slightly different than the non-directional single quote/apostrophe, hence the supposed incompatibility with ASCII.  So I guess LibreOffice makes this automatic adjustment and Wordpad doesn't.

Well alright, that's all well and good, but what about the "\'92" part?  The "s" right after it seems to be the actual "s" in the text which is supposed to print.  However, it doesn't appear unless I insert a space before it so that it says "\'92 s", but then the extra space appears in LibreOffice between the apostrophe and the "s".  So basically it comes down to a difference between where my program implicitly expects there to be a space for it to understand that the previous formatting command has ended and the next letter is to actually be printed, versus where LibreOffice does not expect there to be a space in the same spot!  This is weirdly inconsistent, as though LibreOffice had not held to the standard properly, and not cared whether its documents saved in RTF would be compatible with any other programs!

But then oddly, when I tried copying the text from LibreOffice and pasting it into Wordpad it looked fine.  I saved it and opened it in Notepad and it said "Here\rquote s some text", so it's automatically replacing the "\u8217\'92" with "\rquote " (notice the space at the end which causes Wordpad to realize that the "s" is separate from that command and thus should actually be printed).

Well anyway, I guess I'll be using Wordpad from now on, at least for stuff of this nature.

However, I did try loading in some larger and more complicated documents (admittedly ones that had been created using Word and then translated from DOC to RTF using LibreOffice, because I don't think Wordpad can handle DOC, but I'll test that as well), and when I did so I noticed a lot of missing words, especially at the beginning of paragraphs and items on lists, and possibly the ends of paragraphs.  It was a few days ago and I don't remember, but I intend to test that stuff again now.

I also intend to test whether things that I write programmatically in one program and save as RTF files can be read using the other program without messing them up, which probably comes down to a question of whether the first program is saving them the "normal" way like Wordpad does, or whether it saves them the weird way like LibreOffice does.  But if that's the case then I'm not sure what I'll do, other than perhaps do a search and replace on a bunch of potentially messed up pieces of text, which may be a lot more than just apostrophes!

Well, I'll get to that now...



EDIT: Well now I've tested that stuff and it looks like Wordpad is not compatible with DOC files, but yes my programs are, and it seems that LibreOffice is the odd one that doesn't really match anything else.  Well, that's good to know!
1 week ago