Win a copy of Succeeding with AI this week in the Artificial Intelligence and Machine Learning forum!

Terrance Samson

Ranch Hand
+ Follow
since Mar 11, 2020
Cows and Likes
Cows
Total received
1
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Terrance Samson

Well, thanks for these clarifications.  I don't have anything more specific to say at the moment though.
2 months ago

Stephan van Hulst wrote:
Simply put, the two are unrelated. For a larger block, the key will be stretched or repeated, and for a smaller block I assume that the key will be rotated (otherwise there is no point to supplying a large key). Both of them should be 'large enough', but their relative sizes to each other doesn't really matter.



Well I guess that's what I meant.  If you're going to have a large key (which you should) then you should generally also have a large block size to take full advantage of it.

Stephan van Hulst wrote:
With 128 bit block sizes, you can safely encrypt 256 exabyte with the same key. For comparison, I believe that last year the total amount of storage capacity in the world was around 20.000 exabyte.



I'd be interested in knowing how that number (256) was calculated.  When you say 20.000, are you using the period as a decimal point or just to separate digits?  I know that the usage varies regionally, and I use a period as a decimal point, but it seems like there wouldn't be a need to put three 0s after a decimal point.
2 months ago

Stephan van Hulst wrote:
Not at all. The entire purpose of encryption is to get rid of patterns in the plaintext. And even IF you could somehow glean anything from the ensuing ciphertext, congratulations: You found out that the plaintext was encoded using an encoding that expands it by 33%.



Well, I guess that's a good point.

Stephan van Hulst wrote:
If anything in Java and .NET related to cryptography is portable, it's BouncyCastle.



If you say so, but like I said, I'm not sure what this BouncyCastle thing is, so I don't know why it would necessarily be exceptionally portable, but alright.

Stephan van Hulst wrote:
I'm not sure what block size has to do with key size, unless you use a separate key for each block.



Yeah, I guess I got a little confused.  But often in my experience, the block and key sizes will be identical.  And in any case, if you have absolutely immense keys, how could you have relatively tiny blocks, unless you're just using different pieces of the key, or otherwise the key is much larger than it needs to be for that particular block without necessarily being more secure.  It seems to me that a large key and large block are better than a small key and small block.

Stephan van Hulst wrote:
You can't change it for another reason: Like the padding and feedback modes in your C# code, the block size is also fixed, so you would have to re-encrypt all your files to make them compatible, which as you've said before isn't an option.



Yeah, as I've said many times, I'm not trying to change the algorithm at all, but just re-implement it in a different language - that's all.
2 months ago

Lexi-Mae Erickson wrote:
What kind of data you have on there to explain the air gap? Kims real birthdate? Or information about his nuke program? Other than some root keys for some CAs and maybe some military stuff I can't think about any reason to dev some weird code on an air-gapped system and fearing even sharing parts of it. For me you seem to stuck deep down the "security through obscurity" hole.
Using AES with a good block mode like CFB/OFB (even better: GCM and AEAD - but that's hard to do correctly in Java as there're some flaws in the decryption verification) and securing the random generated key with RSA with a good passphrase and keeping that one secure is proven to be secure enough so that the data can't be deciphered in a reasonable amount of time - that's basically the base of TLS (aside from that "perfect forward secrecy" stuff wich only generates a random DH key for key exchange so breaking the RSA or DSA key doesn't allow for breaking the session keys - only to mock a forged signature). The level of security should only be based on how secure the key is handled - not on hiding the algorithm (that's basically the flaw why Enigma failed - aside from its capturing along with some codebooks and keeping it secret that this mission was successful).



First of all, not to be rude, but what data I'm using is none of your business, which frankly is the entire point of cryptography, isn't it?  As for your assertion about security through obscurity, even Wikipedia says, "Security experts have rejected this view as far back as 1851, and advise that obscurity should never be the only security mechanism."  But notice the word "only".  It's not the only form of security that I use, nor is it even the main one.

As for all of your other suggestions for various algorithms, I would appreciate them if not for the fact that I've already said repeatedly (because nobody listens to me so I have to repeat myself) that I'm locked into the algorithm that I'm using because I already have a C# program that uses it, and I already have thousands of encrypted files!  My question was never, "How can I find a different/better algorithm?"; it was only, "How can I implement the exact same algorithm in Java?"  So all attempts to change the algorithm are completely moot and irrelevant, and only prove that you're not actually reading the questions that I ask, or you just simply don't care about my requirements and would rather tell me irrelevant things.

RSA is a horrible algorithm, because I'm pretty sure that quantum computing will break it fairly easily.

And who says that I'm "hiding the algorithm"?  I simply said that I had to hide the program that I'm making, which happens to contain the algorithm, among other things.  Also, on the same portable USB drive are many other programs that I don't want to risk any spyware seeing, hence another reason why it's not safe to put that drive in the port until the computer is once again air-gapped.

Lexi-Mae Erickson wrote:
I'm currently at a point I accepted it that me just registering to give some unwanted input was a waste of time. There're some known secure ways I and others pointed out - and then there's yours. There's no reason for fearing getting compromised by sharing code when it's already fundamentally flawed by false assumptions. And although it's harsh and some not like to read such lines - but I still doubt your knowledge: The way you treat ASCII and make it the reason of your failure is just one of many points made me just laugh and think: "Does this guy know Wikipedia and how to use Google?". There're several reasons why US-ASCII is just 7 bits - most notably as it comes from teletype paper strips with 3 + 4 bits devided by one continuous line so the paper can be transported. That's also the reason why DEL is the one control character not within the range of the first 32 but actually 127: Because when you made a mistake on paper you punched out all 7 bits wich meant: "This was a mistake and this very character was deleted.". Calling the inventors of this encoding scheme idiots - ok - that's it for me.
Also: There's nothing like "extended" ASCII - there's 7 bit US-ASCII - and then there're ANSI, Windows-125x, unicode, ISO 8859-X and maybe IBM EBCDIC (don't know if IBM mainframes still uses it or if the also switched to some more common ones). These are known as character sets. Base64, Hex and such ones are encoding schemes.



Well once again, the only reason why a lot of your input was unwanted was because it is incompatible with my requirements, which are to reproduce the algorithm precisely identically by translating it from one language to another.  That's it!  That's ALL I wanted, but for some reason you keep telling me to change the algorithm, which is exactly the opposite of what I was asking for!  And if you think my algorithm is insecure, then why don't you explain how it's insecure?  Or do you even know?  Or is it just because you heard that "layering is bad", without understanding the specific reasons why or circumstances to which that statement applies?  I explained enough about the algorithm in general that you should be able to comment on the specific parts of it and why they're not secure, if that were true, but the fact that you don't, nor does anyone else, but rather you just keep saying "it's bad" with no explanation at all, implies that you don't actually know the reason and you're just going along with what people tend to think/say.  But if you encrypt with substitution blocks, then scatter the bits with a transposition cipher so that they no longer reside within the same blocks anymore, and then encrypt again with a different substitution cipher, there's NO possible way that the ciphers can interfere with each other, no matter how many times people tell you that "layering is bad".

As for the 7-bit ASCII, maybe I was a bit harsh calling them idiots (though I don't know why you'd take that personally, unless you're actually one of the people who designed it), but I'm more interested in computing principles themselves than in the history of their development, and consequently I didn't realize the circumstances of how they were made.  I mean, come on - when do you ever work with anything which is 7-bit?  Normally it would be reasonable to assume that 8 bits were used, and in that circumstance it would be pretty stupid to only use values 0 through 127 when there are twice as many that could easily be used.

Lexi-Mae Erickson wrote:
I still don't know why you refuse Base64 as it's the personified devil - it's defined in it's RFC so that there's no ambiguous meaning - and it can in fact even be used to exchange data between the ASCII based world and IBM EBCDIC. Just one reason why MIME and PEM uses it.



As I said before, because:

- It expands the data a lot, and some of my files are already huge and don't need to be expanded even more.
- It's incompatible with the already existing algorithm in C#.
- I don't see any reason to use it, because if encryption works on any old data then why should I have to change the format first to make it less efficient if it could take all 256 possible values of a byte and encrypt them just fine?
- Because it shoves in an extra 2 bits of 00 along with every 6 bits (that's what it does, right?), it seems like it might be less secure (which should concern you, since you're always accusing my algorithm of being insecure), because those two bits are therefore predictable.

Lexi-Mae Erickson wrote:
As one final note before I sign off: If you want some reliable and language independent way of securely encrypting data just use AES with some other mode than ECB and use proper padding, generate random keys and secure it by random RSA - and the keypair then with a good and strong passphrase. By the way: Java only knows either NoPadding or PKCS5Padding - so if you use padding you have to use one of them. Your C# code uses PKCS7 - this is not compatible with Java unless you implement it yourself - wich you should not do but rather use some crypto lib that supports it - give BouncyCastle a try). Otherweise - if you want to stick to your "I do my own crypto" I guess the question "How to do it without Base64?" is out of scope of the base problems. Strings are in Java 16bit unsigned char values (the only unsigned datatype in Java) but crypto only works on 8bit signed bytes - to convert between them there's the method String.getBytes() wich needs a proper charset in wich the data has to be correctly encoded in. So if you have 7bit US-ASCII just set it and it basically only copies the lowest 7bit from the char array to a new byte array - use this as input to the standardized crypto functions (look at the doc to see what base settings are supported). It could had been that easy - same goes for C#. So I guess all your coding was just a waste of time - and if you lose your code your data are also lost.
Btw: Why a block size of 256bit doesn't work? Cause NSA demanded so! Rijndael does support block and key sized of 128, 192 and 256. AES cut it down to only 128 bit block size and 128 or 256 key size (although 192bit keys supported it's not used - and hence not widely supported). Hate to say and I guess this may get moderated - but the way you expressed yourself you should had already known it - or at least should had been able to google up this yourself.
It's not just I don't want to waste any more time - but I can't help you - as you refused what I offered for some odd reason. Best of luck - but they way I see it: data loss incoming by hardware failure.



Well, there's a bunch of different stuff here, so I'll address it all individually:

Again, RSA is garbage when you take into account quantum computing.

Actually, thanks for the info about the Java padding options.

I don't think I want to use the BouncyCastle except as a last resort, because like I said, I want this thing as lightweight and portable as possible, and I'm not sure how using third party additions will affect that.

What do you mean by, "I guess the question 'How to do it without Base64?' is out of the scope of base problems"?  That's exactly the problem that I was asking about in the first place!

As for the idea that if I lose my code (or the compiled program) then my data is lost, yes I suppose that's true, but wouldn't the same be true if I lost a key?  That's the same problem that everyone faces with encrypted data, which is why I have everything redundantly backed up on several drives.

Why doesn't 256-bit block size work?  Because I prefer blocks closer to 256 bytes, and if you add up the size of all my keys, it's several KB.

And I'll absolutely never trust anything the NSA tells me!  They recommended a version of DES in which they deliberately inserted a back door!

I don't think there's a reason why this would be moderated, because although we're arguing, I'm not angry with you in any way, and I hope you're not angry at me either.  I'm a bit frustrated perhaps, only because most people tend to avoid answering my main question and instead tell me all sorts of information which are not compatible with my specific issue (though I think Stephan may have solved it, regardless).

Again, I don't use Google, but DuckDuckGo, though I'm nitpicking.

The "odd" reason why I refused what you're offering, once again, is because it's incompatible with my requirements.

And what do you mean by "data loss incoming by hardware failure"?  As I said before, I have everything backed up redundantly, and wouldn't the same problem apply to any key being potentially lost?  If this isn't what you mean by that, please clarify.
2 months ago
Wow, thanks!  That code looks great, and I'll test it as soon as I have time, but I'm about done for the day.  I'll see if I can get that and the Java to share files.  If so then I'm about as close to the solution as I can possibly get until I get my real code back so I can compare it.  Thanks again!
2 months ago
Well Stephan, I fixed your code until it would compile and run (it never could figure out what a "WrongKeyException" was, but I just replaced it with a BadPaddingException and then it would compile).  Anyway, I have it printing the plain and cipher text and it seems to be doing what I want and actually keeping things the same size!  The only thing is that I'm not sure how to verify that it's actually using one byte per character, though I guess I could just write it into a file and check that, and the fact that it's using ASCII implies that it is (though that may just be for display purposes, and not necessarily imply anything about data storage).  Anyway though, I think this might work, or at least be as close as I'll be able to get before I can compare it to my old code.  Thanks a bunch!

In any case, if anyone else has anything to say or any answers to my questions or any code to post, I'm always happy to read it, but I don't know whether or not it will be necessary, depending on whether I've overlooked anything.  But thanks again, everyone!
2 months ago
And by the way, here's the code that I've been working on for C#.  A lot of good it does though, since it throws an exception having to do with padding.  Anyway, it also is at the moment using base 64 translation.  Why?  I don't know.  I  think I was trying to see if I could get it compatible with Java that way, and then I could maybe change them both together.  And the original example that I found may have done that too.  I don't remember.  Also, I apologize that it's very messy and does things in weird ways, but I really was just trying to cram code in and try to get it to work any way that I could, and it became this monstrosity.  Anyway, here's the C# code example that I made:

2 months ago

Stephan van Hulst wrote:
The idiot that had to work with machines that used 7 bit bytes. You may think that 'byte' is synonymous to '8 bits', but it's not. 'Byte' means: the number of bits that is natural to the machine operating on them. If you want to differentiate between 'byte' and '8 bits', use the word 'octet'.

I believe ASCII evolved from Baudot code, which was used for teleprinters. I can assure you that the committee that designed ASCII put a lot of thought in it.



Seriously?  I mean I know that technically a byte isn't necessarily 8 bits, but in every single instance that I've ever encountered in the 27 years that I've been writing code, I've never once seen a counterexample!  They were really using 7 bits?  Huh.

Stephan van Hulst wrote:
So I guess what we're helping you with is encrypting individual blocks with AES. The rest doesn't matter, because you're mangling it with your own algorithm that does something completely different from AES, and you'll have to implement that yourself in Java.



Essentially yes, that's what it comes down to.  I was trying to simplify the problem to only supply the information which is necessary and relevant to the exact problem, but people kept asking for more information, most of which is irrelevant and only confuses the issue.

Stephan van Hulst wrote:
Because in this entire discussion, you've led us to believe that you interpret the binary data you get after encryption as a string, NOT the plaintext before encryption.



Well I don't know why I would need to do that, in the sense that I can't actually read an encrypted string, but whatever.  I guess maybe it was just a miscommunication.

Stephan van Hulst wrote:
Because YOU have not given us a clear problem description. Everyone in this topic is confused by your requirements. We simply do not know what you want, and we certainly can't help you implement an algorithm that doesn't follow a public specification without knowing what the original looks like.



I've described what I need and answered every question as well as I know how.  I only have a few requirements, like no padding or data expansion, and that it must allow compatibility between languages and between text formats (ASCII vs. Unicode, and for holding RTF text).  Sure, there are a few parts that I'm not sure how I should set, but like I said, if I get it close enough, I can probably nudge the rest into place after I get my code back on here (or at least I could hopefully get it somewhat close so that it will be easier to fix the rest later than it would be if I Just wait until then to do the whole thing), but I can't do that yet, and I'd rather not wait until the code is back on here before I look into how to fix it, because doing so would then require that I run back and forth between two computers in different rooms as I look up information, and copy code by either remembering what it says one line at a time and typing it in, or else having to save everything onto a portable drive and move stuff to this computer so that I can test it.  Either way it's a pain, and it's much easier while I have the Internet on this computer, but that's only possible before I put the other code back on, so unfortunately, there's really no ideal situation and it's sort of a catch-22, but believe me, nobody is more frustrated by that than I am.

Stephan van Hulst wrote:
Here's what I understand you want to do:

- Read an RTF file. The file is assumed to consist of only US-ASCII.
- Encode the RTF to binary plaintext.
- Encrypt the binary plaintext using some unknown algorithm you designed, but which uses AES for individual blocks. Which cipher and padding modes you use are unknown.
- Do something with the binary ciphertext. Write it to a file? Decode it to a string first? Who knows.



That sounds like exactly what I'm trying to do, except that like I said, I'm not using padding, and I can set the cipher mode once I figure out what it's supposed to be, but for now, I'll just set one for each test program that will both match so that they're compatible, and if I have to change both later then so be it.  Also, as to the fourth and final point on the list, I need to be able to display text in the event that what is being decrypted is text, but otherwise I'm just manipulating files.

Stephan van Hulst wrote:
The only thing I can help you with is encrypting the individual blocks. Encoding the RTF should be trivial, and what you do with the binary data after encryption you can decide for yourself. I'm going to assume that the cipher uses CBC mode and no padding, and that you've hard-coded an IV...



The code that you provided looks very good, and seems like it might be promising!  Thank you very much, and I'll test it now.  But keep in mind that it might be a good idea to print the cipher text as well (or even the length of the text as a number) to compare and make sure that it's not actually expanding.  But I'll implement that when I test it.

EDIT: By the way, I get numerous errors for your code, many of which because you didn't list any imports, so I'll try to figure out which ones to use, and it shouldn't be too hard I think.  But you're also using a "var" type, which my IDE is implying is a Java 10 thing and I'm using Java 8, so I'll see if I can explicitly define the types.  Those are the only problems that I see at the moment, but I'll see if I can get it running.  By the way, I didn't realize it was possible to put multiple exception types in a single catch, separated by "|", but that's very interesting and seems convenient!
2 months ago
That's fine.  And no, you didn't interrupt me, because I had actually finished.  The only reason why I posted so much was because there was a lot to which I wanted to reply, and I guess I had a lot to say.  By the way, I hope that you don't think anything I said was too harsh, because I wasn't trying to be rude to anyone, but I just felt a bit frustrated at times.
2 months ago

Lexi-Mae Erickson wrote:
As I noticed I was following this topic for a bit since it was posted and I noticed some, well, I call it "things that don't add up". I also don't want to be offensive, and I noticed that my post my sound a bit harsh, wich I'm sorry about it. But I noticed that Terrance Samson seem to lack some Java basics as lines like this one: ... Primitive datatypes are one of the first things one learns of a new programming language. So that he seems not that have known that says to me he seem not familiar with Java but rather tries to tinker something together to match some existing C# code.



Yes, I know that, and I can see why you would think that I have no experience with Java.  Actually, I used it very extensively a long time ago, but I haven't touched it once since 2008 until just recently, so I guess I got a bit confused or misremembered the data types.  Frankly, I don't usually mess much with unsigned data types (and I'm not sure that I ever needed to with Java back when I used to use it), but when I need them, I need them, so I guess I was just surprised that when I wanted to use them they weren't there!

Stephan van Hulst wrote:
As Lexi-Mae pointed out, this is a super bad idea. You will not find a single crypto expert who will tell you otherwise. Pick a single, well established encryption algorithm, use it, and be done with it. I agree with you that if you use separate keys for your additional cryptographic operations, you won't compromise the security of AES, but it's absolutely pointless as there will be NO additional provable security, and provable security is all that matters.



Yes, I realize that it's an unpopular idea, but I'm curious what you think about using transposition ciphers between substitution ciphers, so that when you do a substitution cipher on a block, the block consists of a bit from one spot in the file, another bit from perhaps several kb later, and another bit from several kb after that, etc., so that after you encrypt the block, the data within the block didn't even have any coherent pattern, similarity or relevance between the different bits.  What do you think of that idea - is that insecure?

And I don't agree that provable security is all that matters.  I DO think that the way that mentality came about is because people feel that they need to scientifically verify that something is secure before using it, because they don't want to risk using something which is not secure, and I COMPLETELY agree with that sentiment.  However, when talking about additional security, it's a whole different scenario.  If I'm using AES as one of the steps, then as long as all of the other steps are done in such a way as to guarantee not to interfere with AES, then that means that in the worst case scenario, my algorithm is at least as secure as AES, but in the best case scenario, it's a hell of a lot more secure!  How is that not worth doing?  If security of your data is important enough to you then you can't possibly do too much security for it, and it's INCREDIBLY important to me that my data remains secure - as in, I would literally rather be skinned alive, burned and endlessly tortured than to have the data compromised!

I know that AES is considered secure by itself, but you never know what the future will bring in terms of cryptanalysis.  What if 30 or 50 years from now, someone figures out how to crack it?  That's NOT acceptable to me, so I need to take EVERY possible precaution to secure my data!  If you have a 10- or 20- or 30-point system, and any single point fails, no big deal, because they'd literally all have to fail in order for the security to be compromised.  And frankly, even though careless layering is very dangerous (we DO agree about that), careful layering only makes any possible pattern, tendency or other potential exploitability within an algorithm become obscured by other algorithms, so that in the end, all of the data looks so random by any possible method of analysis that there's just absolutely no way anyone could even begin to crack it!

Stephan van Hulst wrote:
As long as you tell your clients that you have less provable security than AES with a proper feedback mode, because you're not using the feedback mode properly (on account of the fixed IV).



What makes you think I have clients?  This is not commercial software.

Stephan van Hulst wrote:
PKCS#7 is a standard for cryptographic messages used in public key infrastructures, but its padding mode is often used in symmetric algorithms as well. I believe that in .NET's AES implementation, it used this padding mode by default. You can only remove the padding if you know your data is always an exact multiple of 16 bytes. The only way you can guarantee this is by introducing your own padding and that would be worse than using the established padding modes. Besides, it's already too late because in your own words, you don't want to decrypt and re-encrypt thousands of files that you have already encrypted.



Yes, but it may not necessarily be too late, because it's possible that's actually what I did with C# (I don't remember), and if so then it's what I have to do again with Java.

Stephan van Hulst wrote:
This may well be true, and that's why he asked us for help with the Java version. I kind of understand the requirement of quickly putting something together in a language that you don't have a lot of experience with, while being able to reference code in a different language. This is not the problem. The problem is the lack of provable security in the original code.



No, provable security has nothing to do with the problem, because the problem, which is my original question, is how to get the Java version compatible with the C# version for both text and other data.

Stephan van Hulst wrote:
Anyway, the real question mark for me is what Terrance does with the data after he's encrypted it, because to me it doesn't make a lot of sense to first decode it in ASCII, only to encode it again using some other unknown default encoding during serialization. But he said he'd have a C# sample ready for us soon, so let's wait for that.



I don't know what you're referring to as the "other unknown default encoding during serialization".

Lexi-Mae Erickson wrote:
I'm looking forward to see what's going on in your C# code. If possible I would like to request that you post either as much as possible (if you hit the 10k character limit may consider share a github repo) or at least the relevant parts. A "stipped down" version may could lead to confusion or speculation if parts essential to this construct are left out. If you don't want to (or may can't for any reason like license or other) please try to at least put together a SSCCE so we can work on a common basis.

In the mean time I may get to re-work my code to a all-BounceCastle one so it will be easier to port it over to C# (in the hope that the lib for C# at least somewhat mirrors the Java one I'm at least somewhat familiar with). But don't quote me on this as I currently have a project myself I'd like to get within the next 24 hours.



Well like I said, I can't access any of the original C# code right now, so you'll have to settle for the example I throw together (if I can even get it to work!).  I may be able to get you part of the original code eventually, but it may take a month or two, because I need to get the other computer fixed, so that I can connect it to the Internet, and disconnect this one, so that it will be safe to put the code back on again, but first I need to wait for the stupid coronavirus to go away so that I can have my computer fixed (there are restrictions on operating most types of businesses at the moment).

And like I said before, I'm not entirely sure whether I'll be able to use third-party stuff like the BounceCastle thing, but maybe I'll look into it if for some reason I can't get the regular AES stuff built into Java to work.
2 months ago
I've also looked through your code now and I may be able to use a lot of pieces of it, so thank you, but I really don't think I'll be able to use it as is, because it's importing this BouncyCastle thing, which I don't even know what it is, but I'm guessing it's nothing included with Java.  Sorry but maybe I should have specified before that ideally I'm trying to keep this thing as portable as possible.  Ideally I want it as one single JAR file and it absolutely MUST be compatible with MacOS and Windows.  As I said, I'm very out of practice with Java and I used to use it just for making Websites, so I don't know whether I can trust the process enough to be sure that if I put a bunch of third-party plugins and stuff into it, the ultimate file that I generate won't necessarily be missing something that it needs, or otherwise be incompatible.  Maybe this is a legitimate concern or maybe it isn't, but in any case, I'm just being careful because I don't really have experience with that specific stuff.

Anyway, I also don't generate the keys while I do the encryption; I have them pre-generated and stored as separate files (the way that I need to use this stuff, there's really no other possible way to accomplish it), and especially if I'm generating keys on the fly, then how would it be able to re-open a saved file later, much less have the other version of the program do so?  Anyway, that's alright though, because I guess I can probably adapt that part to the way that I need it, anyway.
2 months ago

Lexi-Mae Erickson wrote:
A quick Google search with the term "java aes example" over the half of the links on the first page are in fact codes without any form of encoding like Base64 or even basic hex string. So let me ask: What terms you used to search for you only came up with Base64 examples?



Well I don't know what to tell you, because I searched pretty much the same keywords I think (it was a while ago so I don't remember the exact keywords) and all of the results that I found used base 64 translations.  It should be noted that I search with DuckDuckGo rather than Google, but it's a good search engine, so I don't think that would make a difference.

Lexi-Mae Erickson wrote:
As you said you want to use it for file encryption, one secure method is to generate a random AES key, use it for the file encryption, and secure it by a RSA key, pretty much like TLS works, and store the private key protected with a passphrase. There're lot of openSSL examples, but here's a complete example in Java. Please note: This example requires the BounceCastle crypto lib:



Well I really can't add RSA into this, because like I said, I'm kind of stuck with the algorithm that I'm already using, unless I want to decrypt and re-encrypt thousands of files.  Besides, RSA is public key which typically is used for two-way communication online, but I'm just encrypting files for storage, which could be sent to other people, but not live while encrypting, anyway, because some of the files are huge, it would be inconvenient, and it wouldn't be necessary for the purposes that I'm using it.  I suppose RSA could theoretically be used with something like this, anyway, but like I said, I really need to stick with the current algorithm which, despite what a lot of people think, is actually secure, but maybe I'm not properly explaining the reasons, or people aren't properly interpreting the explanations.  In any case, the way that I designed the algorithm is irrelevant, because the only that which is relevant to my whole question is how to get the Java version compatible with the C# version for all data, whether text or otherwise.

Lexi-Mae Erickson wrote:
As others already mentioned: Just cause you chain different algorithms togehter doesn't make the whole process any more secure. If you use AES256 you get AES256 security - no matter if you wrap it around or inside other algorithms. Note that the RSA key although encrypted with AES depends on the chosen passphrase. If you only use a short phrase without special characters it can easily bruteforced, but it's a common practice and compatible with openSSL (at least somewhat - the IV part needs to be handled separately).



Again, as I've explained, it's not an option for me to redesign the whole algorithm, nor to add in a password hash thing that you describe, and the actual security of the algorithm has been established whether you agree with it or not.  Why is it that nobody can ever seem to stay on topic?  All of these long digressions are simply irrelevant.  I'm NOT looking to redesign the algorithm - only to re-implement the already existing algorithm using a different language!  Can we PLEASE just stick to the point, for God's sake!

Lexi-Mae Erickson wrote:
TLDR: From what you posted so far it's hard to understand what you try to do / want to accomplish and what your issues are. You state you want something compatible with some C# code - please post it. You also said you layer different crypto - don't do that! You didn't found what you need using google? I doubt that as crypto stuff is like exploded after Snoweden - so I guess there will be answers to your questions as most likely someone already done it. And as it seem you lack some kind of basic knowledge needed to do this (correctly) you shouldn't try to do your own, as I guess it would result in some unsecure DIY crypto - wich, and this isn't me and you but about any crypto, should not be done at all (Don't roll your own crypto!).



Well I guess almost everything in this paragraph is inaccurate or irrelevant, but I suppose you wouldn't have known this until reading this post, but here goes:

- What I'm trying to accomplish is to encrypt/decrypt text or other data using Java and to have a 100% identical effect to what my C# program already does, so that they can both share files.
- I can't post my original C# code because it's not safe for me to plug that drive into this computer, since at the moment, it's not air-gapped.
- I layer different cryptographic protocols, and I will continue to do so, but I don't do it haphazardly without having any idea what I'm doing - I've very carefully analyzed it all and it's safe.  I know that people say it isn't, and in some instances they're correct, but I get the impression that people who say that it's neversafe to do that either don't really understand the specific implementation or just haven't thought it through logically for specific algorithms, and are instead just going by their "knowledge" that it is a fact that it's unsafe because that's what they've heard/read, rather than reasoning it all for specific examples.  I don't mean that as an insult to anyone, but it's just an observation about how a lot of people tend to think, like how when people may use a mathematical formula because they're told that it's the correct one to use in a given situation, but without having any idea what the specific formula does or how it works or why.  Look at my above explanation about varying keys/block sizes and alternating with widespread transposition if you want to understand why it's safe, because I'm growing tired of repeating myself, and nobody seems to even take into account the things that I've said on the matter - they just keep repeating "it's not safe" without providing any specific reason as it would apply to this specific situation.
- I didn't find what I needed using Google because I didn't use Google - I used DuckDuckGo, and no I didn't find it there either.
2 months ago

Lexi-Mae Erickson wrote:Hello,
I've been following this for quite a while now and everytime I get more and bigger question marks in my head. So, I think some of you already had this in mind, but let me ask it: If your crypto has to be "equal" to what you do in C#, and yes, there most likely will be a way to do it, why not just post your C# code in the first place so we can see what you're actually doing? As you see, you're explanations seem to be ambiguous. Also you may seem to lack some basic knowledge about charsets, encodings and cryptography.



I can't post the original C# code because I had it on this computer when it was air-gapped (disconnected from the Internet or any network) for security, so that there would be no possibility for eavesdroppers (I don't want anyone to see my programs/code or anything that I work on, because I only do them anonymously), and though I don't mind sharing the part of the code which does that small piece of cryptography, I had no way of getting it onto this computer without accessing the whole project backed up on a portable USB drive, which isn't an option anymore, because this computer isn't air-gapped at the moment (I backed up everything, erased the hard drive, reinstalled Windows and connected it to the Internet out of temporary necessity - long story), and until I can take it back offline again, I won't be able to safely and securely access those files (at the moment, I have no other way to access the Internet, because my other computer that I usually use for that has died, and I can't repair or replace it until this stupid flu virus is gone and the computer repair shops are re-opened).

But like I said, I'll try to get a new C# example to you, which will do pretty much what I think the other one was doing, as soon as I can get it to work (if I can get it to work).  And if it's not exactly the same, then hopefully it will be close enough that I'll be able to nudge things into place after I can finally get the old C# source code back on this computer.

I do have a lot of knowledge of cryptography, but not the specific way that this algorithm is implemented in each language and why it seems to be causing slight incompatibilities, and I have a reasonable knowledge of charsets, but I guess I was always using some sort of "extended" ASCII without necessarily realizing it.  I mean, what idiot had the idea to only use half of the numbers to represent characters, and throw away the other half, when there were a ton of extra characters that are relatively common and useful?

Lexi-Mae Erickson wrote:
What really worries me is your statements about the input and output sizes and that you're layering multiple crypto schemes. AES as most other algorithms is used as a block cipher for encrypting files, so you will end up with padding. As AES operates on 128bit blocks and depend on the selected padding scheme you will end up with an output size of either the next 16byte boundary or with an entire additional block. So, when you have an input size of 94.820.763 the output would be at least 94.820.768, or 5.926.298 blocks of 16 bytes unless you either specify NoPadding, wich shouldn't be done, is insecure, and in Java only works with ECB mode - wich is even more insecure to use, or you set AES to stream mode instead of block mode. But this isn't as easy done as said.



Well, as for your concern about layering, I've explained before but I'll summarize again: one thing I do before encrypting with AES is that I apply numerous substitution and transposition ciphers of completely different types and different keys, different block sizes with different offsets of where I consider the blocks to begin and end, and I also alternate between substitution ciphers and transposition ciphers, so that I'll do a substitution cipher with a very large block, then a transposition cipher with an even larger block which scatters the bits all over the place into other parts of the file, so that the next substitution cipher will encode a bunch of bits pulled from spots that had previously been a bunch of different blocks throughout the file.  So I don't see how the different algorithms layered together could possibly interfere with each other in any way that could compromise security.  And as I said, if it were that easy then cryptanalysis would be MUCH easier in the first place.

I don't know what you mean by "...which shouldn't be done, is insecure, and in Java only works".  I'm sorry but it seems like either you didn't finish your sentence or the syntax is not as I interpret it to be.  Do you mean that it "only works in Java", meaning that it doesn't work in other languages?

Anyway, I remember now that the way I implemented AES was that I was encrypting individual blocks one at a time, using a variety of keys, so first I broke the data into blocks the size of the encryption block, and then did it all with those blocks, so there was no need for padding.  As for the final left-over part at the end of the file which may be smaller than a whole block, I don't remember what I did with it.  It's possible that I didn't bother encrypting it, because I figured that it's already encrypted in so many other ways, and transposed so that by that time, it's just a bit here and here from a bunch of unrelated spots in the file, so that you can't really crack what any of the data represents, unless you were to crack many of the AES blocks, anyway, so it should be secure enough that way.  However, I think what I actually did is that I either encrypted that final part which is smaller than a block, by borrowing some bits either just before it, or after it (wrapping around to the beginning of the file) which were already encrypted as part of a different block, and then just re-encrypting them as part of the other block but in different bit positions and with a different key.  Does that make sense?  Anyway, that would explain why I was able to do it without any padding at all.

Lexi-Mae Erickson wrote:
I also still struggle why you so focused about using ASCII, as from what I've read so far you seem not to familiar with it, character sets and encodings or that encryption is a binary operation, even when done on text input. So to be short: whenever you are encrypting something with AES, even if it's plain text, the output will be some arbitrary binary data. So, some method that takes in some ASCII (or String for that matter) and will return some ASCII will not work as the binary data returned by the AES cipher most likely will contain data within the range of 0x00 - 0x20 wich are "not printable" as they are known as "control characters". If you want some printable output you need some additional encoding like Base64 or a Hex-dump wich consist of only printable characters. That's why you found a lot of examples using such schemes.



The reason I'm trying to use ASCII is only because that's what C# uses, and that's how my C# program is already implemented, and I'm trying to make Java use the same files.  As I said before, I'm plenty familiar with ASCII, but I don't usually give it a bunch of thought because usually it takes care of itself automatically, so silly me, it slipped my mind that apparently it's designed in a ridiculous way that only uses half of the allowable values, and is therefore incredibly inefficient, so I guess I had confused it with some extended version.  Again, I'm sorry, but can we please get past that now?  Yes, I realize that encryption is binary and therefore can encrypt any data, whatever it may be, whether it's text or not.  But that's my point.  In some contexts I refer to text as binary data because it is, as is all data on a computer.  I mean, it may not necessarily be an array of bytes or chars (though in C/C++ it is), but even as a String, it's still holding bits, just like everything else.  Maybe I didn't make all of that clear before, but trust me, I do understand it.

However, it still poses a potential problem, because if I'm using ASCII in one language and Unicode in another language, then when translating to arrays of numbers (whether bytes, ints, etc.), if for whatever reason they don't turn out as EXACTLY the same arrays of the same data, then when they're encrypted, even if the encryption/decryption algorithms were exactly identical between both programs, the text still wouldn't be legible, because it wouldn't translate properly, so it's an additional concern

As I said, I can't use base 64 because it expands the data, and also because it's not compatible with what I've already implemented in C#, and all of the many files that I've already encrypted.  In any case, I won't need to use base 64 to make it readable, because I don't care whether the ciphertext is readable characters, because I can't read it anyway; I only care that the plaintext is readable, an that's before it has been encrypted (or after it has been decrypted) anyway, so why should there be a problem with that?

MORE IN THE NEXT POST, BUT IT WAS TOO MANY CHARACTERS
2 months ago
Lexi-Mae, thanks a lot for your input and your code!  I've read what you said and skimmed the code a bit, and though I don't have time to write a full reply right now, I intend to do so tomorrow, when I also work on this stuff some more, and try to get you a C# example.

EDIT: Oh, I didn't realize there were several more posts on the next page as well, but I'll look into all that also...
2 months ago

Stephan van Hulst wrote:What I don't understand is, why do you have all these layers in place if you can just encrypt it once and it will be secure? And why is all of this such a hard requirement when you said you added AES as 'an afterthought'?



Well what do you mean encrypt it once and it will be secure?  I don't agree with that.  AES or any single form of encryption that anyone uses by itself is not secure by my standards.  For one thing, they use keys which are like 256 bits, where as I use keys that are more like 256 bytes, and I use about a dozen of them, for substitution and transposition ciphers (alternating between them, by the way, for anyone who thought they'd interfere with each other, because when I transpose, I scatter the bits so wide that they won't fit within the same blocks during the next substitution cipher).  Sure it takes a couple of minutes to encrypt or decrypt anything, but that's a small price to pay for an incredibly high level of security.

And again, even though AES was just used as a bonus after everything else, it's still necessary because like I said, I need the Java version to be compatible with the C# version, and I've already encrypted several thousand files using the C# version, so I really don't want to have to individually decrypt and re-encrypt each and every one of them with the new version (and then I'd actually have to keep two copies of each if I wanted them to stay compatible with both versions).

Stephan van Hulst wrote:
Who of us said that it adds a percentage? That is true for Base64 encoding, not for AES.



Someone said that the C# version of AES which I'm using must not be true AES because the real version always expands each block or something

Stephan van Hulst wrote:
This may have caused some other posters to continue the discussion using fractions. Anyway, the reason you're getting 64 bytes instead of 48 bytes is because PKCS#7 will use an entire block of padding if your data happens to be an exact multiple of the block size. That means that using such a padding mode, the extra amount of data will always be between 1 and 16 bytes for AES.



Oh, alright.  I had assumed that if I used an exact block size (at least initially for testing) then it wouldn't pad at all, and that would be one less thing for me to worry about while I'm trying to diagnose the first test case, but I guess I was wrong.  I'm not sure what exactly PKCS#7 is (I'm a bit rusty on some of the lingo for this stuff) and I don't remember setting anything to that mode, but I'll look into that.  Is there an alternate mode that I could use which wouldn't put extra padding if none is needed?
2 months ago