Forum:

Java in General

UTF-16 encoding decoded as UTF-8

Ranch Hand

Posts: 89

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

Hi I know this sounds strange so I'm hoping someone with a bit more knowledge on encoding can answer this.
I'm trying to pass a String as UTC2 to a third party (They are rendering this as an SMS). I am using UTF-16 as, as far as I know this is a super-set of UTC2.
Unfortunately I have been running into problems, after a lot of trial and error I am getting the String Виталий А Терниевский back as Виталий<corrupted char><corrupted char>Терниевский.

What I don't understand is why.
I am getting this result by encoding the String as UTF-16BE -> %04%12%04%38%04%42%04%30%04%3B%04%38%04%39+%04%10+%04%22%04%35%04%40%04%3D%04%38%04%35%04%32%04%41%04%3A%04%38%04%39

Then decoding this as UTF-8 -> 8B0;89 "5@=852A:89

I thought that this would just have corrupted the data but it is being rendered as something. Does anyone have an idea on what the second String represents?
Cheers,
Barry

Gamini Sirisena

Ranch Hand

Posts: 378

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

What is UTC2 ?
[ October 22, 2008: Message edited by: Gamini Sirisena ]

Barry Higgins

Ranch Hand

Posts: 89

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

Apologies that was a typo I meant UCS-2.
Thanks,
Barry

Gamini Sirisena

Ranch Hand

Posts: 378

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

Can you post some relevant pieces of code? That might help the ranchers here to get an idea..

Barry Higgins

Ranch Hand

Posts: 89

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

In short this will print the String that I am passing

8B0;89 "5@=852A:89=

This seems to "nearly" render correctly.
I have no idea how this can create anything but a garbled mess, but somehow it produces nearly the correct text.

Thanks,
Barry

James Sabre

Ranch Hand

Posts: 781

I like...

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

Originally posted by Barry Higgins:
In short this will print the String that I am passing

8B0;89 "5@=852A:89=

This seems to "nearly" render correctly.
I have no idea how this can create anything but a garbled mess, but somehow it produces nearly the correct text.

import java.net.URLDecoder; import java.net.URLEncoder; public class Test { private static final String russian = "Виталий А Терниевский"; public static void main(final String args[]) throws Exception { String blah = URLEncoder.encode(russian, "UTF-16BE"); System.out.println(URLDecoder.decode(blah, "UTF-8")); } }

Thanks,
Barry

What am I missing? You URL encode using UTF-16BE and then you treat that URL encoded output as if it is URL encoded using UTF-8 to decode it. This does not make sense. If you encode with X then you need to decode with X.

It is not obvious to me what you are trying to do.

Retired horse trader.
Note: double-underline links may be advertisements automatically added by this site and are probably not endorsed by me.

Gamini Sirisena

Ranch Hand

Posts: 378

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

I believe James is right..

Check your program modified below. Most probably your console does not support printing unicode characters. So this program generates an html and hopefully your browser supports unicode display. Comes out nicely for me.

I think following is what you need to do. I think the following explanation is correct.

The String representation of java is always UTF-16.
So when you do..
URLEncoder.encode(russian, "UTF-8")
the String russian is converted to UTF-8 and then URL encoded.
The third party then decodes the URL encoded string knowing and
specifying that it is a UTF-8 string.

Barry Higgins

Ranch Hand

Posts: 89

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

Thank you both for your response. I have probably not been very clear in what I was saying.
I understand that encoding in UTF-16 and then decoding in UTF-8 does not make sense.
My question is more to do with what is the result if one were to do this?
The Strings that are resulting from the code snippet I put up earlier are very close to what the third party provider requires (I know this based on the SMS messages that I have been receiving). However I do not know why anyone would want text in this format i.e. does it represent something formatted differently and if there is a more structured and understood way of attaining this?
I hope this makes a bit more sense.
Cheers,
Barry

Ilja Preuss

author

Posts: 14112

posted 16 years ago

Number of slices to send:

Optional 'thank-you' note:

Send

Originally posted by Barry Higgins:

I understand that encoding in UTF-16 and then decoding in UTF-8 does not make sense.
My question is more to do with what is the result if one were to do this?

Garbage, as far as I can tell...

The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus

With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.