Need Help in parsing Japanese SHIFT JIS Characters in Java

[Deepak Lal]

Hi Ranchers,
Im Back with another problem...yes again it is to do with Parsing a file which has "JAPANESE SHIFT JIS CHARACTERS".

Please find the code below.

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;


/**
 * 
 */

/**
 * @author Deepak Lalchandani 
 * @Tested at Home
 * @Date :18th June 2009
 *
 */
public class CheckJapanese {

	/**
	 * @param args
	 * @throws IOException 
	 */
	public static void ifJapeneseCharactersShiftJIS() throws IOException{
		
	try{
		BufferedReader bufferedReader = null;
		//InputStreamReader isr = null;		      
		//reading the initial file into character Array
		//isr = new InputStreamReader(new FileInputStream(
                //      "C:/Documents and Settings/Administrator/Desktop/JapaneseCharactersSHIFTJIS/APH_HT_18June.txt"),"UTF-8");
		//char finalArray[] = new char[2* 1024];
		//int i = isr.read(finalArray);
		//for(int j=0;j< finalArray.length;j++)
        //{
          //  System.out.println("finalArray["+j+"] --->"+finalArray[j]);
              
       // }
		System.out.println("Testing for Japanese Shift JIS characters.");
		bufferedReader = new BufferedReader(new InputStreamReader(new FileInputStream("C:/Documents and Settings/Administrator/Desktop/JapaneseCharactersSHIFTJIS/APH_HT_18June.txt"),"UTF-8"));                
        System.out.println("After Parsing file --> Line is "+bufferedReader.readLine());
		// After Parsing file --> Line is 400000001;	JPY;	400000001;
//	JPY;	APH;	15;	N;	;	JPY;	11112009;	;	ECCCI13RNAT10;	;	;
//	11112009;	N;	;	20;	01;	;	;	;	;	;	;	032;	?????R?[?h;
//	1234567890123456;	;	;	Address1;	Address2;	Address3;
//	;	;	;	;	;	;	;	;	;	;	;	;	;	;	;	;
//	;	;	;	;	;	;	;	CustRef;	??????????;	Fax Number;	;
//	;	;	02;	;	;	PAyment DEtails...................................xx;	1       
        //hence we observe that when the above line(which is having Japanese Shift JIS characters) is parsed
        // Japanese Shift JIS characters are getting converted to Junk Characters like ? as shown below
        // ?????R?[?h  and further in same line we get ??????????
        
        //THE REQUIREMENT IS THAT I SHOULD RETAIN MY JAPANESE SHIFT JIS CHARACTERS 
        //PLEASE HELP RANCHERS ...
        
		}catch(Exception e ){
			System.out.println("Exception caught ::"+e.getCause()+"e.getMessage ::"+e.getMessage());
		}
		
	}
	
	
	public static void main(String[] args) throws IOException {
		CheckJapanese.ifJapeneseCharactersShiftJIS();
	}

}

Why is Java changing the Japanese SHIFT JIS characters to Junk values as shown in the above program...

Please help......

[Christophe Verré]

You're talking of reading Shift-JIS files, so why are you setting the input stream's encoding to UTF-8 ?

[Deepak Lal]

Please let me know as to what should be done
Im stuck up with this...what charset needs to be set for Japanese SHIFT-JIS scenario ???



Please help me
Stuck up....

[Christophe Verré]

The character set is named "Shift_JIS".

[Deepak Lal]

how do i use it in my code. ???

should i first convert it to UTF-8 as shown in the pasted code and then use SHIFT-JIS charset.

The reason being is we are uploading UTF-8 files as well.so it first checks for UTF-8 encoding and there it is converting to junk characters,Then im using

String newValue = new String(fieldValue.getBytes(en),"SHIFT-JIS")
row.set(fieldName,newValue);

hence we see that newValue is also having Junk characters.

How should i handle such that UTF-8 and SHIFT_JIS works as well.

Please advice.

[Deepak Lal]

Hi All

The flow is

im uploading a file with Japanese SHIFT JIS characters with encoding type --> UTF-8

Then we have The below logic

bufferedReader = new BufferedReader(new InputStreamReader(new FileInputStream("C:/Documents and Settings/Administrator/Desktop/JapaneseCharactersSHIFTJIS/APH_HT_18June.txt"),"UTF-8"));

It parses and then we have a encodeRow method where we are doing as below

String newValue = new String(fieldValue.getBytes(en),"ISO88591") //we are taking a charset related to ISO88591    
row.set(fieldName,newValue);  //here newValue is coming as Junk Characters

hence we see that newValue is also having Junk characters

Please help and advice( I need a solution such that i upload a file(Japanese/chinese/korean) in UTF 8 format and it should not show junk characters....i want to achieve this through code....Please advice Ranchers.




Help provided will be highly appreciated.


Deepak

[Deepak Lal]

Please need help on this

[Deepak Lal]

Hi Ranchers,
Need help on this...
Deepak

[Deepak Lal]

Please help me on this....Please suggest some solution....

[Campbell Ritchie]

Don't know, sorry, but this looks too difficult for the beginners' forum. Moving.

[Deepak Lal]

Please need help java experts.....

im uploading a file with Japanese SHIFT JIS characters with encoding type --> UTF-8

Then we have The below logic

bufferedReader = new BufferedReader(new InputStreamReader(new FileInputStream("C:/Documents and Settings/Administrator/Desktop/JapaneseCharactersSHIFTJIS/APH_HT_18June.txt"),"UTF-8"));  

It parses and then we have a encodeRow method where we are doing as below

String newValue = new String(fieldValue.getBytes(en),"ISO8859-1") //we are taking a charset related to ISO88591       
row.set(fieldName,newValue);  //here newValue is coming as Junk Characters  

hence we see that newValue is also having Junk characters

Please help and advice( I need a solution such that i upload a file(Japanese/chinese/korean) in UTF 8 format and it should not show junk characters....i want to achieve this through code....Please advice Ranchers.

[Steve Luke]

Hi Deepak,

You should not be doing conversions between character sets like you are doing - in memory, using the bytes returned from String directly. You need to do a better job of converting and conserving bytes and byte orders / bytes per character...

Anyway, the best way to do it in memory is to use the java nio CharSetEncoder and CharSetDecoder classes. Their API can be found here:
java.nio.charset.CharsetEncoder
java.nio.charset.CharsetDecoder

When you first read the String, you are best off using the java.nio tools, for example accessing the values via a ByteBuffer, then using a CharSetDecoder created from the Charset the source data is in to convert the ByteBuffer to the 16 bit Unicode CharBuffer and Strings that Java uses. Then when it is time to convert it to another format you use a CharSetEncoder for the new destination Charset, and encode the CharBuffer (String) into the ByteBuffer of the new encoding.

An example of what the code might look like is attached. Note that this is 100% un-tested.

package luke.steven.strings;

import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.CharacterCodingException;
import java.nio.charset.Charset;
import java.nio.charset.CharsetDecoder;
import java.nio.charset.CharsetEncoder;

public class ConversionUtility
{
    public static String getString(String charSet, byte[] bytes) throws CharacterCodingException
    {
        ByteBuffer buffer = ByteBuffer.wrap(bytes);
        Charset charset = Charset.forName(charSet);
        CharsetDecoder decoder = charset.newDecoder();
        CharBuffer output = decoder.decode(buffer);
        return output.toString();
    }

    public byte[] convertToEncoding(String charSet, String input) throws CharacterCodingException
    {
        CharBuffer buffer = CharBuffer.wrap(input);
        Charset charset = Charset.forName(charSet);
        CharsetEncoder encoder = charset.newEncoder();
        ByteBuffer output = encoder.encode(buffer);
        return output.array();
    }

    public byte[] convertBetweenEncodings(byte[] originalBytes, String sourceCharSet, String destCharSet)
            throws CharacterCodingException
    {
        String unicodeString = getString(sourceCharSet, originalBytes);
        byte[] output = convertToEncoding(destCharSet, unicodeString);
        return output;
    }
}

[Steve Luke]

This is crossposted (and answered) over on the Sun Forums.

Hi Deepak, when you cross post please let us (on both forums) know, so as not to waste our time duplicating effort on posts that may already be solved already. I know this one went a while without being answered but the appropriate thing would have been to put a link to your post over on the Sun Forums so we could review what they are saying and combine efforts.

Anyways, I notice a few other of your active posts are also cross posted to both forums. Could you please update them all with links?

Thanks,
Steve

<edit: Sorry, I had the wrong thread linked. >

[Deepak Lal]

Hi Steve and Ranchers,
Thanks for the information and solution.I will test and try it...In case of any clarifications.I will revert back.

Sorry for cross posting in another forums

Will not make this mistake again
Apologies for the same.



Deepak

[Rob Spoor]

Deepak Lal wrote:Sorry for cross posting in another forums

Will not make this mistake again
Apologies for the same.

There's no problem in cross posting, but http://faq.javaranch.com/java/BeForthrightWhenCrossPostingToOtherSites

[Deepak Lal]

Hi Steve,
I had a Clarification,What paramters i need to pass to test the ConversionUtility snippet.Could you please provide inputs ?
In case of conversion from UTF-8 to SHIFT_JIS,??

Deepak Lal

[Steve Luke]

Deepak Lal wrote:Hi Steve,
I had a Clarification,What paramters i need to pass to test the ConversionUtility snippet.Could you please provide inputs ?
In case of conversion from UTF-8 to SHIFT_JIS,??

Deepak Lal

Hi Deepak,

Sorry, I cut the JavaDoc comments out for brevity.
Here is the docs for each method:

    /**
     * Takes the bytes of a String (<b>bytes</b>) and stored in
     * the defined character set (<b>charSet</b>) and converts it to a Java
     * String (which is a UTF-16 character set).
     * 
     * @param charSet
     *            The character set used to store the source <b>bytes</b>. Must
     *            be a valid character set String recognizable by the
     *            {@link java.nio.charset.Charset#forName(String)}. See the
     *            Charset class' JavaDocs for more info.
     * @param bytes
     *            The byte array representation of the String. Must not be null.
     * @return A Java String of the converted bytes (in UTF-16).
     * @throws CharacterCodingException
     *             if the <b>bytes</b> can not be converted using the defined
     *             <b>charSet</b>.
     */
    public static String getString(String charSet, byte[] bytes) throws CharacterCodingException

    /**
     * Takes a String in the Java UTF-16 format and converts it into a byte
     * array in the character set defined by the <b>charSet</b> variable.
     * 
     * @param charSet
     *            The character set used to store the destination bytes. Must be
     *            a valid character set String recognizable by the
     *            {@link java.nio.charset.Charset#forName(String)}. See the
     *            Charset class' JavaDocs for more info.
     * @param input
     *            The String to convert to the desired character set.
     * @return The bytes representing the String in encoded in the desired
     *         character set.
     * @throws CharacterCodingException
     *             if the String can not be converted to the desired character
     *             set.
     */
    public byte[] convertToEncoding(String charSet, String input) throws CharacterCodingException


    /**
     * Converts a byte array (<b>originalBytes</b>) from one character set
     * (<b>sourceCharSet</b>) to another character set (<b>destCharSet</b>).
     * 
     * @param originalBytes
     *            The bytes of to convert from the source to destination
     *            character sets.
     * @param sourceCharSet
     *            The character set used to store the source bytes. Must be a
     *            valid character set String recognizable by the
     *            {@link java.nio.charset.Charset#forName(String)}. See the
     *            Charset class' JavaDocs for more info.
     * @param destCharSet
     *            The character set used to store the destination bytes. Must be
     *            a valid character set String recognizable by the
     *            {@link java.nio.charset.Charset#forName(String)}. See the
     *            Charset class' JavaDocs for more info.
     * @return A byte array for the String in the destination character set.
     * @throws CharacterCodingException
     *             If either the source or destination character sets could not
     *             be used for conversion.
     */
    public byte[] convertBetweenEncodings(byte[] originalBytes, String sourceCharSet, String destCharSet)

I also added one more method so you can convert directly from an NIO ByteBuffer:

    /**
     * Takes a ByteBuffer (<b>bytes</b>) which contains the bytes for a String stored in
     * the defined character set (<b>charSet</b>) and converts it to a Java
     * String (which is a UTF-16 character set).
     * 
     * @param charSet
     *            The character set used to store the source <b>bytes</b>. Must
     *            be a valid character set String recognizable by the
     *            {@link java.nio.charset.Charset#forName(String)}. See the
     *            Charset class' JavaDocs for more info.
     * @param bytes
     *            The ByteBuffer representation of the String. Must not be null.
     * @return A Java String of the converted bytes (in UTF-16).
     * @throws CharacterCodingException
     *             if the <b>bytes</b> can not be converted using the defined
     *             <b>charSet</b>.
     */
    public static String getString(String charSet, ByteBuffer bytes) throws CharacterCodingException
    {
        Charset charset = Charset.forName(charSet);
        CharsetDecoder decoder = charset.newDecoder();
        CharBuffer output = decoder.decode(bytes);
        return output.toString();
    }

[Deepak Lal]

Hi Steve,
Thanks a lot for all your replies,but im getting confused which one to invoke first.
I mean if i had to write a main method in java,which is the first method to be invoked first and what parameters i have to supply in this scenario.
I mean conversion from UTF-8 to SHIFT_JIS Scenario ???


Thanks a lot in advance

Deepak Lal

[Steve Luke]

The three methods are meant for three different purposes. So which one you want to call (or call first) depends on what you have.

1) Say you have bytes that you read from the input, and you want to convert them to a String for display. Then call getString(String, byte[]).
2) Say you have a String, and you want to convert it to bytes so you can store them in a specific format. Then call convertToEncoding(String, String).
3) Say you have bytes in one character set, and you need to store those bytes in another character set. Then call convertBetweenEncodings(byte[], String, String)

[Steve Luke]

Also, I want to repeat this:
"Note that this is 100% un-tested."

I don't claim that this WILL work. I claim this is the strategy to use. I still haven't tested it. If you find it does work, please report back. If not, then it may need to be tweaked a bit. Without a lot of time, or the inputs to be able to thoroughly test I doubt I will be able to do so.