writing unicode to a file

[Jack Lord]

Hi,
I'm writing a program to help me learn the Nigerian language Hausa. It uses lots of non-standard characters (hooked-k etc) and although I can display them in a GUI alright, it all goes wrong when I write and read them to and from a file. Then the non-standard letters just get displayed as ?, so I get lots of words that look like ?ar?ashin, which isn't very helpful. I assumed that since Java uses Unicode internally, it would also use it when writing to a file. Is that wrong, or have I messed up my reader and writer classes?

Writer:

 import java.util.*;
import java.io.*;
 
public class HWrite {
BufferedWriter out;
 
public void write (ArrayList words, String filename) throws IOException {
try {
out = new BufferedWriter(
                           new FileWriter(filename+".hvc", true));
            for (int i=0; i<words.size(); i++) {
out.write("\"" + ((Word)words.get(i)).english + "\"");
out.write('\t');
out.write("\"" + ((Word)words.get(i)).hausa + "\"");
out.write('\t');
out.write((((Word)words.get(i)).male) ? "true" : "false");
out.newLine();
}
out.close();
} catch (FileNotFoundException e)
{}
}
} 

Reader:

 import java.util.*;
import java.io.*;
 
public class HRead {
 

private BufferedReader in;
private StreamTokenizer st;
private String filename;
 
public ArrayList read(String name) {
filename = name + ".hvc";
ArrayList words = new ArrayList();
 
try {
        in = new BufferedReader(new FileReader(filename));
        st = new StreamTokenizer(in);
        int token=st.nextToken();
        while (token != StreamTokenizer.TT_EOF) {
 
            String english = st.sval;
token=st.nextToken();
            String hausa = st.sval;
            token=st.nextToken();
            String gender = st.sval;
            token=st.nextToken();
            Word nextWord = new Word(english, hausa, (gender.equals("true")) ? true : false);
            words.add(nextWord);
        }
        in.close();
    } catch (IOException e) {}
 
    /*For testing
    for (int i=0; i<words.size(); i++) {
Word temp = (Word)words.get(i);
System.out.println(temp.english + "\t" + temp.hausa + "\t" + temp.male + "\n");
}*/
 
    return words;
} 

[Jimmy Die]

Hi,

Thou shalt never assume!

When you write strings to a stream a character data, by default the unicode characters ar converted to the local representation of chars in the host machine, and these are then written to the stream. When you read a string it converts it back.

Check your java documentation regarding the character sets (java.io) you can set in your writing and reading, otherwise the default is making conversions between Unicode and what ever your machine is using.

[Layne Lund]

You probably need to create a FileInputStream and specify the exact character set you wish to use. As Jimmy says, FileReader uses the system's default character set. With a little bit of work, you can still create a BufferedReader from the FileInputStream you use. You'll probably need InputStreamReader somewhere in there.

HTH

Layne

[Jack Lord]

Thanks for pointing me in the right direction. In the end I used

in = new BufferedReader(new InputStreamReader(new FileInputStream(filename), "UTF8"));  

and similar for the writer class. It seems to work, so it'll do fine until someone points out how terrible it is . I was really pleased with the way my program was coming along while I was just testing the input/output stuff on the console - then I hooked it up to the GUI and got very depressed. Now I'm all optimistic again: so thanks for cheering me up!