BufferedWriter with prime quotation marks

[Tai Lo]

I am trying to take multiple files and merge them all in to one new file. Everything to this point is great except for one item. *note i am receiving these files so i have no control over changing anything*

When writing the file get weird symbols replacing the the prime quotation marks that were in the original file. I can't figure out what I am doing wrong and have tried setting the encoding in bufferedWriter but that does not seem to make a difference. Any help would be great. Thanks!

Here is the original file:

3_PatFemale("X")|3_PatSex (”M” or “F”)|

Here is what it looks like in the newly created file:

|3_PatFemale("X")|3_PatSex (�M� or �F�)|

BufferedWriter out =  new BufferedWriter(new OutputStreamWriter(new FileOutputStream(exportFile),"UTF-8"));

for (File f : files) {
        FileInputStream fis;
        try {
            fis = new FileInputStream(f);
            BufferedReader in = new BufferedReader(new InputStreamReader(fis));

            String aLine;
            while ((aLine = in.readLine()) != null) {
                out.write(aLine);
                out.newLine();
            }

            in.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

[Campbell Ritchie]

How do you know the files are encoded in UTF-8? How do you know the quote marks are being read correctly by your reader? Try printing them to the console; if that does not work try the old‑fashioned technique of displaying them on an option pane, so you can verify they are being read correctly.

[Tai Lo]

Both console and option pane print out the following:

[Campbell Ritchie]

Are you reading in UTF-8? If so, it is likely that your input files are not UTF-8 and you are therefore not reading them correctly. Is there amy way to find out the encoding of the files you are reading?
If not, you might try printing out each char as an int (preferably in hexadecimal) and comparing your output with the Unicode values. You might even use the read() method because that gives you the individual characters. The values of \u201c and \u201d are \u201c=“ \u201d=” Here is a table which tells you what Unicode characters are in UTF-8. If you go to the U+2000 block, you find quotes like this:-U+2018 ‘ e2 80 98 LEFT SINGLE QUOTATION MARK U+2019 ’ e2 80 99 RIGHT SINGLE QUOTATION MARK U+201A ‚ e2 80 9a SINGLE LOW-9 QUOTATION MARK U+201B ‛ e2 80 9b SINGLE HIGH-REVERSED-9 QUOTATION MARK U+201C “ e2 80 9c LEFT DOUBLE QUOTATION MARK U+201D ” e2 80 9d RIGHT DOUBLE QUOTATION MARKIf your quotes don't come out as e2809b/e2809c then you aren't using UTF-8.

[Tai Lo]

Thanks for the hint. I completely neglected checking the file's encoding. Turns out it was ANSI (Cp1252). Using this has seemed to solve my issue. Thanks!

[Campbell Ritchie]

Success