Problems Reading UTF-8 File

[Mercurio Savedra]

Hi guys I need clarify this issue

I am working with the SunFtpClient Class in a project that involve
download file contents from Ftp server on Unix Machine. I create some
Files in notepad, write the content and then I save as UTF-8 encoding.

Next I transfer the file content from my machine to the ftp server
in binary mode. Here everything is Ok. But the problem is right here
I execute this piece of code and kaboom the problem appears. Let�s review
the code and next I specify the problem

package Scripts;
import com.banesco.lynx.Encoder;
import com.banesco.lynx.FtpManager;
import com.banesco.lynx.ProcessParametersHelper;
import com.banesco.lynx.common.SunFtpWrapper;
import com.banesco.lynx.config.FileType;
import com.banesco.lynx.exceptions.FtpManagerException;
import java.io.*;
import java.util.ArrayList;
 
public class Script20 {
 
public static void main(String[] args) throws FtpManagerException {
 
//process parmeter helper
ProcessParametersHelper parametersManager=ProcessParametersHelper.newInstance();
 
//Test FTP
SunFtpWrapper ftp = new SunFtpWrapper();
        
        
String serverName = "199.177.4.186";
try {
ftp.openServer(serverName);
 
if (ftp.serverIsOpen()) {
 
System.out.println("Connected to " + serverName);
try {
ftp.login("rr", "rr01");
ftp.binary();
InputStreamReader converter=new InputStreamReader(ftp.get("Message.txt"),"UTF-8");
BufferedReader reader = new BufferedReader(converter);
String line = null;

FileWriter writer=new FileWriter(new File("C:\\LocalMessage.txt"));


while ( (line = reader.readLine()) != null)
{
System.out.println("Content " +line);
writer.write(line);
}

writer.flush();
writer.close();

} catch (Exception ftpe) {
ftpe.printStackTrace();
} finally {
ftp.closeServer();
}
} else {
System.out.println("Unable to connect to" +  serverName);
}
System.out.println("Finished");
} catch (IOException e) {
 
e.printStackTrace();
}
 

}
}

The Message.txt Content is the following
One Two Three Four Five Six

The LocalMessage.txt Content is the following
?One Two Three Four Five Six

SomeBody Could Ask What is the problem?

The problem is that although I use UTF-8 in InputStreamreader as the Convert
Encoding ,the BOM bytes are not filtering and I suppose that the ? character in the content of file LocalMessage.txt is the result of those bytes. Why InputStreamReader converter=new InputStreamReader(ftp.get("Message.txt"),"UTF-8"); is not working well

I appreciate your comments

[Paul Clapham]

You are correct. When Notepad writes a file in UTF-8 encoding, it puts the BOM (byte order mark) at the beginning of the file. This is unnecessary since byte ordering is unambiguous in an 8-bit encoding, but it does it anyway. So the BOM is there.

You would think that a Java Reader that is decoding from UTF-8 would notice that there's a BOM at the beginning of the file, since the UTF-8 specification says it may be there. But no, it doesn't. So it's up to you to read that byte (or character) and ignore it.