In java.io, Readers/Writers are for text and InputStream/OutputStream are for binary data. There are also 2 classes, the InputStreamReader and OutputStreamWriter which are for bridging between byte streams and character data using specified character set encodings.
Part of the problem you can run into with using bytes when
you should be using chars is that it only works for basic ASCII encoding. In java a byte is a signed 8 bit integer type which can represent numbers between -128 and 127 inclusive. A char in java is an unsigned 16 bit type. It can represent many more character encodings other than the basic 128 basic ASCII. If your text file is UTF-8 encoded text lets say...and it contains non-ASCII characters, those aren't going to be represented correctly using a byte[] because a byte is unaware of character encoding. Your 16 bit long UTF-8 characters will come out looking like 2 separate 8 bit values which may or may not correspond to printable ascii characters.
I would say working with binary file types like .jpg, .pdf, etc...you want to be using FileInputStream/FileOutputStream, not FileReader/FileWriter.