Difference between Streams and Writers

[Nicholas Jordan]

Jim, Ernest & Stan,

Like when supermodels think really, really hard ... I am going to try to KISS this. <!-- Keep It Simple, Stupid. A common engineering slang, for intermediate readers who may not be familiar with the acronym. -->

I stripped out about 300+ lines from class Alice this morning, during one cup of coffee no less, to make a serious attempt at doing what you guys have suggested. (compiler gave clean build, the first check to do on that)

Try not to have nightmares on this next one :roll: I took a large Java class from a degreed worker in the field and just changed the type from char to byte to get some other class to work.

Obviously, I have to fix this.

So I dove into that, and ran against byte/char mcbs/latin-1 writer/stream .... flushes on close, the list of issues resists attempts by me to fix the design challenge by reading the sources for java from src.jar

You three have provided meaningful help, so let me enlist further aid.

At this point in prototyping, I need two types of files:

1. (first type of file) Reads text files with translation left to the compiler authors. Should be able to accept re-direction of stdout. Should take care of implementation details such as flush() at close and have decent size buffers, behaving robustly under coding approaches the intermediate coder can understand.

2. (second type of file) Does not try to do any thinking for me .... no char/byte translation,.... I may have to set up my own buffers - might even have to write my own println(), all I want it to do is do what I ask, and not assume I need trainer wheels for ANYTHING. If I forget to do something because I have not read about it, that is my problem. I have highly trained, highly compartmentalized safety barriers, even to the point of having a team member whose sole job it is to go play golf with the owner and http://en.wikipedia.org/wiki/Baccalaureate cs's to check code, neither with the ability to interfere with each other's bailiwick

I can fix the other stuff you workers have pointed out in my work, but this one can consume unproductive hours a hundred at a time, something my prototyping does not need right now.

I know about going back and looking for historical design decisions in the API - I assume the risk.

[ December 16, 2006: Message edited by: Nicholas Jordan ]

[Jim Yingst]

From the title of your post it sounds like you may already have most of your answer already: use InputStreams and Readers.

For the first type of file you probably want a Reader, perhaps something like:

Reader r = new BufferedReader(new InputStreamReader(new FileInputStream(filename), encoding);

Or if another application has redirected their standard output to your Java program, it appears as standard input:

Reader r = new BufferedReader(new InputStreamReader(System.in, encoding);

Regardless of those details, most of your code should probably just assume you're using a Reader, so you can change the details later if you need to. However you may find it more useful to assume a BufferedReader instead, since that allows you to use a readLine() method which is very useful, and unavailable in a plain Reader.

Alternately if you're using JDK 5+ you can just use a Scanner instead, which takes care of most of these details for you. Just look at the options you have among different constructors.

Since you're reading rather than writing, there's no flush() method. When you're done you should close() the reader yourself, but that's not a big deal. It's usually a good idea to use a try/finally to ensure it's closed even if an error occurred.

For the second type of file, you probably want a plain InputStream. Probably a FileInputStream, maybe a FileInputStream wrapped in a BufferedInputStream. But again, most of your code should just assume it's an InputStream - the underlying details aren't usually important, and should generally be hidden from most of the program. And again, call close() when you're done.

Does that help?

[Jesper de Jong]

This looks like a follow-up to some discussion that I haven't read, but what you describe under point 1 is exactly what Readers and Writers are in Java, and what you describe under point 2 is exactly what InputStreams and OutputStreams are.

InputStream and OutputStream are low-level classes for reading and writing data from and to files (and other I/O devices).

Reader and Writer are higher-level classes, built on top of streams, for reading and writing text files and do all the binary/character conversions that are necessary. Writer even automatically calls flush() when you close() it (see the API documentation).
[ December 11, 2006: Message edited by: Jesper Young ]

[Nicholas Jordan]

Originally posted by Jim Yingst:
Does that help?

Definintely.

Streams vs Writers - which is the issue I wanted addressed - has been given treatment.

You guys have been patient with me. This is the next issue to be fixed after the other things pointed out to me in the Saloon, so I will give it deep attention after resting, which has proven essential to effective work. I am exhausted right now.

Short version is Writers are of type char. Streams are of type byte. The JVM may do UNICODE conversion on char, but will leave byte as eight bits per byte everywhere. That is what I will be looking at in your reply and in the src.jar.

[para deleted, not relevant]

I will use fail-fast throughout the app, classes with finally's may confuse the gc, according to memory management docs on leak detection I read Sunday on the Sun website. Generally, I check before entry with a conditional - handling unexpected with an else.

The OS should discontinue file activity if an application exits. The app is heavily threaded. If at any time, there is an unexpected condition ( an exception ) there are too many threads running or it just becomes overly complex to try to do an orderly shutdown. The app cannot hang for any reason, there are several issues involved. One of these, what I think of as a single-threaded 16-bit .dll dirtying up the scene appears to be reflected in System.java - there is no non-blocking read() for System.in. In one of those - are you ready Jim ? - " ... except if the exception is an exceptional exception ... " System.java/FileInputStream.java says throw IOException if attempt to read after end of file, but does not dis-ambiguate this condition from <code> no more data available </code> - what's the difference. ( duh... )  <em>NONE</em>

See Stephen C. Dewhurst for a discussion of exception semantics. In paticular, I see no reason for a programmer not to drop into operations with an

if(obj.valid())

Out of data is not an exceptional condition, it is routine.

I know, it's funny. I'm laughing but it's not funny. This is why some of my posts are hard to read - I try to cram both meanings into one phrase, but then again it is also the type of thing that can lead to ten megabyte posts.

Let me work on this Saturday / Sunday when I am rested.

I will let you know either way.
[ December 16, 2006: Message edited by: Nicholas Jordan ]

[Nicholas Jordan]

Originally posted by Jesper Young:
[ As posted by: Jesper Young ]

This is the answer I was looking for.

Taking Jim's sample code and attempting, I get this:

Writer r = new BufferedWriter(new OutputStreamWriter(System.out, encoding);

What happens when I try to go to code is that I see that out is a Writer, so I try to construct a file on System.out and it just flat will not work any way I try it.

The last build I had noticed this setOut() and rebuilt my standard out redirection to a file and got a clean build using this syntax. For some reason it is not flushing or something, but there are no errors or warnings in the build.

The first approach I will use is to take Jim's code and place it directly where I try to redirect standard out and see what I get with that. Then I will check that it has buffering and a println() - both of those I can construct with my skills.

Essential, central question I have is to separate out file types that compiler/JVM may try to do MCBS conversion for me, so that I know what remaing file types I can use to implement application control files. In those files, I need to write arbitrary binary values directly in and out and do bit-twisting on them as I prefer, such as for implementing file control blocks, at the application layer, separate from the OS's file control circuitry which will be the   bailwick of a board manufacturer and OS vendor.

looks like a follow-up to some discussion that I haven't read

They were responsive in a post I made in Intermediate which asked about removing entries from a collection. It ran to 10,000 keystrokes on the return from the script.

The thing about supermodels is something I read in Harper's Magazine ten years ago. It's just so damn funny. No further meaning.
[ December 14, 2006: Message edited by: Nicholas Jordan ]

[Jim Yingst]

[NJ]: classes with finally's may confuse the gc, according to memory management docs on leak detection I read Sunday on the Sun website.

I think you may be confusing finalize() (a method) with finally (a keyword). Aside from sounding similar they do often serve similar purposes, cleaning things up when you are done with them. Except that finalize() is generally pretty sucky for a variety of reasons, and widely reviled, while finally generally works quite well for what people try to do with it.

[NJ]: " ... except if the exception is an exceptional exception ... "

[Nicholas Jordan]

Originally posted by Jim Yingst:
[JY]: I think you may be confusing finalize() (a method) with finally (a keyword). Aside from sounding similar they do often serve similar purposes, cleaning things up when you are done with them. Except that finalize() is generally pretty sucky for a variety of reasons, and widely reviled, while finally generally works quite well for what people try to do with it.

It may well be that I have a confusion. It is clear in my mind what it is I want to do, programmatically, and it is also clear, conceptually, what the page on memory management was saying. This is critical, later on, so let me do a few rounds - I certainly do not mind reading several html pages of Sun documentation. You demonstrate a deep fluency in antecedent code, just point out the Gotcha's .... that is what I do not want to build into code that may well have to be maintained by others, exactly as you guys were so quick to point out in Removing Entries From a Collection.

IOW - This is where we do that.

1. Program original design does not run for a long time. It's just like you guys were trying to warn me about. Code gets maintained. I do not want to build in memory leaks.

2. Program is heavily threaded. What happens - to the entire application - if an exception in n-th thread occurs. At any time, a file and an action are paired up, that becomes an isolated unit of execution that can be passed off to the n-th processor. No , repeat no, cache-consistency issues. Just none at this time. I made this as a prototyping decision to avoid building slop into the discussion version. You will note in the comments for class Alice a quick and early comment to the effect this is 0.1.0 version or something, and that it is a discussion version. Great designs become great big headaches, something I can tell you guys are trying to prevent. This is the time in prototyping to address this fail-fast design pattern and it's applicability to the application under the point raised.

Thread-N running nicely,...Thread-N+(arbitrarily large number) fails. Program is in hands of operator who is not really interested in runlogs. Discuss your point from the view of a remote sys-admin who may not hear about the failure. What I am going to do is enforce every thread perform well under testing and protoyping, but allow a minor portion of threads to fail under real-world use, just as a practacality matter. But, <teflon> if the exception is an exceptional exception, eg many threads failing</teflon> we want fail-fast behavior and bear in mind I am not shy about going 20,000 keystrokes,... I want fail-fast behaviour of the entire application in that falure mode. Ninja keyboard ready, I want to do the equivalent of pull the plug. I have had winnie hang too many times and have to provide the operator absolute ability to do this in crisis. There are plenty of cs's to review any design decisions we make here.

In general, it is my reading that finally is a catch-chain construct that gets executed in an exception condition, no matter what. Since there is no exception object avaiable, I cannot get a stack trace. Will make a lot of things sucky for me without doing me any good, except pleasing my critics. If I cannot handle things in the catch, then we have the crisis mode failure I describe and let's catch this (npi - no pun intended) before it goes to 10k post. The doc I remember reading probably had a discussion of finally vs finalize() (the method) as an integral intrinsic in the work. It had a graphic showing memory being hogged to the point that the machine would bog. Any machine. It would bog in a pattern generally resembling (in the broad stroke) decline of efficiency of hash-tables. The page demand is handled by a tool first described by Messers G.M. Adelson-Velsky and E.M. Landis an is just a smarter implementation of a solution to the same problem addressed by hash-tables.

<!-- Message edit: Will have to use NIO because Windows, I/O is not interruptable.-->
<!-- Message edit: I notice on p.123 Java Threads that locks should be released in a finally. This may be what you have in mind ultimately. It is difficult to sort out one without the other, still reading.-->
<!-- finally{} at return ? I thought finally only executed on exception condition. This means that closes cannot throw.-->
[ December 16, 2006: Message edited by: Nicholas Jordan ]

[Nicholas Jordan]

Originally posted by Jim Yingst:
From the title of your post it sounds like you may already have most of your answer already: use InputStreams and Readers.

For the first type of file you probably want a Reader, perhaps something like:

Reader r = new BufferedReader(new InputStreamReader(new FileInputStream(filename), encoding);

Or if another application has redirected their standard output to your Java program, it appears as standard input:

Reader r = new BufferedReader(new InputStreamReader(System.in, encoding);

Here, directly from source:

/**
 * Open a session log.
 */
Writer sessionLog = 
  new BufferedWriter(
    new OutputStreamWriter(
      new FileOutputStream("Runlog.txt")));

When you need the application to run in an extremely focused and aggressive manner in the pursuit of a goal there is a common practice in cs to allow out-of-band control information to be written in a human-readable format to a runLog/sessionLog. I gave additional effort to this issue this morning and arrived at the specualtive conjecture that everything that runs down the wire (networking - generally) would be subject to reasons discussed in A:\Character.java - which at 89,000+ characters, sort of explains your point.
My efforts to establish this gets:

Alice.java:187: Method println(java.lang.String) not found in class java.io.Writer.
sessionLog.println(java.lang.Integer.toHexString(lookingGlass.logName.hashCode()));
 
Alice.java:191: Method println(java.lang.String) not found in class java.io.Writer.
sessionLog.println(lookingGlass.getUserName(solarisWorkaround));
 
Alice.java:256: Method println(java.lang.String) not found in class java.io.Writer.
sessionLog.println("Exception:  " + e.getMessage());
 
Alice.java:299: Method println(java.lang.String) not found in class java.io.Writer.
sessionLog.println(e.getMessage());

Note line numbering in editor shows elimination of roughtly 600 lines of code, a small price to pay for the caliber of help I am getting. The posted code is from this morning's efforts.

Regardless of those details, most of your code should probably just assume you're using a Reader, so you can change the details later if you need to. However you may find it more useful to assume a BufferedReader instead, since that allows you to use a readLine() method which is very useful, and unavailable in a plain Reader.

Prototyping decision has been made to do this for now, as text files is something the team can understand - binary mode will only be used for controlling the actual program it's self until greater skills are attained. I am sure I will end up using   NIO   as a consequence of the way my mind works and the way the program is supposed to work. I got through on reading of Java Threads a few minutes ago, admittedly forced to skim to get through one time for overall understanding, but I have the clear belief that it will be necessary to use NIO, unless select() blocks - in which case it's back to the knockin-knoggin

Alternately if you're using JDK 5+ you can just use a Scanner instead, which takes care of most of these details for you. Just look at the options you have among different constructors.

I installed a copy of JDK_5, on Leslie's machine, so that I would have it available if needed. In what folder to I look for the constructors and comments in sources ? I use 1.2 for most of my prototyping, writing and throwing away 600 lines of code at a time requires remarkably fast editors, compilers, and a consumer-grade kernel that is broken.

Not trying to be funny,  truth  is stranger than fiction.

Since you're reading rather than writing, there's no flush() method. When you're done you should close() the reader yourself, but that's not a big deal. It's usually a good idea to use a try/finally to ensure it's closed even if an error occurred.

I need i/o input and output. Both, and one snag that came up this morning was the how to capture e.getMessage() to a run log.

But again, most of your code should just assume it's an InputStream - the underlying details aren't usually important, and should generally be hidden from most of the program. And again, call close() when you're done.

I have to be careful about assumptions.

Does that help?

Definitely.
[ December 16, 2006: Message edited by: Nicholas Jordan ]

[Jim Yingst]

[NJ]: Note line numbering in editor shows elimination of roughtly 600 lines of code

Yay!

The multiple errors whenever you use println() are there because, well, Writer has no println() methods. You can see the API for Writer. If you want to use println() (and why not, it's a convenient method) then you need to use a class that defines println() methods - PrintStreamand PrintWriter .being the two standard choices.
[ December 16, 2006: Message edited by: Jim Yingst ]

[Nicholas Jordan]

Originally posted by Jim Yingst:
[JY]:Yay!
That would be:
- no ?

...Writer. If you want to use println() (and why not, it's a convenient method) then you need to use a class that defines println() methods - PrintStreamand PrintWriter .being the two standard choices.

So I can just make the l-value a PrintWriter and get on with regular expressions. What about NIO b4 moving on ? Does it block anywhere at any time ? Bought enough books to load a small pickup. It appears from quick reading that FileChannel is what I need. I assume PrintWriter will fit into a FileChannel,... and what would be the comparable inFile construct ?

[ December 16, 2006: Message edited by: Nicholas Jordan ]

This is what I got to compile, no testing yet:

PrintWriter sessionLog =
new PrintWriter(
 new BufferedWriter(
   new OutputStreamWriter(
    new FileOutputStream("Runlog.txt"))));

[ December 17, 2006: Message edited by: Nicholas Jordan ]

[Ernest Friedman-Hill]

You can (and should) replace the nested OutputStreamWriter/FileOutputStream pair with a single "FileWriter". I tend to use BufferedWriter in there only when performance requires it; therefore you can probably get away with just a PrintWriter wrapped around a FileWriter.

[Jim Yingst]

Well, I would note that a common reason to use an OutputStreamWriter plus FileOutputStream would be to be able to specify an encoding - something I frequently prefer to relying on platform default encoding. Now in this case, Nicholas is just using the default encoding anyway, so there's really no reason to create the extra stream.

Incidentally, JDK 5 allows you to create a PrintWriter that writes to a file, with a particular encoding, in one step. Which is about ten years overdue, I'd say, but better late than never I suppose.

[Nicholas Jordan]

Originally posted by Jim Yingst:
Well, I would note that a common reason to use an OutputStreamWriter plus FileOutputStream would be to be able to specify an encoding - something I frequently prefer to relying on platform default encoding. Now in this case, Nicholas is just using the default encoding anyway, so there's really no reason to create the extra stream.

Rather than rely on the Blunder Bus of my work, which constructor ?

OutputStreamWriter / FileOutputStream

Incidentally, JDK 5 allows you to create a PrintWriter that writes to a file, with a particular encoding, in one step. Which is about ten years overdue, I'd say, but better late than never I suppose.

I have a coffee'd up answer to ten years overdue, along the lines of not trying to dig out ten years of code archaeology. What I need from JDK-5 right now is the ability to busy-loop on System.in, properly. I understand machine-loading/load-balancing issuues intuitively, I have worked around equipment all of my life - Java Threads (the book) is easier for me to understand than many of the issues you would likely correct me on, I will take those up in the correct fora, separately.

[Jim Yingst]

Well there are various alternatives lying around in different places, but the one that's been available longest is

    new OutputStreamWriter(OutputStream out, String charsetName)

Which allows you to do things like

    PrintWriter out = new PrintWriter(new OutputStreamWriter(new FileOutputstream("myfile.txt"), "UTF-8));

Or as of JDK 5 you can finally use

    new PrintWriter(File file, String csn)

to do the same thing in less convoluted fashion:

    PrintWriter out = new PrintWriter("myfile.txt", "UTF-8");

For input there are similar constructors in InputStreamReader and Scanner, respectively.

[Nicholas Jordan]

Originally posted by Ernest Friedman-Hill:
... there only when performance requires it; therefore you can probably get away with just a PrintWriter wrapped around a FileWriter.

Dr. Friedman-Hill.

I have really tried to make myself comprehensible on this. It just gets ten kb posts, which slows v.51 dialups.

Dov Bulka has spent fifteen years in the trenches of software development delivering large-scale software products to market. He was the performance architect of the IBM Domino-Go Web server that has powered some of the biggest Web sites ever hosted on the Internet, including that of the 1996 Atlanta Olympics. He received his Ph.D. in computer science from Duke University. He gives a thorough, well thought out discussion of the matter of performance in the cited book, a discussion which I feel is almost me talking. It escapes my why anyone would code in any other style, except in rare or limited settings such as the new coder or sources may be (?) - it just escapes me.

Men of honor may disagree. We disagree on this.

It does seem to be camp style though.

[ December 23, 2006: Message edited by: Nicholas Jordan ]

[ Additional material: Wednesday, December 27, 2006]

Java seems to be a naturally flowing linguistic. Fixing many of the inherent weaknesses of C/C++, I chose to code in this on the basis of suggestions from my Team Lead, who stated that is was widely respected by bacculareate cs's with whom he works in other phases of the project.

After acclimatizing to two keywords, I was already back up to the coding speed I had attained in it's legacy linguistic. I looked at your website, you are accomplished - and I have spared no mercy on you guys.

Spare me none now, I want to hear your case for parsiminous code.
(there is no neutral smiley)
[ December 27, 2006: Message edited by: Nicholas Jordan ]