Win a copy of Practical SVG this week in the HTML/CSS/JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic

Reading file splitting data into individual readings

 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi everybody.

I have a list of weather for a city from 1908 till 2013. I am trying to write a program that reads the reading, writes it into an array and gives the average rainfall per year. like the image below


I have problem splitting readings by year. I was gonna use file.reaLine.split(); but I'm not sure how to do it to be able to split data by year.
Any idea?
Thank you
 
Knute Snortum
Sheriff
Posts: 3283
84
Chrome Eclipse IDE Java Postgres Database VI Editor
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Do you know Regular Expressions (regex)? They can be complex but they are designed for this kind of stuff. Look at java.util.regex.Pattern.
 
Piet Souris
Rancher
Posts: 1641
36
  • Likes 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Isn't it very much easier to open this file in excel or libreoffice,
and derive the average rainfall with a couple of mouseclicks?
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Knute Snortum wrote:Do you know Regular Expressions (regex)? They can be complex but they are designed for this kind of stuff. Look at java.util.regex.Pattern.


Do you know anything easier? I'm newbie
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Piet Souris wrote:Isn't it very much easier to open this file in excel or libreoffice,
and derive the average rainfall with a couple of mouseclicks?


you are right but it's my assignment! (
 
Stefan Evans
Bartender
Posts: 1822
10
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok, so what are you going to store in this array?
An object/class?
A number?
In your design, what does each entry in the array contain?

Reading one line as a string and then using the split() method makes sense to me.
That would give you an array of Strings, which presumable the item at index 0 would be the year, index 1 the month etc etc. That entirely depends upon the format of the file.
Is this screenshot an exact example of what the file looks like or a conceptual one? I don't see any rainfall data in it is all :-)

 
Carey Brown
Bartender
Posts: 2265
28
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Your posted image of the data is too small for me to read. Could you cut and paste the first 10 lines of the data file here. Also nice to have would be a cut and paste of 10 data file lines where the year changes.
 
Piet Souris
Rancher
Posts: 1641
36
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I see; no excel/libreoffice then

Well, what Knute suggested would be fine indeed, but a regex might not
be optimal for a newbie.

Next, a Scanner comes in handy. Do you know these? Skip the first
four lines, and then use methods like scnner.nectInt, scanner.nextDouble
and so on.

You could create a class, where every instance represents one observation.
It would then be a doddle to derive means with Java8 streams.

If that is a little futuristic:
I would use a 2D array. First dimension for the year, second dimension
for the month, value will be the rainfall. Think of how you would store
1908, month 1 into array[0][0].

Then, getting a year average should be straightforward.
 
Piet Souris
Rancher
Posts: 1641
36
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Stefan Evans wrote:(...)
Is this screenshot an exact example of what the file looks like or a conceptual one? I don't see any rainfall data in it is all :-)

Missed that! Well, assuming "---" means "nothing at all", that certainly makes
calculating the average a tad easier
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So the user is gonna enter the name of the city (I have 5 cities) and the average rainfall, sunny hours and temperature BY YEAR comes up (for now I'm focusing on one which is rainfall for each year). I'm gonna make a 2D array table for each city with the number of rows and 7 columns as I have 7 columns in every data list and variable rows that goes up to 1257 rows. My biggest problem for now is the years start from 1905 till 2013 so if somehow I can split data by year and write it into a 2D array for example all of the data for 1905, 1906,1907 and son on are split.
I think I'm doing something wrong and there should be a way to calculate the average for every year because the program must work with any file with the same format.

I upload another image for you guys.

https://www.imageupload.co.uk/image/cVDF
 
Carey Brown
Bartender
Posts: 2265
28
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
 
Stefan Evans
Bartender
Posts: 1822
10
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So they point Carey was making with that code sample is

Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file.
Step 2: Analyze the data in memory. e.g. loop through all the data, check if that record pertains to the year of interest, and get its rainfall value (if present)


I'm gonna make a 2D array table for each city with the number of rows and 7 columns as I have 7 columns in every data list and variable rows that goes up to 1257 rows

So I would assume that would be something like


Have you learned about Objects yet?
Personally I would prefer to have a MonthlyRecord[]

Using a List instead of an Array makes things easier, because you don't need to know how large the List is before you create it, you can just add things to it.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Carey Brown wrote:


I really like the idea of arraylist and objects. I have been reading about them so I can do my project this way. I'm gonna give it a try to see if I can do it this way.
Can anybody please them me what is happening in the for each loop(I assume we need the for-each loop for reading the data into araylist? )? what does this if statement mean and what does this mean

Thannk you very much guys
 
Carey Brown
Bartender
Posts: 2265
28
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Line 15 can be read as: for each record in the list of records. Each time through the loop the variable 'record' will contain the next instance of a MonthlyRecord object (not a list).
Line 17: Check to see if the year has changed from the year of the previous record.
Line 24: Save the year of this record for the purpose of checking it on line 17 on the next go around of the loop. 'Year' serves no other purpose than to detect when we have come across a new set of MonthlyRecord's for the next year.

The variable 'year' might have been more clearly written as 'previousYear'.

Line 17 might have been better written as

This would deal with the special case of first time through the loop.

The class 'MonthlyRecord' on line 34 is just a sketch showing the 'year' field. This would need to be flushed out with additional fields, one for each of the columns in your data file. You'll need a constructor that takes a single String that is a line read from the input file, and then parses it into the appropriate fields. In my example I've assumed that when you parse the year from the line of data that you'll convert it to an int.

Edit: "record.getYear()" takes a 'record' object, which in this case is an instance of the MonthlyRecord class, and invokes the class's 'getYear()' method, which in this case returns the object's field 'year'.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Stefan Evans wrote:So the point Carey was making with that code sample is
Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file...

And I'd go even further than that. A MonthlyRecord class should hold ALL the information you need to store, viz:and furthermore, IT should be responsible for pulling it's values from a "record" (presumably a line of text), viz:and then you can add it to your list with something like:
  MonthlyRecord mr = new MonthlyRecord( myFile.readLine() );
  records.add(mr);
and then sort/extract/muck about with it all you want.

HIH

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:
Stefan Evans wrote:So the point Carey was making with that code sample is
Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file...

And I'd go even further than that. A MonthlyRecord class should hold ALL the information you need to store, viz:and furthermore, IT should be responsible for pulling it's values from a "record" (presumably a line of text), viz:and then you can add it to your list with something like:
  MonthlyRecord mr = new MonthlyRecord( myFile.readLine() );
  records.add(mr);
and then sort/extract/muck about with it all you want.

HIH

Winston

Is this the way of adding a file to an array list?

Cause I tried this way but when I print out the array list with this

compiler gives an error and says index:0 size:0 I assume it means it could not add it to array list so there is nothing in there. I tried many different ways.
I tried this but it says The method add(Assignment.MonthlyRecord) in the type List<Assignment.MonthlyRecord> is not applicable for the arguments (String)

I tried a for loop but every time there is something wrong so I'm just wondering if there is something wrong with the structure of code or I'm doing it wrong.
I have one public class followed by a public static void plus these below
 
Carey Brown
Bartender
Posts: 2265
28
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Close, but change it to

If you want to print out all the records you'll need a toString() method in the MonthlyRecord class that formats the record the way you'd like.
Then, loop through all the records like this

Of course you'll need to skip the rows in the file that make up the header.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What about this? does do the job?


If yes then why do I need String[] fields = record.split("\\s+");in my public class as shown above? cause without the split I was getting Format exception.
and another thing is with this structure I have to be able to print the value of the tmax like System.out.println(MonthlyRecord.tmax()); is it right? so I can print it, calculate it or do what ever with it.

Guys sorry for asking too many questions but every time you guys mention something I read about it and I'm really interested to learn java. I learnt a lot of stuff from this forum more than any classes.
 
Knute Snortum
Sheriff
Posts: 3283
84
Chrome Eclipse IDE Java Postgres Database VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This line:

will loop until the end of the file. So the for loop will execute once with i = 0 and then "file" will be at the end of the file for all the other iterations. If you want to read one line, try:

and test line for null with an if statement.
 
Carey Brown
Bartender
Posts: 2265
28
Eclipse IDE Firefox Browser Java MySQL Database VI Editor Windows
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
saeid jamali wrote:What about this? does do the job?


If yes then why do I need String[] fields = record.split("\\s+");in my public class as shown above? cause without the split I was getting Format exception.
and another thing is with this structure I have to be able to print the value of the tmax like System.out.println(MonthlyRecord.tmax()); is it right? so I can print it, calculate it or do what ever with it.

Guys sorry for asking too many questions but every time you guys mention something I read about it and I'm really interested to learn java. I learnt a lot of stuff from this forum more than any classes.

You already have a split() in your MonthlyRecord constructor, you don't need one here.
Your for loop will never execute because records.size() should be zero at this point. You don't need the for() here at all, the while() should suffice except for reading and throwing away a couple of header lines.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ok guys just one more question about reading file and that's it
My code looks alright to me but I'm getting NumberFormatException for some reasons and I can't figure out what's happening.
So I'm reading the file like this in my main

So I trim the leading and trailing space before adding it to my arraylist.

Then in my constructor class I have the following just before parse the records.

So I split the reading with one or more white space and then it goes through a loop and replaces all non digits with "" and I keep "." and "-" for decimal and negative numbers.
This was working fine last night and I don't remember changing anything but all of a sudden it stopped working. I'm so confused cause if there is any sign or letter attached to a number it will works fine and replaces them. but if there is any word on it's own in the header it gives exception. It says NumberFormatException strin "".

Maybe when it replaces a non digit with ("") it causes the exception?? somewhere some String being converted to int but I have no clue.
and the other problem is my no data sign is like "---" if I keep "-" for negatives number it won't be replaced and it gives me exception !!
 
Knute Snortum
Sheriff
Posts: 3283
84
Chrome Eclipse IDE Java Postgres Database VI Editor
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I looks like what you wanted to use was replaceAll(). It takes a regex.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Knute Snortum wrote:I looks like what you wanted to use was replaceAll(). It takes a regex.

Yea I tried it as well I get the same exception. it says NumberFormatException "Location" which location is the first word in my file without All. and it says NumberFormatException: "" with All.
I deleted

It works without if I delete the text on the top of file and this "---" which means empty data in my file.
 
Knute Snortum
Sheriff
Posts: 3283
84
Chrome Eclipse IDE Java Postgres Database VI Editor
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Could you post the stack trace of the exception, the surrounding code, and the data that causes the problem?
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
saeid jamali wrote:Then in my constructor class I have the following just before parse the records...

First: DON'T GUESS.

What you need is to know EXACTLY what every single record in your file looks like, and what order they come in.

For example, as Carey already said, you'll probably need to make sure that your constructor ignores header records - so I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method and have the constructor throw an Exception if you try to call it with a header record.

Then your main() logic might look something like:but before you can do that, you MUST know precisely what you're going to be getting.

Winston
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:... I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method...

Or possibly even better, a:
  public static final boolean isValidData(String record) { ...
method that checks that everything in the record is correctly formatted, eg:I leave the tests up to you.

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:
Winston Gutkowski wrote:... I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method...

Or possibly even better, a:
  public static final boolean isValidData(String record) { ...
method that checks that everything in the record is correctly formatted, eg:I leave the tests up to you.

Winston

but it doesn't ignore the header which is a text does it?
So I uploaded two pictures of how the file looks from beginning to end
top.jpg
[Thumbnail for top.jpg]
but.jpg
[Thumbnail for but.jpg]
 
Campbell Ritchie
Marshal
Posts: 52636
119
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Please don't post pictures but text. Those pictures are illegible on my screen. But neither the text nor the pictures tell us what the correct format for the file is. Only you know that.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
and there is white space between every column of course.
I'm trying to read only numbers and for that I used regex in my constructor so it replaces any non digits to ("") but still I'm getting exception. shouldn't it replace those words on the top and bottom with ("")?
 
Knute Snortum
Sheriff
Posts: 3283
84
Chrome Eclipse IDE Java Postgres Database VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It's hard to tell what the problem is without the code you've written. Can you post your entire code here?
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Knute Snortum wrote:It's hard to tell what the problem is without the code you've written. Can you post your entire code here?

The entire code is a lot of code but the part for reading data is like this.
I read the data like this from the file.


and this is my constructor:



this code does the job. it ignores all the symbols and signs like ("@", "*", "#") except for that ("---") because I kept this sign in my replaceAll method for negative numbers this is the first problem. the second problem is that it should ignores all the alphabet as well but seems like it doesn't so I need to get rid of this letters somehow.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
saeid jamali wrote:and this is my constructor:

Well that's wrong right there isn't it? In fact, practically all your conversions are.

Look at your data. HARD.

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:
saeid jamali wrote:and this is my constructor:

Well that's wrong right there isn't it? In fact, practically all your conversions are.

Look at your data. HARD.

Winston

Do you think so? are you saying this because of "---" sign? if yea this sign can be any where in the data as you can see tmax, tmin, sunny columns also have this ("---") and I have 4 more files formatted exactly the same but this "---" can be found any where. Otherwise it calculates my data perfectly.
 
Winston Gutkowski
Bartender
Posts: 10573
65
Eclipse IDE Hibernate Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
saeid jamali wrote:Do you think so? are you saying this because of "---" sign? if yea this sign can be any where in the data as you can see tmax, tmin, sunny columns also have this ("---") and I have 4 more files formatted exactly the same but this "---" can be found any where...

OK, so what's wrong with:
?

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
EDIT: it doesn't work. it prints the value of the whole line where "---" was found as 0.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think somehow the problem with "---" is sorted now I'm trying to ignore letters while reading the file. I noticed all of my file have the same length of header and the actual numbers in all of them starts from 8th line. So is there any way to ignore first 7 lines?
 
Liutauras Vilda
Marshal
Posts: 3233
127
BSD
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
On the performance side, probably you don't need Double or Integer objects. Should be enough for you to have primitives. Consider using parseInt and parseDouble instead.
 
Liutauras Vilda
Marshal
Posts: 3233
127
BSD
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
saeid jamali wrote:is there any way to ignore first 7 lines?
No. But you can read them and do not do nothing with them. For reading you're using readLine method, so think of how to read 7 lines and do not do nothing with them, and only from line 8 start your processing.
 
Hey cool! They got a blimp! But I have a tiny ad:
the new thread boost feature: great for the advertiser and smooth for the coderanch user
https://coderanch.com/t/674455/Thread-Boost-feature
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!