Reading file splitting data into individual readings

[saeid jamali]

Hi everybody.

I have a list of weather for a city from 1908 till 2013. I am trying to write a program that reads the reading, writes it into an array and gives the average rainfall per year. like the image below


I have problem splitting readings by year. I was gonna use file.reaLine.split(); but I'm not sure how to do it to be able to split data by year.
Any idea?
Thank you

[Knute Snortum]

Do you know Regular Expressions (regex)? They can be complex but they are designed for this kind of stuff. Look at java.util.regex.Pattern.

[Piet Souris]

Isn't it very much easier to open this file in excel or libreoffice,
and derive the average rainfall with a couple of mouseclicks?

[saeid jamali]

 

Knute Snortum wrote:Do you know Regular Expressions (regex)? They can be complex but they are designed for this kind of stuff. Look at java.util.regex.Pattern.

Do you know anything easier? I'm newbie

[saeid jamali]

 

Piet Souris wrote:Isn't it very much easier to open this file in excel or libreoffice,
and derive the average rainfall with a couple of mouseclicks?

you are right but it's my assignment! (

[Stefan Evans]

Ok, so what are you going to store in this array?
An object/class?
A number?
In your design, what does each entry in the array contain?

Reading one line as a string and then using the split() method makes sense to me.
That would give you an array of Strings, which presumable the item at index 0 would be the year, index 1 the month etc etc. That entirely depends upon the format of the file.
Is this screenshot an exact example of what the file looks like or a conceptual one? I don't see any rainfall data in it is all :-)

[Carey Brown]

Your posted image of the data is too small for me to read. Could you cut and paste the first 10 lines of the data file here. Also nice to have would be a cut and paste of 10 data file lines where the year changes.

[Piet Souris]

I see; no excel/libreoffice then

Well, what Knute suggested would be fine indeed, but a regex might not
be optimal for a newbie.

Next, a Scanner comes in handy. Do you know these? Skip the first
four lines, and then use methods like scnner.nectInt, scanner.nextDouble
and so on.

You could create a class, where every instance represents one observation.
It would then be a doddle to derive means with Java8 streams.

If that is a little futuristic:
I would use a 2D array. First dimension for the year, second dimension
for the month, value will be the rainfall. Think of how you would store
1908, month 1 into array[0][0].

Then, getting a year average should be straightforward.

[Piet Souris]

 

Stefan Evans wrote:(...)
Is this screenshot an exact example of what the file looks like or a conceptual one? I don't see any rainfall data in it is all :-)

Missed that! Well, assuming "---" means "nothing at all", that certainly makes
calculating the average a tad easier

[saeid jamali]

So the user is gonna enter the name of the city (I have 5 cities) and the average rainfall, sunny hours and temperature BY YEAR comes up (for now I'm focusing on one which is rainfall for each year). I'm gonna make a 2D array table for each city with the number of rows and 7 columns as I have 7 columns in every data list and variable rows that goes up to 1257 rows. My biggest problem for now is the years start from 1905 till 2013 so if somehow I can split data by year and write it into a 2D array for example all of the data for 1905, 1906,1907 and son on are split.
I think I'm doing something wrong and there should be a way to calculate the average for every year because the program must work with any file with the same format.

I upload another image for you guys.

https://www.imageupload.co.uk/image/cVDF

[Carey Brown]

import java.util.ArrayList;
import java.util.List;
public class RainFall
{
public static void main( String[] args )
{
List<MonthlyRecord> records = new ArrayList<>();
// parse file and create list of records
int year = 0;
for( MonthlyRecord record : records )
{
if( record.getYear() != year )
{
// first time through there will be no data
// calculate averages
// output data
// clear accumulated data
}
year = record.getYear();
// accumulate data
}
// if any data remaining in accumulator
// calculate averages
// output data
}
}
class MonthlyRecord
{
int year;
public int getYear()
{
return year;
}
}

[Stefan Evans]

So they point Carey was making with that code sample is

Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file.
Step 2: Analyze the data in memory. e.g. loop through all the data, check if that record pertains to the year of interest, and get its rainfall value (if present)

I'm gonna make a 2D array table for each city with the number of rows and 7 columns as I have 7 columns in every data list and variable rows that goes up to 1257 rows

So I would assume that would be something like

String[][] weatherData = new String[1257][7];

Have you learned about Objects yet?
Personally I would prefer to have a MonthlyRecord[]

Using a List instead of an Array makes things easier, because you don't need to know how large the List is before you create it, you can just add things to it.

[saeid jamali]

 

Carey Brown wrote:

import java.util.ArrayList;
import java.util.List;
public class RainFall
{
public static void main( String[] args )
{
List<MonthlyRecord> records = new ArrayList<>();
// parse file and create list of records
int year = 0;
for( MonthlyRecord record : records )
{
if( record.getYear() != year )
{
// first time through there will be no data
// calculate averages
// output data
// clear accumulated data
}
year = record.getYear();
// accumulate data
}
// if any data remaining in accumulator
// calculate averages
// output data
}
}
class MonthlyRecord
{
int year;
public int getYear()
{
return year;
}
}

I really like the idea of arraylist and objects. I have been reading about them so I can do my project this way. I'm gonna give it a try to see if I can do it this way.
Can anybody please them me what is happening in the for each loop(I assume we need the for-each loop for reading the data into araylist? )? what does this if statement mean

if( record.getYear() != year ) 

and what does this mean

year = record.getYear();

Thannk you very much guys

[Carey Brown]

Line 15 can be read as: for each record in the list of records. Each time through the loop the variable 'record' will contain the next instance of a MonthlyRecord object (not a list).
Line 17: Check to see if the year has changed from the year of the previous record.
Line 24: Save the year of this record for the purpose of checking it on line 17 on the next go around of the loop. 'Year' serves no other purpose than to detect when we have come across a new set of MonthlyRecord's for the next year.

The variable 'year' might have been more clearly written as 'previousYear'.

Line 17 might have been better written as

if( year != 0 && record.getYear() != year )

This would deal with the special case of first time through the loop.

The class 'MonthlyRecord' on line 34 is just a sketch showing the 'year' field. This would need to be flushed out with additional fields, one for each of the columns in your data file. You'll need a constructor that takes a single String that is a line read from the input file, and then parses it into the appropriate fields. In my example I've assumed that when you parse the year from the line of data that you'll convert it to an int.

Edit: "record.getYear()" takes a 'record' object, which in this case is an instance of the MonthlyRecord class, and invokes the class's 'getYear()' method, which in this case returns the object's field 'year'.

[Winston Gutkowski]

 

Stefan Evans wrote:So the point Carey was making with that code sample is
Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file...

And I'd go even further than that. A MonthlyRecord class should hold ALL the information you need to store, viz:

public final class MonthlyRecord {
private final int year;
private final int month;
private final double tmax;
private final double tmin;
private final int afDays; // ?? no idea whether the type is right,
// because I have no idea what 'af days' means.
private final double rain;
private final double sun;
...
}

and furthermore, IT should be responsible for pulling it's values from a "record" (presumably a line of text), viz:

 ...
public MonthlyRecord(String record) {
String[] fields = record.split("\\s+");
this.year = Integer.valueOf(fields[0]);
this.month = Integer.valueOf(fields[1]);
this.tmax = Double.valueOf(fields[2]);
...
}
...

and then you can add it to your list with something like:
  MonthlyRecord mr = new MonthlyRecord( myFile.readLine() );
  records.add(mr);
and then sort/extract/muck about with it all you want.

HIH

Winston

[saeid jamali]

 

Winston Gutkowski wrote:

Stefan Evans wrote:So the point Carey was making with that code sample is
Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file...

And I'd go even further than that. A MonthlyRecord class should hold ALL the information you need to store, viz:

public final class MonthlyRecord {
private final int year;
private final int month;
private final double tmax;
private final double tmin;
private final int afDays; // ?? no idea whether the type is right,
// because I have no idea what 'af days' means.
private final double rain;
private final double sun;
...
}

and furthermore, IT should be responsible for pulling it's values from a "record" (presumably a line of text), viz:

 ...
public MonthlyRecord(String record) {
String[] fields = record.split("\\s+");
this.year = Integer.valueOf(fields[0]);
this.month = Integer.valueOf(fields[1]);
this.tmax = Double.valueOf(fields[2]);
...
}
...

and then you can add it to your list with something like:
  MonthlyRecord mr = new MonthlyRecord( myFile.readLine() );
  records.add(mr);
and then sort/extract/muck about with it all you want.

HIH

Winston

Is this the way of adding a file to an array list?

MonthlyRecord mr = new MonthlyRecord( myFile.readLine() );
records.add(mr);

Cause I tried this way but when I print out the array list with this

System.out.println(records.get(records.size()-1));

compiler gives an error and says index:0 size:0 I assume it means it could not add it to array list so there is nothing in there. I tried many different ways.
I tried this but it says The method add(Assignment.MonthlyRecord) in the type List<Assignment.MonthlyRecord> is not applicable for the arguments (String)

String line;
While( (line = file.readLine()) !=null){
records.add(line);
}

I tried a for loop but every time there is something wrong so I'm just wondering if there is something wrong with the structure of code or I'm doing it wrong.
I have one public class followed by a public static void plus these below

 class MonthlyRecord
{
private final int year;
private final int month;
private final double tmax;
private final double tmin;
private final int afDays;
private final double rain;
private final double sun;
public MonthlyRecord(String record) {
String[] fields = record.split("\\s+");
this.year = Integer.valueOf(fields[0]);
this.month = Integer.valueOf(fields[1]);
this.tmax = Double.valueOf(fields[2]);
this.tmin = Double.valueOf(fields[3]);
this.afDays = Integer.valueOf(fields[4]);
this.rain = Double.valueOf(fields[5]);
this.sun = Double.valueOf(fields[6]);
}
public int year(){
return year;
}
public int month(){
return month;
}
public double tmax(){
return tmax;
}
public double tmin(){
return tmin;
}
public int afDays(){
return afDays;
}
public double rain(){
return rain;
}
public double sun(){
return sun;
}
}

[Carey Brown]

Close, but change it to

String line;
While( (line = file.readLine()) !=null){
records.add( new MonthlyRecord(line) );
}

If you want to print out all the records you'll need a toString() method in the MonthlyRecord class that formats the record the way you'd like.
Then, loop through all the records like this

for( MonthlyRecord rec : records ) {
System.out.println( rec );
}

Of course you'll need to skip the rows in the file that make up the header.

[saeid jamali]

What about this? does do the job?

for(int i =0; i<records.size(); i++){
while((line = file.readLine()) != null) {
String [] rec = file.readLine().split("\\s+");
records.add(new MonthlyRecord(rec[i]));

If yes then why do I need String[] fields = record.split("\\s+");in my public class as shown above? cause without the split I was getting Format exception.
and another thing is with this structure I have to be able to print the value of the tmax like System.out.println(MonthlyRecord.tmax()); is it right? so I can print it, calculate it or do what ever with it.

Guys sorry for asking too many questions but every time you guys mention something I read about it and I'm really interested to learn java. I learnt a lot of stuff from this forum more than any classes.

[Knute Snortum]

This line:

while((line = file.readLine()) != null) {

will loop until the end of the file. So the for loop will execute once with i = 0 and then "file" will be at the end of the file for all the other iterations. If you want to read one line, try:

line = file.readLine()

and test line for null with an if statement.

[Carey Brown]

 

saeid jamali wrote:What about this? does do the job?

for(int i =0; i<records.size(); i++){
while((line = file.readLine()) != null) {
String [] rec = file.readLine().split("\\s+");
records.add(new MonthlyRecord(rec[i]));

If yes then why do I need String[] fields = record.split("\\s+");in my public class as shown above? cause without the split I was getting Format exception.
and another thing is with this structure I have to be able to print the value of the tmax like System.out.println(MonthlyRecord.tmax()); is it right? so I can print it, calculate it or do what ever with it.

Guys sorry for asking too many questions but every time you guys mention something I read about it and I'm really interested to learn java. I learnt a lot of stuff from this forum more than any classes.

You already have a split() in your MonthlyRecord constructor, you don't need one here.
Your for loop will never execute because records.size() should be zero at this point. You don't need the for() here at all, the while() should suffice except for reading and throwing away a couple of header lines.

[saeid jamali]

Ok guys just one more question about reading file and that's it
My code looks alright to me but I'm getting NumberFormatException for some reasons and I can't figure out what's happening.
So I'm reading the file like this in my main

while((line = file.readLine()) != null)
{
if(line.equals(""))
{
line =line.trim();
}
records.add(new MonthlyRecord(line));
}

So I trim the leading and trailing space before adding it to my arraylist.

Then in my constructor class I have the following just before parse the records.

String[] fields = records.split("\\s+");
for(int i=0; i<fields.length; i++){
fields[i]= fields[i].replace("[^\\d.-]", "");
}

So I split the reading with one or more white space and then it goes through a loop and replaces all non digits with "" and I keep "." and "-" for decimal and negative numbers.
This was working fine last night and I don't remember changing anything but all of a sudden it stopped working. I'm so confused cause if there is any sign or letter attached to a number it will works fine and replaces them. but if there is any word on it's own in the header it gives exception. It says NumberFormatException strin "".

Maybe when it replaces a non digit with ("") it causes the exception?? somewhere some String being converted to int but I have no clue.
and the other problem is my no data sign is like "---" if I keep "-" for negatives number it won't be replaced and it gives me exception !!

[Knute Snortum]

I looks like what you wanted to use was replaceAll(). It takes a regex.

[saeid jamali]

 

Knute Snortum wrote:I looks like what you wanted to use was replaceAll(). It takes a regex.

Yea I tried it as well I get the same exception. it says NumberFormatException "Location" which location is the first word in my file without All. and it says NumberFormatException: "" with All.
I deleted

if(line.equals(""))

It works without if I delete the text on the top of file and this "---" which means empty data in my file.

[Knute Snortum]

Could you post the stack trace of the exception, the surrounding code, and the data that causes the problem?

[Winston Gutkowski]

 

saeid jamali wrote:Then in my constructor class I have the following just before parse the records...

First: DON'T GUESS.

What you need is to know EXACTLY what every single record in your file looks like, and what order they come in.

For example, as Carey already said, you'll probably need to make sure that your constructor ignores header records - so I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method and have the constructor throw an Exception if you try to call it with a header record.

Then your main() logic might look something like:

while((line = file.readLine()) != null) {
if(!MonthlyRecord.isHeader(line))
records.add( new MonthlyRecord(line) );
}

but before you can do that, you MUST know precisely what you're going to be getting.

Winston

[Winston Gutkowski]

 

Winston Gutkowski wrote:... I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method...

Or possibly even better, a:
  public static final boolean isValidData(String record) { ...
method that checks that everything in the record is correctly formatted, eg:

public final class MonthlyRecord {
public MonthlyRecord(String record) {
if (!isValidData(record))
throw new IllegalArgumentException(
"badly formatted record");
String[] fields = record.split("\\s+");
this.year = Integer.valueOf(fields[0]);
...
}
...
// Returns true if, and ONLY if, record is OK to be
// passed to the constructor.
public static final boolean isValidData(String record) {
String[] fields = record.split("\\s+");
// pseudo-pseudocode
return fields.length == 7
&& fields[0] is a 4-digit integer
&& fields[1] is an integer
&& fields[2] is a temperature
&& fields[3] is a temperature
&& fields[4] is an integer
&& fields[5] is "---" or a decimal
&& fields[6] is a decimal;
}
...
}

I leave the tests up to you.

Winston

[saeid jamali]

 

Winston Gutkowski wrote:

Winston Gutkowski wrote:... I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method...

Or possibly even better, a:
  public static final boolean isValidData(String record) { ...
method that checks that everything in the record is correctly formatted, eg:

public final class MonthlyRecord {
public MonthlyRecord(String record) {
if (!isValidData(record))
throw new IllegalArgumentException(
"badly formatted record");
String[] fields = record.split("\\s+");
this.year = Integer.valueOf(fields[0]);
...
}
...
// Returns true if, and ONLY if, record is OK to be
// passed to the constructor.
public static final boolean isValidData(String record) {
String[] fields = record.split("\\s+");
// pseudo-pseudocode
return fields.length == 7
&& fields[0] is a 4-digit integer
&& fields[1] is an integer
&& fields[2] is a temperature
&& fields[3] is a temperature
&& fields[4] is an integer
&& fields[5] is "---" or a decimal
&& fields[6] is a decimal;
}
...
}

I leave the tests up to you.

Winston

but it doesn't ignore the header which is a text does it?
So I uploaded two pictures of how the file looks from beginning to end

[Campbell Ritchie]

Please don't post pictures but text. Those pictures are illegible on my screen. But neither the text nor the pictures tell us what the correct format for the file is. Only you know that.

[saeid jamali]

Braemar Location: 3152E 7914N, 339 metres amsl (1959 to Apr 2005), & 3152E 7919N, 327 metres amsl (Aug 2005 onwards) Estimated data is marked with a * after the value. Missing data (more than 2 days missing in month) is marked by ---. Sunshine data taken from an automatic Kipp & Zonen sensor marked with a #, otherwise sunshine data taken from a Campbell Stokes recorder. yyyy mm tmax tmin af rain sun degC degC days mm hours 1959 1 1.7 -5.7 27 --- 34.2 1959 2 6.2 -3.2 15 --- 68.6 1959 3 7.6 0.8 7 --- 80.9 1959 4 --- --- --- --- 105.0 1959 5 15.6 4.6 1 --- 182.6 1959 6 16.4 7.2 0 --- 164.8 1959 7 18.0 8.5 0 --- 147.4 1959 8 18.6 10.1 0 --- 106.3 1959 9 17.0 5.4 1 --- 148.2 1959 10 13.2 4.8 3 --- 87.8 1959 11 7.0 1.3 8 --- 26.6 1959 12 4.7 -0.6 14 --- 8.2 1960 1 3.8 -2.4 21 --- 28.5 1960 2 3.5 -5.6 23 --- 79.4 1960 3 5.7 0.0 11 --- 50.9 1960 4 10.5 2.4 6 --- 99.2 1960 5 15.5 3.8 3 --- 160.8 1960 6 18.3 8.0 0 --- 190.7 1960 7 16.7 7.7 0 --- 120.9 1960 8 15.9 7.2 0 --- 95.8 1960 9 14.1 5.2 3 --- 106.5 1960 10 9.3 5.2 0 --- 16.3 1960 11 6.3 -0.2 12 --- 25.9 1960 12 3.3 -3.3 18 --- 16.3 1961 1 3.3 -2.7 22 71.8 22.6 2012 12 4.0 -2.1 16 115.8 --- 2013 1 3.9 -1.5 19 81.2 --- Provisional 2013 2 3.6* -3.0* 26* 52.3* --- Provisional 2013 3 2.7 -4.1 25 53.0 --- Provisional 2013 4 7.9 0.1 14 64.0 --- Provisional 2013 5 12.9 3.9 5 93.6 --- Provisional

[saeid jamali]

and there is white space between every column of course.
I'm trying to read only numbers and for that I used regex in my constructor so it replaces any non digits to ("") but still I'm getting exception. shouldn't it replace those words on the top and bottom with ("")?

[Knute Snortum]

It's hard to tell what the problem is without the code you've written. Can you post your entire code here?

[saeid jamali]

 

Knute Snortum wrote:It's hard to tell what the problem is without the code you've written. Can you post your entire code here?

The entire code is a lot of code but the part for reading data is like this.
I read the data like this from the file.

String line;
while((line = file.readLine()) != null){
// if(line.equals(""))
// {
line =line.trim();
// }
records.add(new MonthlyRecord(line));
}
file.close();
} catch (FileNotFoundException e){
System.out.println("Data not found. Please import a file!");
}

and this is my constructor:

public MonthlyRecord(String records) {
String[] fields = records.split("\\s+");
for(int i=0; i<fields.length; i++){
fields[i]= fields[i].replaceAll("[^\\d.-]", "");
}
this.year = Integer.valueOf(fields[0]);
this.month = Integer.valueOf(fields[1]);
this.tmax = Double.valueOf(fields[2]);
this.tmin = Double.valueOf(fields[3]);
this.afDays = Integer.valueOf(fields[4]);
this.rain = Double.valueOf(fields[5]);
this.sun = Double.valueOf(fields[6]);
}

this code does the job. it ignores all the symbols and signs like ("@", "*", "#") except for that ("---") because I kept this sign in my replaceAll method for negative numbers this is the first problem. the second problem is that it should ignores all the alphabet as well but seems like it doesn't so I need to get rid of this letters somehow.

[Winston Gutkowski]

 

saeid jamali wrote:and this is my constructor:

public MonthlyRecord(String records) {
...
this.rain = Double.valueOf(fields[5]);
...
}

Well that's wrong right there isn't it? In fact, practically all your conversions are.

Look at your data. HARD.

Winston

[saeid jamali]

 

Winston Gutkowski wrote:

saeid jamali wrote:and this is my constructor:

public MonthlyRecord(String records) {
...
this.rain = Double.valueOf(fields[5]);
...
}

Well that's wrong right there isn't it? In fact, practically all your conversions are.

Look at your data. HARD.

Winston

Do you think so? are you saying this because of "---" sign? if yea this sign can be any where in the data as you can see tmax, tmin, sunny columns also have this ("---") and I have 4 more files formatted exactly the same but this "---" can be found any where. Otherwise it calculates my data perfectly.

[Winston Gutkowski]

 

saeid jamali wrote:Do you think so? are you saying this because of "---" sign? if yea this sign can be any where in the data as you can see tmax, tmin, sunny columns also have this ("---") and I have 4 more files formatted exactly the same but this "---" can be found any where...

OK, so what's wrong with:

public MonthlyRecord(String records) {
...
if ( ! "---".equals(fields[5]) )
this.rain = Double.valueOf(fields[5]);
...
}

?

Winston

[saeid jamali]

EDIT: it doesn't work. it prints the value of the whole line where "---" was found as 0.

[saeid jamali]

I think somehow the problem with "---" is sorted now I'm trying to ignore letters while reading the file. I noticed all of my file have the same length of header and the actual numbers in all of them starts from 8th line. So is there any way to ignore first 7 lines?

[Liutauras Vilda]

On the performance side, probably you don't need Double or Integer objects. Should be enough for you to have primitives. Consider using parseInt and parseDouble instead.

[Liutauras Vilda]

 

saeid jamali wrote:is there any way to ignore first 7 lines?

No. But you can read them and do not do nothing with them. For reading you're using readLine method, so think of how to read 7 lines and do not do nothing with them, and only from line 8 start your processing.