This week's book giveaway is in the Agile and Other Processes forum.
We're giving away four copies of The Little Book of Impediments (e-book only) and have Tom Perry on-line!
See this thread for details.
Win a copy of The Little Book of Impediments (e-book only) this week in the Agile and Other Processes forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Validating the Data File

 
Sean Keane
Ranch Hand
Posts: 588
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Before you attempt to read from the data file, to what extend did you guys go to validate the data file?

At a very basic level you could just check that the magic cookie is the correct value.

You could check other things like:

* The file length is at least the length of the schema.

* If you pad fields with empty spaces (rather than using the null terminated character) then you could check that the entire file is the length of the schema plus a multiple of an entire record length.
 
Roberto Perillo
Bartender
Posts: 2271
3
Eclipse IDE Java Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Although I think this would be necessary in the real world, I think this isn't really necessary for this assignment... what I did was verify if the magic cookie value present in the file being read was the same as the magic cookie present in the file that was provided to me in the assignment (257, if I make no mistake). But some people passed and didn't even consider doing that because this isn't a requirement. So, if you don't do any validation, then it is ok. Just don't forget to mention your choice in your choices.txt file because (in my opinion) this is an important point of the assignment.
 
Roel De Nijs
Sheriff
Posts: 10662
144
AngularJS Chrome Eclipse IDE Hibernate Java jQuery MySQL Database Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I just verified the value of the magic cookie. Nothing more, nothing less.
 
Oladeji Oluwasayo
Ranch Hand
Posts: 101
Java Netbeans IDE Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I didn't do any validation. The only thing is that my file chooser (in the gui) comes with a filter that allows only .db files. If an invalid database file is chosen, the display will be blank. If the file does not exist (maybe a path was just typed into the database location textfield, the file will be created. I didn't even think of this validation.
 
Sean Keane
Ranch Hand
Posts: 588
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks guys. I think I will just validate the magic cookie and ensure that the file is at least as long as the schema. Then any other issues with the file when reading I will just throw runtime exceptions as I process the file - for example if there are not enough bytes for a record.
 
Roel De Nijs
Sheriff
Posts: 10662
144
AngularJS Chrome Eclipse IDE Hibernate Java jQuery MySQL Database Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
On the server I only checked the magic cookie. At the client I tested a few other things: only .db extension, length bigger than 0, writable, readable,...

When something went wrong on the server while reading the file, an exception is thrown (which is caught by the client and the user gets an appropriate message).
 
Sarah Archer
Greenhorn
Posts: 19
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I took a different view on this.

I didn't check my magic cookie because the instructions did specify which values were valid. It seemed to be more a version number, in hex 0x101, and perhaps compatibility is maintained for minor version number increments - you don't know.

I did check though that the record length = sum of field lengths - sizeof (deleted flag). We have enough information from the specification to know this must hold. You don't have to do it because it is not in the spec, but it is a simple check that will catch a lot of errors including bugs in reading the data file or a corrupt schema.

Finally, I wrote the code in a similar manner to the way I would write production code - lots of logging etc.
 
Sean Keane
Ranch Hand
Posts: 588
Chrome Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sarah Archer wrote:I did check though that the record length = sum of field lengths - sizeof (deleted flag). We have enough information from the specification to know this must hold.

Did you check record lengths when validating the file? I did think about putting this in. But decided against it. I think there was a thread about this before that suggested you can't be guaranteed what the record length is for an individual record, all you can be guaranteed of is its max length, but you aren't guaranteed that a record will always be its max length.

My instructions say:
Data section. 
Repeat to end of file: 
1 byte "deleted" flag. 0 implies valid record, 1 implies deleted record 
Record containing fields in order specified in schema section, no separators between fields, each field fixed length at maximum specified in schema information

But they also say:
All text values, and all fields (which are text only), contain only 8 bit characters, null terminated if less than the maximum length for the field.

Which seems confusing to me. The first set of instructions sound like they are saying that each record will fill out its maximum allowed space. Whereas the second set of instructions sound like records may be stored where the fields are not filled out to the maximum allowed space - they can be shorter than the max allowed space and will in such cases be null terminated.

My data file is set up based on my understanding of the first set of instructions. All fields are padded out with empty spaces in my data file. If I assume this will always be the case then I can easily validate the file by checking that the number of bytes in the file:


 
Roel De Nijs
Sheriff
Posts: 10662
144
AngularJS Chrome Eclipse IDE Hibernate Java jQuery MySQL Database Spring Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sean Keane wrote:I think there was a thread about this before that suggested you can't be guaranteed what the record length is for an individual record, all you can be guaranteed of is its max length, but you aren't guaranteed that a record will always be its max length.

My program expects a fixed record length. If a file is provided where fields are null-terminated and not the length described in the database schema, an error will occur (because database file can't be read) or if it could be read successfully (with a bit of luck ) complete nonsense will be shown to the user.
 
David Byron
Rancher
Posts: 175
Clojure Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I checked the Magic Cookie, read the schema from the file, and then validated the remainder of the file against the schema. Validation checked total file size, correctness of flag on each record, and record count.

The schema-reading and validation were just about the first methods I wrote, and I used them mainly to check whether I was corrupting the database file as I developed the app. (I worked against the file directly rather than via a cache.) When the time came to submit my project, I saw no reason not to leave them as they were.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic