Validating the Data File

[Sean Keane]

Before you attempt to read from the data file, to what extend did you guys go to validate the data file?

At a very basic level you could just check that the magic cookie is the correct value.

You could check other things like:

* The file length is at least the length of the schema.

* If you pad fields with empty spaces (rather than using the null terminated character) then you could check that the entire file is the length of the schema plus a multiple of an entire record length.

[Roberto Perillo]

Although I think this would be necessary in the real world, I think this isn't really necessary for this assignment... what I did was verify if the magic cookie value present in the file being read was the same as the magic cookie present in the file that was provided to me in the assignment (257, if I make no mistake). But some people passed and didn't even consider doing that because this isn't a requirement. So, if you don't do any validation, then it is ok. Just don't forget to mention your choice in your choices.txt file because (in my opinion) this is an important point of the assignment.

[Roel De Nijs]

I just verified the value of the magic cookie. Nothing more, nothing less.

[Oladeji Oluwasayo]

I didn't do any validation. The only thing is that my file chooser (in the gui) comes with a filter that allows only .db files. If an invalid database file is chosen, the display will be blank. If the file does not exist (maybe a path was just typed into the database location textfield, the file will be created. I didn't even think of this validation.

[Sean Keane]

Thanks guys. I think I will just validate the magic cookie and ensure that the file is at least as long as the schema. Then any other issues with the file when reading I will just throw runtime exceptions as I process the file - for example if there are not enough bytes for a record.

[Roel De Nijs]

On the server I only checked the magic cookie. At the client I tested a few other things: only .db extension, length bigger than 0, writable, readable,...

When something went wrong on the server while reading the file, an exception is thrown (which is caught by the client and the user gets an appropriate message).

[Sarah Archer]

I took a different view on this.

I didn't check my magic cookie because the instructions did specify which values were valid. It seemed to be more a version number, in hex 0x101, and perhaps compatibility is maintained for minor version number increments - you don't know.

I did check though that the record length = sum of field lengths - sizeof (deleted flag). We have enough information from the specification to know this must hold. You don't have to do it because it is not in the spec, but it is a simple check that will catch a lot of errors including bugs in reading the data file or a corrupt schema.

Finally, I wrote the code in a similar manner to the way I would write production code - lots of logging etc.

[Sean Keane]

Sarah Archer wrote:I did check though that the record length = sum of field lengths - sizeof (deleted flag). We have enough information from the specification to know this must hold.

Did you check record lengths when validating the file? I did think about putting this in. But decided against it. I think there was a thread about this before that suggested you can't be guaranteed what the record length is for an individual record, all you can be guaranteed of is its max length, but you aren't guaranteed that a record will always be its max length.

My instructions say:

Data section. 
Repeat to end of file: 
1 byte "deleted" flag. 0 implies valid record, 1 implies deleted record 
Record containing fields in order specified in schema section, no separators between fields, each field fixed length at maximum specified in schema information

But they also say:

All text values, and all fields (which are text only), contain only 8 bit characters, null terminated if less than the maximum length for the field.

Which seems confusing to me. The first set of instructions sound like they are saying that each record will fill out its maximum allowed space. Whereas the second set of instructions sound like records may be stored where the fields are not filled out to the maximum allowed space - they can be shorter than the max allowed space and will in such cases be null terminated.

My data file is set up based on my understanding of the first set of instructions. All fields are padded out with empty spaces in my data file. If I assume this will always be the case then I can easily validate the file by checking that the number of bytes in the file:

schemaBytes := number of bytes for schema
recordBytes := number of bytes for a single record
deletedFlagBytes := number of bytes for deleted flag
fileBytes := total number of bytes in file

if (fileBytes - schemaBytes) % (deletedFlagBytes + recordBytes) == 0 then
  we have a valid file
else
  we have an invalid file
endif

[Roel De Nijs]

Sean Keane wrote:I think there was a thread about this before that suggested you can't be guaranteed what the record length is for an individual record, all you can be guaranteed of is its max length, but you aren't guaranteed that a record will always be its max length.

My program expects a fixed record length. If a file is provided where fields are null-terminated and not the length described in the database schema, an error will occur (because database file can't be read) or if it could be read successfully (with a bit of luck ) complete nonsense will be shown to the user.

[David Byron]

I checked the Magic Cookie, read the schema from the file, and then validated the remainder of the file against the schema. Validation checked total file size, correctness of flag on each record, and record count.

The schema-reading and validation were just about the first methods I wrote, and I used them mainly to check whether I was corrupting the database file as I developed the app. (I worked against the file directly rather than via a cache.) When the time came to submit my project, I saw no reason not to leave them as they were.