Bob Matthews

yes - that's the puzzler - line 122 is the middle line of code below

//Normalize the training data        DataNormalization normalizer = new NormalizerStandardize();        normalizer.fit(trainData);              //Collect training data statistics        trainData.reset();

Bob M

Finally all .csv files are correct BUT I now have the following error
which is odd since all my labels are in the range 0 thru 5

Exception in thread "main" java.lang.IllegalStateException: Cannot convert sequence writables to one-hot: class index 6 >= numClass (6). (Note that classes are zero-indexed, thus only values 0 to nClasses-1 are valid)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesSequence(RecordReaderMultiDataSetIterator.java:693)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:369)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:333)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:213)
at org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator.next(SequenceRecordReaderDataSetIterator.java:365)
at org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator.next(SequenceRecordReaderDataSetIterator.java:344)
at org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator.next(SequenceRecordReaderDataSetIterator.java:48)
at org.nd4j.linalg.dataset.api.preprocessor.AbstractDataSetNormalizer.fit(AbstractDataSetNormalizer.java:109)
at org.deeplearning4j.examples.recurrent.seqclassification.UCIExample_2.main(UCIExample_2.java:122)

Process finished with exit code 1

Bob M

Thank you for your advice

Bob M
New Zealand

Why I do not recommend to use your design with Java.util.List: you will need to iterate whole list each time when you need to update a value.

Thank you but I would prefer to stay with ArrayList <String, Integer> or Map<String, Integer> if I can

I am following an example of code for dl4j and ArrayList<> is used further on in the code

I only need to update all the integers once

Bob M

Ha....ha..........
I am a novice programmer, I don't know

List<Pair<String, Integer>> Name = new ArrayList<>();
It is populated by
Name.add(new Pair<>(String_a., Integer_a));
I was looking at ArrayList<Integer> arraylist = new ArrayList<Integer>();
arraylist.add(1); arraylist.add(2); etc to populate the ArrayList and then
arraylist.set(0, 11); arraylist.set(1, 22); etc to update the contents

I have an array list made up of pairs of strings and integers (one of each in each pair)

I also have a .csv column vector with integer values which I wish to update the array list with

How do I accomplish that ?

Bob M

p.s. do I use the arraylist.set(.....) command and if so how do I reference the integer in the pair?

Thank you

Paul: there are many confusing aspects of the dl4j example I am looking at

And your comment is just one of them

details of the example -

https://archive.ics.uci.edu/ml/datasets/Synthetic+Control+Chart+Time+Series

the example produces a huge number of .csv files - none of which are comma separated ???

The example appears to me (a novice) as highly inefficient in its methods of handling data

Bob M

At the moment I have a .csv file

When I look at it it looks like this
1 3 8 4 2 5 4

I would like to change it to
1
3
8
4
2
5
4

How do I do that ?

Bob M

The following code is where the .csv files are populated
and it is the features files that are ending up as row vectors rather than column vectors

//Do a train/test split:        int nTrain = 450;   //70% train, 30% test        int trainCount = 0;        int testCount = 0;        for (Pair<String, String> p : DataAndCats) {            //Write output in a format we can read, in the appropriate locations            File outPathFeatures;            File outPathLabels;            if (trainCount < nTrain) {                outPathFeatures = new File(featuresDirTrain, trainCount + ".csv");                outPathLabels = new File(labelsDirTrain, trainCount + ".csv");                trainCount++;            } else {                outPathFeatures = new File(featuresDirTest, testCount + ".csv");                outPathLabels = new File(labelsDirTest, testCount + ".csv");                testCount++;            }            FileUtils.writeStringToFile(outPathFeatures, p.getFirst(), (Charset) null);            FileUtils.writeStringToFile(outPathLabels, p.getSecond().toString(), (Charset) null);        }

The following code is my current version:-

The idea of the changes is to read the data from files on my computer rather than from the web
In addition, I have shuffled the data rows so that the categories are now random between 0 and 5

//This method loads the data, and converts the "one time series per line" format into a suitable //CSV sequence format that DataVec (CsvSequenceRecordReader) and DL4J can read. private static void downloadUCIData() throws Exception {        if (baseDir.exists()) return;    //Data already exists, don't download it again        // synthetic data        String url1 = "file:///C:/Synthetic Control/data2.data";        String data = IOUtils.toString(new URL(url1), (Charset) null);        String[] lines1 = data.split("\n");        // synthetic categories        String url2 = "file:///C:/Synthetic Control/cats2.csv";        String cats = IOUtils.toString(new URL(url2), (Charset) null);        String[] lines2 = cats.split("\n");        //Create directories        baseDir.mkdir();        baseTrainDir.mkdir();        featuresDirTrain.mkdir();        labelsDirTrain.mkdir();        baseTestDir.mkdir();        featuresDirTest.mkdir();        labelsDirTest.mkdir();        int lineCount = 0;        List<Pair<String, String>> DataAndCats = new ArrayList<>();        for (String line : lines1) {            String transposed = line.replaceAll(" +", "\n");            DataAndCats.add(new Pair<>(transposed, lines2[lineCount]));            lineCount++;        }

Unfortunately, my features .csv files end up as row vectors and they should be column vectors

So I am still wrong somewhere in the code ?

yes - line 33

I have altered my code but still in error ?

A problem with the categories (labels)

Error: NumberFormatException "[Ljava.lang.String;@7fc2413d"
which is what I see in 0.csv in labels

//This method loads the data, and converts the "one time series per line" format into a suitable    //CSV sequence format that DataVec (CsvSequenceRecordReader) and DL4J can read.    private static void downloadUCIData() throws Exception {        if (baseDir.exists()) return;    //Data already exists, don't download it again        // data        String url1 = "file:///C:/MATLAB Script Files/SyntheticControl/data.csv";        String data = IOUtils.toString(new URL(url1), (Charset) null);        String[] lines1 = data.split("\n");        // categories        String url2 = "file:///C:/MATLAB Script Files/SyntheticControl/cats.csv";        String cats = IOUtils.toString(new URL(url2), (Charset) null);        String[] lines2 = cats.split("\n");        //Create directories        baseDir.mkdir();        baseTrainDir.mkdir();        featuresDirTrain.mkdir();        labelsDirTrain.mkdir();        baseTestDir.mkdir();        featuresDirTest.mkdir();        labelsDirTest.mkdir();        int lineCount = 0;        List<Pair<String, String>> DataAndCats = new ArrayList<>();        for (String line : lines1) {            String transposed = line.replaceAll(" +", "\n");            DataAndCats.add(new Pair<>(transposed, lines2[lineCount]));            lineCount++;       }

Hello Liutauras

Thanks for the comment which I will take on board

I have a general question regarding the code from the example.......................

It ends up creating 600 separate .csv files for the data and 600 separate .csv files for the categories

If one wishes to classify huge amounts of data, should I be looking for more efficient data handling processes

Bob M

Hi

The following code is from the UCI Classification Example under the dl4j umbrella
The data is read in from a website via a URL
The data has 600 records, the first 100 are category 1, the next 100 are category 2 etc.

//This method downloads the data, and converts the "one time series per line" format into a suitable    //CSV sequence format that DataVec (CsvSequenceRecordReader) and DL4J can read.    private static void downloadUCIData() throws Exception {        if (baseDir.exists()) return;    //Data already exists, don't download it again        String url = "https://archive.ics.uci.edu/ml/machine-learning-databases/synthetic_control-mld/synthetic_control.data";        String data = IOUtils.toString(new URL(url), (Charset) null);        String[] lines = data.split("\n");        //Create directories        baseDir.mkdir();        baseTrainDir.mkdir();        featuresDirTrain.mkdir();        labelsDirTrain.mkdir();        baseTestDir.mkdir();        featuresDirTest.mkdir();        labelsDirTest.mkdir();        int lineCount = 0;        List<Pair<String, Integer>> contentAndLabels = new ArrayList<>();        for (String line : lines) {            String transposed = line.replaceAll(" +", "\n");            //Labels: first 100 examples (lines) are label 0, second 100 examples are label 1, and so on            contentAndLabels.add(new Pair<>(transposed, lineCount++ / 100));        }

I am changing the code to suit my circumstances
I have two files on my computer - one is data.csv and the other cats.csv
When I try to run my altered code(see below)  I get an error on the line

DataAndCats.add(new Pair<>(transposed, lines2));

The error is "incompatible types"
Can somebody explain what is wrong here

Bob M

//This method loads the data, and converts the "one time series per line" format into a suitable    //CSV sequence format that DataVec (CsvSequenceRecordReader) and DL4J can read.    private static void downloadUCIData() throws Exception {        if (baseDir.exists()) return;    //Data already exists, don't download it again        // data        String url1 = "file:///C:/MATLAB Script Files/SyntheticControl/data.csv";        String data = IOUtils.toString(new URL(url1), (Charset) null);        String[] lines1 = data.split("\n");        // categories        String url2 = "file:///C:/MATLAB Script Files/SyntheticControl/cats.csv";        String cats = IOUtils.toString(new URL(url2), (Charset) null);        String[] lines2 = cats.split("\n");        //Create directories        baseDir.mkdir();        baseTrainDir.mkdir();        featuresDirTrain.mkdir();        labelsDirTrain.mkdir();        baseTestDir.mkdir();        featuresDirTest.mkdir();        labelsDirTest.mkdir();        int lineCount = 0;        List<Pair<String, String>> DataAndCats = new ArrayList<>();        for (String line : lines1) {            String transposed = line.replaceAll(" +", "\n");            DataAndCats.add(new Pair<>(transposed, lines2));        }