Win a copy of Zero to AI - A non-technical, hype-free guide to prospering in the AI era this week in the Artificial Intelligence and Machine Learning forum!

R Jarman

Greenhorn
+ Follow
since Feb 08, 2005
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by R Jarman

I'm reading a ResultSet that gets 1.5 million rows. I put the data into an ArrayList of objects and then do some processing on it. Before the processing, I close the ResultSet. I then get a connection to a different DB server and write the data in the ArrayList out to this DB. I then need to do this process again only reading and processing a different ResultSet. The problem is when I start the second process, I get an out of memory error. I have adjusted the heap size on the VM to -Xmx1024m. I close my ResultSet. I also tried giving back the memory of the ArrayList by parsing it in reverse and doing a remove() for each element. I'm not sure if this works or if I'm depending on the garbage collection. Any ideas on how I can manage the memory better?
10 years ago
I'm a little confused. According to the documentation, a Connection object is in AutoCommit mode by default. So every time I issue a executeUpdate(), it should be doing a commit on its own. Is this not correct? If so, did you turn off AutoCommit and then issue your own commit every 1000 - 10000 rows?

My problem: I have a table with approx 5 million rows. I need to grab the data in a text column in this table, tokenize it, write it back out to the table, and write the first word of that data out to another column in the same table.

My Solution: This only has to be run once so I'm not as concerned with elegance and speed as long as it works. My plan was to retrieve data into a resultset, process it, write the data to the db, then get more data into the resultset and keep going until I've processed the whole table. I have to use a 3rd party tokenize that is written in java so performing this on the server with SPs and SQL isn't an option.

I get the same error on the same row number every time which leads me to believe it is a memory problem but I'm not sure what it is or how to fix it. I can run this code on a table where there are only 500 rows and it executes fine. This is the error I get:


java.io.EOFException
MESSAGE: Can not read response from server. Expected to read 65 bytes, read 47 bytes before connection was unexpectedly lost.

I use two Connection objects, one for reading and one for writing. My understanding is that I have to completely read all of the rows before I can use the connection object for any other resultset since I'm getting a row at a time.

This is how I get the resultset. It is within a try/catch block:



This is how I process it. :



Do I need to instantiate a new PreparedStatement every time I write to the table? I'm using the same statement and simply setting the values in it every time.

I'm the only person using the DB so it isn't that someone else is trying to access the same data I'm accessing. The Java code is executing from my desktop. I do not have access to an application server.

I'm hoping someone can see my error or has an better solution to my problem.
I get the following error when trying to run the file

C:\My Program Files\cTAKES\chunker\scripts\java\data\chunk\genia>java -cp . Geni
a2PTB

Exception in thread "main" java.lang.NoClassDefFoundError: Genia2PTB (wrong name
: data/chunk/genia/Genia2PTB)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$100(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)


This is contents of the directory

C:\My Program Files\cTAKES\chunker\scripts\java\data\chunk\genia>dir
Volume in drive C has no label.
Volume Serial Number is 9883-718B

Directory of C:\My Program Files\cTAKES\chunker\scripts\java\data\chunk\genia

07/27/2009 03:16 PM <DIR> .
07/27/2009 03:16 PM <DIR> ..
07/27/2009 03:16 PM 552 Genia2PTB$1.class
07/27/2009 03:16 PM 1,755 Genia2PTB.class
03/15/2009 07:40 PM 2,579 Genia2PTB.java
3 File(s) 4,886 bytes
2 Dir(s) 37,659,451,392 bytes free

I have tried it with the -cp command and without. I don't have admin rights on the computer I'm using so I can't edit the System classpath. The directory I'm in has the file I'm trying to run. I have checked that and I thought putting the dot "." for the classpath would overwrite whatever is in the system classpath and just use the current directory. What am I doing wrong?
11 years ago
I'm having basically the same problem but I can't get rid of the classpath because I don't have admin rights to the machine. I downloaded a DB and it has a java file that does some configuring. I compiled the .java file fine. The .java and the .class files are in the same directory. I open a CMD prompt window and go to that directory and enter the javac command and get the same error message mentioned previously in the thread. I'm not sure how to fix this.
11 years ago
If it's not designed correctly, how would you go about designing a Log class? I'd like to be able to use this with other code I'm writing.

Thanks
13 years ago
For instance I can write

System.out.println (myInt);

and the system will convert the int to a string and write it out. I guess there is probably an overloaded version of println that takes an int, right?

I have created a toString method for my data object. I can call it explicitly and my log works. If I don't call it, my code doesn't compile. Normally I don't have to explicitly call a toString method. Does this also mean that I need to create a version of my write method that takes an Integer or explicitly call toString on everthing?

Here are my two methods in my Log class. I have taken some of the extra stuff out. I have found this class years ago and kept it until I found a need recently.

public static void init(String _fileName) {

//*******Create system dependent variable***********//

OSName = System.getProperty("os.name");

if (OSName.indexOf("Windows") >= 0) {

dir = "logs";

fileName = "\\" + _fileName;

} else {

dir = "logs";

fileName ="/" + _fileName;

}

}



public static void write(String str) {

//******************first check if the dir exists or not

file = new File(dir);

if (!file.exists()) {

try {

file.mkdirs();

} catch(Exception e) {

System.out.println(e);

}

}



//********Create or open the log File *************//

file = new File(dir + fileName);

if(!file.exists()) {

try {

file.createNewFile();

} catch(Exception e) {

System.out.println(e);

}
}
try {

fileOutputStream = new FileOutputStream(dir+fileName, true);

} catch(Exception e) {

System.out.println(e);

}



//************write the string to the file ************//

try {

str = str + "\n";

fileOutputStream.write(str.getBytes());

fileOutputStream.close();

} catch(Exception e) {

System.out.println(e);

}


If anyone has a more elegant way of writing to a log file without constantly opening and closing, I'm open for suggestions.

Thanks.
13 years ago
Here's what I want to do. I have a recursive method that builds combinations of things. It doesn't matter what. I'm generating data to use in some research. I want to write this data to a single file.

Here's what I tried. I tried creating a very large array of this data and write it out all at once but the array was too large and it was crashing. So I decided to write the objects out to a file as I created them. I created a Log class that has static methods to set the filename and write data to the file. I created them as static methods because I was under the assumption I needed it to be static to use the same file without passing it around. Kind of like using System.out.println.

Log.init ("test.txt");
Log.write ("test data");

Anyway, I can write strings fine. My problem occurs when I try to write out any other data type. It won't compile. I guess the toString method doesn't get called at compile time unless I explicitly call it?

My question is how can I write data to a single file from within my recursive method?
13 years ago
I'm using a ListIterator to parse through an ArrayList. I understand how to use it to parse the entire array but what if I'm looking for something in the array? Once I find the object I want to do some processing on it and I don't want to parse the rest of the array. How can I punch out of an ListIterator once a condition has been met or am I better off using the old way with for loops or while loops?
13 years ago
No, I'm using a PrintWriter which is supposed to encapsulate instantiating all of the other writer classes. It doesn't have an append option in it constructor.
13 years ago
I have a file that I am using as a log file. I want to be able to write to it from within different parts of my program. I don't want to clear the file everytime I open it. I created a method that takes two strings as arguments, one is the filename and the other is the string to write to the file. I instantiate a PrintWriter for the filename argument and writes the other string to my file. Since I instantiate a new PrintWriter everytime I call the method, it creates or overwrites the file, right? I use this method also to write other information to another file. Is there a way to tell the PrintWriter not to overwrite the existing file? If not, any suggestions on how I can write to a log file from several different classes?
13 years ago
I attached a quicksort I use. Part of your problem is the "2" at the beginning. This should be different everytime it is called. My quicksort is something I found on the internet and modified for my use. I sort an arraylist of objects and instead of checking the value in the array, I check a value in the object in the array. That's what my getDeliveryDate() method calls are. Hope this helps.

private void quickSort(ArrayList<Bid> elements, int lowIndex, int highIndex)
{
int lowToHighIndex;
int highToLowIndex;
int pivotIndex;
int pivotValue;
Bid lowToHighValue;
Bid highToLowValue;
Bid parking;
int newLowIndex;
int newHighIndex;
int compareResult;

lowToHighIndex = lowIndex;
highToLowIndex = highIndex;

/** Choose a pivot, remember it's value
* No special action for the pivot element itself.
* It will be treated just like any other element.
*/

pivotIndex = (lowToHighIndex + highToLowIndex) / 2;
pivotValue = elements.get(pivotIndex).getDeliveryDate();

/** Split the Vector in two parts.
*
* The lower part will be lowIndex - newHighIndex,
* containing elements <= pivot Value
*
* The higher part will be newLowIndex - highIndex,
* containting elements >= pivot Value
*/

newLowIndex = highIndex + 1;
newHighIndex = lowIndex - 1;

// loop until low meets high
while ((newHighIndex + 1) < newLowIndex) // loop until partition complete
{
// loop from low to high to find a candidate for swapping
lowToHighValue = elements.get(lowToHighIndex);
while (lowToHighIndex < newLowIndex && lowToHighValue.getDeliveryDate() < pivotValue )
{
newHighIndex = lowToHighIndex; // add element to lower part
lowToHighIndex++;
lowToHighValue = elements.get(lowToHighIndex);
}

// loop from high to low find other candidate for swapping
highToLowValue = elements.get(highToLowIndex);

while (newHighIndex <= highToLowIndex && highToLowValue.getDeliveryDate() > pivotValue)
{
newLowIndex = highToLowIndex; // add element to higher part
highToLowIndex--;
highToLowValue = elements.get (highToLowIndex);
}

// swap if needed
if (lowToHighIndex == highToLowIndex) // one last element, may go in either part
{
newHighIndex = lowToHighIndex; // move element arbitrary to lower part
}
else if (lowToHighIndex < highToLowIndex) // not last element yet
{
if (lowToHighValue.getDeliveryDate() >= highToLowValue.getDeliveryDate()) // low >= high, swap, even if equal
{
parking = lowToHighValue;
elements.set (lowToHighIndex, highToLowValue);
elements.set (highToLowIndex, parking);

newLowIndex = highToLowIndex;
newHighIndex = lowToHighIndex;

lowToHighIndex++;
highToLowIndex--;
}
}
}

// Continue recursion for parts that have more than one element
if (lowIndex < newHighIndex)
{
this.quickSort(elements, lowIndex, newHighIndex); // sort lower subpart
}
if (newLowIndex < highIndex)
{
this.quickSort(elements, newLowIndex, highIndex); // sort higher subpart
}
}
13 years ago
OK, that all makes sense. I thought Java handled all of the memory deallocation unlike C++.

If I have an arraylist that I'm not using should I iterate through it calling remove() to release the memory or leave it alone?

So do I enter the options before the program name or after?

java -Xms256m -Xmx512m MyProg

or

java MyProg -Xms256m -Xmx512m
13 years ago
I'm building a lot of objects dynamically. I read info from a text file into an arraylist. My program does fine for 200 items in the file. It crashes at 500 items. Once I have the items into the program, I parse the data into a 2 dimensional arraylist. Some of this data is removed because it doesn't meet certain conditions. Combinations are then made from the data items. So where I might have started with an arraylist of 100, my arraylist could now be 2200 or more. This is where it seems to crash. The goal is to then process these combinations and reduce the arraylist to a dozen combinations or so.

When I call arraylist item.remove() it removes the item from the arraylist. Doesn't the garbage collection run and clear those objects out of memory? Is there a trick help the garbage collector clean up all of those removed objects?
13 years ago
While running I'm getting the following exception.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Is there something I can enter when I run my program to increase the heap size?
13 years ago