Emilio Gagliardi

Greenhorn
+ Follow
since Jun 02, 2004
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Emilio Gagliardi

Loren, thank you very much for your advice and support. I will check out safari and hold my breath for 1.5

cheers,
17 years ago
Hi Loren!
Thax for that.
What does Safari mean in the context you provided it? I don't mind reading anything online, I'm just not sure what you are refering to there. :/

Also, we are talking mac here, so we don't have access to 1.5 and so I should continue programming as though nothing exceeds the largest integer value? Last question, the 2 GB limit, by what you've said I am restricted to 2^31-1 objects to a collection, whose total memory may exceeed 2 GB. So, I could create 1 variable that requires 6GB worth of memory and java will be able to manage the memory?

Cheers,
17 years ago
Hi there, I'm about to start working on a G5 that has a 64-bit processor and 8 GB or RAM. What should I be aware of from the start about how to program Java in this environment? In particular, are there settings in the JVM I need to modify? Do i do anything different with my code? For example, if I need a data structure that is 6GB do I address it like normal? I'm trying to track down the "Mac OS X for Java Geeks" book, does it contain any material on this topic?

Cheers,
17 years ago

Make sure your classpath is set accordingly and just use the javac and java commands on the command line (or better yet, Ant and Ant scripts).

The note merely states that you should have the JAVA_HOME environment variable set appropriately as some software installations expect it to be set.



I'm not sure how to do any of that actually. I don't know what an Ant script is. I would like to run my program from the command line on a mac. Where should I put my source files, where should my compiled code go and where do I stick third party jar files so that it meets Mac standards?

Any advice is grately appreciated!
17 years ago
Thank you very much for that! Can I ask a further question? Can you explain why I can't set a FileChannel's position to greater than Integer.MAX_VALUE and yet I can pass 19 gigs worth of doubles through the ByteBuffer and FileChannel to a file in the first place? That is the original problem that brought me here. The code I posted was my debugging efforts to figure out how to access portions of a single 19 gig file.

Since then I have reduced my files to hold no more than Integer.MAX_VALUE worth of bytes but it is a real hack job and still doesn't work. I keep getting IllegalArgumentException. Grrr...

Thank you for any suggestions.
17 years ago
Hi there,
I've been searching all day trying to understand how Java works on Mac and I'm getting lost. I've located the directories that everything is supposed to happen in System/library/frameworks/javavm.framework but after that I'm stumped about what everything is and what to modify. The first thing I need to do is install JDK 1.5 beta. But Mac won't let me create a folder under Versions? Also, there wasn't any download at Sun for a "mac" version so I Downloaded the unix version, except the unix version is .bin file.

JAVAHOME

Some applications look for Java�s home directory (JAVA_HOME) on the users system, especially during installation. If you need to explicitly set this, in a shell script or an installer, set it to /Library/Java/Home. Setting it to the target of that symbolic link can result in a broken application for your customers. Programatically you can use System.getProperty("java.home"), as you would expect.

/Library/Java/Home also contains the /bin subdirectory where symbolic links to command-line tools like java and javac can be found. These tools are also accessible through /usr/bin.

Hoe does this relate to running Java programs from the command line? That I should put my .java files in Java/Home and all other archive files in Extensions? Good grief. And I thought Macs were user friendly...
Cheers,
17 years ago
OK, this is getting weirder!
In an attempt to solve the read problem I modified the write code as follows

Now, when I run the above code fileSize_0 starts at zero, and increases by 390784 bytes as expected.
However, when the file reaches Integer.MAX_VALUE in size, instead of returning the fileSize as expected this is the output I get:
and fileSize_0 continues to decrease in magnitude.

Can anyone decipher this?? This is beyond my understanding. What I don't understand is that in my original code, I wrote 50000 word vectors to file which created the original 19 gig file. How can the size of the file as reported by the FileChannel get smaller when in fact its getting bigger??? Could this be part of the reason I can access locations larger than Integer.MAX_VALUE?

Cheers,

[ June 16, 2004: Message edited by: Emilio Gagliardi ]
[ June 16, 2004: Message edited by: Emilio Gagliardi ]
17 years ago
Hi there!
I am attempting to use a FileChannel to read the contents of a file that is 18.6416 gigs of doubles. The specific error is an IllegalArgumentException generated at this line:

where j is a integer set in a for statement (it is never negative) which means that position was being set to a number larger then MAX_VALUE = 2147483647 the point in execution where the error occurs when there is an attempt to set the position value to larger than 2147748864.
What confuses me is that I wrote this file with a fileChannel in the first place. I wasn't manually setting position though.

Is there a simple workaround to this or do I need to write files smaller than MAX_VALUE in size in order to manually set where in the file I want to read? I need to be able to specify the position because I am looping over the file and I want to read at specific intervals (390764 bytes) that do not overlap. Is there another class that can handle large file sizes? Or is it that i'm asking too much to store all this data in one file?

Thanks for reading!~!
17 years ago
HI Tim;
I think after this project I'll owe you a beer or three

I need to calculate the Euclidian distance between each vector. The unique part comes from the fact that the distance between vector m and n is the same as the distance between vector n and m. Meaning, once I've computed the distance between any two vectors (a compared to b), I don't need to calculate the distance from the perspective of the other vector (b compared to a) nor do I need to calculate the distance between a and a!

How exactly do you compute the hashcode of an array?? That sounds promising! I'm curious about how to compute hashcodes but I don't really get what is hashed.

Cheers,
17 years ago
I am working on a project where one of the goals is to allow us to compare arrays of doubles. The data is currently stored in a file as 50000*50000 doubles; think of it as 50000 vectors with 50000 elements in each.
I need to process each vector to every other vector but I don't need to process a vector against itself or repeats. Futhermore, I can't store all the vectors in memory. What I need to do is this:
OUTER LOOP
read vector[i]//all from same file
INNER LOOP
read vector[j] //all from same file
compare and store value
i + j would look like this:[0,1][0,2][0,3][0,4][1,2][1,3][1,4][2,3][2,4]...etc, progressively iterating over the file until no more unique comparisons are possible.

Now the part I don't really understand is how I can keep track of which pairs I have compared so I'm not comparing [0,0][1,0][1,1][2,0][2,1][2,2] because either they already exist (0,1 is the same as 1,0) or I don't want to make the comparison (1,1). At first i thought I would simply load all the vectors into memory, but then realized I can't. But that leaves the problem of keeping track of which ones I have compared. Part of the problem is that I need to keep track of the values I produce when I compare the vectors so that I can sort them later.

That leads me to a whole new problem of how to store the vector comparisons, but that is further down the road.

SO I called this architectural advice because at this point I'm not sure how to think about processing the data and which data structures I should investigate utilizing. One glimor of an idea is that I could use a MappedByteBuffer but I'm not really sure how to implement that to solve my problem.

Any advice or suggestions are greatly appreciated.
[ June 10, 2004: Message edited by: Emilio Gagliardi ]
17 years ago
This is probably a dumb question but are CSV files written as char?
As in, if I want to store 123456 as an actual integer it would take 4 bytes, as opposed to 12 bytes to store the six characters...is that right? If so, then I must write the numbers and commas and end-of-line characters.

Thank you for your input. =)

Cheers,
17 years ago
Hi just a quick question, what is the best way to store primitive types when writing to a file so that other programs can read/import the data? For example, excel or SPSS.

Cheers,
Emilio
17 years ago
Thanks for your suggestions! I think I can perform all my calculations on a row by row basis, so I only ever need to load one or two rows.
If we have to do more fancy math then I'm sure identifying the math will reveal further requirements.

Cheers,
Emilio
17 years ago
Hi all;
I'm sure this is simple!
I just installed java 1.5.0 beta 2 and want to verify that my IDE is using the newest java runtime. The install directory was: C:\utilities\j2sdk1.5.0_beta2
so I added the following line to my user and system PATH variable:
C:\utilities\j2sdk1.5.0_beta2\bin

Is that all I need to do? I'm using netbeans 3.6. I just need to make sure I'm using 1.5.0 because of a library that I found uses the 1.5.0 JDK.

Thank you very much.
17 years ago
Hi Stefan and Tim; Thank you for your insights!
In my limited computing knowledge, I really don't know how to do much. Simultaneous writing to disk? I naively thought that IFF I knew that each logical chunk consisted of 50000 integers, I would know ahead of time which byte range I could allocate to an object in charge of writing and so long as the data was put into the correct region it wouldn't matter *when*. But order does matter as we need to know which 50000 integers belong to what key.

Now, on the topic of keys, I hadn't even considered storing a key with the data. I just presumed that if the order was known ahead of time I didn't need to store a key. In terms of performance, can I achieve better performance if I don't restrict myself to a sequential OI scheme? I have to admit that the notion of creating 50000 files on my computer scares the bajeebaz out of me :/

Tim, I'm not sure how to coordinate my IO, data structure algorithm at this point because the math is opened. The first math I need to do is simple: I need to aggregate 50000 2D matrices into 50000 1D matrices. Currently, I'm just summing the contents of the cells up as I go. I also need to normalize each 1D matrix, but I don't know if that means against itself or the entire 50000 aggregated matrices. For example, one method I can use is to simply divide each cell in a 1D matrix by the largest value in that 1D matrix =OR= by the largest number from the 50000 aggregated matrices? Finally, I need to calculate the Euclidian distance between each 1D matrix. So, I was going to do some of my bookkeeping on the ByteBuffer as there are some utilitt classes that I found to operate on ByteBuffers directly, then I was going to use the underlying array as the container to pass the data to my file writing method. Thats it!

Can you explain how one can implement non-naively?
The matrices will be mostly sparse so I guess thats good? Finally, I ran a test last night and it took 181 seconds to write 10000 1D matrices with 50000 integers in each. The resulting file was almost 2 gigs. Truth be known, I have no idea if thats bad, or how bad that is...and I imagine that the time will change dramatically between machines and JVMs. The computer this will ultimately run on is a G5 with dual processors and 160 gigs of hardrive space. But I have no idea about Macs at all.

Thanks again for your advice!

Cheers,
Emilio
17 years ago