Win a copy of Head First Go this week in the Go forum!

Tan Ban

Greenhorn
+ Follow
since Jul 24, 2016
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
2
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Tan Ban

I'm trying to query Mongocollection from Java Program. Connection is LDAP. Everything looks OK, till I create a MongoCursor object. But at the line where MongoCursor object is created, I'm getting socket exception as follows

INFO: Closed connection [connectionId{localValue:2}] to "Host:Port" because there was a socket exception raised by this connection.

Message-Exception authenticating MongoCredential{mechanism=PLAIN, userName='uname', source='dbname', password=hidden, mechanismProperties={}}



I'm wondering, Am I following the right way to query a collection, to retrieve documents (here I'm using 'create date' criteria). Why Am I getting socket excpetion?

Here is what I tried so far-

3 weeks ago
@Dave Tolls

Is this a two-way comparison?
That is, you need to show what has changed in matching PK lines between the two files, but also when one file has a PK that is not represented in the other?



Yes it is. When one file has a PK that is not present in the other file, I need to highlight this in the end report.
10 months ago

Well, one step hich would halve the memory used is that you only actually need to store one of the files in memory, using your current technique.
You can then cycle through the other one a line at a time.



I will give a try and see how it works.

Dump one file into a database, whether SQL or NoSQL?



Sounds interesting. Could you please elaborate more...
10 months ago
Unfortunately this is not a line to line compare program, but a primary key column based compare. Meaning, Line1 in File1 can be at Line10,000 in File2 . My program should be intelligent enough to identify the row and compare both the lines. That is the reason why I store the file data in Lists. Also, i store the compare result in a HashMap with <Row, Compare output> and write to an excel file as Report in the end.

My code absolutely works fine with few thousands of records. Problem comes with Larger Inputs.

10 months ago
My requirement is to read and compare two CSV / TSV files with 10 million records each. Here is the methods I implemented for reading the files



Program runs fine for smaller inputs i.e, when input file size are few thousands of records. But when the input size grows big, I get java.lang.OutOfMemoryError, java heap exception.
What is the memory efficient way of reading the flat files?
Note: Earlier I faced similar issue with excel files. Resolved the issue using Excel Streaming Reader.


10 months ago
Compilation Error
Encountered when Java compiler cannot compile your code often because of bad syntax in the program ( I said often to make things simple but can't generalize).

Exception
Your program will pass compilation without any issues, but at some point in run time, program crashes due to poor logic. Eg:- Attempting to divide by zero, Accessing illegal array indexes
10 months ago
@Tim Moores
Thank you for pointing me to this. Sounds interesting.
I'm working on that API. Will let you know after implementing.
10 months ago
@Carey Brown
Thank you for your reply. By looking at your answer,I'm happy that you understood my program completely because all your suggestions are already implemented and tested with no luck.

Does your ExcelUtilities.readExcel(path) method convert a spreadsheet into lists and then delete the raw spreadsheet from memory?
Yes. After reading excel to List object, in the very next statement I've set workbook objects to null.

before calling writeToExcel() you already have the HashMap and no longer need dataSheet1 and 2, you could set them to null so that they could be GC'd
Done this as well.
10 months ago
@Carey Brown
I totally agree with you. Excel is not the right choice to store huge volume of data. But this is not in my hands. I do get input files from another source, in the form of excel.

If I understand correctly, are you asking me to convert Excel sheets to Database tables programmatically and compare? Is that the approach you are suggesting?
10 months ago
I get java.lang.OutOfMemoryError: GC overhead limit exceeded exception while comparing two excel sheets like this-

The above code is working absolutely fine with few thousands of excel records. But when size of the input sheets increases to, say 50k records in each excel sheet, program is running into java.lang.OutOfMemoryError: GC overhead limit exceeded
I understand -
  • java.lang.OutOfMemoryError: GC is thrown when Garbage collector is taking an excessive amount of time and recovers very little memory in each run
  • I'm trying to read huge volumes of data into two ArrayList<ArrayList<String>> objects which could be the root cause of this issue
  • To solve this, one option is to increase heap size mentioning the Heap size I want to extend as  Xmx - max heap size

  • But this is not an elegant solution. I may need this program to compare two excel sheets with one million records each by making it as an executable jar.
    So I'm planning to rewrite the entire program with right data structures and an efficient algorithm . Any help on this?
    10 months ago
    I have a JTable populated from database. My need is to add a column in the table with a checkbox for each row. User should be able to select any number of rows by checking the chekcboxes. Also a checkbox should be present on the column header to select All rows.

    How can I do this with Eclipse WindowBuilder?
    2 years ago