We have an application which performs a very heavy operation which consumes a very High memory , cpu and takes a hell lot of time.
We have a hierarchical sets of data which is in a file, we read that file and create java objects from that and store in solr index.
Next time when we modify that file we again give to the application, which will read the file create java objects, it will fetch the appropriate data from solr.
We compare the new data with the old which we retrieved from solr and update the solr data for any modified or new data.
Say if there are around 30 lakhs of such java objects, it takes around 30 minutes to process and the memory consumption is around 35 GB, the cpu usage varies from 10% to 60%.
Can any one please suggest what would be the best approach to process such huge hierarchical data?