• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Issue in reading large file on server, Servlet request time out

 
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am trying to upload large text file. Text file size would be around 100 MB (around 1500000 lines). I have to process each line from the file and have to store it into DB.
I am reading file on server(in a servlet ) using below code.


----- Above code works fine if I process file with around 10000 records, but when I process large file I am getting out of memory issue. I want to split large file into small file on server , store it in temp directory and wanted to load each file but when I am splitting the file sometimes half of the last line in file will present in one file and remaining line will present in another file first line(Example – File1-> ‘How are you’, File1->’How’ File2->’Are You’).
Also processing of file take lot of time and servlet request is getting timed out.

Please suggest any solution for this problem. If possible please give some example.
Thanks in advance for your help.
Yogesh…
 
Bartender
Posts: 2856
10
Firefox Browser Fedora Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Firstly from the design perspective, I think that the file upload servlet should not be doing the DB update, have a separate utility class for that to keep things simple.
If your problem is only the time-out, putting a small AJAX to simply META-REFRESH the page should solve the problem.
 
Amit Ghorpade
Bartender
Posts: 2856
10
Firefox Browser Fedora Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Also take a look at https://coderanch.com/t/496463/Servlets/java/EE-Session-Expires-Large-File
 
yogesh goswami
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Amit. I will try with 'AJAX META-REFRESH'

Could you please suggest something to split file on server, I am able to split files but issue is , file splitting using size instead of ’ No. of lines’ and because of that last line is also splitting((Example – File1-> ‘How are you’, File1->’How’ File2->’Are You’)).
I am placing file splitting code please suggest if anything can be done?

 
Rancher
Posts: 2759
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You don't really need to split file. You are adding unnescary complication to the code. You can read lines in batches (of let's say 1000 lines each) process them, then read the next batch
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Are you required to use a browser on the client side?

If not, a custom Java client would be much more flexible - for example it could send batches of 100 lines and keep track of how many lines have been processed so that you could resume processing if the connection was broken.

Seems to me the chance of a connection problem in the course of sending 100MB with DB processing on the server must be pretty substantial.

Bill
 
reply
    Bookmark Topic Watch Topic
  • New Topic