• Post Reply Bookmark Topic Watch Topic
  • New Topic

Web App and large statistical data

 
frank jardillier
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi !

i have to develop a web application managing statistical data. The amount of data is counted in one hundred million lines of numbers. The objective is to display html pages within some tables, charts and maps. These pages would be generated on the fly. Data used in these pages could be computed (formula) using others data (like ratio, evolution, percentage ...). There is a formula for each displayed information.
The data is stored in database, of course. One of the objective is to allow the final user update data and add data into the database. Then this information could be used into the formula from the final html pages. These new data are negligible in quantity but they are an important functionality.

Well...my questions are simple... is j2ee enough to build that? I think so, but what kind of api i have to use? ORM with JPA? using EJB for the business layer? MySql, Oracle or PostgreSql? Just a single MVC framework, struts or jsf? using RIA (javafx) or just simple Htpp application is enough? well.... i really dont know how to begin, what tools use, how to measure and anticipate the performance.... if you have any information, way to guide me, web ressources, books, idea of implementing this...you are welcome thak you !
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Massive number crunching does NOT belong in a web application. A stand alone application -or- a distributed grid should handle the number crunching, graphics generation, etc. etc.

The web side should only provide a user interface to do queries, updates, requests for graphics etc.

Do you have an estimate on the time to do a single run through this hundred million line number set? Is the job so large that a computing grid might be useful?

Bill
 
frank jardillier
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thank you for your answer.

well, exactly i dont know if this project can is possible to develop or not. I think this is very big objective near to be dream. So, i want to approach expert who can help me (us) to know if the project could be real or not.
Your response helps me. Grid is a way to explore , but it requires experts, we dont have any so i have to work to know more about this.

An architecture which we have thougt could be :
- optimizing database only for select requests from a web gui
- develop a batch process (like background task) to update/insert data during the night or weekend; so no real-time integrating data.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
An architecture which we have thougt could be :
- optimizing database only for select requests from a web gui
- develop a batch process (like background task) to update/insert data during the night or weekend; so no real-time integrating data.


I think you are on the right track. It is an interesting problem and many people are dealing with similar situations. When you make some progress, please use this forum to let us know how you are doing.

Bill
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!