Forums Register Login

Web App and large statistical data

+Pie Number of slices to send: Send
Hi !

i have to develop a web application managing statistical data. The amount of data is counted in one hundred million lines of numbers. The objective is to display html pages within some tables, charts and maps. These pages would be generated on the fly. Data used in these pages could be computed (formula) using others data (like ratio, evolution, percentage ...). There is a formula for each displayed information.
The data is stored in database, of course. One of the objective is to allow the final user update data and add data into the database. Then this information could be used into the formula from the final html pages. These new data are negligible in quantity but they are an important functionality.

Well...my questions are simple... is j2ee enough to build that? I think so, but what kind of api i have to use? ORM with JPA? using EJB for the business layer? MySql, Oracle or PostgreSql? Just a single MVC framework, struts or jsf? using RIA (javafx) or just simple Htpp application is enough? well.... i really dont know how to begin, what tools use, how to measure and anticipate the performance.... if you have any information, way to guide me, web ressources, books, idea of implementing this...you are welcome thak you !
+Pie Number of slices to send: Send
Massive number crunching does NOT belong in a web application. A stand alone application -or- a distributed grid should handle the number crunching, graphics generation, etc. etc.

The web side should only provide a user interface to do queries, updates, requests for graphics etc.

Do you have an estimate on the time to do a single run through this hundred million line number set? Is the job so large that a computing grid might be useful?

Bill
+Pie Number of slices to send: Send
thank you for your answer.

well, exactly i dont know if this project can is possible to develop or not. I think this is very big objective near to be dream. So, i want to approach expert who can help me (us) to know if the project could be real or not.
Your response helps me. Grid is a way to explore , but it requires experts, we dont have any so i have to work to know more about this.

An architecture which we have thougt could be :
- optimizing database only for select requests from a web gui
- develop a batch process (like background task) to update/insert data during the night or weekend; so no real-time integrating data.
+Pie Number of slices to send: Send
 

An architecture which we have thougt could be :
- optimizing database only for select requests from a web gui
- develop a batch process (like background task) to update/insert data during the night or weekend; so no real-time integrating data.



I think you are on the right track. It is an interesting problem and many people are dealing with similar situations. When you make some progress, please use this forum to let us know how you are doing.

Bill
I guess everyone has an angle. Fine, what do you want? Just know that you cannot have this tiny ad:
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com


reply
reply
This thread has been viewed 794 times.
Similar Threads
How do you set up Tomcat as a web server?
JSP to Access 2000 database
Dynamic includes vs. duplication of data
how to choose into the j2ee for a web application with large amount of data
New web site design technologies??
Thread Boost feature
More...

All times above are in ranch (not your local) time.
The current ranch time is
Mar 29, 2024 02:10:06.