• Post Reply Bookmark Topic Watch Topic
  • New Topic

Dazed and Confused

 
William Zimmerman
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I posted this on the intermediate Java forum, and they suggested I ask here as well:
I am designing a data acquisition, processing and display system that will use several processing nodes (linux pc) to distribute the load, and are considering using Java. The system will run stand-alone (not on the www), will have lots of scientific plots (x-y, color contour, waterfall) and performance is important. I have a rudimentary knowledge of Java as a programming language, and am aware of concepts such as EJB and JMS, but I am overwhelmed with the options available to me (and my team of 4 s/w engineers). My thoughts are to use Jbuilder and JClass as my development environment, but before I hurtle head-first into a design and development effort, I am wondering the following:
* What IDE, (IDEA, JBuilder, Eclipse) if any is appropriate for this type of system?
* Should I be concerned about MVC and other Patterns?
* What about Struts?
* What is the best way to integrate TCP/IP into Java?
* Are there any good references to using Java for stand-alone apps?
* What Java technologies should I be looking at?
I realize that I am asking a lot. I am looking for guidance as to what and where to look for information, not necessarily answers. Thanks.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Setting aside for a moment the GUI interface problems, if this was my problem I would want to first think about the data gathering architecture. Questions like:
What is the rate of data point aquisition per node?
How big (memory wise) is a single data point?
How precise does the timing have to be?
What fraction of dropped data points is acceptable?
How should raw data points be logged?
Is the node network fixed or flexible?
Bill
 
William Zimmerman
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
>What is the rate of data point aquisition per node?
more than 300 channels of 24-bit data (~18 MB/sec aggregate rate) comming in a single ATM/SONET spigot.
>How big (memory wise) is a single data point?
Data ensembles will be at least 300K values
>How precise does the timing have to be?
Latency is not a big issue, but data skipping is frowned upon
>What fraction of dropped data points is acceptable?
1 in 10^9 would be nice. This system will only be running the aforementioned task, so we may be able to get away with UDP even though there are no guarantees with this protocol. I have thought of TCP, Corba and NDDS, but I am concerned that if I set up guaranteed delivery, they will not be able to keep up.
>How should raw data points be logged?
Raw snapshots of data may be accumulated in a raid, and then backed up to a SAIT tape drive
>Is the node network fixed or flexible?
It will be fixed, once we determine what we need.
 
Chris Shepherd
Ranch Hand
Posts: 286
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Wow, when I started reading your description I thought, "He's trying to duplicate MY project". When I read more closely, I can see the differences, but I have already done something sort of like what you want to do.
We read up to 24 channels of data every half second, pulling in ~20k of data. Your data rate is alot higher, but I do a heck of a lot of data massaging every half second. We run our server on an ETX(pc on a single board) board with custom hardware mated to it to do the data reading from the probes. We had a driver written for linux to allow us to read the data from the hardware. This system is totally stand-alone. When the linux box starts up the software starts up and shutting down the software shuts down linux. We provide polar,X-Y, orbit,Frequency scan,and waterfall plots, all usable to review both saved data and real-time data. I have used CodeGuide and moved to IDEA about a year ago. I created all the plots with JClass 5(6 is the most recent I think). I don't distribute the collected data for load balancing as its all done on the server, but the server can distribute all of its data via the network and it can be run as a standalone unit. The client side of the program connects to the server using RMI. It collects the live data from the server and does all the display and analysis. The server hardware runs a client and remote PCs can also run a client and both see the exact same real-time data at almost the same time(network lag time is the only difference).
What you are trying to do, sounds quite doable, as I've done something similar to it. I can say that using java for stand alone apps works quite well. You may have some display slowdowns if you are trying to display large amounts of data on the plots. I was able to speed things up by using filtering on some of the worst offenders, and I have some really quite usable plots that are quite responsive to user interface.
Oh What kind of storage are you planning to use? I use firstSQL, which is a native java database. It is written in java which lets it port between OS with absolutely no problem. Iv tarballed a DB on the linux hardware, used putty to pull it into my windows box, unzipped it into place for my program and started up the program on windows. No problems.
My suggestion to you is to start small. Build a core of the data collection and make sure that works. Then just layer on the additional features. Probably the most important thing to get right from the start tho is to create your data structures correctly(both in your DB and in memory). I have had to rework my data structures 3 times since I started my project, and each time has been a major pain. If you get that right from the start, adding the features you want should be alot easier.
Oh, and threads. Gotta use threads everywhere it makes sense to split the processing up. You don't want your data display to get in the way of your data collection. One other thing to keep in mind(man this just keeps getting longer) for display purposes. There is one (and only one) display update thread. Use it wisely and keep extra junk out of its thread. Spawn a separate thread to handle lenghthy calculations if you need to. If the display thread poluted/stalled with extra gunk, it can slow down your display updating which causes terrible looking screens after a window popup call or drag.
Man this got long. Hope there's something here to help you get started..
Chris
[ May 04, 2004: Message edited by: Chris Shepherd ]
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
more than 300 channels of 24-bit data (~18 MB/sec aggregate rate) comming in a single ATM/SONET spigot.

That is a pretty impressive data rate - I suspect you are not going to be able to use any of the message/object passing APIs and will end up with permanently open socket based binary transfer of big data blocks. Also you may be re-using byte[] alot rather than creating new ones.
Are you able to specify multi-cpu hardware? Java Threads can work very happily with multiple CPUs if carefully designed.
Bill
 
William Zimmerman
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We are pretty free to spec out the system. Will probably go with dual 2.4 GHz Xeon 1U servers. There is concern about heat dissapation, or we would go faster. Possible use of dual-G5, but we have to wait and see.
What exactly do you mean by "permanently open socket based binary transfer of big data blocks"? Anything other than UDP?
[ May 05, 2004: Message edited by: William Zimmerman ]
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13078
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'm talking about using java.net.ServerSocket and Socket to open a connection and then working with the InputStream and OutputStream. Thats about the most primitive network connection level you can work with in Java. I have been trying to find an example I did, but that was several years ago.
Bill
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!