• Post Reply Bookmark Topic Watch Topic
  • New Topic

Max threads supported by OS?  RSS feed

 
kriti sharma
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am designing a java application which uses flat XML files as database. To deal with the concurrency issue, I have written a multithreaded database server. Each time a client connects to the database approximately 30 threads are spawned. This java application is designed for the use of a maximum 100 clients. If we take the worst case that all 100 clients connect at a time i.e. 100*30 =3000 threads. Will a P4 system be able to support so many java threads at a time?are there any memory constraints?if so please tell me, i will arrange for that amount of RAM or hard-disk. Please tell me definitively. Is my design ridiculous? I have no other option. Please help.
 
Ernest Friedman-Hill
author and iconoclast
Sheriff
Posts: 24217
38
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why do you spawn 30 threads for each client? I can't think of a scenario where this makes sense. Can you explain?

The maximum number of threads is platform-dependent. On at least some versions of Windows it's 2000.
 
kriti sharma
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i am giving one thread for access to each table which is a xml file. if a transaction involves simultaneously reading and writing to 10-15 tables(which may be the worst case of database design ) 10-15 threads will be made. also each read/write thread is called from another thread of the base class. the reason being that if 10 read/write threads are to be created and even a single one of them has to wait for table access(in a while loop) no next thread can come to existence till this thread unblocks,kinda risky situation. i tried to give a thread cover to each read/write thread so that even if a read/write thread blocks,the base thread has no other operation to get blocked.this funda worked. hence 30 threads per client. i have yet to design the database and write the read/write methods in JDOM, so am not very sure if the number of threads per client will be more or less than 30.DOES MY DESIGN SOUND RIDICULOUS???
 
kriti sharma
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i have read surfacially about stack and heap size setting of JVM.but i do not know how to modify or view existing settings. i am having both jdk1.3 and 1.4 versions.

i have read that default stack size for a thread maybe 256kb,512kb or max 1mb.if we assume that the jvm setting is 1 mb,then 3000X1mb=3GB.if we set the heap size of jvm to say 5 GB, my database server will work on the machine?or are there any other constraints with respect to so many threads.

also what should be other essential system configurations to support this database server, P4-2.8 GHz, 512 MB RAM?anything else.

Thanks,please do answer my questions.
 
Yaroslav Chinskiy
Ranch Hand
Posts: 147
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Kriti,

I am sorry to say it by 3000 threads sounds rediculous. I dont know all the req. for ur project but there is definatly a way to implement it with less threads. For example you can use NIO.

In ur app. do u need to notify the client about the request complition? U can have set of threads that control DB and each of them will have a request Q to process changes sent from the clients. If you need to send the reply then client can wait for it (based of the socket dicussion they are connected all the time). You can have request/reply objects that will allow ASYNC implementation with fewer threads.
 
kriti sharma
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yaroslav :

tell me what do commercial rdbms providers do to give concurrency and transaction management. they too use so many or even more threads.but they do it in c/c++ or any other language other than java, at bit/byte level, that is why the memory overhead is less.am i right?

the problem is that i have technology crisis and i cannot write the server in c/c++. i just want to make a decently robust application.thats it. so even if 3000 threads sound funny,digest it.more so becoz it is the worst case scenario only for testing purpose. please tell me about the stack and heap sizes and system configuration, as i have mentioned above.i just want to go on with the design.

i did like your idea about the request queue. i will implement it to line up requests for similar operations.this will definitely cut down the number of threads.
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'd rethink the threading design for sure. Heavily loaded systems find that one thread per incoming request can be a burden, so 20 or so per request will be a problem at 5% of the load.

Think about one thread per file that reads a queue of read and write requests in FIFO order. Look for literature on Command and Active Object patterns. Or maybe a configurable number threads that serve a queue per file in round robin fashion.

Did you see the little program I posted in response to your other question on threading the read and write requests? I was able to demonstrate that if a lot of read requests come in quickly the write requests might never execute. That's why I'm changing my tune to FIFO this time around.

http://www.surfscranton.com/files/Locker.zip
 
Stefan Wagner
Ranch Hand
Posts: 1923
Linux Postgres Database Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I found a page about Threads from SUN, how to modify the memory-usage of threads. (I didn't went through, nor am I experienced in this question).

http://java.sun.com/docs/hotspot/threads/threads.html

I ask, whether xml is a good decision of implementing a database.
I don't believe classical rdbms systems need that much threads, to perform a query over 20 tables.
Why don't you use a classical database?
 
Stefan Wagner
Ranch Hand
Posts: 1923
Linux Postgres Database Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Well - I couldn't resist to write a small test-programm.
ENV:
- linux 2.6.5
- j2sdk 1.4.2-beta-b19
- 256 MB RAM, 512 MB swap

Tried to create 3000 (lean) threads, resultet in a few more than 2000 threads being created, then: OutOfMemoryError. This didn't change mentionable by using -server -Xms -Xmx switches. A small benefit was encountered by freeing as much ram as possible (run from console, no X, no Mozilla 2024 -> 2060 Threads.

Maybee I will test it on my desktop, with 1GB RAM and 2GB swap

If your Hardware allows, or the users don't use your server simultanously, it might work .
You could use a Threadpool to prevent a Thread-overrun.

But all in all, I would use a real rdbms, if possible.

update:
ENV:
- linux 2.6.5
- j2sdk 1.4.2_02-b03
- 1 GB RAM, 2 GB swap

created 54.000 Threads without problem
64.000 Threads: Out of Memory.

(edit post revisited)
shame on me!

Most of the Threads had finished, before the last one was started
What a luck, nobody responded till now!
Either nobody was interested.
Or everybody was too impressed.

Here is a more conservative result for the 1GB desktop:
3300 Threads running simultanously.

Conclusio: Make a quick 'n' dirty test, but not too dirty.

[ May 18, 2004: Message edited by: Stefan Wagner ]

[ May 18, 2004: Message edited by: Stefan Wagner ]
[ May 18, 2004: Message edited by: Stefan Wagner ]
 
kriti sharma
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thanks Stefan for your efforts, actually i was busy at the office hence cud not respond early.

i have to abstain from using an rdbms because i do not want my client to buy it for lakhs of currency. my application will be sold in thousands of bucks only.

i cud not understand, did out of memory come after just 64 threads?and what is the 2GB swap you are talking of?i just know about hard-disk and RAM.

to make a dirty test of 3000 threads i will also have to ensure that they do some task that takes time and keeps them alive simultaneously. did you do that? or else you will never know if the "EXISTED" simultaneously.


can you tell me how to change the stack and heap sizes for JVM , the -xms and -xmx thing?
 
Stefan Wagner
Ranch Hand
Posts: 1923
Linux Postgres Database Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by kriti sharma:
i have to abstain from using an rdbms because i do not want my client to buy it for lakhs of currency. my application will be sold in thousands of bucks only.

Postgresql and hsqldb are rdbms which don't cost anything. (beside time, handbook, ...)


i cud not understand, did out of memory come after just 64 threads?and what is the 2GB swap you are talking of?i just know about hard-disk and RAM.
After 64 thousand threads (but here the measurement was inaccurate, because the first threads finished, before the last ones where created. There where only about 3000 Threads running at the same time.
Swap is part of the harddrive, which is reserved for pseudo-ram. When RAM is low, unused parts are written to disk, and restored from disk when needed again. It slows down your performance, since real RAM is more than 10 times faster, but it's a cheap solution to handle peeks of high memory usage.


to make a dirty test of 3000 threads i will also have to ensure that they do some task that takes time and keeps them alive simultaneously. did you do that? or else you will never know if the "EXISTED" simultaneously.

Yes, in the last step, I modified the programm to wait with exiting until all threads where created, which leads from 54.000 to 3.300 Threads. And they didn't do much - created a random number and decided to sleep, or to exit.

I could quote it here, if you like, it's about 100 lines.

can you tell me how to change the stack and heap sizes for JVM , the -xms and -xmx thing?

I can write java -Xms128m -Xmx256m ThreadTest 3300, but cannot explain anything, which you don't find on the docs, for linux it's here:http://java.sun.com/j2se/1.4.2/docs/tooldocs/linux/java.html
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!