• Post Reply Bookmark Topic Watch Topic
  • New Topic

I/O Throttle limitation

 
Marx Villegas
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Ranchers,
I have a question, and don't know how silly can it be. The thing is that I'm required to write an application that basically moves files from one disc volume to another. In this system, there is already a file server application running.
The hard part comes when I'm asked to regulate the IO throttle of it. Basically what I need to achieve is that this "file move" application does not consume much IO compared to the other live & highly-stressed file server application.

I've thought about simply using thread mechanisms, but I'm not sure that could help.
Any suggestions?

As always, many thanks
XM
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How is "much IO" defined?

I'm assuming the code knows the size of the files it moves, and if it times the file transfers it could calculate a crude MB/s ratio. Whenever the process is in danger of going over a certain limit, the file copying process can sleep for a bit.
 
Mike Simmons
Ranch Hand
Posts: 3090
14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Most file copying in Java uses either FileInputStream and FileOutputStream, or FileChannels. Etiher way, the methods you will use allow you to limit the mximum number of bytes to transfer at one time. Consider making this a configurable parameter, max.transfer.block.size or whatever, which can be read from a properties file. Another parameter might be transfer.block.delay.time, the time you're required to sleep between blocks. So depending on how you configure it, a transfer of 1000000 bytes might be done as 10 transfers of 100000-byte blocks, with 0.5 seconds of delay between blocks. Or it might be done as 100 transfers of 10000-byte blocks, with a different delay. Or 500 transfers of 2000 bytes. Whatever. Let the person running the program adjust the parameters until they find a combination that works well on their system.

Using this plan, it may be better to sleep for transfer.block.delay.time milliseconds before each block, including the first one - not afterwards. Otherwise it would be possible for a large number of small copy jobs to swamp the system, keeping it very busy with no sleeping.

Also, I once solved a similar problem simply by inserting a call to Thread.yield() after each block, with no sleep(). In many cases this may have no effect, but on systems it may be exactly what you need. Give it a try, at least.
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am not so sure but isnt assigning a low thread priority to the file moving thread give you something close to what you want?
Since, java has long bid adeau to green threads and every thread transaltes to an OS thread, i believe if the application threads have a high priority, they will always get more CPU and I/O then the low priority file moving thread.
Is this what you want?
 
Marx Villegas
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Actually Mike's idea about the Stream size and speed (sleeping) constraints looks like a good option, I'm definitely trying it.
Now, I'm not completely sure about this, but thread management (including yields, etc) will only work inside of one application (AFAIK java threads have nothing to do with OS threads). So, this probably won't solve my problem
Many thanks guys, I'm very grateful
XM
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Marx:
AFAIK java threads have nothing to do with OS threads


That was the case when java used green threads

Since, most of the jvms(sun's vm , jrocket) do not use green threads but use native threads instead, so each java thread actually is a OS thread and the kernel takes care of all the scheduling/time-slicing, etc.
However, thread priorities in java do not directly map to the OS priorities.

This discussion has details about the difference between green and native threads.
 
Ulf Dittmer
Rancher
Posts: 42970
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Twiddling with thread priorities, or talking about different kinds of threads will not solve this problem, if I understand it correctly, because we're talking about code running in different processes. The process doing the file copying is not the same as the file server whose I/O requirements are to be respected.
 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ulf:
Twiddling with thread priorities, or talking about different kinds of threads will not solve this problem, if I understand it correctly, because we're talking about code running in different processes. The process doing the file copying is not the same as the file server whose I/O requirements are to be respected.


Thats why i said "I am not so sure" in my original post

Thanks for pointing this out Ulf. Actually, I was thinking that since each java thread is actually an OS thread, if the two threads in question have different priorities we can achieve what we want.
However, I missed the point that each thread belongs to a process and thread scheduling happens based on thread priorities inside a process and not across processes. (Correct me if my understanding is wrong)

I came across this post while googling for this. It may be of interest for some.
 
Marx Villegas
Ranch Hand
Posts: 94
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thread management (including yields, etc) will only work inside of one application (AFAIK java threads have nothing to do with OS threads)


Actually I mischose my wording (Sorry ). What I meant to say was that a thread.yield() won't yield the CPU execution of another application.

Thanks a lot to everyone, you guys are cool
Best
XM
 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!