• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Rob Spoor
  • Devaka Cooray
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
  • Tim Holloway
Bartenders:
  • Jj Roberts
  • Al Hobbs
  • Piet Souris

weird Java performance on linux server

 
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
2 company servers on which I can build my (relatively simple) app using javac. On one, it always takes under 2 seconds. On the other, ~30% of the builds take longer - 20% are 2-3 seconds, 7% are 3-4 seconds, and the remainder are longer.

(I see similar behavior when running the app, hence I believe it's a general java problem rather than something specific to javac.)

I'm using the linux/gnu 'time' command to investigate this, and including the CPU percentage (%P) in the output. I could understand it if the 'slow' builds had lower CPU% values, but they don't - the 'quick' <2 second builds are typically 200-250%, but the >7 second build I recorded earlier had a CPU usage of 1000%.

So if there's a resource conflict with one or more other processes, it doesn't appear to be lack of CPU that's the problem. And if it's something else, why doesn't java/javac just idle until it becomes available? Is it busy-waiting?

Any insights gratefully received. If it helps, java -version = 1.7.0_85; linux 2.6.32 (uname reports 'GNU/Linux'). From my perspective, the servers are 'the same' - just a change in the last digit of the address I ssh to.

Cheers
John
 
Bartender
Posts: 9626
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Is the full output from java -version identical?  I used to see a significant performance penalty when I used OpenJDK installed by default on a lot of Linux distributions.  I think it has gotten better in recent years, but I always install the Oracle JDK just in case.  
 
Saloon Keeper
Posts: 13379
295
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So the servers have the same hardware, the same services running, the same configuration, etc?

How do the servers access the code that needs to be built?
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for the replies.

Is the full output from java -version identical?


Yes (checked with tkdiff). Both reporrt:

So the servers have the same hardware, the same services running, the same configuration, etc?


I'm afraid I don't know; they 'feel' the same, but I appreciate that doesn't mean much. I have asked our IT department if they can throw any light on what's going on, and I would hope they would have told me if there were any significant differences that might explain it. Anything in particular I should check (and how)?

How do the servers access the code that needs to be built?


Sorry, not sure what you mean. That is, the code appears as files in the linux file system, but I suspect you're look for more detail than that which I probably won't be able to provide you with (unless you tell me which commands to enter).

Cheers
John
 
Joe Ess
Bartender
Posts: 9626
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:Thanks for the replies.



As I said, the Oracle JDK may give you better performance across the board.  

I'm afraid I don't know; they 'feel' the same



Definitely follow up with IT to see if there's any differences between the systems. That would be a very likely cause. I've also used benchmark programs like DeCapo to look at (close to) real-world performance of my servers.

Sorry, not sure what you mean.  



I think Stephan wants to know if you are using a source control system to manage your code or if you manage the source by hand.  If you aren't using source control, it's possible that you have file variations between your systems that will be difficult to identify.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

As I said, the Oracle JDK may give you better performance across the board.


I think the only way I could attempt to change that would be to put a request in to IT, and I don't rate my chances.

I've also used benchmark programs like DeCapo to look at (close to) real-world performance of my servers.


Ok, I'll try that later. But I can't control what other users (on other continents) are doing on those servers, which I guess is going to affect those benchmark results?

If you aren't using source control, it's possible that you have file variations between your systems that will be difficult to identify.


The source is in git, but I don't think that's relevant here - the servers see the same filesystem as far as the directories where I'm working are concerned. That is, if I'm working on one server and make a change to a file, I see that change immediately in (what appears to be) the same file at the same filesystem location on the other server.

Cheers
John
 
Stephan van Hulst
Saloon Keeper
Posts: 13379
295
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
That, or if the source is accessed from a repository that's not on those servers, network latency may also be a factor.
 
Stephan van Hulst
Saloon Keeper
Posts: 13379
295
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:That is, if I'm working on one server and make a change to a file, I see that change immediately in (what appears to be) the same file at the same filesystem location on the other server.


It matters though. A file system is just an abstraction. Even if both servers "see" the same thing, the physical location may influence compilation times.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Stephan van Hulst wrote:

John Matthews wrote:That is, if I'm working on one server and make a change to a file, I see that change immediately in (what appears to be) the same file at the same filesystem location on the other server.


It matters though. A file system is just an abstraction. Even if both servers "see" the same thing, the physical location may influence compilation times.


Yes, that makes sense. But would you expect to see high CPU usage while Java is waiting for the network? I would expect slow compilation with low CPU usage, not high.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:I would expect slow compilation with low CPU usage, not high.


(if the slow compilation is caused by network latency)
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Regarding hardware differences, I've just done lscpu. The server where I see the slow builds 'only' has 24 'CPUs' compared to the other's 40, and 12288K L3 cache vs 25666K. But if these differences were the cause of different build times, wouldn't the symptoms be consistent? For 'normal' builds, the times are pretty similar; it's only the odd build on the one server that's slow.
 
Saloon Keeper
Posts: 24517
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:the >7 second build I recorded earlier had a CPU usage of 1000%.



Alarm bells.

Actually, even the 250% number is disturbing. Compiling used to be a big system load, but that was back when computers were less powerful than today's cellphones.

Actually compilers tend to be more I/O bound than CPU bound.

For that much CPU overhead, I'd check the paging rates. Virtual swaps eat a surprising amount of CPU. Use vmstat to monitor.

Also, my own experience for good performance is that you want the compiler task and the files being compiled on the same machine.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:For that much CPU overhead, I'd check the paging rates. Virtual swaps eat a surprising amount of CPU. Use vmstat to monitor.


Thanks Tim - just to confirm, that's something that's going to change significantly from build to build, resulting in orders of magnitude differences in build times and CPU usage?

Using top, I didn't see big changes in other activity on the server(s) while recording my results, but that's not to say there weren't any.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
...and sorry, how should I use vmstat?
 
Tim Holloway
Saloon Keeper
Posts: 24517
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:...and sorry, how should I use vmstat?





The "so" is swap out rate, si is swap in, swpd is pages swapped and "man vmstat" has detailed info.

Ideally, you're not swapping at all. The higher the swap rate, the less efficiently you're using memory and you probably need to either offload something or IBM it (Install Bigger Memory).

You can set vmstat to periodically sample and print, which gives an ongoing view.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Right - some more data for you

Results of timing some (clean) builds earlier. The time command I'm using is

but I also did a 'date' before the build, and have prefixed time output lines with timestamps (and added a load of spaces to make the columns line up):

You can see how the times vary, and long times correspond to high CPU%. I was also running vmstat in a background process, with timestamps; here is its output around the first long build time (real=42.95, starts at 57:23) including 'normal' builds before and after. Timestamps are on the ends of the lines.

So in the long build vmstat shows that the 'in' value shoots up; in = 'the number of interrupts per second, including the clock'. And it looks more cores are brought in to handle the load(?).

So do you think the interrupts are the cause, or just a symptom? Any idea what they indicate?
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:So in the long build vmstat shows that the 'in' value shoots up


...along with 'sy' = 'time spent running kernel code (system time)'.
 
Tim Holloway
Saloon Keeper
Posts: 24517
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You seem to have "bursty" events, possibly about every 30 seconds, although the sample size isn't large enough to be sure.

It's possible that there's some other process that kicks in at intervals and eats the machine, leaving little for the compiler process.

A crude way of checking this would be to run "top", although you might have to adjust its sampling interval to get a good idea. And of course, top itself is going to take periodic bites as it wakes up, samples, and displays.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:You seem to have "bursty" events, possibly about every 30 seconds


Sorry Tim - can you explain where you get that from?

It's possible that there's some other process that kicks in at intervals and eats the machine, leaving little for the compiler process.
A crude way of checking this would be to run "top", although you might have to adjust its sampling interval to get a good idea.


I have performed the tests while watching top and not noticed anything usual. This morning I suspect I was the only human user on the server.

Any suggestions as to what sort of process I should be looking for ie. what might cause all those interrupts?
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Update - just re-run the build tests, and out of 96 builds, there were only 8 that took >2 seconds, and just one build took longer than 3 seconds. But they all had relatively low CPU% figures, which is what I would expect if something took longer than expected ie. they had to wait for some resource. So whatever was happening this morning isn't happening now.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:But they all had relatively low CPU% figures


'they' = the (slightly) longer builds
 
Tim Holloway
Saloon Keeper
Posts: 24517
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

John Matthews wrote:

Tim Holloway wrote:You seem to have "bursty" events, possibly about every 30 seconds


Sorry Tim - can you explain where you get that from?



From looking at CPU usage down the table. It shows periods of low usage interrupted by periods of higher usage.

The "top" system utility can help pin down what's causing that.
 
John Matthews
Rancher
Posts: 343
11
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:

John Matthews wrote:

Tim Holloway wrote:You seem to have "bursty" events, possibly about every 30 seconds


Sorry Tim - can you explain where you get that from?



From looking at CPU usage down the table. It shows periods of low usage interrupted by periods of higher usage.


But isn't that just my builds?

The "top" system utility can help pin down what's causing that.


I've just tried doing ps before each build to capture CPU usage. But after eliminating processes common to each (pre-)build, there wasn't anything left, at least in the top 35 I was recording (of which the last 4 or 5 measured 0% CPU).

As mentioned previously, I have watched top while the builds are going on and haven't spotted any difference in process activity between quick and slow builds.

I think the most promising line of inquiry is the interrupt activity recorded by vmstat; I'm going to have a look at /proc/interrupts...
 
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic