Ahead of Time vs Just in time compilers

[john price]

I've read a small amount about each. If you could explain the downsides and benefits of each, that would be great. You could also say common times when to use which. I don't see anything wrong with AOT, based on what I've read. Why doesn't the Java JDK use it?
Thanks,
cc11rocks

[Alexander Kober]

Giving you a complete answer to this question would probably take a few weeks, this topic is fairly complicated.

The shortest (though very, very incomplete) explanation i can give for the motivation behind JIT compilers (besides the platform-independence topic) is this:

With AOT, you compile your code according to the available state of the art. That is, if you were compiling code in the early 90s before SSE was available, you'd probably get a binary that iteratively goes over your data and processes one element at a time. Tun the same code today, and it will still just iterate. With a JIT, you run your old code on a modern JVM. The compiler can, in a lot of cases, identify what your doing and find a better way of doing it. So the same code fragment may be JIT-compiled to use SSE and suddenly run 16 times faster.

Another (less abstract) example: The latest versions of the JVM support a feature called lock elision. For historical reasons, the StringBuffer class was synchronized, which means that in the following code segment, a lock is acquired and released three times (with every call to append):

public String makeText(){
  final StringBuffer buffer = new StringBuffer();
  buffer.append("Foo");
  buffer.append("Bar");
  buffer.append("Foobar")
  return buffer.toString();
}

A modern JVM will correctly identify that the lock is not needed since the buffer instance is only used locally in the context of the enclosing method. A modern JIT will therefore simply remove the locking, thus increasing performance. This is a feature that was not available before, but your old code still gains from the new technology.

[Pat Farrell]

There is a very old rule for performance, probably goes back to the Eniac: 90% of the performance issues are in 10% of the code. Often its worse, more like 99% of issues in 1% of code.

With a traditional compiler/optimization system, you have to guess which parts are important. With a JIT optimization, you can first measure which parts are actually used, and optimize that.

But the key is that Java was designed from the start to run with the exact same byte-code on lots of machines and operating systems. And it delivers. It could not do that if it was optimized at compile time. In the mid-90s, there were lots of competing architectures, Sparc, MIPS, Alpha, Itanium, X386, etc, and X386 was no where near the clear winner. Even today, moving to an ARM machine is a lot easier with Java than it is with C/C++ or traditional compilers.

[john price]

So the short answer is that AOT is better for short term, while JIT is better for long term?

John Price

[Matthew Brown]

That's one of the short answers. Another is that AOT can be better if your compiler fully understands the target machine. But if you want portability and performance, JIT is better.

[john price]

I see. Thank you guys for your responses!

John Price

[Pat Farrell]

john price wrote:So the short answer is that AOT is better for short term, while JIT is better for long term?

I think that to answer that, we'd need a definition of your short and long term. I'm so old that I'm used to code being in production for a decade or more. That is long term. You may have a very different answer.

In practice, the JIT hot-spot optimizations are very very good, and do all the standard things that good optimizing compilers do.

As we move away from the near brain-dead X386 architecture, optimization becomes critical. Even with it, you can lose 90% of your performance with badly laid out code that causes lots of pipeline stalls and cache misses.

[Tim Holloway]

There are actually (at least!) 3 different types of "Just In Time".

The first, and least sophisticated is to simply compile everything the first time you need it and that's all she wrote. Python interpreters tend to do this.

The second one, which I understand to be the standard for current JVMs is to interpret the bytecodes the first time or 6 you execute and then, once you've seen the need for speed, to compile to native code.

The third one is extremely evil. In addition to monitoring the number of times the code is executed, the JVM also monitors HOW it's executed.

Here's a primitive example: Back when machines operated in mathematically-determined cycles, a branch instruction typically had 2 different timings. One for when the branch was taken, one for when the branch wasn't taken. Usually, the no-branch case was faster. Obviously, this meant that you could optimize logic based on the knowledge of which case was more common (subject to the constraint that what we "know" isn't always what's actually the case).

JIT recompilation takes this one step further. It monitors the actual real-time usage, and flips the logic based on what actually is the most common case. And flip it back again if the workload shifts. Statically-compiled code can't do this. That's how Java can sometimes out-perform hand-tuned assembly/C/C++ code.

Of course, to even do that, you have to spend a lot of time measuring, so this fails on simple one-shot programs like "Hello, World!". But when you're scaling up, eventually it reaches a point where it pays for itself with interest.

[Alex Major]

I never understood why no one created a JSR* to have a new annotation or new keyword prependable to a method that tells the JVM to compile the method either ASAP or before program execution. Only the programmer knows beforehand where "real-time" or faster performance is required in the code.

* http://en.wikipedia.org/wiki/Java_Specification_Request

[Pat Farrell]

Alex Major wrote:Only the programmer knows beforehand where "real-time" or faster performance is required in the code.

No, that is the point. the programmer does not know. The programmer may think they know, but they simply can't. How can the programmer know what platform (x386, ARM, etc.) that the bytecode will execute on.

[Tim Holloway]

Alex Major wrote:I never understood why no one created a JSR* to have a new annotation or new keyword prependable to a method that tells the JVM to compile the method either ASAP or before program execution. Only the programmer knows beforehand where "real-time" or faster performance is required in the code.

* http://en.wikipedia.org/wiki/Java_Specification_Request

Probably because what the programmer "knows" is wrong. I've demonstrated that repeatedly over the years. The real bottleneck usually isn't even close to where you "know" it is, once you start measuring it.

Circa-1985 C compilers came with a raft of tweaking keywords, starting with "register" and moving on from there. For the most part, these constructs have become useless/ignored. Although there were certain cases, such as OS/hardware interfaces where the normal interfaces had to be overridden, the constructs having to do with hardware optimization failed because A) the hardware changed and no one could afford to re-tweak the code (as mentioned above) and B) because the compiler finally got better at the task of optimization than the programmer. Especially optimization-in-the-large, which is where you get your real speed boosts.

[Alex Major]

I'm writing music notation software. My initial prototype could play the fastest 32nd notes and refresh the display on my old 867 Mhz Mac from 2001, under Java 1.4.2. But I will need to playback and send note data to other music programs via Rewire* at very low latency. I'm hoping multi-core CPUs, which are the norm, will minimize the lag from bytecode interpretation and the JVMs on-the-fly-compilation. I use object pools so that the garbage collector finds nothing to remove. Memory is not an issue anymore. I still have a nagging fear that Java won't cut the mustard. Too bad Excelsior JET* only works with Windows and Linux.

* http://en.wikipedia.org/wiki/Rewire
* http://en.wikipedia.org/wiki/Excelsior_JET

[Pat Farrell]

Alex Major wrote:I'm writing music notation software. My initial prototype could play the fastest 32nd notes

I don't see how a pure Java solution will solve your problem. Playback of media (audio is easier than video, but its all the same) demands real time continuous output. There have been decades of work on low latency drivers for Windows and OSX.

I think you should plan on having the stream playback be done by a separate process, written in C. You can have your Java program drive the stream process. You do not want a thread, as the Java GC will stop the whole thread.

But this has very little to do with JIT compilers or even hand optimization. Its a very specialized need, and it needs specialized engineering.

[Tim Holloway]

Actually, Project X does video in Java quite nicely.

But on Linux, at least, there's a bus subsystem package specifically designed for audio "patchcord" use, so I'd probably be interfacing with it on that platform.

[Phil Freihofner]

Alex Major wrote:I'm writing music notation software. My initial prototype could play the fastest 32nd notes and refresh the display on my old 867 Mhz Mac from 2001, under Java 1.4.2. But I will need to playback and send note data to other music programs via Rewire* at very low latency. I'm hoping multi-core CPUs, which are the norm, will minimize the lag from bytecode interpretation and the JVMs on-the-fly-compilation. I use object pools so that the garbage collector finds nothing to remove. Memory is not an issue anymore. I still have a nagging fear that Java won't cut the mustard. Too bad Excelsior JET* only works with Windows and Linux.

I've been dabbling with Java sound, and am discovering that many of the speed & response problems I've had have been due to less than optimal programming rather than problems with the language being slow. Maybe some of that can be laid at the feet of the complexity of the API and what seems to me to be a less than optimal presentation of it in the official Tutorials. But as I get a better grasp conceptually, I've been able to have some good successes. Yes, learning to work with object pools has been a help! I don't know what is needed with Rewire, but here are two examples of issues I have had some success with.

1) Latency when dealing with sending Swing GUI real time controls to playback. I was having a heck of a time getting things like sliders to work with the javax.sound.sampled SourceDataLine "controls". Finally made decent progress when I wrote my own controls to work at the per-sample level, including a LinkedBlockingQueue implementation that stores GUI events and feeds them to the per-buffer processing loop, based upon their timing. (The JVM doesn't really process the per-buffer data in "real time" but works ahead of the playback by an amount linked to the size of the buffer, so the events also have to be packaged and dealt with in a non-real-time sort of way.) More discussion, and an example of a crude Theremin program can be found via this link: http://hexara.com/VSL/JTheremin.htm. The GUI response is "not bad."

2) Use of Thread.sleep or util.Timer can be problematic as it's accuracty depends on the OS implementation (e.g., WinXP resolution of 15 milliseconds). But a ScheduledThreadPoolExecutor has some improvements over Timers, AND, there are ways to get a granularity or accuracy of 1 millisecond from Thread.sleep on WindowsXP. There's some discussion of this over at a thread on http://www.java-gaming.org/index.php/topic,24311.0.html. See reply #16!

And, there's lots of cool stuff with concurrency programming to learn about that have great potential for optimizing shipping data from here to there. (I'm just now reading "Java Concurrency in Practice" and am learning a great deal.)

I understand these tidbits might not be sufficient for your goals. And I should say, I've not yet been working with Java for a full two years. I DID hear that the library was rewritten and improved for Java 7 if that is worth anything. In summary, I wouldn't underestimate Java's potential, although it may take some real expertise to realize this potential, and for practical purposes might be easier to go the JNI "native code" route.

[Alex Major]

Pat Farrell wrote:
But this has very little to do with JIT compilers or even hand optimization. Its a very specialized need, and it needs specialized engineering.

Such great replies from Tim and Phil, and Pat graciously hints we are off topic. I can address what Pat said, then how about I start another topic for you guys in Java -> Threads and Synchronization called "Programming Near Real-Time Apps"? Can you propose a better name?

Tim Holloway and Phil Freihofner are saying Java desktop applications are fast enough to seem "real-time" without AOT compilation with millisecond resolution, for any use. Perhaps, desktop apps no longer need AOT compilation.

For enterprise apps that run for months without shutting down, AOT compilation doesn't matter, the JVM JIT compiler has compiled every possible bytecode by the end of the first few days. depending on your -XX:CompileThreshold setting. And, if I'm not mistaken, it keeps it all compiled until the JVM is shutdown.

[Alex Major]

Tim Holloway wrote:Actually, Project X does video in Java quite nicely.

I ran it on my Mac but I haven't found a valid .avi, .mp4 or .mov that it does not complain about. Then it complains of "not enough data in buffer". Can't find the solution anywhere but I believe you. It has good reviews everywhere. sourceforge participants love it.

[davian mcllm]

Can someone elaborate, why it's necessary to compile that intermediate format just-in-time instead of, say, ahead-of-time during the installation phase.

[Alex Major]

davian mcllm wrote:Can someone elaborate, why it's necessary to compile that intermediate format just-in-time instead of, say, ahead-of-time during the installation phase.

This might answer your question: Why is Java faster when using a JIT?

[Alex Major]

Why doesn't the JVM "cache" the compiled bytecodes for the next run of an application? Why recompile at every run?

[Stephan van Hulst]

Probably for security reasons. If the cached machine code is saved to disk, anyone can edit it and feed it back to the JVM bypassing security checks.