there are some java libraries that are free for non commercial use but require to purchase a license for commercial applications.
My question is : what if some one use this library in commercial app without purchasing licence ? how can the owner of the librairy control the use of his work. and if you use it for commercial purposes what will happen ?
Well if the programmer is really sneaky he can add code that makes his java program phone home from time to time since these days most computers are connected to the internet.
In short, there's no simple answer. The problem is unrelated to java (so this should probably be in another forum) and has to do with general good practices for protected property. Usually java code is used in binary form when used in commercials applications and the source is barred from viewing unless this is a module the developer wants extended.
Most will tell you though that java is considered a *very* unsafe language (in that you can decompile binary code pretty easily) but there are means of compiling to exe type files.
Really, it comes down to the company and the value of the software. If the software is really good, people will buy it and if the company is really smart, they'll make sure people pay.
Perhaps in your country you will not be sued, but if you visit a country where it is, it might be an expensive experience.
Pay the license, or develop a similar license yourself, if that's cheaper.
If the library is only used for optional tasks, you might ship your application without that library, and only document how your customers could get the library, and let them decide to download the library and buy a license or not.