• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Running a Python script via runtime.exec

 
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Good day! This is my first post here, and I'm afraid I come here with a problem that I have no idea how to solve.

See, I have thousands of Python scripts I need to run, each with their own peculiarity. When I run each of them through a command line in a Linux terminal, they work perfectly, maybe only taking a little while to finish, but they DO finish. When I run them through a home-made scheduler using cron4j (each script has it's own class, to contain some information that's specific to each), they hiccup. Several of them stop doing what they're supposed to do - I assume their thread (that's how cron4j works) is stopping. Naturally, I thought about starting each script separately, as if I was running it through a terminal. Again, through a command line execution, the script in question runs PERFECTLY. When I run it through runtime.exec, however, it, without fail, stops working at the same exact spot. I have no idea why this is.

I tried cutting the script down in sizes and trying them all out - every piece works perfectly when separate from the rest. When they're all together, though, no dice. Again, I think the problem here is with threads. Any idea what I can do to fix this conundrum? This is really driving me crazy.

Thanks in advance!
-- A. Bettega.

Edit - To add more information, this is the code I'm working with:



It's missing some stuff because some of that I can't show, and some of it was removed (like the logging, for instance) for testing purposes. Whatever's missing is irrelevant to what is up there.
 
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Likes 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Aroldo Bettega wrote:Good day! This is my first post here, and I'm afraid I come here with a problem that I have no idea how to solve.


Hi. I split up the really long lines in your code, as they were screwing up the windowing here.

As to your problem, I'm not quite sure, but you might want to read this (a real oldie, but worth reading; some of the problems may have been fixed in subsequent releases). If anything else leaps out at me, I'll post further.

Winston
 
Aroldo Bettega
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:

Aroldo Bettega wrote:Good day! This is my first post here, and I'm afraid I come here with a problem that I have no idea how to solve.


Hi. I split up the really long lines in your code, as they were screwing up the windowing here.

As to your problem, I'm not quite sure, but you might want to read this (a real oldie, but worth reading; some of the problems may have been fixed in subsequent releases). If anything else leaps out at me, I'll post further.

Winston


Wow, that actually solved my problem perfectly! If anyone gets here in the future, the problem apparently was with the buffer size - it was too small to read the Process output. Page four of that "guide" has the solution that worked for me - a "StreamGobbler".

Cheers, mate! Thanks!
 
Winston Gutkowski
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Aroldo Bettega wrote:Wow, that actually solved my problem perfectly! If anyone gets here in the future, the problem apparently was with the buffer size - it was too small to read the Process output. Page four of that "guide" has the solution that worked for me - a "StreamGobbler".

Cheers, mate! Thanks!


You're very welcome. A grey hair shared is a grey hair halved.

Winston
 
Aroldo Bettega
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Aw, crud. Nevermind. It worked on a single script. When I tried running several of them, they all bugged out. For instance, I modified the StreamGobbler a bit to create a bit of logging. Well, actually, let me explain the process as a whole.

I have a "Script.java" class that defines the actual script. Script.java holds things like the InputStream the script ID, and the java.io.File for the logging. This Script.java extends Task.java from the cron4j library. Script.java also declares the StreamGobbler internal class.

Then, let's take Script_1.java for instance. It extends Script.java, and every information created by Script.java. In the declaration of StreamGobbler, one of the parameters is Script, so that I can send Script_1, 2, etc. to it. StreamGobbler is, thus, also responsible for the logging part of the system.

When I tried this out with a single script, it worked like a charm. When I tried starting several scripts concurrently (which will happen often on the actual system), instead of creating a log file in the "log_<script_id>.txt", it created and repeatedly wrote over "log_0.txt".

Script.java:


Something like that.

This is Script_1. Remember, all 6 thousand scripts I have are very similar - they all run a separate python file. I ommited some things both from Script.java and Script_1.java, but nothing that actively gets on the way of understanding.



This is basically how the system is working. The ONLY idea I have on what's the issue here is that using Threads is really harming me here. But, of course, there's no other way to go. Any thoughts on what I could do?

Edit - I should probably mention that these scripts in python are actually quite lengthy - we're working on converting them to java, but since there are thousands, that's going to take just a little while.

Edit 2 - Ok, I managed to solve the ID problem. The remaining issue, though, is with the scripts never finishing. I started my BufferedReader with the second parameter at 8096 - let's see what happens.
 
Aroldo Bettega
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Alright, I think it's a different problem altogether.

I tried scheduling the running of several scripts at once using Linux's cron tool. They all finished running in 1 and a half minute, two tops. When I try running them all through cron4j, they take maybe three, four minutes to finish. I can only assume it's because each execution is a separate thread. What can I do to speed the process up? This thing is not exactly doing it's job if it takes over twice the regular time to run properly.

Oh, and worst of all! Some times, out of the blue, some scripts just... freeze! Then they take upwards to 10, 20 minutes to finish the execution. What could be the issue here?
 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:

Aroldo Bettega wrote:Good day! This is my first post here, and I'm afraid I come here with a problem that I have no idea how to solve.


Hi. I split up the really long lines in your code, as they were screwing up the windowing here.

As to your problem, I'm not quite sure, but you might want to read this (a real oldie, but worth reading; some of the problems may have been fixed in subsequent releases). If anything else leaps out at me, I'll post further.

Winston



This saved my life!!! Thanks for the link !
 
Marshal
Posts: 79151
377
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Welcome to the Ranch That article is still probably the classic about its subject. ProcessBuilder objects make the procedure a bit simpler; they weren't available when Daconta write the article.
 
Could you hold this puppy for a sec? I need to adjust this tiny ad:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic