• Post Reply Bookmark Topic Watch Topic
  • New Topic

How do I run spark job server as a service which can be connected to spark master but no as local[*]  RSS feed

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
How to run spark job server as a service which will be connected to spark master but not as local[*]?
Spark configuration steps:

        Installation steps..
    1) Java ( version 1.8 )
    2) Spark ( version 1.6.2) also latest version
    3) Scala ( version 2.10.5)
   
        spark started by - ' ./sbin/start-all.sh '
        job server started by - server_start.sh
   
        1. Now Spark is running on 8080 port and worker is running on 8081 port
        2. Spark job-server is running on 8090 port.
        When we start spark job-server with the help of SBT command and place spark master URL with local[*] then it's working fine. But when we replace local[*] with spark://master:7077 and checked with our own example like WordCount example.

The command is used for word count example is :

     a) Uploaded the jar by curl command like :-
        {curl --data-binary @SparkTest.jar localhost:8090/jars/JarTest}
   
        The jar is uploaded successfully on spark job-server.
   
        b) Trying to execute WordCount java program using curl command:
          { curl -d "input.filename = data.txt" 'localhost:8090/jobs?appName=JarTest&classPath=com.pro.WordCount'}
   
        the job can be seen running on spark (8080) but the status of the job is always 'RUNNING' mode only and the result is not able to get back in response.
   
        C) I have also tried to install Livy server on my local system and spark is running on the remote server at 8080 port. I run spark job using Livy URL in java program from my system through Livy gateway but did not get the result back after submitting the job on spark master spark://master:7077.
   
        I want to execute our job with an interactive session or with in the same spark context but got failed to execute with this all kind of approach. Please help us where we might be missing in the configuration?
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!