Win a copy of Real-World Software Development: A Project-Driven Guide to Fundamentals in Java this week in the Agile and Other Processes forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Liutauras Vilda
  • Knute Snortum
  • Bear Bibeault
Sheriffs:
  • Devaka Cooray
  • Jeanne Boyarsky
  • Junilu Lacar
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
  • salvin francis
Bartenders:
  • Tim Holloway
  • Piet Souris
  • Frits Walraven

java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize in Spark DecisionTreeClassification

 
Ranch Hand
Posts: 1361
8
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am getting error message java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize while trying to run the below Spark ML DecisionTreeClassification program:

https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala


The command to run : spark-submit --class org.apache.spark.examples.mllib.DecisionTreeClassificationExample --master local[4] /home/cloudera/scala-workspace/DecisionTreeClassification/target/DecisionTreeClassification-0.0.1-SNAPSHOT.jar



The input data is the data at this path which I have copied to sampleInput.txt
https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_libsvm_data.txt

Code:




Error: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize


Stacktrace:

[cloudera@quickstart bin]$ spark-submit --class org.apache.spark.examples.mllib.DecisionTreeClassificationExample --master local[4] /home/cloudera/scala-workspace/DecisionTreeClassification/target/DecisionTreeClassification-0.0.1-SNAPSHOT.jar
WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/lib/spark) overrides detected (/usr/lib/spark).
WARNING: Running spark-class from user-defined location.
20/03/18 08:12:36 INFO spark.SparkContext: Running Spark version 1.6.0
20/03/18 08:12:37 INFO spark.SecurityManager: Changing view acls to: cloudera
20/03/18 08:12:37 INFO spark.SecurityManager: Changing modify acls to: cloudera
20/03/18 08:12:37 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cloudera); users with modify permissions: Set(cloudera)
20/03/18 08:12:38 INFO util.Utils: Successfully started service 'sparkDriver' on port 38834.
20/03/18 08:12:38 INFO slf4j.Slf4jLogger: Slf4jLogger started
20/03/18 08:12:38 INFO Remoting: Starting remoting
20/03/18 08:12:39 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.0.2.15:36496]
20/03/18 08:12:39 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.0.2.15:36496]
20/03/18 08:12:39 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 36496.
20/03/18 08:12:39 INFO spark.SparkEnv: Registering MapOutputTracker
20/03/18 08:12:39 INFO spark.SparkEnv: Registering BlockManagerMaster
20/03/18 08:12:39 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-aff0c510-375c-4f95-b640-8b668acc30a0
20/03/18 08:12:39 INFO storage.MemoryStore: MemoryStore started with capacity 530.0 MB
20/03/18 08:12:39 INFO spark.SparkEnv: Registering OutputCommitCoordinator
20/03/18 08:12:40 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
20/03/18 08:12:40 INFO ui.SparkUI: Started SparkUI at http://10.0.2.15:4040
20/03/18 08:12:40 INFO spark.SparkContext: Added JAR file:/home/cloudera/scala-workspace/DecisionTreeClassification/target/DecisionTreeClassification-0.0.1-SNAPSHOT.jar at spark://10.0.2.15:38834/jars/DecisionTreeClassification-0.0.1-SNAPSHOT.jar with timestamp 1584544360287
20/03/18 08:12:40 INFO executor.Executor: Starting executor ID driver on host localhost
20/03/18 08:12:40 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44453.
20/03/18 08:12:40 INFO netty.NettyBlockTransferService: Server created on 44453
20/03/18 08:12:40 INFO storage.BlockManager: external shuffle service port = 7337
20/03/18 08:12:40 INFO storage.BlockManagerMaster: Trying to register BlockManager
20/03/18 08:12:40 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:44453 with 530.0 MB RAM, BlockManagerId(driver, localhost, 44453)
20/03/18 08:12:40 INFO storage.BlockManagerMaster: Registered BlockManager
20/03/18 08:12:42 INFO scheduler.EventLoggingListener: Logging events to hdfs://quickstart.cloudera:8020/user/spark/applicationHistory/local-1584544360372
20/03/18 08:12:42 INFO spark.SparkContext: Registered listener com.cloudera.spark.lineage.ClouderaNavigatorListener
20/03/18 08:12:43 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 295.7 KB, free 529.7 MB)
20/03/18 08:12:44 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 25.7 KB, free 529.7 MB)
20/03/18 08:12:44 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:44453 (size: 25.7 KB, free: 530.0 MB)
20/03/18 08:12:44 INFO spark.SparkContext: Created broadcast 0 from textFile at MLUtils.scala:71
20/03/18 08:12:44 INFO mapred.FileInputFormat: Total input paths to process : 1
20/03/18 08:12:44 INFO spark.SparkContext: Starting job: reduce at MLUtils.scala:105
20/03/18 08:12:44 INFO scheduler.DAGScheduler: Got job 0 (reduce at MLUtils.scala:105) with 1 output partitions
20/03/18 08:12:44 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (reduce at MLUtils.scala:105)
20/03/18 08:12:44 INFO scheduler.DAGScheduler: Parents of final stage: List()
20/03/18 08:12:44 INFO scheduler.DAGScheduler: Missing parents: List()
20/03/18 08:12:44 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[5] at map at MLUtils.scala:103), which has no missing parents
20/03/18 08:12:44 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.9 KB, free 529.7 MB)
20/03/18 08:12:44 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.1 KB, free 529.7 MB)
20/03/18 08:12:44 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:44453 (size: 2.1 KB, free: 530.0 MB)
20/03/18 08:12:44 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1004
20/03/18 08:12:44 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[5] at map at MLUtils.scala:103) (first 15 tasks are for partitions Vector(0))
20/03/18 08:12:44 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
20/03/18 08:12:44 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, ANY, 2247 bytes)
20/03/18 08:12:44 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
20/03/18 08:12:44 INFO spark.ExecutorAllocationManager: New executor driver has registered (new total is 1)
20/03/18 08:12:44 INFO executor.Executor: Fetching spark://10.0.2.15:38834/jars/DecisionTreeClassification-0.0.1-SNAPSHOT.jar with timestamp 1584544360287
20/03/18 08:12:44 INFO util.Utils: Fetching spark://10.0.2.15:38834/jars/DecisionTreeClassification-0.0.1-SNAPSHOT.jar to /tmp/spark-706a354a-9f16-4ca9-8aad-d7d88d591f4a/userFiles-e66b3b96-33c7-4ecd-bcc1-91ebbcf2eefa/fetchFileTemp8429674972166985312.tmp
20/03/18 08:12:44 INFO executor.Executor: Adding file:/tmp/spark-706a354a-9f16-4ca9-8aad-d7d88d591f4a/userFiles-e66b3b96-33c7-4ecd-bcc1-91ebbcf2eefa/DecisionTreeClassification-0.0.1-SNAPSHOT.jar to class loader
20/03/18 08:12:44 INFO spark.CacheManager: Partition rdd_4_0 not found, computing it
20/03/18 08:12:44 INFO rdd.HadoopRDD: Input split: hdfs://quickstart.cloudera:8020/user/cloudera/sampleInput.txt:0+104736
20/03/18 08:12:45 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
20/03/18 08:12:45 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
20/03/18 08:12:45 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
20/03/18 08:12:45 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
20/03/18 08:12:45 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
20/03/18 08:12:45 INFO storage.MemoryStore: Block rdd_4_0 stored as values in memory (estimated size 166.7 KB, free 529.5 MB)
20/03/18 08:12:45 INFO storage.BlockManagerInfo: Added rdd_4_0 in memory on localhost:44453 (size: 166.7 KB, free: 529.8 MB)
20/03/18 08:12:45 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 2585 bytes result sent to driver
20/03/18 08:12:45 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1165 ms on localhost (executor driver) (1/1)
20/03/18 08:12:45 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
20/03/18 08:12:45 INFO scheduler.DAGScheduler: ResultStage 0 (reduce at MLUtils.scala:105) finished in 1.191 s
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Job 0 finished: reduce at MLUtils.scala:105, took 1.450203 s
20/03/18 08:12:45 INFO spark.SparkContext: Starting job: take at DecisionTreeMetadata.scala:110
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Got job 1 (take at DecisionTreeMetadata.scala:110) with 1 output partitions
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (take at DecisionTreeMetadata.scala:110)
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Parents of final stage: List()
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Missing parents: List()
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[10] at map at DecisionTreeMetadata.scala:110), which has no missing parents
20/03/18 08:12:45 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.7 KB, free 529.5 MB)
20/03/18 08:12:45 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 2.5 KB, free 529.5 MB)
20/03/18 08:12:45 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:44453 (size: 2.5 KB, free: 529.8 MB)
20/03/18 08:12:45 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1004
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[10] at map at DecisionTreeMetadata.scala:110) (first 15 tasks are for partitions Vector(0))
20/03/18 08:12:45 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
20/03/18 08:12:45 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 2247 bytes)
20/03/18 08:12:45 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1)
20/03/18 08:12:45 INFO storage.BlockManager: Found block rdd_4_0 locally
20/03/18 08:12:45 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 2063 bytes result sent to driver
20/03/18 08:12:45 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 66 ms on localhost (executor driver) (1/1)
20/03/18 08:12:45 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
20/03/18 08:12:45 INFO scheduler.DAGScheduler: ResultStage 1 (take at DecisionTreeMetadata.scala:110) finished in 0.067 s
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Job 1 finished: take at DecisionTreeMetadata.scala:110, took 0.105416 s
20/03/18 08:12:45 INFO spark.SparkContext: Starting job: count at DecisionTreeMetadata.scala:114
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Got job 2 (count at DecisionTreeMetadata.scala:114) with 1 output partitions
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (count at DecisionTreeMetadata.scala:114)
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Parents of final stage: List()
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Missing parents: List()
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[9] at retag at RandomForest.scala:135), which has no missing parents
20/03/18 08:12:45 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 4.3 KB, free 529.5 MB)
20/03/18 08:12:45 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 2.3 KB, free 529.5 MB)
20/03/18 08:12:45 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:44453 (size: 2.3 KB, free: 529.8 MB)
20/03/18 08:12:45 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1004
20/03/18 08:12:45 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[9] at retag at RandomForest.scala:135) (first 15 tasks are for partitions Vector(0))
20/03/18 08:12:45 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
20/03/18 08:12:45 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 2247 bytes)
20/03/18 08:12:45 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2)
20/03/18 08:12:45 INFO storage.BlockManager: Found block rdd_4_0 locally
20/03/18 08:12:46 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 2). 2058 bytes result sent to driver
20/03/18 08:12:46 INFO scheduler.DAGScheduler: ResultStage 2 (count at DecisionTreeMetadata.scala:114) finished in 0.039 s
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Job 2 finished: count at DecisionTreeMetadata.scala:114, took 0.074757 s
20/03/18 08:12:46 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 39 ms on localhost (executor driver) (1/1)
20/03/18 08:12:46 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
20/03/18 08:12:46 INFO spark.SparkContext: Starting job: collectAsMap at DecisionTree.scala:1043
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Registering RDD 12 (flatMap at DecisionTree.scala:1040)
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Got job 3 (collectAsMap at DecisionTree.scala:1043) with 1 output partitions
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Final stage: ResultStage 4 (collectAsMap at DecisionTree.scala:1043)
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 3)
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 3)
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[12] at flatMap at DecisionTree.scala:1040), which has no missing parents
20/03/18 08:12:46 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 14.2 KB, free 529.5 MB)
20/03/18 08:12:46 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 6.8 KB, free 529.5 MB)
20/03/18 08:12:46 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:44453 (size: 6.8 KB, free: 529.8 MB)
20/03/18 08:12:46 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1004
20/03/18 08:12:46 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[12] at flatMap at DecisionTree.scala:1040) (first 15 tasks are for partitions Vector(0))
20/03/18 08:12:46 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
20/03/18 08:12:46 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, executor driver, partition 0, PROCESS_LOCAL, 2345 bytes)
20/03/18 08:12:46 INFO executor.Executor: Running task 0.0 in stage 3.0 (TID 3)
20/03/18 08:12:46 INFO storage.BlockManager: Found block rdd_4_0 locally
20/03/18 08:12:46 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on localhost:44453 in memory (size: 2.3 KB, free: 529.8 MB)
20/03/18 08:12:46 INFO spark.ContextCleaner: Cleaned accumulator 3
20/03/18 08:12:46 INFO storage.BlockManagerInfo: Removed broadcast_2_piece0 on localhost:44453 in memory (size: 2.5 KB, free: 529.8 MB)
20/03/18 08:12:46 INFO spark.ContextCleaner: Cleaned accumulator 2
20/03/18 08:12:46 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:44453 in memory (size: 2.1 KB, free: 529.8 MB)
20/03/18 08:12:46 INFO spark.ContextCleaner: Cleaned accumulator 1
20/03/18 08:12:47 INFO executor.Executor: Finished task 0.0 in stage 3.0 (TID 3). 2236 bytes result sent to driver
20/03/18 08:12:47 INFO scheduler.DAGScheduler: ShuffleMapStage 3 (flatMap at DecisionTree.scala:1040) finished in 1.394 s
20/03/18 08:12:47 INFO scheduler.DAGScheduler: looking for newly runnable stages
20/03/18 08:12:47 INFO scheduler.DAGScheduler: running: Set()
20/03/18 08:12:47 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 4)
20/03/18 08:12:47 INFO scheduler.DAGScheduler: failed: Set()
20/03/18 08:12:47 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[14] at map at DecisionTree.scala:1042), which has no missing parents
20/03/18 08:12:47 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 1393 ms on localhost (executor driver) (1/1)
20/03/18 08:12:47 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
20/03/18 08:12:47 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 19.3 KB, free 529.5 MB)
20/03/18 08:12:47 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 8.2 KB, free 529.5 MB)
20/03/18 08:12:47 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:44453 (size: 8.2 KB, free: 529.8 MB)
20/03/18 08:12:47 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1004
20/03/18 08:12:47 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[14] at map at DecisionTree.scala:1042) (first 15 tasks are for partitions Vector(0))
20/03/18 08:12:47 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
20/03/18 08:12:47 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, executor driver, partition 0, NODE_LOCAL, 1978 bytes)
20/03/18 08:12:47 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 4)
20/03/18 08:12:47 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
20/03/18 08:12:47 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 16 ms
20/03/18 08:12:48 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 4). 333028 bytes result sent to driver
20/03/18 08:12:48 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 1128 ms on localhost (executor driver) (1/1)
20/03/18 08:12:48 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
20/03/18 08:12:48 INFO scheduler.DAGScheduler: ResultStage 4 (collectAsMap at DecisionTree.scala:1043) finished in 1.115 s
20/03/18 08:12:48 INFO scheduler.DAGScheduler: Job 3 finished: collectAsMap at DecisionTree.scala:1043, took 2.630181 s
20/03/18 08:12:48 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 40.0 B, free 529.5 MB)
20/03/18 08:12:48 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 24.0 B, free 529.5 MB)
20/03/18 08:12:48 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:44453 (size: 24.0 B, free: 529.8 MB)
20/03/18 08:12:48 INFO spark.SparkContext: Created broadcast 6 from broadcast at DecisionTree.scala:601
20/03/18 08:12:49 INFO spark.SparkContext: Starting job: collectAsMap at DecisionTree.scala:651
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Registering RDD 17 (mapPartitions at DecisionTree.scala:622)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Got job 4 (collectAsMap at DecisionTree.scala:651) with 1 output partitions
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (collectAsMap at DecisionTree.scala:651)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 5)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 5)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[17] at mapPartitions at DecisionTree.scala:622), which has no missing parents
20/03/18 08:12:49 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 461.9 KB, free 529.0 MB)
20/03/18 08:12:49 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 113.3 KB, free 528.9 MB)
20/03/18 08:12:49 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:44453 (size: 113.3 KB, free: 529.7 MB)
20/03/18 08:12:49 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1004
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[17] at mapPartitions at DecisionTree.scala:622) (first 15 tasks are for partitions Vector(0))
20/03/18 08:12:49 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
20/03/18 08:12:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, executor driver, partition 0, PROCESS_LOCAL, 2236 bytes)
20/03/18 08:12:49 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 5)
20/03/18 08:12:49 INFO spark.CacheManager: Partition rdd_16_0 not found, computing it
20/03/18 08:12:49 INFO storage.BlockManager: Found block rdd_4_0 locally
20/03/18 08:12:49 INFO storage.MemoryStore: Block rdd_16_0 stored as values in memory (estimated size 189.9 KB, free 528.7 MB)
20/03/18 08:12:49 INFO storage.BlockManagerInfo: Added rdd_16_0 in memory on localhost:44453 (size: 189.9 KB, free: 529.5 MB)
20/03/18 08:12:49 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 5). 2816 bytes result sent to driver
20/03/18 08:12:49 INFO scheduler.DAGScheduler: ShuffleMapStage 5 (mapPartitions at DecisionTree.scala:622) finished in 0.294 s
20/03/18 08:12:49 INFO scheduler.DAGScheduler: looking for newly runnable stages
20/03/18 08:12:49 INFO scheduler.DAGScheduler: running: Set()
20/03/18 08:12:49 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: failed: Set()
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[19] at map at DecisionTree.scala:642), which has no missing parents
20/03/18 08:12:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 293 ms on localhost (executor driver) (1/1)
20/03/18 08:12:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
20/03/18 08:12:49 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 200.6 KB, free 528.5 MB)
20/03/18 08:12:49 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 43.0 KB, free 528.5 MB)
20/03/18 08:12:49 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:44453 (size: 43.0 KB, free: 529.5 MB)
20/03/18 08:12:49 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1004
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[19] at map at DecisionTree.scala:642) (first 15 tasks are for partitions Vector(0))
20/03/18 08:12:49 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
20/03/18 08:12:49 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, executor driver, partition 0, NODE_LOCAL, 1978 bytes)
20/03/18 08:12:49 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 6)
20/03/18 08:12:49 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
20/03/18 08:12:49 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
20/03/18 08:12:49 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 6). 1461 bytes result sent to driver
20/03/18 08:12:49 INFO scheduler.DAGScheduler: ResultStage 6 (collectAsMap at DecisionTree.scala:651) finished in 0.270 s
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Job 4 finished: collectAsMap at DecisionTree.scala:651, took 0.694949 s
20/03/18 08:12:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 274 ms on localhost (executor driver) (1/1)
20/03/18 08:12:49 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
20/03/18 08:12:49 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 40.0 B, free 528.5 MB)
20/03/18 08:12:49 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 24.0 B, free 528.5 MB)
20/03/18 08:12:49 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:44453 (size: 24.0 B, free: 529.5 MB)
20/03/18 08:12:49 INFO spark.SparkContext: Created broadcast 9 from broadcast at DecisionTree.scala:601
20/03/18 08:12:49 INFO spark.SparkContext: Starting job: collectAsMap at DecisionTree.scala:651
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Registering RDD 20 (mapPartitions at DecisionTree.scala:622)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Got job 5 (collectAsMap at DecisionTree.scala:651) with 1 output partitions
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Final stage: ResultStage 8 (collectAsMap at DecisionTree.scala:651)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 7)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 7)
20/03/18 08:12:49 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 7 (MapPartitionsRDD[20] at mapPartitions at DecisionTree.scala:622), which has no missing parents
20/03/18 08:13:33 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[14] at map at DecisionTree.scala:1042), which has no missing parents
20/03/18 08:13:33 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 19.3 KB, free 529.5 MB)
20/03/18 08:13:33 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 8.2 KB, free 529.5 MB)
20/03/18 08:13:33 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:59293 (size: 8.2 KB, free: 529.8 MB)
20/03/18 08:13:33 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1004
20/03/18 08:13:33 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[14] at map at DecisionTree.scala:1042) (first 15 tasks are for partitions Vector(0))
20/03/18 08:13:33 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
20/03/18 08:13:33 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, executor driver, partition 0, NODE_LOCAL, 1978 bytes)
20/03/18 08:13:33 INFO executor.Executor: Running task 0.0 in stage 4.0 (TID 4)
20/03/18 08:13:33 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
20/03/18 08:13:33 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 6 ms
20/03/18 08:13:34 INFO executor.Executor: Finished task 0.0 in stage 4.0 (TID 4). 325941 bytes result sent to driver
20/03/18 08:13:34 INFO scheduler.DAGScheduler: ResultStage 4 (collectAsMap at DecisionTree.scala:1043) finished in 1.321 s
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Job 3 finished: collectAsMap at DecisionTree.scala:1043, took 2.658891 s
20/03/18 08:13:34 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 1323 ms on localhost (executor driver) (1/1)
20/03/18 08:13:34 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
20/03/18 08:13:34 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 40.0 B, free 529.5 MB)
20/03/18 08:13:34 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 24.0 B, free 529.5 MB)
20/03/18 08:13:34 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:59293 (size: 24.0 B, free: 529.8 MB)
20/03/18 08:13:34 INFO spark.SparkContext: Created broadcast 6 from broadcast at DecisionTree.scala:601
20/03/18 08:13:34 INFO spark.SparkContext: Starting job: collectAsMap at DecisionTree.scala:651
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Registering RDD 17 (mapPartitions at DecisionTree.scala:622)
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Got job 4 (collectAsMap at DecisionTree.scala:651) with 1 output partitions
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (collectAsMap at DecisionTree.scala:651)
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 5)
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 5)
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[17] at mapPartitions at DecisionTree.scala:622), which has no missing parents
20/03/18 08:13:34 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 452.7 KB, free 529.0 MB)
20/03/18 08:13:34 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 111.2 KB, free 528.9 MB)
20/03/18 08:13:34 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:59293 (size: 111.2 KB, free: 529.7 MB)
20/03/18 08:13:34 INFO spark.SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1004
20/03/18 08:13:34 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[17] at mapPartitions at DecisionTree.scala:622) (first 15 tasks are for partitions Vector(0))
20/03/18 08:13:34 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
20/03/18 08:13:34 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, executor driver, partition 0, PROCESS_LOCAL, 2236 bytes)
20/03/18 08:13:34 INFO executor.Executor: Running task 0.0 in stage 5.0 (TID 5)
20/03/18 08:13:34 INFO spark.CacheManager: Partition rdd_16_0 not found, computing it
20/03/18 08:13:34 INFO storage.BlockManager: Found block rdd_4_0 locally
20/03/18 08:13:34 INFO storage.MemoryStore: Block rdd_16_0 stored as values in memory (estimated size 181.6 KB, free 528.8 MB)
20/03/18 08:13:34 INFO storage.BlockManagerInfo: Added rdd_16_0 in memory on localhost:59293 (size: 181.6 KB, free: 529.5 MB)
20/03/18 08:13:35 INFO executor.Executor: Finished task 0.0 in stage 5.0 (TID 5). 2816 bytes result sent to driver
20/03/18 08:13:35 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 275 ms on localhost (executor driver) (1/1)
20/03/18 08:13:35 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
20/03/18 08:13:35 INFO scheduler.DAGScheduler: ShuffleMapStage 5 (mapPartitions at DecisionTree.scala:622) finished in 0.274 s
20/03/18 08:13:35 INFO scheduler.DAGScheduler: looking for newly runnable stages
20/03/18 08:13:35 INFO scheduler.DAGScheduler: running: Set()
20/03/18 08:13:35 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 6)
20/03/18 08:13:35 INFO scheduler.DAGScheduler: failed: Set()
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[19] at map at DecisionTree.scala:642), which has no missing parents
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 196.0 KB, free 528.6 MB)
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 42.1 KB, free 528.5 MB)
20/03/18 08:13:35 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:59293 (size: 42.1 KB, free: 529.5 MB)
20/03/18 08:13:35 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1004
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[19] at map at DecisionTree.scala:642) (first 15 tasks are for partitions Vector(0))
20/03/18 08:13:35 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
20/03/18 08:13:35 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, executor driver, partition 0, NODE_LOCAL, 1978 bytes)
20/03/18 08:13:35 INFO executor.Executor: Running task 0.0 in stage 6.0 (TID 6)
20/03/18 08:13:35 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
20/03/18 08:13:35 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
20/03/18 08:13:35 INFO executor.Executor: Finished task 0.0 in stage 6.0 (TID 6). 1461 bytes result sent to driver
20/03/18 08:13:35 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 316 ms on localhost (executor driver) (1/1)
20/03/18 08:13:35 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
20/03/18 08:13:35 INFO scheduler.DAGScheduler: ResultStage 6 (collectAsMap at DecisionTree.scala:651) finished in 0.315 s
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Job 4 finished: collectAsMap at DecisionTree.scala:651, took 0.713543 s
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 40.0 B, free 528.5 MB)
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 24.0 B, free 528.5 MB)
20/03/18 08:13:35 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:59293 (size: 24.0 B, free: 529.5 MB)
20/03/18 08:13:35 INFO spark.SparkContext: Created broadcast 9 from broadcast at DecisionTree.scala:601
20/03/18 08:13:35 INFO spark.SparkContext: Starting job: collectAsMap at DecisionTree.scala:651
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Registering RDD 20 (mapPartitions at DecisionTree.scala:622)
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Got job 5 (collectAsMap at DecisionTree.scala:651) with 1 output partitions
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Final stage: ResultStage 8 (collectAsMap at DecisionTree.scala:651)
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 7)
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 7)
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 7 (MapPartitionsRDD[20] at mapPartitions at DecisionTree.scala:622), which has no missing parents
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 453.0 KB, free 528.1 MB)
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 111.4 KB, free 528.0 MB)
20/03/18 08:13:35 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:59293 (size: 111.4 KB, free: 529.4 MB)
20/03/18 08:13:35 INFO spark.SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:1004
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 7 (MapPartitionsRDD[20] at mapPartitions at DecisionTree.scala:622) (first 15 tasks are for partitions Vector(0))
20/03/18 08:13:35 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 1 tasks
20/03/18 08:13:35 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, localhost, executor driver, partition 0, PROCESS_LOCAL, 2236 bytes)
20/03/18 08:13:35 INFO executor.Executor: Running task 0.0 in stage 7.0 (TID 7)
20/03/18 08:13:35 INFO storage.BlockManager: Found block rdd_16_0 locally
20/03/18 08:13:35 INFO executor.Executor: Finished task 0.0 in stage 7.0 (TID 7). 2236 bytes result sent to driver
20/03/18 08:13:35 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 124 ms on localhost (executor driver) (1/1)
20/03/18 08:13:35 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool
20/03/18 08:13:35 INFO scheduler.DAGScheduler: ShuffleMapStage 7 (mapPartitions at DecisionTree.scala:622) finished in 0.126 s
20/03/18 08:13:35 INFO scheduler.DAGScheduler: looking for newly runnable stages
20/03/18 08:13:35 INFO scheduler.DAGScheduler: running: Set()
20/03/18 08:13:35 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 8)
20/03/18 08:13:35 INFO scheduler.DAGScheduler: failed: Set()
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[22] at map at DecisionTree.scala:642), which has no missing parents
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 196.0 KB, free 527.8 MB)
20/03/18 08:13:35 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 42.1 KB, free 527.7 MB)
20/03/18 08:13:35 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on localhost:59293 (size: 42.1 KB, free: 529.3 MB)
20/03/18 08:13:35 INFO spark.SparkContext: Created broadcast 11 from broadcast at DAGScheduler.scala:1004
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[22] at map at DecisionTree.scala:642) (first 15 tasks are for partitions Vector(0))
20/03/18 08:13:35 INFO scheduler.TaskSchedulerImpl: Adding task set 8.0 with 1 tasks
20/03/18 08:13:35 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 8, localhost, executor driver, partition 0, NODE_LOCAL, 1978 bytes)
20/03/18 08:13:35 INFO executor.Executor: Running task 0.0 in stage 8.0 (TID 8)
20/03/18 08:13:35 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
20/03/18 08:13:35 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
20/03/18 08:13:35 INFO executor.Executor: Finished task 0.0 in stage 8.0 (TID 8). 1461 bytes result sent to driver
20/03/18 08:13:35 INFO scheduler.DAGScheduler: ResultStage 8 (collectAsMap at DecisionTree.scala:651) finished in 0.100 s
20/03/18 08:13:35 INFO scheduler.DAGScheduler: Job 5 finished: collectAsMap at DecisionTree.scala:651, took 0.332057 s
20/03/18 08:13:35 INFO rdd.MapPartitionsRDD: Removing RDD 16 from persistence list
20/03/18 08:13:35 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 8) in 109 ms on localhost (executor driver) (1/1)
20/03/18 08:13:35 INFO storage.BlockManager: Removing RDD 16
20/03/18 08:13:35 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool
20/03/18 08:13:35 INFO tree.RandomForest: Internal timing for DecisionTree:
20/03/18 08:13:36 INFO tree.RandomForest:   init: 3.170973948
 total: 4.54518002
 findSplitsBins: 2.822473871
 findBestSplits: 1.342828208
 chooseSplits: 1.333617008
20/03/18 08:13:36 WARN util.ClosureCleaner: Expected a closure; got org.apache.spark.examples.mllib.DecisionTreeClassificationExample$$$Lambda$13/1555453155
20/03/18 08:13:36 WARN util.ClosureCleaner: Expected a closure; got org.apache.spark.examples.mllib.DecisionTreeClassificationExample$$$Lambda$14/200141801
20/03/18 08:13:36 INFO spark.SparkContext: Starting job: count at DecisionTreeClassificationExample.scala:58
20/03/18 08:13:36 INFO scheduler.DAGScheduler: Got job 6 (count at DecisionTreeClassificationExample.scala:58) with 1 output partitions
20/03/18 08:13:36 INFO scheduler.DAGScheduler: Final stage: ResultStage 9 (count at DecisionTreeClassificationExample.scala:58)
20/03/18 08:13:36 INFO scheduler.DAGScheduler: Parents of final stage: List()
20/03/18 08:13:36 INFO scheduler.DAGScheduler: Missing parents: List()
20/03/18 08:13:36 INFO scheduler.DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[24] at filter at DecisionTreeClassificationExample.scala:58), which has no missing parents
20/03/18 08:13:36 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 7.3 KB, free 527.9 MB)
20/03/18 08:13:36 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 3.9 KB, free 527.9 MB)
20/03/18 08:13:36 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on localhost:59293 (size: 3.9 KB, free: 529.5 MB)
20/03/18 08:13:36 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1004
20/03/18 08:13:36 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[24] at filter at DecisionTreeClassificationExample.scala:58) (first 15 tasks are for partitions Vector(0))
20/03/18 08:13:36 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 1 tasks
20/03/18 08:13:36 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 9, localhost, executor driver, partition 0, PROCESS_LOCAL, 2247 bytes)
20/03/18 08:13:36 INFO executor.Executor: Running task 0.0 in stage 9.0 (TID 9)
20/03/18 08:13:36 ERROR executor.Executor: Exception in task 0.0 in stage 9.0 (TID 9)
java.io.IOException: unexpected exception type
at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1682)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1254)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2075)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2175)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1248)
... 40 more
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
at org.apache.spark.examples.mllib.DecisionTreeClassificationExample$.$deserializeLambda$(DecisionTreeClassificationExample.scala)
... 50 more
Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
... 51 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.LambdaDeserialize
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 51 more
20/03/18 08:13:36 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 9.0 (TID 9, localhost, executor driver): java.io.IOException: unexpected exception type
at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1682)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1254)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2075)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2175)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1248)
... 40 more
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
at org.apache.spark.examples.mllib.DecisionTreeClassificationExample$.$deserializeLambda$(DecisionTreeClassificationExample.scala)
... 50 more
Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
... 51 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.LambdaDeserialize
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 51 more

20/03/18 08:13:36 ERROR scheduler.TaskSetManager: Task 0 in stage 9.0 failed 1 times; aborting job
20/03/18 08:13:36 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool
20/03/18 08:13:36 INFO scheduler.TaskSchedulerImpl: Cancelling stage 9
20/03/18 08:13:36 INFO scheduler.DAGScheduler: ResultStage 9 (count at DecisionTreeClassificationExample.scala:58) failed in 0.066 s due to Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9.0 (TID 9, localhost, executor driver): java.io.IOException: unexpected exception type
at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1682)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1254)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2075)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2175)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1248)
... 40 more
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
at org.apache.spark.examples.mllib.DecisionTreeClassificationExample$.$deserializeLambda$(DecisionTreeClassificationExample.scala)
... 50 more
Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
... 51 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.LambdaDeserialize
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 51 more

Driver stacktrace:
20/03/18 08:13:36 INFO scheduler.DAGScheduler: Job 6 failed: count at DecisionTreeClassificationExample.scala:58, took 0.129475 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9.0 (TID 9, localhost, executor driver): java.io.IOException: unexpected exception type
at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1682)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1254)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2075)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2175)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1248)
... 40 more
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
at org.apache.spark.examples.mllib.DecisionTreeClassificationExample$.$deserializeLambda$(DecisionTreeClassificationExample.scala)
... 50 more
Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
... 51 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.LambdaDeserialize
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 51 more

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1457)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1445)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1444)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1444)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1668)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1627)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1616)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1862)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1875)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1888)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1959)
at org.apache.spark.rdd.RDD.count(RDD.scala:1157)
at org.apache.spark.examples.mllib.DecisionTreeClassificationExample$.main(DecisionTreeClassificationExample.scala:58)
at org.apache.spark.examples.mllib.DecisionTreeClassificationExample.main(DecisionTreeClassificationExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: unexpected exception type
at java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1682)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1254)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2075)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2175)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2284)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2208)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2066)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1570)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:430)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.lang.invoke.SerializedLambda.readResolve(SerializedLambda.java:230)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1248)
... 40 more
Caused by: java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
at org.apache.spark.examples.mllib.DecisionTreeClassificationExample$.$deserializeLambda$(DecisionTreeClassificationExample.scala)
... 50 more
Caused by: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
... 51 more
Caused by: java.lang.ClassNotFoundException: scala.runtime.LambdaDeserialize
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 51 more
20/03/18 08:13:36 INFO spark.SparkContext: Invoking stop() from shutdown hook
20/03/18 08:13:36 INFO ui.SparkUI: Stopped Spark web UI at http://10.0.2.15:4040
20/03/18 08:13:36 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/03/18 08:13:36 INFO storage.MemoryStore: MemoryStore cleared
20/03/18 08:13:36 INFO storage.BlockManager: BlockManager stopped
20/03/18 08:13:36 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
20/03/18 08:13:36 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/03/18 08:13:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
20/03/18 08:13:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
20/03/18 08:13:36 INFO spark.SparkContext: Successfully stopped SparkContext
20/03/18 08:13:36 INFO util.ShutdownHookManager: Shutdown hook called
20/03/18 08:13:36 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-57932eae-5fbc-4e79-b8e2-f1a2ae7deeb6





The POM.xml




How can I resolve this issue? Thanks










 
I don't get it. A whale wearing overalls? How does that even work? It's like a tiny ad wearing overalls.
Java file APIs (DOC, XLS, PDF, and many more)
https://products.aspose.com/total/java
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!