Impossibile distribuire il lavoro Spark locale, il worker non riesce con EndPointAssociationError

Question

Ho scaricato Spark 1.1.0 e lo ho creato con "sbt assembly". Si tenta di eseguire l'esempio Spark-Cassandra project:

import org.apache.spark.{SparkContext, SparkConf} 
import org.apache.spark.SparkContext._ 
import com.datastax.spark.connector._ 

object Test { 
    def main (args: Array[String]) { 
    val conf = new SparkConf(true) 
     .set("spark.cassandra.connection.host", "127.0.0.1") 

    val sc = new SparkContext("spark://127.0.0.1:7077", "test", conf) 

    val rdd = sc.cassandraTable("test", "kv") 
    println(rdd.count) 
    println(rdd.first) 
    println(rdd.map(_.getInt("value")).sum) 
    } 
} 

Il master scintilla viene avviato eseguendo il ./start-master.sh nella directory sbin. Quindi avviare uno slave sulla stessa macchina con il comando:

./start-slave.sh 0 scintilla: //127.0.0.1: 7077

Dove la scintilla: // .. indirizzo è quello prelevato da localhost: 8080 (dashboard spark).

Questo va tutto bene, la dashboard vede l'operatore. Quindi eseguo il programma Scala e ottengo ClassNotFoundException. Questo sembra essere un po 'fuorviante, in quanto i registri dei lavoratori segnalare questo errore:

14/10/29 12:23:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@localhost:33137] -> [akka.tcp://sparkExecutor@localhost:37279]: Error [Association failed with [akka.tcp://# 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@localhost:37279] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: localhost/127.0.0.1:37279 
] 

Quando avvio il programma di esempio c'è questo avvertimento:

14/10/29 12:22:31 WARN util.Utils: Your hostname, bas-HP-EliteBook-8530w resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0) 
14/10/29 12:22:31 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address 

Non so perché questo avvertimento è Là. Il mio file hosts è il seguente:

127.0.0.1 localhost 
127.0.0.1 bas-HP-EliteBook-8530w 

# The following lines are desirable for IPv6 capable hosts 
::1  ip6-localhost ip6-loopback 
fe00::0 ip6-localnet 
ff00::0 ip6-mcastprefix 
ff02::1 ip6-allnodes 
ff02::2 ip6-allrouters 

conf/spark-env.sh ha queste opzioni abilitate:

SPARK_LOCAL_IP=127.0.0.1 
SPARK_MASTER_IP=127.0.0.1 

registro lavoratore completa:

Spark Command: java -cp ::/home/bas/Downloads/spark-1.1.0.backup/conf:/home/bas/Downloads/spark-1.1.0.backup/assembly/target/scala-2.10/spark-assembly-1.1.0-hadoop1.0.4.jar -XX:MaxPermSize# 
======================================== 

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
14/10/29 12:21:16 INFO Worker: Registered signal handlers for [TERM, HUP, INT] 
14/10/29 12:21:16 INFO SecurityManager: Changing view acls to: bas, 
14/10/29 12:21:16 INFO SecurityManager: Changing modify acls to: bas, 
14/10/29 12:21:16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bas,); users with modify permissions: Set(bas,) 
14/10/29 12:21:17 INFO Slf4jLogger: Slf4jLogger started 
14/10/29 12:21:17 INFO Remoting: Starting remoting 
14/10/29 12:21:17 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@localhost:33137] 
14/10/29 12:21:17 INFO Utils: Successfully started service 'sparkWorker' on port 33137. 
14/10/29 12:21:17 INFO Worker: Starting Spark worker localhost:33137 with 2 cores, 2.8 GB RAM 
14/10/29 12:21:17 INFO Worker: Spark home: /home/bas/Downloads/spark-1.1.0.backup 
14/10/29 12:21:17 INFO Utils: Successfully started service 'WorkerUI' on port 8081. 
14/10/29 12:21:17 INFO WorkerWebUI: Started WorkerWebUI at http://localhost:8081 
14/10/29 12:21:17 INFO Worker: Connecting to master spark://127.0.0.1:7077... 
14/10/29 12:21:18 INFO Worker: Successfully registered with master spark://127.0.0.1:7077 
14/10/29 12:22:34 INFO Worker: Asked to launch executor app-20141029122234-0000/0 for test 
14/10/29 12:22:35 INFO ExecutorRunner: Launch command: "java" "-cp" "::/home/bas/Downloads/spark-1.1.0.backup/conf:/home/bas/Downloads/spark-1.1.0.backup/assembly/target/scala-2.10/spark-a# 
14/10/29 12:23:05 INFO Worker: Asked to kill executor app-20141029122234-0000/0 
14/10/29 12:23:05 INFO ExecutorRunner: Runner thread for executor app-20141029122234-0000/0 interrupted 
14/10/29 12:23:05 INFO ExecutorRunner: Killing process! 
14/10/29 12:23:05 INFO Worker: Executor app-20141029122234-0000/0 finished with state KILLED exitStatus 1 
14/10/29 12:23:05 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkWorker/deadLetters] to Actor[akka://sparkWorker/sy# 
14/10/29 12:23:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@localhost:33137] -> [akka.tcp://sparkExecutor@localhost:37279]: Error [Association failed with [akka.tcp://# 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@localhost:37279] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: localhost/127.0.0.1:37279 
] 
14/10/29 12:23:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@localhost:33137] -> [akka.tcp://sparkExecutor@localhost:37279]: Error [Association failed with [akka.tcp://# 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@localhost:37279] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: localhost/127.0.0.1:37279 
] 
14/10/29 12:23:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkWorker@localhost:33137] -> [akka.tcp://sparkExecutor@localhost:37279]: Error [Association failed with [akka.tcp://# 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@localhost:37279] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: localhost/127.0.0.1:37279 
] 

registro Master Full:

park Command: java -cp ::/home/bas/Downloads/spark-1.1.0.backup/conf:/home/bas/Downloads/spark-1.1.0.backup/assembly/target/scala-2.10/spark-assembly-1.1.0-hadoop1.0.4.jar -XX:MaxPermSize# 
======================================== 

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
14/10/29 12:20:52 INFO Master: Registered signal handlers for [TERM, HUP, INT] 
14/10/29 12:20:52 INFO SecurityManager: Changing view acls to: bas, 
14/10/29 12:20:52 INFO SecurityManager: Changing modify acls to: bas, 
14/10/29 12:20:52 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bas,); users with modify permissions: Set(bas,) 
14/10/29 12:20:53 INFO Slf4jLogger: Slf4jLogger started 
14/10/29 12:20:53 INFO Remoting: Starting remoting 
14/10/29 12:20:53 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkMaster@127.0.0.1:7077] 
14/10/29 12:20:53 INFO Utils: Successfully started service 'sparkMaster' on port 7077. 
14/10/29 12:20:54 INFO Master: Starting Spark master at spark://127.0.0.1:7077 
14/10/29 12:20:54 INFO Utils: Successfully started service 'MasterUI' on port 8080. 
14/10/29 12:20:54 INFO MasterWebUI: Started MasterWebUI at http://localhost:8080 
14/10/29 12:20:54 INFO Master: I have been elected leader! New state: ALIVE 
14/10/29 12:21:18 INFO Master: Registering worker localhost:33137 with 2 cores, 2.8 GB RAM 
14/10/29 12:22:34 INFO Master: Registering app test 
14/10/29 12:22:34 INFO Master: Registered app test with ID app-20141029122234-0000 
14/10/29 12:22:34 INFO Master: Launching executor app-20141029122234-0000/0 on worker worker-20141029122117-localhost-33137 
14/10/29 12:23:05 INFO Master: akka.tcp://sparkDriver@192.168.122.1:40211 got disassociated, removing it. 
14/10/29 12:23:05 INFO Master: Removing app app-20141029122234-0000 
14/10/29 12:23:05 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/sy# 
14/10/29 12:23:05 INFO Master: akka.tcp://sparkDriver@192.168.122.1:40211 got disassociated, removing it. 
14/10/29 12:23:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@127.0.0.1:7077] -> [akka.tcp://sparkDriver@192.168.122.1:40211]: Error [Association failed with [akka.tcp:/# 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@192.168.122.1:40211] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.122.1:40211 
] 
14/10/29 12:23:05 INFO Master: akka.tcp://sparkDriver@192.168.122.1:40211 got disassociated, removing it. 
14/10/29 12:23:05 INFO Master: akka.tcp://sparkDriver@192.168.122.1:40211 got disassociated, removing it. 
14/10/29 12:23:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@127.0.0.1:7077] -> [akka.tcp://sparkDriver@192.168.122.1:40211]: Error [Association failed with [akka.tcp:/# 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@192.168.122.1:40211] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.122.1:40211 
] 
14/10/29 12:23:05 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@127.0.0.1:7077] -> [akka.tcp://sparkDriver@192.168.122.1:40211]: Error [Association failed with [akka.tcp:/# 
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkDriver@192.168.122.1:40211] 
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /192.168.122.1:40211 
] 
14/10/29 12:23:05 INFO Master: akka.tcp://sparkDriver@192.168.122.1:40211 got disassociated, removing it. 
14/10/29 12:23:05 WARN Master: Got status update for unknown executor app-20141029122234-0000/0 

IntelliJ stacktrace:

/lib/idea_rt.jar com.intellij.rt.execution.application.AppMain Test 
    14/10/29 12:22:31 WARN util.Utils: Your hostname, bas-HP-EliteBook-8530w resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0) 
    14/10/29 12:22:31 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address 
    14/10/29 12:22:31 INFO spark.SecurityManager: Changing view acls to: bas, 
    14/10/29 12:22:31 INFO spark.SecurityManager: Changing modify acls to: bas, 
    14/10/29 12:22:31 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(bas,); users with modify permissions: Set(bas,) 
    14/10/29 12:22:31 INFO slf4j.Slf4jLogger: Slf4jLogger started 
    14/10/29 12:22:31 INFO Remoting: Starting remoting 
    14/10/29 12:22:32 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.122.1:40211] 
    14/10/29 12:22:32 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@192.168.122.1:40211] 
    14/10/29 12:22:32 INFO util.Utils: Successfully started service 'sparkDriver' on port 40211. 
    14/10/29 12:22:32 INFO spark.SparkEnv: Registering MapOutputTracker 
    14/10/29 12:22:32 INFO spark.SparkEnv: Registering BlockManagerMaster 
    14/10/29 12:22:32 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20141029122232-f3bf 
    14/10/29 12:22:32 INFO util.Utils: Successfully started service 'Connection manager for block manager' on port 34325. 
    14/10/29 12:22:32 INFO network.ConnectionManager: Bound socket to port 34325 with id = ConnectionManagerId(192.168.122.1,34325) 
    14/10/29 12:22:32 INFO storage.MemoryStore: MemoryStore started with capacity 470.3 MB 
    14/10/29 12:22:32 INFO storage.BlockManagerMaster: Trying to register BlockManager 
    14/10/29 12:22:32 INFO storage.BlockManagerMasterActor: Registering block manager 192.168.122.1:34325 with 470.3 MB RAM 
    14/10/29 12:22:32 INFO storage.BlockManagerMaster: Registered BlockManager 
    14/10/29 12:22:32 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-eea6f1f3-8f69-4900-87c7-6da17f1f3d76 
    14/10/29 12:22:32 INFO spark.HttpServer: Starting HTTP Server 
    14/10/29 12:22:32 INFO server.Server: jetty-8.1.14.v20131031 
    14/10/29 12:22:32 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:33688 
    14/10/29 12:22:32 INFO util.Utils: Successfully started service 'HTTP file server' on port 33688. 
    14/10/29 12:22:33 INFO server.Server: jetty-8.1.14.v20131031 
    14/10/29 12:22:33 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 
    14/10/29 12:22:33 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 
    14/10/29 12:22:33 INFO ui.SparkUI: Started SparkUI at http://192.168.122.1:4040 
    14/10/29 12:22:33 INFO client.AppClient$ClientActor: Connecting to master spark://127.0.0.1:7077... 
    14/10/29 12:22:33 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 
    14/10/29 12:22:34 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20141029122234-0000 
    14/10/29 12:22:34 INFO client.AppClient$ClientActor: Executor added: app-20141029122234-0000/0 on worker-20141029122117-localhost-33137 (localhost:33137) with 2 cores 
    14/10/29 12:22:34 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20141029122234-0000/0 on hostPort localhost:33137 with 2 cores, 512.0 MB RAM 
    14/10/29 12:22:34 INFO client.AppClient$ClientActor: Executor updated: app-20141029122234-0000/0 is now RUNNING 
    14/10/29 12:22:34 INFO cql.CassandraConnector: Connected to Cassandra cluster: Test Cluster 
    14/10/29 12:22:34 INFO core.Cluster: New Cassandra host /127.0.0.1:9042 added 
    14/10/29 12:22:34 INFO cql.LocalNodeFirstLoadBalancingPolicy: Adding host 127.0.0.1 (datacenter1) 
    14/10/29 12:22:34 INFO cql.LocalNodeFirstLoadBalancingPolicy: Adding host 127.0.0.1 (datacenter1) 
    14/10/29 12:22:34 INFO cql.LocalNodeFirstLoadBalancingPolicy: Adding host 127.0.0.1 (datacenter1) 
    14/10/29 12:22:34 INFO cql.LocalNodeFirstLoadBalancingPolicy: Adding host 127.0.0.1 (datacenter1) 
    14/10/29 12:22:35 INFO cql.CassandraConnector: Disconnected from Cassandra cluster: Test Cluster 
    14/10/29 12:22:35 INFO spark.SparkContext: Starting job: count at Test.scala:23 
    14/10/29 12:22:35 INFO scheduler.DAGScheduler: Got job 0 (count at Test.scala:23) with 1 output partitions (allowLocal=false) 
    14/10/29 12:22:35 INFO scheduler.DAGScheduler: Final stage: Stage 0(count at Test.scala:23) 
    14/10/29 12:22:35 INFO scheduler.DAGScheduler: Parents of final stage: List() 
    14/10/29 12:22:35 INFO scheduler.DAGScheduler: Missing parents: List() 
    14/10/29 12:22:35 INFO scheduler.DAGScheduler: Submitting Stage 0 (CassandraRDD[0] at RDD at CassandraRDD.scala:47), which has no missing parents 
    14/10/29 12:22:36 INFO storage.MemoryStore: ensureFreeSpace(4224) called with curMem=0, maxMem=493187235 
    14/10/29 12:22:36 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.1 KB, free 470.3 MB) 
    14/10/29 12:22:36 INFO storage.MemoryStore: ensureFreeSpace(2338) called with curMem=4224, maxMem=493187235 
    14/10/29 12:22:36 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.3 KB, free 470.3 MB) 
    14/10/29 12:22:36 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.122.1:34325 (size: 2.3 KB, free: 470.3 MB) 
    14/10/29 12:22:36 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0 
    14/10/29 12:22:36 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (CassandraRDD[0] at RDD at CassandraRDD.scala:47) 
    14/10/29 12:22:36 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 
    14/10/29 12:22:38 INFO cluster.SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@localhost:37279/user/Executor#2049453845] with ID 0 
    14/10/29 12:22:38 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, NODE_LOCAL, 23151 bytes) 
    14/10/29 12:22:38 INFO storage.BlockManagerMasterActor: Registering block manager localhost:37704 with 265.4 MB RAM 
    14/10/29 12:22:39 INFO network.ConnectionManager: Accepted connection from [192.168.122.1/192.168.122.1:36717] 
    14/10/29 12:22:39 INFO network.SendingConnection: Initiating connection to [localhost/127.0.0.1:37704] 
    14/10/29 12:22:39 INFO network.SendingConnection: Connected to [localhost/127.0.0.1:37704], 1 messages pending 
    14/10/29 12:22:39 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:37704 (size: 2.3 KB, free: 265.4 MB) 
    14/10/29 12:22:40 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.ClassNotFoundException: com.datastax.spark.connector.rdd.partitioner.CassandraPartition 
      java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
      java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
      java.security.AccessController.doPrivileged(Native Method) 
      java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
      java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
      java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
      java.lang.Class.forName0(Native Method) 
      java.lang.Class.forName(Class.java:270) 
      org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59) 
      java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612) 
      java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) 
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) 
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 
      java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 
      java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) 
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 
      java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) 
      org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) 
      org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) 
      org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) 
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
      java.lang.Thread.run(Thread.java:745) 
    14/10/29 12:22:40 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, localhost, NODE_LOCAL, 23151 bytes) 
    14/10/29 12:22:40 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) on executor localhost: java.lang.ClassNotFoundException (com.datastax.spark.connector.rdd.partitioner.CassandraPartition) [duplicate 1] 
    14/10/29 12:22:40 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, localhost, NODE_LOCAL, 23151 bytes) 
    14/10/29 12:22:40 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2) on executor localhost: java.lang.ClassNotFoundException (com.datastax.spark.connector.rdd.partitioner.CassandraPartition) [duplicate 2] 
    14/10/29 12:22:40 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, localhost, NODE_LOCAL, 23151 bytes) 
    14/10/29 12:22:40 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3) on executor localhost: java.lang.ClassNotFoundException (com.datastax.spark.connector.rdd.partitioner.CassandraPartition) [duplicate 3] 
    14/10/29 12:22:40 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 
    14/10/29 12:22:40 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
    14/10/29 12:22:40 INFO scheduler.TaskSchedulerImpl: Cancelling stage 0 
    14/10/29 12:22:40 INFO scheduler.DAGScheduler: Failed to run count at Test.scala:23 
    Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, localhost): java.lang.ClassNotFoundException: com.datastax.spark.connector.rdd.partitioner.CassandraPartition 
      java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
      java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
      java.security.AccessController.doPrivileged(Native Method) 
      java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
      java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
      java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
      java.lang.Class.forName0(Native Method) 
      java.lang.Class.forName(Class.java:270) 
      org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:59) 
      java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612) 
      java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) 
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) 
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 
      java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) 
      java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) 
      java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) 
      java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) 
      java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) 
      org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) 
      org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) 
      org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159) 
      java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
      java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
      java.lang.Thread.run(Thread.java:745) 
    Driver stacktrace: 
     at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173) 
     at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
     at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) 
     at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) 
     at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688) 
     at scala.Option.foreach(Option.scala:236) 
     at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688) 
     at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391) 
     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) 
     at akka.actor.ActorCell.invoke(ActorCell.scala:456) 
     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) 
     at akka.dispatch.Mailbox.run(Mailbox.scala:219) 
     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) 
     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) 
     at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) 
     at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) 
     at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 

    Process finished with exit code 1 

build.sbt:

name := "sparktest" 

version := "1.0" 

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3" withSources() withJavadoc() 

libraryDependencies += "org.apache.spark" % "spark-core" % "1.1.0" 

versione Scala-compiler = 2.10.4

versione Java 1.7.0_67 =

ho provato a fissare la scintilla IP locale come ad esempio:

System.setProperty("SPARK_LOCAL_IP", "127.0.0.1") 
println(System.getenv("SPARK_LOCAL_IP")) 

e simili t his:

scala.util.Properties.envOrElse("SPARK_LOCAL_IP", "127.0.0.1") 
println(System.getenv("SPARK_LOCAL_IP")) 

Ma stampa semplicemente null.

Grazie in anticipo per eventuali suggerimenti.

Edit: Modifica della configurazione scintilla come segue risolve il problema:

val conf = new SparkConf(true) 
     .set("spark.cassandra.connection.host", "127.0.0.1") 
     .set("spark.executor.extraClassPath", "/home/bas/Downloads/spark-cassandra-connector/spark-cassandra-connector-java/target/scala-2.10/spark-cassandra-connector-java-assembly-1.2.0-SNAPSHOT.jar") 

Si noti che questo non ha funzionato con il file jar SBT della scintilla cassandra-connettore, ho dovuto costruirlo dalla fonte. In effetti, ho rimosso tutte le dipendenze SBT e aggiunto la dipendenza spark come anche una build di origine.

Answer 1

Il motivo per cui si sta verificando l'errore è dovuto al fatto che Spark non ha spark-cassandra-connector sul classpath del lavoratore.

È possibile utilizzare uno dei plug-in sbt one-jar come sbt-assembly per creare assembled jar incluse tutte le dipendenze. E usarlo to submit vostro lavoro per Spark:

./bin/spark-submit \ 
    --class <main-class> 
    --master <master-url> \ 
    <assembled jar> \ 
    [application-arguments] 

Oppure si può utilizzare l'opzione Runtime Environment Spark spark.executor.extraClassPath e impostare classpath per l'applicazione a livello globale.