Cos'è la scintilla della libreria di versioni SparkSession

Question

Codice Spark con SparkSession.

import org.apache.spark.SparkConf 
    import org.apache.spark.SparkContext 

    val conf = SparkSession.builder 
    .master("local") 
    .appName("testing") 
    .enableHiveSupport() // <- enable Hive support. 
    .getOrCreate() 

Codice pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>com.cms.spark</groupId> 
    <artifactId>cms-spark</artifactId> 
    <version>0.0.1-SNAPSHOT</version> 
    <name>cms-spark</name> 

    <pluginRepositories> 
     <pluginRepository> 
      <id>scala-tools.org</id> 
      <name>Scala-tools Maven2 Repository</name> 
      <url>http://scala-tools.org/repo-releases</url> 
     </pluginRepository> 
    </pluginRepositories> 

    <dependencies> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.10</artifactId> 
      <version>1.6.0</version> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-sql_2.10</artifactId> 
      <version>1.6.0</version> 
     </dependency> 

     <dependency> 
      <groupId>com.databricks</groupId> 
      <artifactId>spark-csv_2.10</artifactId> 
      <version>1.4.0</version> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-hive_2.10</artifactId> 
      <version>1.5.2</version> 
     </dependency> 

     <dependency> 
      <groupId>org.jsoup</groupId> 
      <artifactId>jsoup</artifactId> 
      <version>1.8.3</version> 
     </dependency> 

    </dependencies> 

    <build> 
     <plugins> 
      <plugin> 
       <artifactId>maven-assembly-plugin</artifactId> 
       <version>2.5.3</version> 
       <configuration> 
        <descriptorRefs> 
         <descriptorRef>jar-with-dependencies</descriptorRef> 
        </descriptorRefs> 
       </configuration> 
       <executions> 
        <execution> 
         <id>make-assembly</id> <!-- this is used for inheritance merges --> 
         <phase>install</phase> <!-- bind to the packaging phase --> 
         <goals> 
          <goal>single</goal> 
         </goals> 
        </execution> 
       </executions> 
      </plugin> 
     </plugins> 

    </build> 
</project> 

ho qualche problema. Creo la scintilla del codice con SparkSession, iam ottenere problemi SparkSession non trova nella libreria SparkSql. Quindi iam non può lanciare la scintilla del codice. Iam domanda qual è la versione per trovare SparkSession nella libreria Spark. Do codice pom.xml.

Grazie.

Answer 1

È necessario Spark 2.0 per utilizzare SparkSession. E 'disponibile in Maven repository snapshot di centrale come per ora:

groupId = org.apache.spark 
artifactId = spark-core_2.11 
version = 2.0.0-SNAPSHOT 

La stessa versione deve essere specificato per altri manufatti Spark. Nota che 2.0 è ancora in beta e dovrebbe essere stabile tra circa un mese, AFAIK.

Aggiornamento . In alternativa, è possibile utilizzare Cloudera fork di Spark 2.0:

groupId = org.apache.spark 
artifactId = spark-core_2.11 
version = 2.0.0-cloudera1-SNAPSHOT 

repository Cloudera deve essere specificato nella lista repository Maven:

<repository> 
    <id>cloudera</id> 
    <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> 
</repository> 

Answer 2

<dependency> 
    <groupId>org.apache.spark</groupId> 
    <artifactId>spark-core_2.10</artifactId> 
    <version>2.0.0</version> 
</dependency> 

Answer 3

è necessario entrambi i manufatti di base e SQL

<repositories> 
    <repository> 
     <id>cloudera</id> 
     <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url> 
    </repository> 
</repositories> 
<dependencies> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_2.11</artifactId> 
     <version>2.0.0-cloudera1-SNAPSHOT</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-sql_2.11</artifactId> 
     <version>2.0.0-cloudera1-SNAPSHOT</version> 
    </dependency> 
</dependencies>