葛亭错误的NoClassDefFoundError：org.apache.spark.internal.Logging卡夫卡星火流

Question

我对下面的Java代码运行在获得运行时的错误，

是否有任何依赖关系必须包括用于记录像log4js或什么？

为什么这个错误不会出现在编译的时候，这样更容易..

这里是我的Java代码，

SparkConf sparkConf = new SparkConf().setAppName("JavaKafkaWordCount11").setMaster("local[*]"); 
     sparkConf.set("spark.streaming.concurrentJobs", "3"); 

     // Create the context with 2 seconds batch size 
     JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(3000)); 

     Map<String, Object> kafkaParams = new HashMap<>(); 
     kafkaParams.put("bootstrap.servers", "x.xx.xxx.xxx:9092"); 
     kafkaParams.put("key.deserializer", StringDeserializer.class); 
     kafkaParams.put("value.deserializer", StringDeserializer.class); 
     kafkaParams.put("group.id", "use_a_separate_group_id_for_each_stream"); 
     kafkaParams.put("auto.offset.reset", "latest"); 
     kafkaParams.put("enable.auto.commit", true); 

     Collection<String> topics = Arrays.asList("topicName"); 

     final JavaInputDStream<ConsumerRecord<String, String>> stream = KafkaUtils.createDirectStream(jssc, 
       LocationStrategies.PreferConsistent(), 
       ConsumerStrategies.<String, String>Subscribe(topics, kafkaParams)); 

     stream.mapToPair(new PairFunction<ConsumerRecord<String, String>, String, String>() { 
      @Override 
      public Tuple2<String, String> call(ConsumerRecord<String, String> record) throws Exception { 

       System.out.println("file data"); 
       return new Tuple2<>(record.key(), record.value()); 
      } 
     }); 

依赖使用，

<dependency> 
      <groupId>log4j</groupId> 
      <artifactId>log4j</artifactId> 
      <version>1.2.17</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.10</artifactId> 
      <version>1.6.2</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming-kafka_2.10</artifactId> 
      <version>1.6.2</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-mllib_2.10</artifactId> 
      <version>1.6.2</version> 
     </dependency> 
     <!-- <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId> 
      <version>0.10.0.1</version> </dependency> --> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming-kafka-0-10_2.10</artifactId> 
      <version>2.0.0</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming-twitter_2.10</artifactId> 
      <version>1.6.2</version> 
     </dependency> 
     <dependency> 
      <groupId>org.restlet.jee</groupId> 
      <artifactId>org.restlet</artifactId> 
      <version>2.0.10</version> 
     </dependency> 

获得下面的错误，

Exception in thread "main" java.lang.NoClassDefFoundError: org.apache.spark.internal.Logging 
    at java.lang.ClassLoader.defineClassImpl(Native Method) 
    at java.lang.ClassLoader.defineClass(ClassLoader.java:346) 
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:154) 
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:727) 
    at java.net.URLClassLoader.access$400(URLClassLoader.java:95) 
    at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:1182) 
    at java.security.AccessController.doPrivileged(AccessController.java:686) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:602) 
    at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:846) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:825) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:325) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:805) 
    at org.apache.spark.streaming.kafka010.ConsumerStrategies.Subscribe(ConsumerStrategy.scala) 
    at spark.KafkaConsumerDirectStream.main(KafkaConsumerDirectStream.java:45) 
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:607) 
    at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:846) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:825) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:325) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:805) 
    ... 14 more 

Answer 1

问题得到了通过建立上述提到的依赖关系解析进入下面的命令，

<dependency> 
      <groupId>log4j</groupId> 
      <artifactId>log4j</artifactId> 
      <version>1.2.17</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.10</artifactId> 
      <version>2.0.0</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming-kafka-0-10_2.10</artifactId> 
      <version>2.0.0</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-mllib_2.10</artifactId> 
      <version>2.0.0</version> 
     </dependency> 

Answer 2

import org.apache.spark.Logging，它在Spark版本1.5.2或更高版本中可用，所以我建议您使用1.5.2或更高版本的spark。

你正在使用哪种spark版本？

还有另一个解决此问题的依赖项，并且与Spark 2.x兼容。

对于SBT，使用这种依赖性：

“org.apache.bahir” %% “火花流，Twitter的” ％ “2.0.0”