mercredi 4 mars 2015

IncompatibleClassChangeError when running spark from (spring) servlet

When running a simple spark job from a (spring) web app, I get an IncompatibleClassChangeError. It is probably due to some incompatible dependencies, but I cannot find which.


To put it simply, this is the issue with exception stack.


Here is the failing code:



// file conversion with spark
// --------------------------
JavaRDD<String> inputRdd = sparkContext.textFile(inputFile).cache();

Function fct = new Function<String,String>() {
@Override
public String call(String line) throws Exception {
return line.toUpperCase();
}
};
JavaRDD<Strineg> outputRdd = inputRdd.map(fct); // *** fail ***

outputRdd.saveAsTextFile(outputDir);


And it is called from a simple spring servlet:



@RequestMapping(method = RequestMethod.GET, value="/upper")
public @ResponseBody boolean upper(@RequestParam(required = true) String inputFile,
@RequestParam(required = true) String outputDir,
@RequestParam(required = false) String master,
@RequestParam(required = false) String namenode) {
if(master==null) master = "local";
SparkUpper.upper(inputFile, outputDir, master, namenode);

return true;
}


And here is the (maven) dependencies:



<dependencies>
<!-- spring -->
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>${spring.version}</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>${spring.version}</version>
</dependency>

<!-- spark & hadoop -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.5.2</version>
</dependency>

</dependencies>


The strange thing is that is basically do the same, with same spark dependencies but called from a servlet, as in this project that works fine.


Any idea would be greatly appreciated.


Aucun commentaire:

Enregistrer un commentaire