Alok February 2016

NoClassDefFoundError in Spark Submit

I am working on a spark application . I am using a third party jar. I added it in my sbt file. The compilation was successful

> libraryDependencies += "org.springframework.security" %
> "spring-security-web" % "3.0.7.RELEASE

But when i submit spark job

spark-submit --master yarn-client --class MyClass target/scala-2.10/my-app.jar

It fails with following error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to 
stage failure: Task 0 in stage 48.0 failed 1 times, most recent failure: Lost task 0.0 in stage 48.0 (TID 3216, localhost): 
`java.lang.NoClassDefFoundError:` org/springframework/security/web/util/IpAddressMatcher

I also tried explicitly passing jar in spark-submit jar

spark-submit --master yarn-client --jars third-party-jars/spring-security-web-3.0.7.RELEASE.jar --class MyClass target/scala-2.10/my-app.jar

But then it fails with following error:

java.lang.NoClassDefFoundError: org/springframework/util/StringUtils
    at org.springframework.security.web.util.IpAddressMatcher.<init>(IpAddressMatcher.java:33)

Its similar error but on another class.

Any suggestion on how to resolve this?

Thanks

Answers


Thoram Mastero February 2016

Also you can use IntelijIDE. You will not need to adjust it always. You can download here for Ubuntu. https://apps.ubuntu.com/cat/applications/intellij-idea-ce/

Post Status

Asked in February 2016
Viewed 3,608 times
Voted 11
Answered 1 times

Search




Leave an answer