Home Ask Login Register

Developers Planet

Your answer is one click away!

Alok February 2016

NoClassDefFoundError in Spark Submit

I am working on a spark application . I am using a third party jar. I added it in my sbt file. The compilation was successful

> libraryDependencies += "org.springframework.security" %
> "spring-security-web" % "3.0.7.RELEASE

But when i submit spark job

spark-submit --master yarn-client --class MyClass target/scala-2.10/my-app.jar

It fails with following error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to 
stage failure: Task 0 in stage 48.0 failed 1 times, most recent failure: Lost task 0.0 in stage 48.0 (TID 3216, localhost): 
`java.lang.NoClassDefFoundError:` org/springframework/security/web/util/IpAddressMatcher

I also tried explicitly passing jar in spark-submit jar

spark-submit --master yarn-client --jars third-party-jars/spring-security-web-3.0.7.RELEASE.jar --class MyClass target/scala-2.10/my-app.jar

But then it fails with following error:

java.lang.NoClassDefFoundError: org/springframework/util/StringUtils
    at org.springframework.security.web.util.IpAddressMatcher.<init>(IpAddressMatcher.java:33)

Its similar error but on another class.

Any suggestion on how to resolve this?



Thoram Mastero February 2016

Also you can use IntelijIDE. You will not need to adjust it always. You can download here for Ubuntu. https://apps.ubuntu.com/cat/applications/intellij-idea-ce/

Post Status

Asked in February 2016
Viewed 3,608 times
Voted 11
Answered 1 times


Leave an answer

Quote of the day: live life