My code is:

import org.apache.spark.SparkContext

It can run in interactive mode, but when I use scalac to compile it, I got the following error message:

object apache is not a member of package org

This seems to be the problem of path, but I do not know exactly how to configure the path.

5 Answers

You need to specify the path of libraries used when compiling your Scala code. This is usually not done manually, but using a build tool such as Maven or sbt. You can find a minimal sbt setup at

1

I had this issue because I had the wrong scope for my spark dependency in my pom.xml file. This is wrong:

<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>${spark.version}</version> <scope>test</scope> <!-- will not be available during compile phase -->
</dependency>

Changing test -> provided will work and will not include spark in your uberjar which is what you will almost certainly want:

<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>${spark.version}</version> <scope>provided</scope>
</dependency>

2

One easy way (if you're using the Play Framework) is to look up the LibraryDependacy in the Maven Repository, choose the version, choose SBT and then add it to the bottom of your project/build.sbt file, like so:

//
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.2"

Afterwards, you'll want to enter reload into the sbt console and then compile. This can feel a little foreign if you're coming from pip or js but the Maven Repo is your friend.

I was facing this issue in the sbt interactive session.

Resolved the is by simple executing reload in the session.

Hope this helps!

I have had the same issue when running scala wordcount program on spark frame work. I was using eclipse as IDE and maven as build tool. I just removed the scope of spark frame work in POM file from "test"-->"provided", like below. It worked. provided

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.