Class SparkContext

Create a new SparkContext.

Parameters:

• master - Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
• jobName - A name for your job, to display on the cluster web UI
• sparkHome - Location where Spark is installed on cluster nodes.
• pyFiles - Collection of .zip or .py files to send to the cluster and add to PYTHONPATH. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.
• environment - A dictionary of environment variables to set on worker nodes.
• batchSize - The number of Python objects represented as a single Java object. Set 1 to disable batching or -1 to use an unlimited batch size.

Overrides: object.__init__

Distribute a local Python collection to form an RDD.

>>> sc.parallelize(range(5), 5).glom().collect()
[[0], [1], [2], [3], [4]]

Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.

Create an Accumulator with the given initial value, using a given AccumulatorParam helper object to define how to add values of the data type if provided. Default AccumulatorParams are used for integers and floating-point numbers if you do not provide one. For other types, a custom AccumulatorParam can be used.

Add a file to be downloaded with this Spark job on every node. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

To access the file in Spark jobs, use SparkFiles.get(path) to find its download location.

>>> from pyspark import SparkFiles
>>> path = os.path.join(tempdir, "test.txt")
>>> with open(path, "w") as testFile:
...    testFile.write("100")
>>> sc.addFile(path)
>>> def func(iterator):
...    with open(SparkFiles.get("test.txt")) as testFile:
...        fileVal = int(testFile.readline())
...        return [x * 100 for x in iterator]
>>> sc.parallelize([1, 2, 3, 4]).mapPartitions(func).collect()
[100, 200, 300, 400]

Add a .py or .zip dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

Set the directory under which RDDs are going to be checkpointed. The directory must be a HDFS path if running on a cluster.

If the directory does not exist, it will be created. If the directory exists and useExisting is set to true, then the exisiting directory will be used. Otherwise an exception will be thrown to prevent accidental overriding of checkpoint files in the existing directory.

defaultParallelism

Default level of parallelism to use when not given by user (e.g. for reduce tasks)

Get Method:

unreachable.defaultParallelism(self) - Default level of parallelism to use when not given by user (e.g.