pyspark.RDD.withResources

RDD.withResources(profile)[source]

Specify a pyspark.resource.ResourceProfile to use when calculating this RDD. This is only supported on certain cluster managers and currently requires dynamic allocation to be enabled. It will result in new executors with the resources specified being acquired to calculate the RDD.

New in version 3.1.0.

Notes

This API is experimental

previous

pyspark.RDD.variance

next

pyspark.RDD.zip