Updating Spark Configuration Values

After a brief summer break, I started exploring the Spark UI. The first steps I took are exploring changing Spark configuration parameters such as spark.sql.shuffle.partitions.

I can update the default value by adding a config() statement in the Scala code:

val spark = SparkSession
    .builder()
    .appName("optimization")
    .config("spark.sql.shuffle.partitions", "300")  // <-- Here
    .master("local[*]")
    .getOrCreate()

Querying the value of this parameters, shows that it’s indeed 300.