I was experimenting with the new Scala 3 macro syntax and getting errors even though my code was clearly correct:

val msg = Expr("Hello")
val printHello = '{ print($msg) }
println(printHello.show) // print("Hello")

I soon figured out that the Scala version in my Spark sbt project was 2.12.17 - so version 2. Looking at the Spark compatibility matrix I saw that Spark doesn’t support Scala 3 yet!

This is a known issue, but someone already found a workaround using .map(_.cross(CrossVersion.for3Use2_13)) in my build.sbt file:

Datasets

Upgrading to Scala 3 lead to the following error:

This could be solved by adding the following dependency in my build.sbt file:

I updated my datasets.scala code as follows:

case class Usage(uid: Int, user: String, usage: Int)
case class UsageCost(uid: Int, user: String, usage: Int, cost: Double)

// Correct way to create TypeTags in Scala 3
import scala.reflect.runtime.universe._

implicit val usageTypeTag: TypeTag[Usage] = typeTag[Usage]
implicit val usageCostTypeTag: TypeTag[UsageCost] = typeTag[UsageCost]

// Create explicit encoders (these should work now with TypeTags)
implicit val usageEncoder: Encoder[Usage] = Encoders.product[Usage]
implicit val usageCostEncoder: Encoder[UsageCost] = Encoders.product[UsageCost]

The code now compiles but I’m encountering an error:

Datasets¶

Datasets