Today at work I had a discussion about using serverless compute for Databricks jobs. The question was, whether this was cheaper in the end or not. Working with serverless compute in DABs is certainly easy: you don’t have to specify any value for your clusters (serverless will be used by default).
Is it actually cheaper, though?
Researching several forum posts leads to conflicting information. Some users report a significant cost increase, while others see clear benefits of using serverless.
A lot will likely depend on the type of workloads. If your jobs run for hours, t he benefits of serverless seem marginal. Serverless seems to shine where you run many short-lived jobs. This is because idle time is reduced: clusters are shut down more readily after a jobs finished running.
In the end, the actual benefit can probably only be determined by calculating and comparing the cost of serverless versus classic compute. In my opinion, the lack of worry around infrastructure allows me to focus exclusively on the code and data, while leaving the infrastructure optimization to Databricks. That in itself is a lofty improvement.