Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses
Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses
Databricks is a quickly evolving platform with several compute options available for users, leaving many with a difficult choice. In this blog post, we look at three popular options for scheduled jobs using Databricks own’ TPC-DI benchmark suite. By the way, kudos to the Databricks team for creating such a fantastic test package. We highly
Jeffrey Chou
10 Dec 2024
![]()
Blog
Best Practices to Manage Databricks Clusters at Scale to Lower Costs
Best Practices to Manage Databricks Clusters at Scale to Lower Costs
Managing Databricks clusters efficiently is a critical challenge for many organizations, especially as they scale on the platform. As the amount of data organizations process grows rapidly, so do compute costs. Most organizations tackle this by having their data engineers manually optimize clusters, which quickly becomes a bottleneck as the organization scales from dozens to
Noa Shavit
08 Aug 2024
![]()
Blog, Videos
Top 9 Lessons Learned about Databricks Jobs Serverless
Top 9 Lessons Learned about Databricks Jobs Serverless
To much fanfare, Databricks announced their wide release of serverless compute across all of their platforms at the Data + AI 2024 conference. It’s quite clear that Databricks’ vision is to own the compute layer to make life easier for end users, so they don’t have to worry about annoying cluster or versions details. We
Jeffrey Chou
02 Jul 2024
![]()
Blog
Rethinking Serverless: The Price of Convenience
Rethinking Serverless: The Price of Convenience
As is the case with many concepts in technology, the term Serverless is abusively vague. As such, discussing the idea of “serverless” usually invokes one of two feelings in developers. Either, it’s thought of as the catalyst for this potential incredible future, finally freeing developers from having to worry about resources or scaling concerns, or
Vinoo Ganesh
09 Feb 2024
![]()
Blog
