Choosing the right Databricks cluster: Spot instances vs. on-demand clusters, All-Purpose Compute vs. Jobs Compute
Choosing the right Databricks cluster: Spot instances vs. on-demand clusters, All-Purpose Compute vs. Jobs Compute
According to Wavestone’s 2024 Data and AI Leadership Executive Survey, about 82.2% of data and AI leaders report that their organizations are increasing investments in data and analytics. As companies increasingly rely on big data, the significance of efficient data processing solutions and optimal configuration of clusters become even more crucial. Choosing the correct cluster
Noa Shavit
17 Dec 2024
![]()
Blog
Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses
Databricks Compute Comparison: Classic Jobs vs Serverless Jobs vs SQL Warehouses
Databricks is a quickly evolving platform with several compute options available for users, leaving many with a difficult choice. In this blog post, we look at three popular options for scheduled jobs using Databricks own’ TPC-DI benchmark suite. By the way, kudos to the Databricks team for creating such a fantastic test package. We highly
Jeffrey Chou
10 Dec 2024
![]()
Blog
Integrating Gradient with Terraform
Integrating Gradient with Terraform
Infrastructure as Code (IaC) has revolutionized how we manage cloud resources, with Terraform emerging as the leading tool for this approach. The ability to define, version, and automate infrastructure through code has transformed operations from manual, error-prone processes into systematic, repeatable workflows. However, cluster configurations defined in Terraform are typically over-provisioned to guarantee performance and
Kartik Nagappa
09 Dec 2024
![]()
Blog, Releases
Unlock Databricks cost transparency
Unlock Databricks cost transparency
In the world of big data and cloud computing, managing costs effectively is a significant challenge. While Databricks provides powerful tools for data engineers and analysts, understanding the complete cost picture can be complex. Databricks customers receive two separate bills – one for their Databricks usage and another from their cloud provider where clusters were
Kartik Nagappa
18 Nov 2024
![]()
Blog
AdTech company saves 300 eng hours, meets SLAs, and saves $10K with Gradient
AdTech company saves 300 eng hours, meets SLAs, and saves $10K with Gradient
Today, balancing performance requirements with cost efficiency is a critical challenge for both data engineers and executives. A leading advertising analytics platform faced this significant challenge. With strict customer-facing runtime Service Level Agreements (SLAs) to meet efficiently, the company needed a solution that could optimize their data infrastructure for both performance and compute costs. This
Noa Shavit
30 Oct 2024
Blog, Case Study
A new approach to managing compute resources: Insights from Sync
A new approach to managing compute resources: Insights from Sync
Today, data engineers face considerable challenges optimizing the performance of their data pipelines while managing ever rising cloud infrastructure costs. On a recent episode of the Unapologetically technical podcast, Sync Co-founder and CEO, Jeff Chou, shed light on the innovative solutions that are reshaping how we approach these challenges. Read on for the key insights,
Noa Shavit
10 Sep 2024
![]()
Blog, Videos
