Choosing the right Databricks cluster: Spot instances vs. on-demand clusters, All-Purpose Compute vs. Jobs Compute
Choosing the right Databricks cluster: Spot instances vs. on-demand clusters, All-Purpose Compute vs. Jobs Compute
According to Wavestone’s 2024 Data and AI Leadership Executive Survey, about 82.2% of data and AI leaders report that their organizations are increasing investments in data and analytics. As companies increasingly rely on big data, the significance of efficient data processing solutions and optimal configuration of clusters become even more crucial. Choosing the correct cluster
Noa Shavit
17 Dec 2024
![]()
Blog
Integrating Gradient with Terraform
Integrating Gradient with Terraform
Infrastructure as Code (IaC) has revolutionized how we manage cloud resources, with Terraform emerging as the leading tool for this approach. The ability to define, version, and automate infrastructure through code has transformed operations from manual, error-prone processes into systematic, repeatable workflows. However, cluster configurations defined in Terraform are typically over-provisioned to guarantee performance and
Kartik Nagappa
09 Dec 2024
![]()
Blog, Releases
AdTech company saves 300 eng hours, meets SLAs, and saves $10K with Gradient
AdTech company saves 300 eng hours, meets SLAs, and saves $10K with Gradient
Today, balancing performance requirements with cost efficiency is a critical challenge for both data engineers and executives. A leading advertising analytics platform faced this significant challenge. With strict customer-facing runtime Service Level Agreements (SLAs) to meet efficiently, the company needed a solution that could optimize their data infrastructure for both performance and compute costs. This
Noa Shavit
30 Oct 2024
Blog, Case Study
A new approach to managing compute resources: Insights from Sync
A new approach to managing compute resources: Insights from Sync
Today, data engineers face considerable challenges optimizing the performance of their data pipelines while managing ever rising cloud infrastructure costs. On a recent episode of the Unapologetically technical podcast, Sync Co-founder and CEO, Jeff Chou, shed light on the innovative solutions that are reshaping how we approach these challenges. Read on for the key insights,
Noa Shavit
10 Sep 2024
![]()
Blog, Videos
Databricks 101: An Introductory Guide on Navigating and Optimizing this Data Powerhouse
Databricks 101: An Introductory Guide on Navigating and Optimizing this Data Powerhouse
In an era where data reigns supreme, the tools and platforms that businesses utilize to harness and analyze their data can make or break their competitive edge. Among these, Databricks stands out as a powerhouse, yet navigating its complexities often feels like deciphering cryptic code. With businesses generating an average of 2.5 quintillion bytes of
Noa Shavit
16 Jul 2024
![]()
Blog
What is Databricks Unity Catalog (and Should I Be Using It)?
What is Databricks Unity Catalog (and Should I Be Using It)?
To help in your understanding, we wrote this blog post to address the question, “What is Databricks Unity Catalog?” and whether users should be using it (the answer is yes). We walk through a precise technical answer, and then dive into the details of the catalog itself, how to enable it and frequently asked questions.
Jeffrey Chou
29 May 2024
![]()
Blog
