How to evaluate the performance of your Databricks Jobs clusters with Gradient
How to evaluate the performance of your Databricks Jobs clusters with Gradient
Many data engineers and platform managers at companies want to understand “How are my pipelines doing from a cost and performance perspective?” It turns out this very benign and simple question is riddled in complexity. Fundamentally there are so many things that can change from run to run, that even determining if it’s the same
Jeffrey Chou
03 Sep 2024
Blog, Case Study
How Abnormal Reduced Databricks Costs by 38% with Gradient
How Abnormal Reduced Databricks Costs by 38% with Gradient
Who is Abnormal? Abnormal is a hypergrowth company in the email security space that helps companies worldwide prevent email attacks while automating security operations. They rely on Databricks extensively to help process terabytes of data across thousands of jobs daily, translating to an enormous amount of daily Databricks usage. From ETL jobs, streaming, SQL, to
Jeffrey Chou
18 May 2024
Case Study
How Forma.ai improved their Databricks costs quickly and easily with Gradient
How Forma.ai improved their Databricks costs quickly and easily with Gradient
Forma.ai is a B2B SaaS startup based in Toronto, Canada building an AI powered sales compensation system for enterprise. Specifically, they seamlessly unify the design, execution, and orchestration of sales compensation to better mobilize sales teams and optimize go-to-market performance. Behind the scenes, Forma.ai deploys their pipelines on Databricks to process sales compensation pipelines for their
Jeffrey Chou
12 Feb 2024
Case Study
How a Disney Senior Data Engineer Obtained 80% Cost Savings using Gradient
How a Disney Senior Data Engineer Obtained 80% Cost Savings using Gradient
Sr. Data Engineer at Disney Streaming In the self-written blog post below, a Sr. Data Engineer chronicles his experience with the Spark Gradient for EMR. In the blog post we helped accelerate a job from 90 to 24 minutes, which was amazing to see! The first job I put into the gradient went from processing
Jeffrey Chou
15 Jan 2024
Case Study
5 Lessons learned from testing Databricks SQL Serverless + DBT
5 Lessons learned from testing Databricks SQL Serverless + DBT
Databricks’ SQL warehouse products are a compelling offering for companies looking to streamline their production SQL queries. However, as usage scales up, the cost and performance of these systems become crucial to analyze. In this blog we take a technical deep dive into the cost and performance of their serverless SQL warehouse product by utilizing
Jeffrey Chou
17 Oct 2023
Blog, Case Study
How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
How to Use the Gradient CLI Tool to Optimize Databricks / EMR Programmatically
Introduction: The Gradient Command Line Interface (CLI) is a powerful yet easy utility to automate the optimization of your Spark jobs from your terminal, command prompt, or automation scripts. Whether you are a Data Engineer, SysDevOps administrator, or just an Apache Spark enthusiast, knowing how to use the Gradient CLI can be incredibly beneficial as
Pete Tamisin
11 Jul 2023
Blog, Case Study
