Today we are excited to announce our next major product update for Gradient to help companies optimize their Databricks Jobs clusters. This update isn’t just a simple UI upgrade…
We upgraded everything from the inside out!
Without burying the lead, here’s a screenshot of the new project page for Gradient above.
Back in the last week of June of this year (2023), we debuted our first release of Gradient. In the past few months we gathered all of the user feedback on how we can make the experience even better.
So what were the high level major feature requests that we learned in the past few months?
- Visualizations – Visual graphs which show the cost and runtime impact of our recommendations to see the impact and ROI of Gradient
- Easier integration – Easier “one-click” installation and setup experience with Databricks
- More gains – Larger cost savings gains custom tailored to the unique nature of each job
- Azure support – A large percentage of Databricks users are on Azure, and obviously they wanted us to support them
Those features requests weren’t small and required pretty substantial changes from the backend to the front, but at the end of the day we couldn’t agree more with the feedback. While a sane company would prioritize and tackle these one by one, we knew each one of these were actually interrelated behind the scenes, and it wasn’t just a simple matter of checking off a list of features.
Here’s our high level demo video to see the new features in action!
So we took the challenge head on and said “let’s do all of it!” With all of that in mind, let’s walk through each awesome new features!
Feature #1: See Gradient’s ROI with cost and runtime optimizations
- Monitor your jobs total costs across both DBUs and cloud fees in real-time to stay informed
- Ensure your job runtimes and SLAs are met
- Learn what anomalies are impacting your jobs’ performances
- Visualize Gradient’s value in by watching your cost and runtime goals being met
Feature #2: Cluster integrations with AWS and Azure
- Granular compute metrics are obtained by retrieving cluster logs beyond what Databricks exposes in their system tables
- Integrate with Databricks Workflows or Airflow to plug Gradient into how your company runs your infrastructure
- Easy metrics gathering as Gradient does the heavy lifting for you and automatically compiles and links information across both Databricks and cloud environments
Feature #3: A new machine learning algorithm that custom learns each job
- Historical log information is used to train custom models for each of your jobs. Since no two jobs are alike, custom models are critical to optimizing at scale.
- Accuracy is ensured by training on real job performance data
- Stability is obtained with small incremental changes and monitoring to ensure reliable performance
Feature #4: Auto-import and setup all of your jobs with a single click
- Gradient connects to your Databricks workspace behind the scenes to make importing and setting up job clusters as easy as a single click
- Non-invasive webhook integration is used to link your environment with Gradient without any modifications to your code or workflows
Feature #5: View and approve recommendations with a click
- View recommendations in the Gradient UI for your approval before any changes are actually made
- Click to approve and apply a single recommendation so you are always in control
Feature #6: Change your SLA goals at any time
- Runtime SLA goals ultimately dictate the cost and performance of your jobs. Longer SLAs can usually lead to lower costs, while shorter SLAs could lead to higher costs.
- Goals change constantly for your business, Gradient allows your infrastructure to keep up at scale
- Business lead infrastructure allows you to start with your business goals and work backwards to your infrastructure, not the other way around
Feature #7: Enable auto-apply for self-improving jobs
- Focus on business goals by allowing Gradient to constantly improve your job clusters to meet your ever changing business needs
- Optimize at scale with auto apply, no need to manually analyze individual jobs – just watch Gradient get to work across all of your jobs
- Free your engineers from manually tweaking cluster configurations and allowing them to focus on more important work
Try it yourself!
We’d love to get your feedback on what we’re building. We hope these features resonate with you and your use case. If you have other use cases in mind, please let us know!
To get started – see our docs for the installation process!
Connect with us now via booking a demo, chatting with us, or emailing us at email@example.com.