News

Gradient New Product Update Q4 2023

  • Jeffrey Chou
  • 11.27.2023

Today we are excited to announce our next major product update for Gradient to help companies optimize their Databricks Jobs clusters.  This update isn’t just a simple UI upgrade…

We upgraded everything from the inside out! 

Without burying the lead, here’s a screenshot of the new project page for Gradient above.

Back in the last week of June of this year (2023), we debuted our first release of Gradient.  In the past few months we gathered all of the user feedback on how we can make the experience even better.

So what were the high level major feature requests that we learned in the past few months?

  • Visualizations – Visual graphs which show the cost and runtime impact of our recommendations to see the impact and ROI of Gradient
  • Easier integration – Easier “one-click” installation and setup experience with Databricks
  • More gains – Larger cost savings gains custom tailored to the unique nature of each job
  • Azure support – A large percentage of Databricks users are on Azure, and obviously they wanted us to support them

Those features requests weren’t small and required pretty substantial changes from the backend to the front, but at the end of the day we couldn’t agree more with the feedback.  While a sane company would prioritize and tackle these one by one, we knew each one of these were actually interrelated behind the scenes, and it wasn’t just a simple matter of checking off a list of features.

Here’s our high level demo video to see the new features in action!

So we took the challenge head on and said “let’s do all of it!”   With all of that in mind, let’s walk through each awesome new features!

Feature #1:  See Gradient’s ROI with cost and runtime optimizations

  • Monitor your jobs total costs across both DBUs and cloud fees in real-time to stay informed
  • Ensure your job runtimes and SLAs are met
  • Learn what anomalies are impacting your jobs’ performances
  • Visualize Gradient’s value in by watching your cost and runtime goals being met

Feature #2:  Cluster integrations with AWS and Azure

  • Granular compute metrics are obtained by retrieving cluster logs beyond what Databricks exposes in their system tables
  • Integrate with Databricks Workflows or Airflow to plug Gradient into how your company runs your infrastructure
  • Easy metrics gathering as Gradient does the heavy lifting for you and automatically compiles and links information across both Databricks and cloud environments

Feature #3:  A new machine learning algorithm that custom learns each job

  • Historical log information is used to train custom models for each of your jobs.  Since no two jobs are alike, custom models are critical to optimizing at scale.
  • Accuracy is ensured by training on real job performance data
  • Stability is obtained with small incremental changes and monitoring to ensure reliable performance

Feature #4: Auto-import and setup all of your jobs with a single click

  • Gradient connects to your Databricks workspace behind the scenes to make importing and setting up job clusters as easy as a single click
  • Non-invasive webhook integration is used to link your environment with Gradient without any modifications to your code or workflows

Feature #5:  View and approve recommendations with a click

  • View recommendations in the Gradient UI for your approval before any changes are actually made
  • Click to approve and apply a single recommendation so you are always in control

Feature #6:  Change your SLA goals at any time

  • Runtime SLA goals ultimately dictate the cost and performance of your jobs.  Longer SLAs can usually lead to lower costs, while shorter SLAs could lead to higher costs.
  • Goals change constantly for your business, Gradient allows your infrastructure to keep up at scale
  • Business lead infrastructure allows you to start with your business goals and work backwards to your infrastructure, not the other way around

Feature #7:  Enable auto-apply for self-improving jobs

  • Focus on business goals by allowing Gradient to constantly improve your job clusters to meet your ever changing business needs
  • Optimize at scale with auto apply, no need to manually analyze individual jobs – just watch Gradient get to work across all of your jobs
  • Free your engineers from manually tweaking cluster configurations and allowing them to focus on more important work

Try it yourself!

We’d love to get your feedback on what we’re building.  We hope these features resonate with you and your use case.  If you have other use cases in mind, please let us know! 

To get started – see our docs for the installation process!

Connect with us now via booking a demo, chatting with us, or emailing us at support@synccomputing.com.

Sync is now SOC2 Type I Compliant!

  • Jeffrey Chou
  • 10.02.2023

Introduction

In the world of Data, iron clad security is table stakes when considering 3rd party vendors to work with.  Here at Sync, we take our customer’s security seriously and want to assure that their sensitive information is handled with utmost care. This is why we are thrilled to announce that Sync has successfully achieved SOC 2 Type I compliance, a significant milestone in our commitment to data security and privacy.

To request our SOC2 Type I report, please see our security portal in the documentation

What is SOC 2 Type I Compliance?

SOC 2 (System and Organization Controls) is a widely recognized auditing standard developed by the American Institute of CPAs (AICPA). It focuses on the security, availability, processing integrity, confidentiality, and privacy of customer data. Achieving SOC 2 compliance demonstrates a company’s dedication to implementing and adhering to strict information security policies and procedures.

A SOC 2 Type I report is the initial step in the compliance process. It attests that the organization’s security controls are in place and have been effectively designed to meet the criteria specified in the Trust Service Criteria. Type I reports evaluate the operational effectiveness of these controls at a point in time, whereas a Type II report evaluates these controls over a period of time, typically six months or more.

Why SOC 2 Type I Compliance Matters

Enhanced Data Security: Achieving SOC 2 Type I compliance signifies a rigorous commitment to safeguarding sensitive information. It ensures that our systems and procedures have been thoroughly scrutinized and meet the highest standards of data security.

Customer Trust and Confidence: In an era where data breaches are commonplace, customers are becoming increasingly vigilant about the companies they choose to do business with. SOC 2 compliance provides assurance that we take data protection seriously and are willing to invest in the necessary safeguards.

Competitive Advantage: SOC 2 compliance is a differentiator in the market. It sets us apart from competitors who may not have undergone such stringent assessments. It becomes a clear signal to potential clients that we prioritize their data security.

Reduced Risk of Security Incidents: The rigorous auditing process required for SOC 2 compliance often reveals areas for improvement in an organization’s security posture. Addressing these issues reduces the risk of potential security incidents, which can have serious repercussions for both the company and its customers.

Streamlined Vendor Relationships: Many organizations now require their vendors to demonstrate SOC 2 compliance as a prerequisite for doing business. By achieving this certification, we eliminate a potential barrier to entry in the marketplace and can establish partnerships with companies that prioritize security.

The Road to Compliance

Achieving SOC 2 Type I compliance was no small feat for Sync. It required meticulous planning, dedication, and collaboration across various departments. Here’s a glimpse into the journey:

Risk Assessment and Gap Analysis: A primary step was to conduct a thorough risk assessment to identify potential vulnerabilities along with a gap analysis to determine where our existing security controls could be improved to reduce risk.

Policy and Procedure Development: We developed and implemented new policies and procedures to address control deficiencies. These covered a wide range of areas, including access controls, encryption, incident response, and more.

Employee Training and Awareness: Our employees are our first line of defense against security threats. Extensive training programs were implemented to ensure that every team member understands their role in maintaining the security of our systems and data.

Continuous Monitoring and Testing: Achieving compliance is not a one-time event. It requires ongoing vigilance and testing to ensure that security controls remain effective over time. Regular audits and assessments are now a permanent part of our security strategy.

Conclusion

Achieving SOC 2 Type I compliance is a significant milestone for Sync, one that underscores our unwavering commitment to data security and privacy. It is a testament to the hard work and dedication of our team members across the organization.

As we move forward, we will continue to invest in and prioritize data security to ensure that our customers can trust us with their most sensitive information. SOC 2 compliance is not the end of our journey; it is a foundation upon which we will build even stronger security practices to meet the evolving challenges of the digital landscape.

We are excited to embark on this new chapter and look forward to providing our customers with the highest level of confidence in the security of their data. Thank you for being a part of this journey with us.

To view our SOC2 Type 1 report, see our security portal in the documentation.

Infrastructure Optimization Innovator Sync Computing Raises $15.5M

  • Jeffrey Chou
  • 08.16.2022

Infrastructure Optimization Innovator Sync Computing Raises $15.5M to Transform How Developers Control Data and Machine Learning Cloud Infrastructure

Already in use in Fortune 500 companies, Sync’s novel Gradient drastically reduces data processing cloud costs while optimizing performance

BOSTON, Aug. 16, 2022 (GLOBE NEWSWIRE) — Sync Computing, a deep tech, cloud infrastructure optimization company, today announced its $15.5M Series A funding, led by Costanoa Ventures, with support from existing investors The Engine, Moore Strategic Ventures, and National Grid Partners. As part of this latest fundraise, Sync has also received the vote of confidence and funding support from angel investors Satyen Sangani, CEO and co-founder of Alation; Ron Avnur, formerly with Google, MongoDB, and Dremio; and Rebecca Li, formerly with Databricks. Harnessing the power of mathematical optimization, Sync automatically orchestrates and provisions cloud infrastructure to achieve cost and runtime goals with one click. This radically transforms the way developers navigate and control their cloud footprint for data and machine learning workloads.

Alongside its Series A, Sync also announced today the public API launch for its Apache Spark Gradient, which enables data engineers to automatically apply optimization solutions across all of their data jobs. Already field-tested with large enterprise customers such as Duolingo, the world’s #1 language learning platform serving more than 40 million monthly active users, the Gradient has gotten traction across engineering teams at global leaders Disney and multiple other Fortune 500 companies. Matt Weingarten, data engineer at Disney Streaming Services, shared in this post: “For data engineers who are tired of trying to figure out their optimal Spark configurations by trial and error, I definitely recommend Sync Computing’s Gradient tool. It’s worked for our team, and I want to make sure this application gets the recognition it deserves.”

Initially launched as a self-serve product for Amazon EMR and Databricks on AWS infrastructure, the Sync Gradient for Apache Spark has been expanded with the release of the public API. It not only enables data engineering teams to programmatically apply Gradient’s intelligence to all of their jobs, but it also allows users to enable continuous monitoring and alerts through custom integration with AWS EMR or Databricks.

Amalgam Insights, in a recent Preferred Vendor profile in the Amalgam Insights SmartList, recommends that organizations spending at least $100,000 a month or a minimum of $1.2 million per year on Amazon Web Services should consider Sync Computing as a cloud cost and optimization vendor candidate. The analyst firm further explains that “Sync’s proprietary and unique approach to cloud optimization and the implementation of these recommendations provides an opportunity to increase savings or performance above and beyond what is possible in the market at large.”

“We are excited to be bringing Sync Computing into our portfolio of companies rethinking infrastructure for data, machine learning and artificial intelligence,” said Tony Liu, partner at Costanoa Ventures. “Sync’s seasoned cloud and data veterans are building something that hasn’t been done before, using a novel method steeped in deep mathematics. Sync Computing is flipping the cloud infrastructure world on its head.”

“Amid fears of an impending economic recession, now more than ever, cloud computing costs are under the microscope. Companies are in urgent need to curb cloud spend and quickly increase their efficiency without hiring new people or driving the burden upon their team,” explains Jeff Chou, co-founder and CEO of Sync Computing. “Sync empowers data-driven companies to do more on the cloud with less. We could not be more thrilled about our latest round of investment and the continued confidence in our team to drive innovation in the cloud ecosystem.”

Spun out of MIT Lincoln Laboratory by Jeff Chou and Suraj Bramhavar, Activate Fellows (Cohort 2020) supported by DARPA’s Microsystems Technology Office, and MIT Startup Exchange Accelerator veterans, Sync’s technology empowers organizations who run thousands of data pipelines daily to automatically optimize low-level compute resources to make running the cloud easier, faster, and cheaper. A Gartner Cool Vendor in the Augmented Data Management category, Sync has been steadily growing its team, counting over a dozen experienced engineers from IBM, Intel, Cloudability/Apptio, Microsoft, and Netflix, and globally respected academic institutions such as MIT, Berkeley, and Harvard.

About Sync Computing
Sync Computing is the first company to harness the power of mathematical optimization to radically enhance the way developers control cloud infrastructure to achieve business goals for data and machine learning workloads. More at: https://www.synccomputing.com/, LinkedIn, Twitter, Medium, YouTube.

Media Contact:
Claire Cashdan
Scratch Marketing + Media for Sync Computing
claire@scratchmm.com

Distributed Cloud Infrastructure Innovator Sync Computing Emerges from Stealth, Brings in $6.1 Million

  • Jeffrey Chou
  • 01.27.2022

First to launch predictive gradient solution for big data jobs in the cloud, wins $1M contract from the Department of Defense

BOSTON, Jan. 25, 2022 (GLOBE NEWSWIRE) — Sync Computing, a deep tech, distributed cloud infrastructure company, came out of stealth mode today, revealing its initial products, customer traction, and $6.1 million funding. Moore Strategic Ventures and National Grid Partners led the round, joining existing investor The Engine. Alongside active pilots with both public and private enterprise customers in SaaS, finance, and data sectors, the company recently was awarded a $1M contract from the Department of Defense for large, distributed workload optimization. Sync Computing is using the new capital to advance its leadership in the modern data infrastructure space and support further development of its accelerated data infrastructure engine and solution lines.

Spun out of MIT Lincoln Laboratory by Jeff Chou and Suraj Bramhavar, Activate Fellows (Cohort 2020) supported by DARPA’s Microsystems Technology Office, and MIT Startup Exchange Accelerator veterans, Sync Computing is harnessing the computational power of physics to find mathematically the best way to provision cloud infrastructure for data, machine learning, and scientific workloads. Sync’s technology empowers organizations who run thousands of data pipelines daily to automatically optimize low level compute resources to make running the cloud easier, faster, and cheaper.

The technology challenges addressed by Sync Computing are a fundamental part of the massively growing data infrastructure market that Gartner values at over $66B. Cloud costs are exploding – with annual enterprise spending on cloud infrastructure services estimated at $130B (Synergy Research), and over 36% of enterprises spend more than $12 million per year on public clouds, 61% of organizations plan to optimize existing use of cloud (cost savings), making it the top initiative for the fifth year in a row (Flexera). Solving for the problem of managing growing big data workloads against their optimal cloud compute is an urgent, formidable, and yet-to-be-solved challenge until now.

“Companies with data-intensive cloud workflows struggle to hire data engineers while their engineers spend unproductive time manually configuring and tuning rather than important work to move the needle on the business top line. We believe Sync automation can add thousands of hours of data engineering productivity every year,” said Reed Sturtevant, general partner at The Engine. “What Sync offers is transformative – an automatic configuration and cost-performance optimization solution for distributed cloud applications. Previously it was thought to be too mathematically complex and dynamic to solve this challenge, but Sync has figured it out.”

“With today’s constantly expanding use of large-scale cloud computing, we’ve seen companies who have only a dozen engineers responsible for managing up to 10,000 data pipelines per day – it’s physically impossible to optimize cloud infrastructure at such large scales – until now,” said Jeff Chou, co-founder and CEO of Sync Computing. “We’ve essentially converted large scale cloud infrastructure into a math problem, and then solve it in seconds. We are also excited to do our part in reducing the wasteful use of cloud resources and its impact on global carbon footprint. We are bullish on what 2022 will bring for us, for our customers, and for the cloud space.”

As part of its public launch, Sync is announcing two solutions – the Sync Gradient for Apache Spark, and the Sync Orchestrator – a major step in large-scale cloud multi-tenant orchestration of complex data pipelines, inspired by Sync cofounders’ PhD research on solving complex combinatorial optimization problems, which was published in Nature. The Sync Gradient for Apache Spark eliminates provisioning friction for EMR and Databricks on AWS infrastructure, while reducing runtimes and job costs dramatically.

The Sync Gradient has already been field-tested and its performance validated by a number of marquee customers, leading to key partnerships. Duolingo, the world’s #1 language learning platform, serving more than 40 million monthly active users, cut daily data job costs on the cloud in half with the Sync Gradient, with only a negligible increase in run time. “We run many big data jobs, so optimizing our cloud processing performance is critical to our success as a business,” said Kevin Wang, analytics engineer at Duolingo. “We were impressed by Sync Computing’s delivery and appreciated their ability to predict optimized results, even before running our jobs. We are confident they can continue to help us forecast performance and reduce costs for our Spark workloads.”

The Spark AWS Gradient is available today and the Sync Orchestrator is available for preview upon request. Interested parties can sign up here.

About Sync Computing

Sync Computing is the first company to harness the computational power of physics to find mathematically the best way to provision cloud infrastructure for data, machine learning, and scientific distributed-based workloads. Sync provides the only solution, which globally optimizes and automates application configurations, cloud infrastructure, and scheduling to achieve business goals of performance or cost. Coming out of MIT and co-founded by PhDs Jeff Chou (CEO) and Suraj Bramhavar (CTO), with a growing team joining from MIT, UC Berkeley, Harvard, Stanford, IBM, Intel, and Cloudability, the company is working with leading enterprises to optimally tune big data processing across its optimal cloud compute. More at: https://www.synccomputing.com/ (LinkedIn, Twitter).

About The Engine

The Engine is a Cambridge, MA-based venture capital firm. It was conceived and created by MIT in 2016 to address the unmet need for sustained support for startups with the potential to solve complex, existential challenges and make a material, positive impact on society. The Engine provides access to long-term capital, knowledge, connections, as well as the infrastructure these Tough Tech companies need to thrive. For more information, visit www.engine.xyz.

About Moore Strategic Ventures

Moore Strategic Ventures, LLC is the privately held investment company for Louis M. Bacon, founder and CEO of Moore Capital Management, LP.

About National Grid Partners

National Grid Partners (NGP) is the venture investment and innovation arm of National Grid plc., one of the largest investor-owned energy companies in the world. NGP invests for strategic and financial impact and leads company-wide culture transformation efforts. The organization provides a multi-functional approach to building startups, including innovation (new business creation), incubation, corporate venture capital, business development, and culture acceleration. NGP is headquartered in Silicon Valley and has offices in Boston, London, and New York. Visit ngpartners.com or follow us on Twitter @ngpartners_ and LinkedIn.