System Initiative: A DevOps Makeover by Ex-Chef Adam Jacob
It’s time to rethink DevOps, according to Adam Jacob, co-founder and former CTO of Chef. He and co-founder Mahir Lupinacci have been working on a new vision, which he concedes is a hard problem, since 2019. The result, System Initiative, is launching in private beta.
“Our aspirations in the DevOps movement were right, but too many of us are stuck, not achieving those goals,” he said. “And now we’re kind of at risk of having those aspirations fade, and [this] becoming a thing that sounded good, but was just hype. … And I think that would be a tragedy. That would be a real loss.”
Adam Jacob
Mahir Lupinacci
He recounted a meeting with a room full of tech execs at a Fortune 50 company toward the end of his tenure at Chef when he was asked to solve a certain problem. He went to the whiteboard and did, and the company’s CTO gave him a high five.
“I knew I was right, that if he did everything that I said, it would work, and that it would change their environment for the better,” he said. “And also knew that he wasn’t going to succeed, right? The number of things that had to go right and the amount of tiny details that needed to work, there was no way that they were going to get all that right,” he said.
He maintains that it’s not a problem of tools, which he describes as “pretty amazing,” that have grown up in this space. It was more of a systemic problem and the number of “tiny paper cuts” that break developers out of their flow.
“The actual problem was how we were stitching all these tools together and the way we were asking people to work just didn’t deliver,” he said.
Multiplayer Games and Digital Twins
“People got stuck in the middle of that transformation and couldn’t really make it to the end,” he said. “Ultimately, it’s because we took all the best practices and tools from this handful of innovative startups, and eventually giant tech companies, and we put them in a blender with each organization’s problem,” he said.
“And while each individual thing we’ve really optimized well on its own… if you look at any individual thing we use in the DevOps toolchain, they’re good, and in fact, in many cases have improved dramatically since we started, but the outcomes still don’t sort of line up.” He cited four key reasons:
- Long feedback loops — It can take hours to find out whether the change you make will work.
- Low intelligence in the system — It can’t use data from one part to inform another part.
- Really cumbersome collaboration — It’s mainly done by pull request, where one person does something, then another person does something in response.
- Context switching is brutal — There are just so many tools to juggle between.
So they sought extensive input and set out to reimagine DevOps from the ground up. They took inspiration from multiplayer games and the concept of digital twins to enable DevOps teams to build detailed interactive simulations of their infrastructure and use them to rapidly update their production environments.
They started with the UI and the realization they were building a simulation.
“So [there was the] insight that said, ‘Hey, we just need to model everything, like in full fidelity with no abstractions.’ And then the other was the realization that what it was fundamentally, was basically like a big reactive hypergraph of functions. So you would just take all this code, and you would stick it in this big graph, and every value for every object or every action you want to take, needed to be reactive to its inputs. And when you made it happen that way, the result is that you can build these highly customizable simulations that allow you to really easily and quickly customize it all from sort of within the application. So that’s how we did it. And it was hard. It’s still not done, [there’s still a lot of] work to do.”
Users describe application components like services, databases and load balancers and specify the computing requirements (compute, network, storage, etc.) Then the system deploys and manages those resources and environments.
Multiple people can work on it simultaneously. When one person makes a change, it also changes on everyone else’s screen. Because it’s a digital twin, users can see immediately whether a change will work. Every configuration parameter, qualification and action you can take is a TypeScript function sitting on a hypergraph. Each function takes inputs from elsewhere on the graph and if anything on the graph changes, it automatically reruns the function for you and stores the results.
Jacob describes the multiplayer visual interface as “like Figma for infrastructure.”
It also offers:
- Intelligent automation that uses the relationships between high-fidelity models to infer configuration dynamically and then automatically writes the necessary code.
- Fast feedback loops through which each model is qualified for use in real time, providing immediate insight into the viability of a configuration. There are no state files to track and no
planvs.applystages, only real-time design and action. - Full customization through writing TypeScript functions.
“System Initiative seeks to replace today’s complex DevOps pipelines and toolchains with simple real-time collaboration workspaces where DevOps teams can create new app stacks without having to worry about the numerous integration points with the rest of the enterprise infrastructure,” said Torsten Volk, managing research director at Enterprise Management Associates (EMA).
“Creating digital twins to simulate this infrastructure could make it much easier and faster to implement new technologies based on existing compliance guardrails and automatically integrated within the latest enterprise context. This enables platform engineers, DevOps engineers, developers, security engineers and all other relevant personas to take responsibility for their part of the stack without forcing developers to wait. This is a true paradigm shift toward a collaborative and real-time version of DevOps that will require DevOps teams to let go of their old, sequential, habits and processes and replace them with this new simulation-based approach.
“The fact that some DevOps practitioners are sufficiently frustrated with the current state of DevOps to talk about the return of the ‘good bits’ of the traditional waterfall model shows that we are now at a point where we do need disruption.”
Going up against Infrastructure as Code
With the focus so far on managing infrastructure, it’s targeting DevOps folks currently doing Infrastructure as Code, and its closest competitors would be IaC companies like Terraform and Pulumi.
However, “Obviously, we have big ambitions to sort of rethink the way that the whole toolchain works,” Jacob said.” And over time, it’ll expand and grow into more at the application-level abstractions, more of the platform-level abstractions.”
He said he and his co-founders did extensive usability research and went through some brutal sessions where users didn’t understand what they had built, but seemed impressed once they did. Users’ most pressing questions have been about how it will scale and how the project will grow over time.
He also shared a story about trying to find the language to describe working with Infrastructure as Code, so he posed that as a question on Twitter.
“Look, it was Twitter, so you’re always gonna get snarky responses. But all I got was snarky responses. And they were like, sadness, pain, tears, you know, heartbreak. When you think about [doing] the things you love, those aren’t usually the responses you come up with,” he said, adding immediately that it’s not about the tools, “in the end, not because they’re bad, but because the system is wrong.”
Is DevOps Merely Evolving?
ESG senior analyst Paul Nashawaty said he does not agree that DevOps has failed to live up to its promise, but that it is constantly evolving.
“The DevOps world is changing, and organizations are looking for ways to adopt new approaches to meet the ever-changing business needs. In order to meet these new business KPIs [key performance indicators], adjustments in the DevOps processes need to be considered. System Initiative provides an approach that seems to align to the new business goals while optimizing how organizations operate and collaborate,” he said.
System Initiative just landed $15 million in Series A funding led by Scale Venture Partners, adding to the $3 million in seed funding led by Amplify Partners.
“The possibilities for System Initiative are vast — unlike anything we have ever seen in the DevOps space,” said Ariel Tseitlin, partner at Scale Venture Partners. “System Initiative’s focus on keeping engineers in that flow state through visual modeling and faster feedback loops is a better way of interacting with infrastructure. Mix in the ability to customize and extend the system fully, and you have the beginnings of a transformation in how folks package and share best practices or understand their compliance posture.”
You can join the private beta on the website. Jacob said the beta might be short-lived; once a couple of features around customization and sharing are in place, the whole thing will be open sourced.
