I’ve seen many job postings over the last year by Salesforce ISV Partners looking to hire a DevOps Engineer to solve the challenges they face throughout their product lifecycle. I’m here to implore those ISVs to consider a better alternative: building a culture of DevOps across your entire product organization.
In this post I’m focusing on the specific DevOps challenges and opportunities for Salesforce ISV Partners. There are many great articles about this subject in the more general software world, but I believe the argument is much stronger in the Salesforce space where developers are inherently focused on deploying their code already.
As I’ve talked with Salesforce AppExchange ISVs over the years, I’ve found that the vast majority of DevOps Engineers at ISVs spend far more of their times on operations than engineering. Why is this? It’s because without a culture of DevOps across the entire product team, DevOps Engineers often become the dumping ground for anything that’s not “feature work” or QA. Need a new QA org spun up? Ask DevOps! Need a demo org of the upcoming release? Ask DevOps! Need to figure out why deployment to the packaging org is failing? Ask DevOps! Need a branch created in the repo? Ask DevOps!
What you find when you look at the types of work typically thrown to DevOps Engineers is a lot of manual operations work, none of which requires the skills in your DevOps Engineer job description. DevOps becomes this magical resource and dumping ground for patching over the tech debt inherent in your processes. That work is exhausting and demoralizing for someone hired with “engineer” in their title, creating additional risk of burnout and ultimately attrition of the one person in your organization who knows how to do the things no one else wanted to learn.
Your product dev team prides itself on being Agile, but then throws an unknown and not estimated amount of operations work over the wall to DevOps with a fixed amount of time to implement it. While product dev works Agile, DevOps works in the worst kind of waterfall every release cycle.
The alternative is to build a culture of DevOps throughout your entire product organization by enabling and empowering everyone on the team, including everyone beyond the developers themselves such as QA, Doc, UX, PM, etc. Creating a culture is a team-wide initiative that can’t be solved by adding a single new person. It needs buy-in and nurturing from leadership to be viewed as a part of the product organization’s special sauce. It needs investment of resources to train the skills across the team.
In a healthy culture of DevOps, the automation necessary to deliver usable environments of a product is viewed as part of the product itself. That means developers take ownership over the deployment and configuration of new features as they build them. QA can both self-provision orgs and own the unique configuration requirements for specific test scenarios. It means that automation is part of the estimation process for new features, not an afterthought tossed into the magical DevOps resource pool.
A healthy culture of DevOps is inherently resilient to the challenges of attrition, both through balancing of work across the team to avoid burnout and spreading knowledge throughout the team to avoid loss of knowledge if someone does leave for another job or to their new private island after winning the lottery.
The real engineering involved in DevOps is tooling engineering. Well designed and engineered tooling and processes enable product teams to own their configuration and implementation of the tooling for their products. After all, they are the experts in how their product should be configured and delivered to environments.
Fortunately, there’s already great tooling available for the specific needs of Salesforce ISVs saving them from the need to invest much in tooling engineering themselves. I clearly have my biases, but I know firsthand that Salesforce.org’s open source CumulusCI and the tools built on it that make up Cumulus Suite provides a comprehensive and mature set of tooling to implement a culture of DevOps for AppExchange product teams at massive scale.
The tooling already exists to build a fully automated pipeline from development to release to delivery. You don’t need in-house resources to engineer those tools. With a fully automated pipeline, the need shifts to having someone wear the “release manager hat” to ensure all the necessary approvals are in place and click the button to start the automated release process.
I come at this topic from the perspective of both having been an individual contributor with the title Release Engineer and having run theRelease Engineering team at Salesforce.org for Nonprofit Cloud and Education Cloud for 8 years. My first approach in building my team was to focus on embedding members of my team into the product teams so they could discover the automation needs of the product as features were being planned and built, and then help implement the automation for the team. We generally expected each product scrum team would need 1/2 of an embedded release engineer.
That embedded model is resource inefficient. Either the 1/2 release engineer for the team has too little work leading to underutilization or they have too much work leading to bottlenecks for the team and burnout for the engineer. We were able to recapture the underutilization by allocating it to tooling engineering, but the bottlenecks were a constant source of friction with product teams.
After a few budget cycles of more products and scrum teams without additional funding for embedded release engineers, our desired 2:1 ratio of embedded release engineers to scrum teams became more like 5:1. Faced with these resource constraints, I created a new charter to fundamentally change the way my team worked with product teams.
We shifted our team to a tooling engineering team model focused on building the tools and automated process to allow our product teams to own the delivery of their products. We provided enablement services and escalation support around our tools but shifted the actual implementation of product specific automation to the product scrum teams.
I fully expected I’d receive blowback from what could easily be perceived as me dumping my team’s work onto the product scrum teams. 6 months into our new charter, I hadn’t received any complaints from product teams about the change. I was honestly shocked, but in retrospect it makes perfect sense. Salesforce developers are already intimately familiar with metadata and deploying it and are the experts in their product already. When a developer learned how to implement a new CumulusCI task in their repository’s cumulusci.yml file, they could implement the automation themselves much faster than they could explain to their team’s embedded release engineer what was needed and then wait until they had time to implement it.
The result was that DevOps was being handled in a truly Agile way because it was part of the estimation process and anyone on the team could do it.
When I started MuseLab in 2022, my vision was to help Salesforce ISVs implement the processes and tooling we created, refined, and enhanced in my time at Salesforce so they too could experience the benefits of a whole-team DevOps culture throughout the Development to Delivery Experience (D2X). Like the digital transformation journey often discussed in the Salesforce world, building a culture of DevOps is its own transformation.
MuseLab’s 6-month D2X Success Service is designed to guide Salesforce AppExchange ISVs through that transformational journey by a process of Discovery, Design, Planning, Enablement, and Support. The goal is to align your whole team around the transformation and guide them through the implementation instead of just doing it for them. The overused teaching someone to fish vs giving them fish analogy is quite apt here.
If you’re interested in learning more, book a 1-hour free consultation with me at https://calendly.com/muselab/free.