In a world of shrinking budgets, changing priorities, and staff attrition, how can you most effectively build products that are resilient to the shifting world around them? The answer ultimately comes down to making it as easy as possible to spin up new products and to onboard and swap staff between products. One of the biggest challenges in doing this is product-specific knowledge that persists only in documentation — or worse, in a team member's head.
This is an area that has fascinated me for the last decade, starting when I joined Salesforce.org and was tasked with designing the DevOps processes that would allow rapid growth of both our team and product portfolio. Even more inspiration came from the subsequent challenge of lowering the barriers to community contributions and extensions to the open source managed package products that came to be the foundations of Nonprofit Cloud and Education Cloud.
As has often been the case for me, the solutions we found were not the result of divinely inspired up-front design. They came through trial and error grounded in a commitment to continuous improvement based on feedback from team members and the community.
Reflecting back, I can identify three principles that guided the overall success of our ability to scale our team, products, and community:
We baked these principles into CumulusCI, the open source tooling we created to solve our needs, but they also apply more generally. I'll use the example of CumulusCI to illustrate my points, but you could also implement these same principles with well designed shell scripts.
Working on a new product shouldn't require learning a whole new way of working. Creating a common developer experience across all products makes it easier to move people around as priorities shift. Ideally your staff needs to remember a handful of common commands that allow them to accomplish their day-to-day work.
This principle is at the core of CumulusCI. Every project starts from a universal configuration with a common set of tasks, flows, and orgs. This means that every project provides a common set of commands for doing the day to day work.
For example, creating a fully configured dev scratch org in any project using CumulusCI is accomplished with the command:
$ cci flow run dev_org --org dev
Likewise, creating a fully configured scratch org for beta testing is accomplished with the command:
$ cci flow run install_beta
There's no need to remember option flags or entirely different commands based on the product you're working on making it far easier to move between products.
Obviously, a one-size-fits-all approach is far too simplistic for the complexity of software development so the universal configuration is just a starting point to create a common developer experience. It's essential that each product is able to fully customize the configuration of the common developer experience to bake in the unique product specific requirements of building fully usable environments throughout the product lifecycle.
You can think of this as an interface (universal config) and an implementation of the interface (product config). In CumulusCI, individual product repositories contain a cumulusci.yml file that specifies all the product's overrides and extensions of the universal configuration. This allows a product team full flexibility to meet the unique needs of their product while keeping the developer experience familiar to new team members.
For example, the dev_org flow in CumulusCI's "contract" is that it will orchestrate the setup of a fully usable development environment. CumulusCI can't know what each product's needs are to meet that goal so each product's developers define their own custom tasks and add them to the dev_org flow to fulfill the contract.
For example, if I have a permission set named MyPermissionSet that I need to assign to the current user, I could add that into the dev_org flow with the following in my product's cumulusci.yml file:
flows:
# Extend the config_dev flow which is called at the end of the dev_org flow
config_dev:
steps:
# config_dev has two steps by default, add a third
3:
task: assign_permission_sets
options:
api_names:
- MyPermissionSet
With this simple change in the product's cumulusci.yml file, every developer automatically gets the necessary permission set assigned when they create an org. This is a very basic example that could easily be accomplished without CumulusCI too. The real power of CumulusCI is its ability to handle far more complex sequences of automation logic but that's beyond the scope of this post.
The final principle for resiliency is about helping people overcome the fear of breaking things. Fully automating the compliance process in your CI/CD pipeline is a great solution here. Human error happens. I've come to accept that I'm incredibly error prone and even embrace that fact as a force that has helped me learn many things over the course of my career. It also makes me want to design systems and processes that isolate the impact of my errors.
Your CI/CD pipelines should protect your team members from their own mistakes which are inevitable. They should also establish the basic standards for work to be merged or released so the team members have flexibility to meet those requirements.
A key point here is to only enforce what really matters. It's an easy instinct to want to bake the entire "ideal" process into your CI/CD pipeline instead of just what needs to be enforced. The problem with that approach is that it limits the flexibility and creativity of team members.
Many years ago we had a raging debate between my release engineering team and our product teams about our bi-weekly release schedule and at what point we needed a code freeze for final regression testing. I advocated for what seemed like the ideal process that balanced all concerns when a member of my team pulled me aside and asked me what we really needed. It turned out that what we needed was just a beta release tag that was passing build and signed off by QE by 9AM on Tuesday morning. From that perspective, we didn't care about when code freeze was and could leave each team the flexibility to decide on their own. If they didn't meet the requirement, they didn't release that cycle.
Building resilient products in a world of limited resources and shifting priorities requires three guiding principles: a common developer experience, per project customization, and pipeline enforcement. By creating a common set of commands and tasks that apply across all products, teams can easily transition and adapt as needed. Allowing customization at the project level ensures that each product can meet its unique requirements while maintaining a familiar developer experience. Implementing automated compliance processes within CI/CD pipelines protects against human error and establishes essential standards without stifling team creativity. By following these principles, teams can successfully scale their products, teams, and communities while navigating the challenges of a changing landscape.