I often hear developers who use sfdx or sf express confusion about how CumulusCI would fit into their workflow. Their confusion is often based on the false impression that CumulusCI is somehow an alternative to the Salesforce CLI. Instead, CumulusCI is a complimentary layer on top of the Salesforce CLI that provides a comprehensive framework for portable automation recipes to deliver complete products, solutions, or features to Salesforce orgs. I like to think of CumulusCI as an alternative to Bash for developers using Salesforce CLI.
The CumulusCI documentation does a good job of explaining CumulusCI's place in the toolchain:
So what does using CumulusCI look like in the day-to-day world of a developer currently using Salesforce CLI? In this post, I'll cover the high level tasks involved in implementing CumulusCI for an existing Salesforce CLI-based project then dig deep into the interaction between CumulusCI and Salesforce CLI, show what's novel in CumulusCI.
CumulusCI and the Product Delivery Model
Before we dive in, let's briefly discuss the Product Delivery Model, a middle ground between the Org Development Model and the Package Development Model. In the Product Delivery Model, your job as a developer is to build automation recipes in version control that can repeatably deliver complete features to orgs. CumulusCI is built to support the Product Delivery Model, but that doesn't mean you have to adopt the Product Delivery Model fully to use CumulusCI. You can integrate its concepts into your work iteratively.
The fundamental challenge every Salesforce developer faces is that their work can't be tested during the development process. It has to be promoted to high level shared environments. Why is that? Because, as the Salesforce Well-Architected Framework refers to it: "Environments are only as useful as the kind of production fidelity they do (and do not) provide." Development environments are rarely faithful to production.
The Product Delivery Model and CumulusCI provide developers with the tools to automate the configuration of all environments for their work with the goal of being able to fully test work in scratch orgs before it's merged or promoted. Even if you don't get all the way there, testing 80% of your work before merge is still a huge improvement!
CumulusCI + Salesforce CLI: Interaction Points
Once you've implemented CumulusCI for a project, your day-to-day experience as a developer doesn't change significantly though there are many improvements and new possibilities. Are you building a complex 1GP extension package architecture and want to build in parallel across dependency layers? CumulusCI makes that possible and easy. Do you have an innovative solution you want to share with the community as open source? CumulusCI makes that easy and efficient.
Consider the lifecycle of a developer working on a feature. They need a development environment (an org), tooling to do development (an IDE), tooling to extract changes from an org (IDE, cli), and a source-control based process to commit their work for review and merge.
CumulusCI's role is mainly focused on the beginning and end of that lifecycle. CumulusCI automates the process of creating a fully-configured scratch org as a development environment. If the project has dependencies, CumulusCI handles preinstalling them for you. Think of the checklist (written down or in your head) of all the things you do when you spin up a new scratch org for development. Then, imagine running a single command and having all that done for you. Better yet, rather than a document that goes out of sync, all that automation is in version control!
Once CumulusCI creates and configures the scratch org, you pick up and use whatever tooling you want to develop and capture your changes. CumulusCI has some options for retrieving metadata changes and for developing smart, safe metadata transformations, but those are purely optional.
When your development work is captured and ready to commit, CumulusCI provides reusable and customizable workflows for the entire CI/CD process and everyone involved in the development lifecycle.
If that sounds difficult, consider that the entire lifecycle for a developer can be summed up with the following commands:
# Create a new feature branch to isolate work
$ git checkout -b feature/trigger-handlers main
# Create a fully configured scratch org for development
$ cci flow run dev_org --org dev
# Open the org (cci or sf)
$ cci org browser dev
$ sf org open -o MyProject__dev
# Retrieve source tracked changes (cci or sf)
$ cci task run retrieve_changes --org dev
OR
$ sf force source pull -o MyProject__dev
$ git add --all
$ git commit -m "Initial prototype"
$ git push
# Build process creates a 2GP skip validation package ready for QA
$ cci flow run qa_org_2gp --org qa
Not so scary huh?
Implementing CumulusCI for a Salesforce CLI Project
CumulusCI is built to nicely wrap around existing Salesforce CLI projects. To get started, all you need to do is install CumulusCI then run one simple command from your repo root. I usually recommend doing this in a new branch called feature/cumulusci
$ git checkout -b feature/cumulusci
$ cci project init
You'll be walked through a series of options and then your repository will be configured for CumulusCI. You'll find some new files and directories were created. The most important are the cumulusci.yml file and the orgs/ directory which contains the scratchdef files for the different scratch org profiles.
The first step you'll likely want to do is to make sure the scratchdefs under orgs contain the necessary features and settings for your project.
Then, continue to the next sections to attempt to create your first dev scratch org using CumulusCI.
Org Management
Like Salesforce CLI, CumulusCI maintains a keychain for storing org credentials for orgs connected or scratch orgs created. CumulusCI's CLI is tightly integrated with Salesforce CLI's keychain:
- CumulusCI uses Salesforce CLI for all scratch org creation, info, and deletion
- CumulusCI can import an org from the Salesforce CLI keychain (cci org import)
- Setting a default org in CumulusCI also sets the org as default for the project in Salesforce CLI
- CumulusCI creates a predictable Salesforce CLI org alias name using the pattern ProjectName__orgname such as MyProject__dev
There are some key differences between CumulusCI's keychain and the Salesforce CLI keychain:
- All CumulusCI projects come preconfigured with five scratch org profiles (dev, feature, qa, beta, and release) which can be customized by each project.
- Scratch orgs in CumulusCI are stored as a profile to create an org that includes the config file, expiration days, and whether the org is namespaced.
- CumulusCI scratch org profiles are lazy. When you run something against them, they wake up and create a scratch org if one is not already created and connected to the profile. If the org expires, cci will offer to recreate it for you.
- CumulusCI scratch org profiles can be cloned to create additional profiles in the keychain with different settings but the same config file.
Org Configuration
With Salesforce CLI, post-deploy org configuration beyond loading a simple dataset is often done either manually or with a set of Bash scripts. If you work across many projects in many repositories, perhaps you might implement a common naming convention for these Bash scripts in an attempt to maintain consistency across projects. CumulusCI provides a mature, robust, and flexible framework for defining the configuration automation needed to build fully usable and testable scratch orgs for a feature.
CumulusCI's Documentation provides extensive details on how to customize the automation recipes to meet the needs of the project. I'll provide a brief primer here.
CumulusCI's project configuration is defined in a cumulusci.yml file in the root of the project's GitHub rpeository. But it's by no means a blank slate. The project configuration is built from CumulusCI's universal configuration yaml file that defines the default configuration for all projects. Your project's cumulusci.yml can then override and extend the universal configuration how it needs.
The universal configuration provides many tasks and flows (named automation orchestrations) to meet common needs throughout the project lifecycle. For example, the dev_org flow creates and configures a development scratch org. Projects can customize the dev_org flow through their cumulusci.yml to add all project specific needs to create fully usable scratch orgs.
By defining the configuration needs of a project in a declarative yaml framework like CumulusCI, your automation becomes portable and reusable by different applications throughout the development lifecycle, well beyond your CI server.
Source Tracking
At the end of all CumulusCI flows for configuring an org, CumulusCI resets the source tracking state for both CumulusCI and Salesforce CLI. This means you automatically ignore all the changes made through automation in the repository so source tracking only picks up the work you build on top of it.
Source Format
CumulusCI can natively work seamlessly with projects in either sfdx (force-app) or mdapi source format (src/) and automatically handles any necessary conversion between formats during its deployment flows. CumulusCI uses Salesforce CLI to do conversion to and from sfdx source format. The default source format for CumulusCI projects is sfdx. For example, if building a 1GP package set to sfdx source format, the flows to deploy to the packaging org automatically handle conversion from sfdx to mdapi format.
CumulusCI's dx Task
CumulusCI contains a built-in task that can wrap any sfdx command. At the time of this writing, the CumulusCI development is actively working on a migration to use the sf command instead, but for now you can do anything sfdx can do from your CumulusCI flows.
Here's a basic example of adding a task to the end of the config_dev flow which is run at the end of the dev_org flow to print out the org's API limits:
flows:
config_dev:
steps:
3: # config_dev has two steps by default
task: dx
options:
command: org list limits
With CumulusCI, you can continue to use all the awesome commands and plugins for Salesforce CLI to stitch together the automation recipes for your project. Better yet, you gain access to all the prebuilt process automation and novel features of CumulusCI!
Novel Features in CumulusCI
So what are those novel features of CumulusCI? Beyond a different but complimentary approach to managing and configuring orgs, CumulusCI includes a rich set of additional functionality to use in your projects. Remember these can all be iteratively integrated into your product recipes — you don't have to do everything all at once!
Dependency Management
CumulusCI was built from the beginning to support the complexities of extension package development. Even if you're not building extension packages, you still likely find yourself developing things with dependencies on AppExchange, open source, or internal packages. CumulusCI's dependency management allows you automate installation of packages by namespace and version number (1GP managed only), version id (1GP and 2GP), and more powerfully, other GitHub repositories using CumulusCI which are recursively inspected and resolved.
For example, the following lines in a cumulusci.yml will automatically install the latest version of Salesforce.org's Nonprofit Success Pack (NPSP), a set of six managed packages and some unmanaged metadata:
project:
dependencies:
- github: https://github.com/SalesforceFoundation/NPSP
Once you've properly defined your project's dependencies in cumulusci.yml, all orgs CumulusCI creates will automatically have dependencies preinstalled.
CumulusCI's GitHub repository dependencies can also be configured with different dependency resolution strategies such as latest production release all the way to 2GP skip validation packaged versions from feature branches of a 1GP managed package developed in a GitHub repository with a matching branch name to unlock parallel cross-repository package development.
Composable Cross-Repository Automation
I mentioned portable automation as a concept briefly before, but let's explore that more. You've built a set of functionality in a GitHub repository (Project A) and automated the full configuration with CumulusCI. Now you want to build another project (Project B) that depends on Project A. If Project A is packaged, you can handle the dependency by manually editing package verison ids in the sfdx-project.json file of Project B. But that just installs the package and isn't great for operations overhead if you frequently release Project A.
With CumulusCI, Project B would define a dependency on Project A and automatically get the right dependency version for the selected resolution strategy without any file edits after releases of Package A. But that's not the biggest win. You can also source in all the automation you built up in Project A. Let's say you built a flow called config_common that contains a set of tasks of common configuration for your project and default sample dataset using capture_sample_data. Your Project B cumulusci.yml can source in Project A's automation and wire it into its automation:
sources:
projecta:
github: https://github.com/YourCompany/ProjectA
flows:
config_dev:
steps:
3:
flow: projecta:config_common
4:
task: projecta:load_sample_data
With automation that is portable and composable, the investment in automation is more naturally part of the development process.
Deep Integration with GitHub
Beyond the ability to reference GitHub projects as dependencies, CumulusCI contains many tasks and flows that automate common operations on GitHub including:
- Release Operations Flows: Out of the box flows for complete release opeations of 1GP managed, 2GP managed, and 2GP unlocked packages including creation of annotated git tags and GitHub Releases.
- Release Notes Generation: Automatically parse pull request bodies for specific sections to be published in release notes by finding all merged pull requests since the past release, merging content together under headings, and publishing to the body of the GitHub Release.
- Automated Merge-Down: CumulusCI is set up by default for a fully automated branching model with the main branch representing the release train and all development occurring in branches starting with feature/. CumulusCI's branching model also supports parent/child feature branches where the child is designed to merge into the parent like feature/parent__child. CumulusCI's automated merge down tasks handle merging main commits to all standalone and parent feature branches and merging from parent branches to their children.
Datasets and Synthetic Data
CumulusCI can both capture and load complex relational datasets to and from Salesforce orgs and generate complex synthetic relational datasets using Snowfakery which are then loaded into Salesforce orgs with a highly efficient and optimized data load capable of loading hundreds of millions of records.
Metadata Transformations
CumulusCI introduced the concept of Metadata ETL or Metadata Transformations. The idea is to automation deployment operations that are often too difficult to deploy a static metadata to be safe and reliable. Consider you're working on a feature that requires a new picklist value in a field. But, you don't know what other picklist values might have been created on that field in other orgs where your feature needs to be deployed. CumulusCI's add_picklist_entries task can retrieve the existing field, check if the value exists, add it if not, then deploy it back to the org, safely.
I wrote about this in CumulusCI's Metadata Transformations.
2GP Packaging, Improved
CumulusCI includes a completely reimplemented approach to 2GP packages that eliminates the need to edit and commit files after a release to record the creation of a release and adds functionality by tapping into CumulusCI's built-in dependency management.
I wrote more about this in Advantages of CumulusCI's 2GP Implementation.
Browser Test Automation
CumulusCI comes pre-installed with Robot Framework, an open source testing framework. CumulusCI includes a custom Salesforce, CumulusCI, and Page Objects library for Robot Framework that provides common keywords and base classes to build robust and maintainable functional tests of Salesforce via APIs and the browser with Selenium.
Marketing Cloud Deployment
CumulusCI's keychain can connect Marketing Cloud accounts and the deploy_marketing_cloud_package task can be used to deploy a set of configuration to Marketing Cloud.
MetaDeploy
Extend your project's portable automation built with CumulusCI to the end user by hosting an instance of MetaDeploy, the open source web app that powers Salesforce.org's package installer at https://install.salesforce.org. CumulusCI includes built in tasks to publish new automation plans to MetaDeploy.
Metecho
Invite declarative-only developers into source-control based collaboration by hosting an instance of Metecho, an open source web app that provides a simple web based user experience for creating new tasks (branches), spinning up a dev scratch org using CumulusCI's dev_org flow, logging into the scratch org to make declarative changes, capturing those changes to version control, submitting for review, spinning up a new scratch org for testing, and creating a pull request with tester approval.
MetaDeploy and Metecho provide great examples of the power of portable automation being applied to different user experiences throughout the Development-to-Delivery Lifecycle.
D2X
While not directly a part of CumulusCI, MuseLab's own open source D2X framework is built as a layer on top of CumulusCI to more deeply integrate it with GitHub including a maintained Docker image with Salesforce CLI and CumulusCI, reusable GitHub Actions workflows to automation feature branch testing and beta and production releases, and a devcontainer configuration for use with the VS Code Devcontainer Extension or GitHub Codespaces.
Our free D2X Launchpad lets anyone launch a fully configured GitHub repository for preconfigured with D2X and CumulusCI in minutes.
Conclusion
CumulusCI offers a wealth of functionality designed to make your life as a Salesforce developer easier. Using it doesn't require abandoning Salesforce CLI, quite to the contrary! But, learning a new tool can be scary and frustrating. Hopefully you've seen how you can start exploring how CumulusCI could be integrated into your current and future projects. I'd love to hear your thoughts and future success stories.
Comments