Are you using Scratch Org Snapshots? Are you using them to their fullest potential?
Check out our comprehensive guide to planning how to integrate snapshots to optimize your package development lifecycle.
With Scratch Org Snapshots, Salesforce teams working on packages, particularly extension packages, can generate new scratch orgs up to 6 times quicker. This results in accelerated development org setup, faster CI builds, quicker test environment provisioning, and expedited feedback loops.
The example from this graphic is a real world example from one of our current consulting engagements with an AppExchange ISV Partner. When trying to work through a complex build issue that takes multiple attempts at resolution, getting build results in ~12 minutes instead of ~74 minutes is a significant boost to developer productivity. And best of all, those benefits flow all the way through your entire product lifecycle.
What is a Scratch Org Snapshot?
A snapshot is a point-in-time copy of a scratch org that includes installed packages, features, limits, licenses, metadata, and data.
Scratch Org Snapshots are a new feature of Salesforce that is part of the Dev Hub. Snapshots capture a scratch org's state as a template that can be used to create new scratch orgs up to 6x faster than running the automation against a new scratch org to install packages, deploy code, insert data, etc. Snapshots can be referenced by name in scratch org definition files, configuring the scratch orgs to start from the snapshot instead of an edition like Developer or Enterprise Edition.
Salesforce has great documentation available for Scratch Org Snapshots. I highly recommend reading through this documentation as part of learning about snapshots. This guide does not attempt to recreate Salesforce's documentation. Instead, I'll focus on the lessons learned in planning and implementing snapshots to optimize the package development lifecycle in our D2X Transformation Success consulting engagements with Salesforce partners and customers. All the stuff the documentation doesn't cover.
How Snapshots Streamline the Product Lifecycle
Faster on-demand creation of scratch orgs saves time and shortens feedback cycles throughout the entire development-to-deliver lifecycle of the product.
- Developers save time creating dev orgs
- QA saves time creating test orgs
- Continuous Integration builds complete faster, providing developers with more immediate feedback on changes
- Faster org provisioning for internal review by Product, UX, Doc, Sales, etc. creating faster feedback cycles
- Operations savings with bulk org provisioning for enablement and implementations
The Constraints of Scratch Org Snapshots
Snapshots are incredibly powerful, but there are some important constraints to keep in mind:
- Snapshot limits for active snapshots and daily snapshots are equal to your DevHub's ActiveScratchOrgLimit (40 for Enterprise Edition by default)
- Snapshot names are limited to only 15 alphanumeric characters and cannot start with numeric (this is undocumented but verified)
- Source Tracking State is not stored with a snapshot. This means the sf project deploy start command will always do a full deployment the first time, minimizing the value of snapshotting pre-deployed unmanaged package source to speed up developer org creation.
- You cannot snapshot namespaced scratch orgs. You can create namespaced clones of a snapshot created from a non-namespaced org.
- Snapshots are limited to 200MB in data, the same limits as non-partner edition scratch orgs.
The Snapshot Anti-Pattern
A snapshot isn’t meant to replace source-driven development or a version control system.
My biggest fear about snapshots is that they'll be used exactly the way Salesforce's documentation warns not to use them. The correct way to use snapshots is to treat them as a means of compiling the results of automation in source control to make those results repeatable more easily and quickly.
The clear way to know if you're breaking doing this is to look for any manual work done in the scratch org before the snapshot is created. All that manual work should be automated. Ensure that everything that goes into your snapshot is defined and automated in source control and only build snapshots used as part of the product lifecycle through automated build pipeline such as GitHub Actions.
Proposed Snapshot Naming Convention
For an average AppExchange ISV Partner with a few managed packages, including extension packages, coming up with a naming convention for the 40+ snapshots you can have in your DevHub while sticking to the constraints of 15 alphanumeric characters is the first hurdle.
We've come up with a proposed standard for naming snapshots that we use in our consulting engagements. It goes as follows:
PC{2} + ImageName{1:11} + OrgTypeCode[U|M] + BuildSuffix{1}
Where:
PC: A two-character product code, both uppercase. Second character can be numeric.
ImageName: A PascalCased alphanumeric string up to 11 characters
OrgTypeCode: The purpose of the org: U for orgs meant for unmanaged package source deployment, M for orgs meant for package installation
BuildSuffix: One character reserved for use in temporary names by builds
For example, assuming the Dreamhouse app were a managed package with dependencies on other packages, the Dreamhouse app might use the product code DH and have the following snapshot names:
- DHBaseDepsU: All common dependencies pre-installed in a scratch org with the shape for deploying unmanaged package source for development
- DHBaseDepsM: All common dependencies pre-installed in a scratch org with the shape for installing a packaged version
- DHLatestBeta: Fully functional org with the latest beta release. Omits the U/M because this is only used for managed
- DHLatestProd: Fully functional org with the latest production release. Omits the U/M because this is only used for manged
We designed this naming convention to ensure snapshot names are as readable and useful as possible. The use of PascalCase, for example, was selected because of the lack of support for special characters that could be used as delimiters like - or _. Developing your own standard for ImageNames is encouraged.
Automating Snapshot Builds
Assuming you've automated everything you need to prepare the scratch org for a snapshot, the next step is figuring out how to automate building those snapshots so snapshot maintenance doesn't become yet another operations burden on your already overworked operations role/team. This is about saving people time!
Automating snapshot creation is pretty simple using Salesforce CLI:
sf org create snapshot \
--name <name, 15 alphanumeric characters max> \
--description <text, 255 characters max> \
--source-org <ID or alias of scratch org>
# If you don't have a target-dev-hub set up or want to use another DevHub, add:
--target-dev-hub <username or alias of Dev Hub org>
The Salesforce DX Developer Guide provides good coverage for the SF CLI commands related to snapshots. In short, they are create, list, get, and delete. The remainder of this guide focuses on important planning and implementation details we found during our implementations.
The OrgSnapshot Object
The key missing feature to properly automate snapshots is the ability to handle building a new snapshot under a temporary name and renaming it. Given the current implementation of snapshots, the way to "publish" a newly built snapshot to an existing name is to delete the current snapshot and then update the new snapshot to the deleted snapshot's name. There's a risk of a second or two here where the snapshot doesn't exist that could cause a build failure. Unfortunately that might just need to be an error you handle in scratch org creation during build scripts.
Currently SF CLI doesn't provide a mechanism for Update of the OrgSnapshot object, the REST API object SF CLI is interacting with for you in the snapshot commands. However, SF CLI is just making some API calls, there's no need to be constrained by SF CLI's lack of an update command for snapshots.
I won't cover in detail here, but even a simple curl command could manage the naming swap. Or, perhaps a contribution to SF CLI, a new plugin?
Or, maybe you prefer some other language or framework. I happen to love Python.
The API calls are quite simple to use in any language and are clearly documented at:
https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_orgsnapshot.htm
You can also review SF CLI's implementation in Typescript at:
https://github.com/salesforcecli/plugin-signups/blob/main/src/commands/org/delete/snapshot.ts
and
https://github.com/salesforcecli/plugin-signups/blob/main/src/commands/org/create/snapshot.ts
Using Snapshots with CumulusCI
CumulusCI is an open source framework optimized for package based Salesforce projects on GitHub. It was built by Salesforce to run the release operations Salesforce Industries products and is still actively used and developed by Salesforce and the community.
One of CumulusCI's major features is a powerful dependency resolution system designed for cross-repository extension package development. For example, Salesforce.org's Nonprofit Success Pack is a collection of six managed packages, each developed independently in their own GitHub repository.
This feature makes two things possible that were previously very difficult:
- Automate the installation of all the latest dependencies for a project with no operational overhead
- Define dependency resolution strategies for different org type, enabling features like cross-repository, cross-package parallel development of new features, again with no operational overhead
If you're building extension packages and have not explored CumulusCI, specifically its dependency management and release operations features, you really should.
With CumulusCI's dependency management configured in your repository, you can create a snapshot of your project's dependencies with a little bit of YAML in the cumulusci.yml file. The example below uses a convention I like to use of having two scratchdefs for a project be default: orgs/unmanaged.json for the shape of developer orgs and orgs/managed.json for the shape of orgs to test packaged versions.
orgs:
scratch:
dev:
# For unmanaged development of package source
config_file: orgs/unmanaged_snapshot.json
feature:
# For CI unit testing of 2GP packaged versions
config_file: orgs/managed_snapshot.json
qa:
# For integration and functional tests of 2GP packaged versions
config_file: orgs/managed_snapshot.json
beta:
# For all testing and review of the latest beta
config_file: orgs/beta_snapshot.json
release:
# For all testing and review of the latests release
config_file: orgs/release_snapshot.json
snapshot_unmanaged:
# Shape for building unmanaged snapshots
config_file: orgs/unmanaged.json
snapshot_managed:
# Shape for building managed snapshots
config_file: orgs/managed.json
flows:
snapshot_dependencies_unmanaged:
description: Installs dependencies and snapshot for unmanaged orgs
group: Snapshot Management
steps:
1:
flow: dependencies
2:
task: dx
options:
command: org snapshot create --name PCBaseDepsU --description Base Dependencies for Unmanaged Orgs
This YAML serves as a starting point to integrating snapshots into a CumulusCI project. It doesn't handle the logic around building the snapshot under a temporary name, monitoring the build, reporting the results, deleting the current snapshot, and renaming the new one. For that...
Coming Soon: Automated Snapshot Pipelines with D2X
We're actively working on a contribution for CumulusCI to implement snapshot management as part of our ongoing consulting engagements. Expect to see that functionality built into CumulusCI to automate everything covered in this article and reusable GitHub Actions workflows in D2X to automate your scratch org snapshot pipelines.
Conclusion: Are you convinced yet?
In conclusion, automating snapshot creation and maintenance is crucial for streamlining the development process in Salesforce projects. By following the proposed naming convention and leveraging tools like Salesforce CLI and CumulusCI, you can ensure that your snapshots are consistently built and managed with minimal manual intervention.
The upcoming Automated Snapshot Pipelines with D2X will further enhance this process, offering a comprehensive solution for automating your scratch org snapshot pipelines. Stay tuned for more updates on how to optimize your Salesforce development workflow and make the most out of your snapshot management.
Embrace automation, standardization, and efficiency in your Salesforce projects to drive success and innovation. Keep exploring new tools and methodologies to enhance your development process further. Let's continue to build better, smarter, and more efficient solutions in the world of Salesforce development.
If you're interested in exploring how snapshots can change your product lifecycle, grab a 1-hour free consultation with me to talk about your specific challenges and potential solutions.
Comments