It’s quite common for Salesforce ISVs to wait until they’ve cut a production version of a package to fully regression test it. This is due to one primary constraint of Salesforce’s packaging infrastructure: Beta versions cannot be upgraded. That means once you install a beta version into an org, that org is useless after the beta is tested because the package can’t ever be upgraded. ISVs who rely on persistent QA orgs can’t just install a pre-release beta version to test it.
For clarification, “pre-release” in this article refers to a pre-release version of the ISV product instead of the Salesforce platform pre-release preview window.
In this post, I’ll examine 3 approaches to pre-release testing for Salesforce ISV-style development:
- Sandbox per Release: Maintain a persist production org for QA, create a sandbox per release to test beta versions
- Alternate Namespace: Publish the package code in an alternate namespace and push upgrade to QA environment
- On-Demand QA Environments: Automate creation of QA environments on-demand so they are disposable after installing a beta
Why Pre-Release Testing?
Before diving into the approaches, let’s briefly discuss the challenges of only testing after creating a production package version.
The first consideration is manageability rules in managed packages which restrict certain changes to released metadata. If you’re only fully testing after creating a production package version, you risk introducing breaking changes that cannot be reverted.
Next, consider the impact on the scalability of your development lifecycle. Fully testing only after a production release is the opposite of the best practice of shifting left. It means issues are found very late in the process and typically when a release date has already been set. This can easily lead to a scramble around release time. For more, check out my post about The Release Subway.
Sandbox per Release
The most traditional of the 3 approaches is to maintain a single production QA org always running on the latest production release. When a new release beta is available, create a sandbox from the production org and use that sandbox to test the beta version. Since sandboxes can be deleted or refreshed, there is less risk installing a beta version. Plus, each created sandbox starts from a copy of all the production QA org’s metadata (and potentially data), saving effort in automating those.
This approach is ideal for an ISV looking to quickly roll out a process for pre-release testing of beta versions without much investment in automating the configuration of QA orgs.
Alternate Namespace (1GP) or Package (2GP)
Many ISVs use an alternate namespace to create “production” versions of their package under a different namespace. This is most common with first-generation packages (1GP) where each package must have a separate namespace.
Using an alternate namespace allows the creation of released package versions in the alternate namespace that can be installed, upgraded, and push upgraded into any org.
However, testing your package code in a different namespace introduces its own level of risk. Your persistent QA orgs must all use the alternate namespace rather than the production one.
There is also a minor risk in missing some manageability rule checks if the alternate namespace package doesn’t have the exact same version history as the production namespace package. This can easily happen if an issue is discovered in the “production” version of the alternate namespace package that gets locked in by the manageability rules requiring creation of another alternate namespace.
Second-generation packages (2GP) make this approach easier by allowing the creation of a second package in the same namespace. The challenges of manageability rules still exist.
On-Demand QA Environments
If you’ve read my other posts such as A Product is More Than a Package, you can probably guess that this is my recommended approach.
By investing in fully automating the entire configuration of a fully testable org, it becomes possible to use on-demand scratch orgs as pre-release beta version QA environments. Since it’s in a disposable scratch org, using a beta isn’t a concern.
CumulusCI includes a default flow called regression_org designed to facilitate this type of pre-release testing by simulating a push upgrade. It first installs the latest production version and then upgrades it to the beta being tested. The upgrade to the beta version is done using options on the PackageInstallRequest object in the Tooling API to simulate the behavior of a push upgrade.
This approach requires investing in automation which can be a significant investment depending on the complexity of your QA environments. However, that investment pays off throughout the development to delivery lifecycle of your product. For example, that same automation makes it possible to test feature branches pre-merge since it allows you to break out of the constraints of persistent orgs altogether.
Conclusion
Pre-release testing is a key component to a best practices based development to delivery lifecycle for Salesforce ISVs. Any of the 3 approaches outlined in this article are far superior to no pre-release testing at all.
If you’re looking for a quick fix only for the challenges of pre-release testing of a 1GP package, consider the Sandbox per Release approach.
If you’re looking for a quick fix for pre-release testing a 2GP package, consider the Alternate Package approach.
If you want to make a broader investment to streamline in your overall development to delivery lifecycle, consider the On-Demand QA Environments approach. The investment required to fully automate the configuration of a QA org is also highly useful in other areas such as demo environments and customer implementations.
Need help deciding which approach is best for you or want to dig in deeper on any of these approaches? Grab some time for us to chat at https://calendly.com/muselab
Comments