Securing Salesforce DevOps: Multi-Job Workflows in GitHub Actions

Getting it working is not the same thing as getting it working securely! While seeing automation work for the first time can be exhilarating, it's important to remember to take a step back and consider the security of your implementation.

One nuanced yet critical challenge I've encountered over the past two years is securely splitting GitHub Actions workflows into multiple jobs that interact with the same Salesforce org. This post delves into this challenge, why it matters, and how we can address it securely.

This article was inspired by multiple posts I've seen from people trying to split up CI/CD workflows across multiple jobs, often misusing build system features like artifacts and caching against security guidance from the vendor and best practices in general.

Not a GitHub user? No problem! Most of this post applies to your CI/CD system of choice. Almost all CI/CD systems have a concept of Workflows, Jobs, and Steps and some mechanism for passing information between them. Just map those words to your platform and read away.

Overview: Understanding Workflows, Jobs, and Steps in GitHub Actions

In GitHub Actions, a workflow is an automated process triggered by events. Each workflow comprises one or more jobs, which are collections of steps executed on the same runner. Here's a quick breakdown:

• Workflow: The entire automation process, defined in a YAML file.
• Job: A set of steps executed in sequence on the same runner.
  • Run in their own isolated containers, including custom Docker images
  • Can specify dependencies on other Jobs (needs) and run in parallel
  • Can use matrix inputs for parallel testing of multiple org setups
  • Can be individually rerun on failure
• Step: An individual task, like running a command or an action.

Jobs are clearly a critical component of designing Workflows. Being stuck in a single Job constrains our ability to fully use all of GitHub's features around Actions.

The Challenge

Both Salesforce CLI (sf) or CumulusCI (cci) store org credentials in local files. Most build scripts start with connecting the target org(s) to the sf or cci keychain.

When splitting workflows, persisting data between jobs becomes necessary. Specifically, for Salesforce workflows, sharing the ~/.sfdx and ~/.cumulusci directories—which contain org authentication details—is essential, especially for workflows that create a scratch org and need to pass that org to another Job. However, doing this securely without violating GitHub's security guidelines is challenging.

Security Implications

As part of our focus on Securing Salesforce DevOps, let's take a deeper dive into the security implications hiding below the surface of that working automation...

The Sensitivity of CI/CD Tokens

CI/CD tokens for Salesforce orgs are highly sensitive. They often grant extensive permissions, and mishandling them can lead to significant security breaches. Referencing our previous post on Least Privilege Access Control, it's clear that over-privileged credentials are a risk.

"But It's Just a Scratch Org!"

It's tempting to downplay the risk because scratch orgs are temporary. However:

• Potential Data Exposure: Scratch orgs may contain sensitive IP like metadata or configurations.
• Credential Leakage: If the ~/.sfdx directory contains other org credentials (like the DevHub), inadvertently sharing it exposes more than just the scratch org.
• Compliance Violations: Using methods that contravene GitHub's security guidelines can lead to non-compliance with industry regulations.

GitHub's Security Guidance

Specifically regarding passing sensitive data between jobs, GitHub provides a skeleton code example of using an external secrets store, along with this note:

"If you want to pass a masked secret between jobs or workflows, you should store the secret in a store and then retrieve it in the subsequent job or workflow."
Example: Masking and passing a secret between jobs or workflows - GitHub Documentation

Ignoring this guidance not only puts your data at risk but also goes against best practices.

Secure Options for Data Persistence Between Jobs

Unfortunately, GitHub doesn't provide us many options here. In fairness, it wasn't really built to handle this type of challenge and their documentation clearly says to use an external secrets manager.

Option 1: Using a Single Job

The most straightforward solution is to keep all steps within a single job. Since jobs run on the same runner instance, the ~/.sfdx and ~/.cumulusci directories persist throughout the job's execution.

I know... this isn't solving the original problem. This is pretty much where we've been stuck in designing D2X's reusable workflows.

Insecure  or Unworkable Options

Here's a brief rundown of the options we've explored and decided against.

Using GitHub Secrets

Secrets are GitHub's recommended way of handling passing sensitive values to workers. If you can use Secrets, use them! Unfortunately, in this case we can't:

• Limitation: Secrets are read-only and cannot be modified by jobs.
• Risk: Jobs cannot write back updated credentials (like refreshed tokens) or create new secrets, and attempting to misuse secrets can lead to security vulnerabilities.

Artifacts

Warning: Do not store any sensitive data in artifacts. Artifacts are accessible by anyone with read access to the repository.
Storing workflow data as artifacts - GitHub Docs

• Persistence: Artifacts persist beyond the job and can be accessed by anyone with read access to the repository.

Caches

"Warning: Do not store any sensitive data in the cache. The cache is stored on GitHub servers and is accessible to anyone with read access to the repository."
Caching dependencies to speed up workflows - GitHub Docs

• Not Secure for Secrets: Caches are designed for dependencies, not sensitive data.
• Risk of Exposure: Caches can be restored in unintended contexts, leading to credential leakage.

Environment Variables and Outputs

• Size Limitations: Cannot handle large data like entire directories.
• Security Concerns: Outputs are logged and may inadvertently expose sensitive information.
• Secrets Masking: Passing structured data like json can cause GitHub's secrets masking functionality to fail to mask a secret.

An Example: D2X's Release 2GP Workflow

D2X is Muselab's own open source CI/CD framework for Salesforce package development on GitHub. A key feature of D2X is its set of reusable GitHub Actions workflows to automate much of the CI/CD process CumulusCI was built to enable.

We've spent a lot of time at Muselab researching how to optimize those workflows. Here's an overview of how D2X's Release 2GP workflow could be optimized if it could be split into multiple jobs:

This example provides just one instance where solving the challenge of passing generated credentials between Jobs in a Workflow would improve efficiency and user/developer experience. Every one of D2X's reusable workflows could benefit too!

Coming Soon: Muselab Platform

We're excited to announce that the upcoming Muselab Platform is designed to address this exact challenge. Our integrated secure cloud org keychain allows you to:

• Create, Connect, and Share Orgs Securely: Manage org credentials with full auditability.
• Fine-Grained Access Control: Define who can access what, minimizing the risk of unauthorized access.
• Seamless Integration: Works with your existing CI/CD pipelines to provide secure credential management without sacrificing flexibility.

By offloading credential management to the Muselab Platform, you can safely split workflows into multiple jobs, harnessing the full power of GitHub Actions while adhering to security best practices.

Conclusion

Balancing efficiency and security in Salesforce DevOps is a complex task. While splitting workflows into multiple jobs offers significant benefits, it's crucial to handle data persistence securely. Until solutions like the Muselab Platform become available, using a single job or integrating an external secrets manager are the secure options.

Remember, the convenience of cutting corners isn't worth the potential security risks. Always prioritize safeguarding your credentials and comply with platform guidelines to protect your organization's assets.

Stay tuned for updates on the Muselab Platform and more insights into securing your Salesforce DevOps processes.