Leveraging Consistency for Safer GitOps Practices

As 2021 begins, DevOps toolchains are becoming more complex and broader as organizations continue to mature in their practices. To keep up, teams are improving their toolchains by avoiding overlaps, conflicts, and functionality gaps.

One way to do this is through GitOps. GitOps centralizes code and configuration, creating a single source of truth for the application's underlying code and configuration.

A step in the right direction, for sure. But there is room for improvement. GitOps can be error-prone as it doesn’t validate the content of commits. It also presents issues with authorization and security—you don’t necessarily want everyone on your team to have access to git repositories.

How can organizations leverage GitOps more safely and consistently to accelerate their DevOps practices? Here’s a closer look at how it works, what the challenges are, and how to make GitOps more trustworthy and consistent.

What Is GitOps?#

GitOps is a practice which uses a developer mentality and toolset to drive operational processes. As the name suggests, the version control system Git is essential to this workflow pattern. However, simply storing files in a Git repository for shared access doesn’t mean an organization is practicing GitOps.

Site reliability engineering architect Rob Scott, in a CI/CD eBook by The New Stack, offers a strong interpretation of GitOps: “The fundamental theorem of GitOps is that if you can describe it, you can automate it. And if you can automate it, you can control and accelerate it. The goal is to describe everything—policies, code, configuration and monitoring—and then version control everything.”

The ability to describe and codify a state or process is essential. In this case, “description” means codifying information in a specific and predictable format so that a pipeline of tools can process that information and act based on it. Formats such as YAML and JSON are examples of this concept, as parsing and manipulating them is well understood.

As these GitOps descriptions evolve, changes are committed to the git repository. These commits can be gated by a notification, review, and approval process (a “pull request”). This is fundamental to GitOps; which has also been described as Operations by Pull Request. Git also tracks the state of the repository at every commit, creating a history along with a way to return to the state of a previous commit.

Many of the canonical GitOps examples revolve around Kubernetes or infrastructure-as-code systems like Terraform. Generally, a continuous deployment system monitors the Git repository for changes to description file(s), and deploys the latest version when updates are detected.

Why Doesn't Everyone Do This?#

In a word: trust. To put GitOps into practice, multiple layers of trust and process are required.

First, a team must be able to trust the input to the process—the format and content of the “descriptions.” With other tools set to take action based on the data provided to a GitOps process, the principle of “Garbage In, Garbage Out” can easily apply if errors or incorrectly formatted descriptions are committed. A simple typo can result in a variety of errors, some of which can negatively impact production environments. Speed without verification isn’t a great solution.

Also, organizations need to be able to trust their users to add or modify information in the correct repositories for GitOps processes to work correctly. Many organizations grant users access to sets of git repositories rather than authorizing one at a time, giving them control of more repositories than they need, which introduces risk. Setting up approvers to pull requests can help alleviate this challenge, but it adds a human step to the process.

Finally, organizations have to build trust in their tooling for GitOps processes to succeed. If the tools watching the git repository aren’t capable of digesting the data in the new commit and acting on it, the other portions of the process have no impact. Examples of tools that successfully “watch” and trigger off of commits include CI/CD pipelines and systems which can “hot reload” configuration without a restart.

A Better Way#

Despite the challenges, GitOps processes can be both safe and consistent for companies of any size. I’ll illustrate this point with an example from my team.

At Transposit, we use GitOps processes whenever a new customer environment (a Transposit “team”) is created. This file controls environment authentication as well as enabling specific feature flags. We’re able to limit access to this process to members of the Transposit sales engineering team by exposing it as a human-in-the-loop workflow initiated through a Transposit runbook. This workflow targets a single file in a git repository hosted on GitHub. This way, my teammates and I can create a targeted pull request whether or not they have full access to the repository.

Rather than expecting the user to manually create a pull request with correctly formatted changes, this workflow asks the user for only two text values in an input prompt. The workflow then gets the current content of the configuration file and uses the two strings gathered from the user to modify the file. These modifications are predictable and consistent because their format is programmatically defined within the Transposit workflow.

After modifying the file, Transposit creates a pull request with the committed changes within the repository. The pull request is merged into the repository after a set of validation tests are executed against the updated configuration. Once the merge is complete, the updated configuration file is loaded by the Transposit application, making the new environment accessible.

Safer GitOps With Transposit#

Transposit provides layers of assurance as organizations adopt and improve their GitOps processes.

• Human-in-the-loop workflows offer a safe interface to create consistent pull requests with verified and sanitized input data.
• As a native API integration platform, Transposit can integrate with multiple git repository providers and abstract that complexity from the user.
• Transposit workflows can be customized and modified through the developer platform to enable organizations to rapidly onboard new GitOps flows, or iterate on existing ones.

For an additional resource on navigating GitOps, check out our webinar Scaling GitOps for Humans.