5 Best Practices for GitOps in Kubernetes

What is GitOps?

GitOps is an operational framework that takes DevOps best practices used for application development such as version control, collaboration, compliance, and CI/CD tooling, and applies them to infrastructure automation. While the software development lifecycle has been automated, infrastructure has remained a largely manual process that requires specialized teams. With the demands made on today’s infrastructure, it has become increasingly crucial to implement infrastructure automation. Modern infrastructure needs to be elastic so that it can effectively manage cloud resources that are needed for continuous deployments.

Modern applications are developed with speed and scale in mind. Organizations with a mature DevOps culture can deploy code to production hundreds of times per day. DevOps teams can accomplish this through development best practices such as version control, code review, and CI/CD pipelines that automate testing and deployments.

GitOps is used to automate the process of provisioning infrastructure. Similar to how teams use application source code, operations teams that adopt GitOps use configuration files stored as code (infrastructure as code). GitOps configuration files generate the same infrastructure environment every time it’s deployed, just as application source code generates the same application binaries every time it’s built.

The benefits of applying GitOps best practices are far reaching and provide:

Increased Productivity
Continuous deployment automation with an integrated feedback control loop speeds up Mean Time to Deployment. Your team can ship 30-100 times more changes per day, increasing overall development output 2-3 times.

Enhanced Developer Experience
Push code and not containers. Developers can use familiar tools like Git to manage updates and features to Kubernetes more rapidly without having to know the internal of Kubernetes. Newly on-boarded developers can get quickly up to speed and be productive within days instead of months.

Improved Stability
When you use Git workflows to manage your cluster, you automatically gain a convenient audit log of all cluster changes outside of Kubernetes. An audit trail of who did what, and when to your cluster can be used to meet SOC 2 compliance and ensure stability.

Higher Reliability
With Git’s capability to revert/rollback and fork, you gain stable and reproducible rollbacks. Because your entire system is described in Git, you also have a single source of truth from which to recover after a meltdown, reducing your meantime to recovery (MTTR) from hours to minutes.

Consistency and Standardization
Because GitOps provides one model for making infrastructure, apps and Kubernetes add-on changes, you have consistent end-to-end workflows across your entire organization. Not only are your continuous integration and continuous deployment pipelines all driven by pull request, but your operations tasks are also fully reproducible through Git.

Stronger Security Guarantees
Git’s strong correctness and security guarantees, backed by the strong cryptography used to track and manage changes, as well as the ability to sign changes to prove authorship and origin is key to a secure definition of the desired state of the cluster.

After seeing the benefits of implementing the best practices in GitOps, below we list the 5 best practices for GitOps in Kubernetes:

1. Use Kubernetes Operators where possible

Best practice dictates using existing Operators whenever appropriate as opposed to ad-hoc, manual scripting. This is largely because managing stateful applications or other complex services in Kubernetes is quite challenging and requires a lot of knowledge that comes from experience.

Kubernetes Operators, which use Custom Resource Definitions (CRDs), are a lot more efficient than in-house solutions. Re-using vetted operators developed by the community or trusted vendors ensures that you standardize on the best tools for the job and benefit from the contributor’s experience in the specific service you’re deploying. For example, using an existing Operator to deploy a relational database on your Kubernetes cluster may be a better approach then re-scripting the entire configuration and deployment process yourself, from scratch.

2. Branches over repositories

While there are advantages to having a multiple-repository setup, (clear ownership, etc.) this model gets unnecessarily complicated as you scale up. Additionally, since each team essentially has separate repositories for development, staging, and production, it is hard to get a good overview of the system as a whole. When each team is only concerned with a single service, people with knowledge of the whole platform become a scarcity.

Reducing the number of repositories benefits developers considerably, as it not only saves time but also makes the tasks of sharing code, debugging and giving feedback a lot more straightforward.

Best practice with regard to GitOps is to have one repository per team; and where there’s a need for additional repositories, resort to branches instead. Of course, there’s no one-size-fits-all when it comes to the number of repositories, but it helps to give each team its own repository so that development is not restricted. At the same time, use branches to ensure you don’t end up with repository sprawl.

3. Pull over Push

Gitops is all about using pull requests to publish new code. A pull request is basically like an application you submit in order to have your changes advanced through the workflow, and is typically accompanied by a lot of supporting information. This additional information pertaining to why the changes were made, and by whom, makes the process of debugging and troubleshooting a lot easier.

A best practice is to avoid pushing anything directly to masters, unless absolutely necessary and standardize on Pull requests for the above reasons. In addition, push requests unnecessarily exposes your cluster credentials over the network, while pull requests cause Kubernetes Operators to deploy new images from inside the cluster. The Operator also watches for changes to configuration parameters and begins “convergence” once it finds one.

4. Automate

When it comes to GitOps and tasks that need to be repeated “exactly” the same way, it’s a best practice to keep the “human element” out of the picture. A hands-off approach here is critical, and every single action needs to be performed by a script that has already been tested for bugs. This is accomplished by Git webhooks, which fire in response to application changes that have passed the screening process.

5. Observability + GitOps

By monitoring the system and the state of your clusters and applications for any anomalies, and tying this data to trigger processes from Git – such as in the case of a failure, bottleneck or other incidents – you can automate response to Production issues and accelerate them to recovery, as well as auto-rollback if needed.

GitOps is an incredibly powerful workflow pattern for managing modern cloud infrastructure. Though primarily focused on Kubernetes cluster management, the DevOps community is applying and publishing GitOps solutions to other non-Kubernetes systems. GitOps can bring many benefits to an engineering team including improved communication, visibility, stability, and system reliability. If you want to benefit and implement the best GitOps practices for your project, contact us.

Sources: GitLab, weave.works