This blog post provides insight into our latest project: building the platform necessary to operate and maintain our new managed services. The platform integrates multiple tools for monitoring and secrets management, all deployed on AWS cloud infrastructure. Our goal was to establish a fully descriptive setup, leveraging Infrastructure as Code (IaC) and Continuous Deployment (CD), enabling a comprehensive GitOps approach to set up and manage the platform.

Platform overview

Our services platform is designed to monitor and manage our new managed service offerings. The applications that build our platform run all on the Elastic Kubernetes Service (EKS) from AWS.

On the EKS cluster, we run the next generation of our 4Dmetrics observability stack, which collects metrics and logs, all visualized in Grafana. Additionally, the platform includes a custom HashiCorp Vault distribution for secure, centralized secret management for our services. Furthermore, the platform includes Harbor as our private image and Helm chart registry.

The platform also supports essential tools for our GitOps approach, such as ArgoCD and Crossplane. Finally, the Cilium service mesh is deployed on the cluster to secure and control the networking.

As you can see, the platform is extensive, requiring the deployment and configuration of numerous components. All of this has been automated using a GitOps approach, enabling us to deploy, configure, and even destroy the entire platform through a single GitLab CI pipeline, with no manual intervention required.

How did we achieve this?

Deploying the platform

To deploy the different parts that make up our platform, we rely on different descriptive methods, such as IaC and CD. The platform can be divided into the following three parts:

1. Cloud infrastructure
2. Service mesh & ArgoCD
3. Applications managed by ArgoCD

By using a GitLab CI pipeline, we can fully automate the deployment of the platform.

Cloud infrastructure

Our cloud infrastructure is deployed and managed using standard IaC practices with OpenTofu to define static cloud resources, such as the EKS cluster, IAM permissions, and roles. The infrastructure code is executed via GitLab CI jobs in the deployment pipeline.

For more dynamic resources, such as EBS volumes, Route53 records, and EC2 instances, we rely on operators within the EKS cluster to provision and maintain them as Kubernetes manifests. These manifests are applied later by ArgoCD.

The image below shows the services platform after the “validate”, “plan”, and “apply” stages of the pipeline.

Service mesh & ArgoCD

All applications and tools, except the service mesh, are deployed and managed on the EKS cluster using ArgoCD. Why exclude the service mesh?

Deploying a service mesh is invasive to the cluster and is best handled before installing any other application. Therefore, we use the GitLab CI pipeline to deploy and configure the Cilium service mesh via Helm right after the cluster has been created with IaC.

Once the service mesh is in place, the final pipeline step deploys ArgoCD. The image below shows the state of the services platform after the “servicemesh” and “argocd” stages of the pipeline.

Applications managed by ArgoCD

All applications and tools running on the EKS cluster are deployed and maintained using ArgoCD, this also includes ArgoCD itself. For this purpose, we utilize a customized ArgoCD distribution that we call “Argo-Platform,” which is based on the ArgoCD AutoPilot project.

This allows all the applications on the EKS cluster to be deployed autonomously once ArgoCD is installed via the GitLab pipeline. The applications are deployed using Helm, Kustomize, or Kubernetes manifests, all stored in a central GitLab project.

The image below shows the fully deployed services platform with all the different applications.

Managing the platform

Up to this point, all of the required infrastructure components, as well as all the specified applications, are defined as code and stored in a GitLab project. This enables us to follow the GitOps approach, using protected branches and features like CODEOWNERS to ensure changes are reviewed and approved before being deployed to production.

Now, one last part is missing to fully automate the platform: the configuration of HashiCorp Vault and Harbor. To solve it, we use Crossplane.

Crossplane, like Terraform or Pulumi, is an IaC tool. However, the main difference is that Crossplane leverages Kubernetes to apply the infrastructure code and to preserve the configuration state. Using Crossplane providers for HashiCorp Vault and Harbor allows us to configure them as Kubernetes Custom Resource Definitions (CRDs). These CRDs are stored as Kubernetes manifests alongside the deployment manifests in GitLab.

As a result, the entire platform – from its infrastructure setup to the application deployments as well as their configurations – is consolidated in a single GitLab project.

Summary

We managed to completely automate the entire setup and teardown of our services platform. All the needed parameters and settings are version-controlled, ensuring that all changes are reviewed and approved before being applied to the production environment.

The use of the GitOps principles for our platform has proven to be an ideal solution for deploying and operating our Kubernetes workloads. Our environment now hosts over 40 applications managed by ArgoCD and Crossplane. It was fully provisioned and configured from scratch in less than an hour, providing a highly available and stable platform on the AWS cloud infrastructure.

Any feedback or questions are very welcome. If you’re curious about all the details or need help applying GitOps in your organization, don’t hesitate to contact us.