Welcome to the second blog post about our new 4DIdentity service. In the first blog post, we reviewed our motivation to create this internal service and its architecture. Brief summary: We use it to enable single sign-on for internal applications and future managed services. 4DIdentity is based on the Amazon Cognito service from AWS. We consciously chose to run it completely serverless, highly resilient, fully manageable and built using Infrastructure as Code (IaC).
In this post, we first outline how we deploy and configure the 4DIdentity service in a fully automated manner using GitOps. Then, we discuss the Infrastructure as Code setup that enables the consistent and automated configuration of the different AWS services. Next, we show how we use GitLab to keep development costs low by automatically cleaning up the infrastructure used during development. Finally, we describe how we implemented a basic permission management solution using GitOps principles.
Deployment
We focused mainly on automation and IaC when we developed this service. We used Terraform as our IaC tool and GitLab CI pipelines to apply the code. As a result, we can deploy the infrastructure to different environments automatically. As mentioned in the first blog post, the services used in 4DIdentity include EventBridge for monitoring, Simple Email Service (SES), and the Amazon Cognito user pools. We also outlined that we apply the Pilot Light disaster recovery strategy, which requires deploying a dormant setup into a failover region. The screenshot below displays the needs diagram of the main GitLab CI pipeline to deploy / configure the different services for the development environment. Especially, nice about this setup is, that deploying the failover infrastrucutre is simply an additional CI job after the primary infrastructure has been deployed.
The main pipeline consists only of trigger jobs, ensuring that the different services are deployed in the correct order, to address the dependencies between them.
Each trigger job, triggers a downstream pipeline that then applies the code by always running through the same four stages: validate, test,
plan, and apply, with an additional cleanup stage that can be executed manually to remove the provisioned infrastructure.
We then leverage the different pipeline types in GitLab, such as branch pipelines, merge request pipelines, and tag pipelines to apply the infrastructure for the various environments:
- Branch pipelines are used to deploy the development environment. Thus, when a development branch is created, a new development environment is deployed automatically. Merging the development branch to the “main” branch then triggers a deployment to the pre-production environment.
- Merge request pipelines are used to preview the changes that will be applied to the pre-production and production environments before the code is merged into the “main” branch.
- Tag pipelines are triggered when the code is tagged in GitLab and are used to deploy the production environment.
Additionally, we use GitLab to store the Terraform state file to ensure its consistency. Finally, we use the GitLab environment feature, which allows us to separate different environments in GitLab that are linked to specific pipelines, keeping the target environments up-to-date automatically.
IaC structure
For our use case, we expect to have several different user pools deployed as part of our 4DIdentity solution. Hence, it made sense to create the different user pools using a Terraform module. However, building and maintaining a module requires additional effort. To reduce this effort, we automated the process as well using GitLab.
In our main pipeline, which contains the trigger jobs for the deployment of our infrastructure, we also included a job to automatically validate and build our Terraform module for setting up the user pools. This job is executed only when the module code has been updated. While building the module, the job also uploads it to the Terraform module registry of GitLab and takes care of the correct semantic versioning. For the semantic versioning, we again leverage the different pipeline types that GitLab offers. The different versions are updated as follows:
- The patch version is upgraded whenever the module is built from a standard branch pipeline.
- The minor version is upgraded whenever the module is built from a tag pipeline.
- For the major version upgrade, we have created a dedicated flag to be set, as there won’t be many major upgrades expected.
Ephemeral vs. persistent environments
As we want to have different environments to ensure the maintainability and availability of the service. However, we also want to keep an eye on our cloud costs and avoid running obsolete infrastructure components. Even though it’s not a big issue, we still prefer to keep our development environment clean and only have services deployed when needed, hence having an ephemeral development environment.
With GitLab, there is a neat way to achieve this by using the built-in feature that terminates an environment, if it has been created from a pipeline
triggered by the creation of a development branch. If the cleanup job is linked to the on_stop action of such an environment, the cleanup job is automatically
triggered when the development branch is merged into the main branch. As a result, the infrastructure used during development is automatically removed,
keeping the cloud environment clean and costs under control.
Group membership management
4DIdentity acts as an identity and access management (IAM) service. Accordingly, it also provides the possibility to manage permissions in applications and tools connected to it. The permissions granted to users are managed depending on a person's group membership rather than on the individual person. In Amazon Cognito, the concept of groups has been adopted as well. Also, the group membership of users can be automated with either AWS Lambda functions or again IaC.
To manage the permissions of our internally used tools, we use a simple IaC setup to manage groups and memberships. A GitLab repository containing the Terraform code that is used to manage the group membership and automatically apply the code to the different environments in the same manner as we deploy the 4DIdentity service.
This allows us to manage the permissions for our internal applications by making a merge request (MR) to the code. Utilizing the CodeOwner feature of GitLab ensures that the responsible person reviews the MR, following the regular review and approval process. Finally, due to GitLab's built-in versioning system, these methods provide audit capabilities out-of-the-box.
Conclusion
This concludes this blog post as well as our series about our 4DIdentity service. In this post, we focused on how we structure our infrastructure code as well as how we apply the code in GitLab pipelines to deploy to different environments.
As always, please feel free to reach out to us for any feedback or questions.