Serrio Mal | Provisioning WebApp Infrastructure with Terraform, Vercel, and GitLab

The web app presenting this article to you is a static site built using Astro. Because it’s just a frontend, it doesn’t need much in terms of infrastructure to run it, but I thought it would be a good learning exercise to see how far I could take the automation of its configuration management, in addition to that of deployments.

Infrastructure Components

There are three infrastructure-related areas of the app that can be managed programmatically - hosting, domains, and monitoring.

For hosting, I chose Vercel. They have a solid free-tier and free preview environments. Deploying a frontend app to Vercel is simple, their CLI tool can build and deploy frameworks such as NextJS, Astro with a single command, or automatically build and deploy from a target Git repo branch upon change.
CloudFlare acts as a CDN proxy between the site and the Internet At Large, and also manages domain configuration, automated SSL/TLS certificates, DNS records, and nameservers. Again, their free-tier gives you a lot of functionality and control.
Checkly provides E2E and API monitoring via Playwright. Their free-tier quota is sufficient for simple uptime monitoring, and can be extended with Playwright E2E scripts.
Sentry provides error tracing, performance monitoring, and alerting functionality.

All scripts for managing infrastructure configuration are hosted in a GitLab repo alongside the webapp project.

Managing Infrastructure with Terraform

Download the Terraform binary for your platform

Terraform is a provisioning tool created by HashiCorp that allows you to define and manage resources such as cloud infrastructure, network configuration, or policies, as code (AKA IaC). These configurations can then be versioned in an SCM, and automated as part of a CI/CD process. Terraform is commonly used to manage cloud provider infrastructure such as AWS, GCP, and Azure, but any remote service that has an API and a Terraform Provider can be managed with Terraform. In this case, Vercel, CloudFlare, and Checkly all offer Terraform Providers to interact with their APIs.

While I’ve used configuration management tools like Ansible in the past, Terraform was new to me. Fortunately, it does not have a super steep learning curve to grok the basics.

A Terraform project can be written in HCL; or in C#, Go, etc., using CDKTF. A project consists of one or more .tf files in which are declared resources, data, and variables. Terraform reads all files in the project and assembles resources into a dependency tree. This means everything can be lumped into one .tf file, or broken down into separate files without having to worry about imports or dependencies between files. The tree starts with the terraform {} entry-point, which defines any required Terraform providers, remote state backends, etc.

Terraform State

GitLab menu — View TF state objects in GitLab’s Infrastructure menu.

Terraform has a concept of State, which allows it to compare the desired resource configuration against the actual configuration, and to make necessary changes to match that state. By default, Terraform stores state locally in a .tfstate file, which is unencrypted.

GitLab provides a free encrypted remote state service for Terraform, as well as full support for Terraform build tools on their platform. GitLab’s Terraform remote state is available to all projects and is viewable in the project’s menu list. Hashicorp also provides a free remote state service via their Terraform Cloud product.

Terraform Providers

A Terraform Provider acts as an abstraction over a third party API. This project uses the following providers:

Fleshing out the main terraform statement defines the GitLab remote backend and provider dependencies:

# main.tf
terraform {
  backend http {}

  # It's recommended to pin providers to a version or version range.
  required_providers {
    cloudflare  = { source  = "cloudflare/cloudflare" }
    vercel      = { source  = "vercel/vercel" }
    checkly     = { source  = "checkly/checkly" }
    sentry      = { source  = "jianyuan/sentry" }
    gitlab      = { source  = "gitlabhq/gitlab" }
  }
}

The backend {} statement uses Terraform HTTP environment variables behind the scenes to communicate with the Gitlab remote store. These variables are defined in the project’s CI/CD environment when using GitLab’s Terraform CI/CD yaml include.

The following is needed to interact with GitLab’s Terraform remote state:

A Gitlab project to host the Terraform code and state. Make note of the project ID.
A Gitlab API-scoped access token.

During planning or execution, Terraform also needs environment variables defined for the providers specified below. Environment variables prefixed TF_VAR_ are automatically assigned to matching Terraform variable declarations when Terraform is ran.

Importing Existing Infrastructure

Terraform can only manage what it knows about, attempting to define a new resource for something that already exists remotely can cause a planning error. The current solution is to use Terraform CLI’s import subcommand to put in data from an external API endpoint. Its functionality is currently a bit cumbersome - you need to query the service’s API endpoints manually using an API tool such as Insomnia, note their ids, create matching resource definitions, then run import against those endpoints.

HashiCorp aims to eventually have import create the .tf resource definitions automatically.

For example, to import an existing Sentry.io configuration:

terraform import sentry_organization.app org-slug
terraform import sentry_team.app org-slug/team-slug
terraform import sentry_project.app org-slug/proj-slug
terraform import sentry_key.app org-slug/proj-slug/key-id

terraform plan shows the existing resources imported from the external API, and any changes.

# sentry_key.app will be updated in-place
~ resource "sentry_key" "app" {
      id           = "a3abc6abcdefghijklmnopq"
    ~ name         = "Default" -> "App Key"
      # (9 unchanged attributes hidden)
 }
# sentry_project.app will be updated in-place
~ resource "sentry_project" "app" {
      id                = "app"
    ~ name              = "Default" -> "App Name"
    ~ resolve_age       = 0 -> 480
      # (12 unchanged attributes hidden)
}

Setting Provider Variables

The following environment variables need to be set, so Terraform can utilize them:

Variable	Value
`GITLAB_USERNAME`	GitLab username
`GITLAB_TOKEN`	GitLab API scoped access token
`TF_VAR_gitlab_group_id`	The number of the GitLab group containing this project.
`TF_VAR_vercel_team_id`	Vercel Team ID that will contain the app deployment
`VERCEL_API_TOKEN`	Vercel API token scoped to the team
`CLOUDFLARE_API_TOKEN`	CloudFlare API token scoped to the domain zone with DNS and Settings edit rights
`TF_VAR_domain`	Root domain, e.g. `domain.com`
`TF_VAR_host_prod_ip`	`76.76.21.21`, Vercel’s production IP
`TF_VAR_host_dns_addr`	`cname.vercel-dns.com`
`TF_VAR_checkly_account_id`	Checkly account ID
`TF_VAR_checkly_api_key`	Checkly API key
`SENTRY_AUTH_TOKEN`	Sentry.io API token

Initializing GitLab Remote State

The remote state can be initialized using the terraform init CLI command with the switches given below.

PROJECT_ID is the numeric ID of the GitLab repo hosting the Terraform code.
STATE_NAME is the name chosen for the Terraform state object.
GITLAB_TOKEN is an API-scoped access token.

GITLAB_ROOT="https://gitlab.com/api/v4/projects"
STATE_URL="$GITLAB_ROOT/$PROJECT_ID/terraform/state/$STATE_NAME"
terraform init \
    -backend-config="address=$STATE_URL" \
    -backend-config="lock_address=$STATE_URL/lock" \
    -backend-config="unlock_address=$STATE_URL/lock" \
    -backend-config="username=$GITLAB_USERNAME" \
    -backend-config="password=$GITLAB_TOKEN" \
    -backend-config="lock_method=POST" \
    -backend-config="unlock_method=DELETE" \
    -backend-config="retry_wait_min=5"

After executing the command, a state object should be listed under the Infrastructure > Terraform menu within the GitLab project.

Refer to GitLab’s Terraform documentation for more details.

Configuring Terraform Providers

Configuring the Vercel Provider

The Vercel Terraform provider can both provision environments and also handle deployments to them. However, at the time of writing v0.10.3 of the provider cannot create a production environment from an initial deployment, instead it will create a preview environment. For this reason, I chose to not have Terraform handle deployments, and instead delegated that duty to the web app’s CI/CD pipeline. Logically this also made sense for keeping infrastructure configuration separate from app configuration.

vercel.tf shown below ensures:

A Vercel project called webapp exists under the Vercel Team specified in TF_VAR_vercel_team_id.
Configure a redirect from serriomal.com to www.serriomal.com.
The output declaration instructs Terraform to return the Vercel project ID when it is generated. The value is saved in the Terraform remote state.

# vercel.tf
provider "vercel" {
  team = var.vercel_team_id
}

resource "vercel_project" "app" {
  name    = "webapp"
  team_id = var.vercel_team_id
}

resource "vercel_project_domain" "www" {
  project_id = vercel_project.app.id
  team_id    = var.vercel_team_id
  domain     = "www.${var.domain}"
}

resource "vercel_project_domain" "root" {
  project_id           = vercel_project.app.id
  team_id              = var.vercel_team_id
  domain               = var.domain
  redirect             = vercel_project_domain.www.domain
  redirect_status_code = 301
}

output "vercel_project_id" {
  description = "The id of the current Vercel project entity"
  value       = vercel_project.app.id
  sensitive   = true
}

Configuring the CloudFlare Provider

The CloudFlare Terraform provider allows for programmatic management of a domain.

cloudflare.tf ensures that:

The domain’s DNS A and CNAME records point to Vercel.
HTTP traffic is proxied through CloudFlare.
The zone settings are correctly configured, for example, to always use HTTPS and compression.

The following is an example CF resource definition.

# cloudflare.tf
provider "cloudflare" {
  max_backoff = 8
  min_backoff = 2
  retries     = 3
}

variable "domain" {
  type     = string
  nullable = false
}

variable "host_dns_addr" {
  type     = string
  nullable = false
}

data "cloudflare_zones" "website" {
  filter {
    name = var.domain
  }
}

resource "cloudflare_record" "root" {
  zone_id = data.cloudflare_zones.website.zones[0].id
  name    = var.domain
  type    = "A"
  value   = var.host_prod_ip
  proxied = true
  ttl     = 1
}

resource "cloudflare_record" "www" {
  zone_id = data.cloudflare_zones.website.zones[0].id
  name    = "www"
  type    = "CNAME"
  value   = var.host_dns_addr
  proxied = true
  ttl     = 1
}

resource "cloudflare_zone_settings_override" "website" {
  zone_id = data.cloudflare_zones.website.zones[0].id
  settings {
    always_use_https         = "on"
    automatic_https_rewrites = "on"
    brotli                   = "on"
    min_tls_version          = "1.2"
    minify {
      css  = "on"
      js   = "on"
      html = "on"
    }
    ssl                      = "full"
    tls_1_3                  = "on"
  }
}

For the cloudflare_zone_settings_override resource type, Terraform will use CloudFlare’s default settings for any rules not specified in the resource declaration.

Configuring the Checkly Provider

The Checkly Terraform provider allows health checks to be defined for API endpoints or end to end functionality. The checkly-health.js script referenced below is a basic Playwright script Checkly provides that can be used as a simple health check. Alerts can also be managed via the Terraform provider, but do not yet support all alert channels that are available via their UI, such as Telegraph.

provider "checkly" {
  api_key    = var.checkly_api_key
  account_id = var.checkly_account_id
}

resource "checkly_check" "health" {
  name                      = "Health Checks"
  type                      = "BROWSER"
  activated                 = true
  group_id                  = checkly_check_group.app.id
  frequency                 = 30
  should_fail               = false
  use_global_alert_settings = true
  script = file("${path.module}/scripts/checkly-health.js")
}

resource "checkly_check_group" "app" {
  name         = "Check Group"
  runtime_id   = "2022.02"
  activated    = true
  concurrency  = 2
  muted        = false
  double_check = true
  tags = [
    "prod"
  ]
  locations = [
    "us-west-1", "us-east-1", "eu-west-1"
  ]
}

Configuring the Sentry.io Provider

The Sentry.io Terraform Provider is a community project officially sponsored by Sentry. The plan below creates an organization, team, javascript project, and a DSN key. Alerting rules can also be defined and managed. The public DSN is output for injection into the webapp’s build pipeline.

# Auth passed via SENTRY_AUTH_TOKEN env var
provider "sentry" { }
resource "sentry_organization" "org-name" {
  name        = "Your Org"
  slug        = "org-slug"
  agree_terms = true
}

resource "sentry_team" "team-name" {
  organization  = sentry_organization.org-name.id
  name          = "Team Name"
  slug          = "team-slug"
}

resource "sentry_project" "proj-name" {
  organization  = sentry_organization.org-name.id
  teams         = [sentry_team.team-name.id]
  name          = "Project Name"
  slug          = "proj-slug"
  # See Sentry list of supported platforms
  platform      = "javascript"
  resolve_age   = 480
}

resource "sentry_key" "dsn" {
  organization  = sentry_organization.org-name.id
  project       = sentry_project.proj-name.id
  name          = "DSN Key"
}

# Save the DSN for later use
output "sentry_dsn_key" {
  description = "Public DSN key"
  value       = sentry_key.dsn.dsn_public
  sensitive   = true
}

Configuring the GitLab Provider

GitLab also has a Terraform Provider, which allows for management of aspects of a project or group’s repos and the GitLab instance in general.

gitlab.tf ensures that:

Group-level GitLab environment variables are set with the output from the Sentry and Vercel providers. Project-level variables can be set instead

# gitlab.tf
# Source from $GITLAB_TOKEN
provider "gitlab" { }

variable "gitlab_group_id" {
  type    = number
  default = 123456 # Set to GitLab Group numeric ID
}

resource "gitlab_group_variable" "vercel_project" {
  group             = var.gitlab_group_id
  key               = "VERCEL_PROJECT_ID"
  value             = vercel_project.app.id
}

resource "gitlab_group_variable" "sentry_dsn" {
  group             = var.gitlab_group_id
  key               = "SENTRY_DSN"
  value             = sentry_key.dsn.dsn_public
}

An advantage of using GitLab’s TF provider to update the project’s environment variables is we don’t need to write Bash scripts in the CI/CD pipeline to parse and manipulate the output of terraform output, nor write them back to the GitLab API ourselves. Everything is handled within Terraform’s execution cycle.

Terraform GitLab CI File

The GitLab-provided CI template is a good starting point to build out a pipeline to apply Terraform to the infrastructure resources. When added to gitlab-ci.yml, the template provides a number of hidden jobs that map to Terraform’s CLI commands such as fmt, plan, and apply. The TF_STATE_NAME variable should match the name of the remote state object previously created in the project. After apply has executed, the Terraform output variables can also be written to group-level variables for use in other projects using GitLab’s API.

include:
    - template: Terraform/Base.gitlab-ci.yml
    - template: Jobs/SAST-IaC.gitlab-ci.yml

variables:
    TF_INIT_FLAGS: "-lockfile=readonly"
    TF_STATE_NAME: provision
    TF_CACHE_KEY: provision

stages:
    - validate
    - test
    - build
    - deploy

lint:
    stage: validate
    image: ubuntu:jammy
    before_script:
        - apt update && apt install -y curl unzip
        - curl -s https://raw.githubusercontent.com/terraform-linters/tflint/master/install_linux.sh | bash
    script:
        - tflint

# terraform fmt
fmt:
    extends: .terraform:fmt
    needs: []

# terraform validate
validate:
    extends: .terraform:validate
    needs: []

# terraform plan
build:
    extends: .terraform:build

# terraform apply - only runs on push to default branch
deploy:
    extends: .terraform:deploy
    dependencies:
        - build
    environment:
        name: $TF_STATE_NAME

The pipeline will execute the validate and plan stages for all branches. On the default branch, it will also apply against the external infrastructure resources.

Serrio-Mal infrastructure diagram — High level outline of project infrastructure.

Provisioning WebApp Infrastructure with Terraform, Vercel, and GitLab