How to use Terraform to Add/Update the Google Search Ownership

I wanted to use Terraform to set the verification token on the DNS and as an environment variable on the pages deployment.

Table of contents

The Problem

To add a site to Google to manage the search you need to verify the domain and then add the code to the Astro-Pages site as an environment variable on the GitLab Pages instance.

1. How to add the site to Google
2. How to get the verification code for the domain from Google
3. How to create the record on the DNS (in my case CloudFlare).
4. How to get the verification code that gets added to the site as a tag
5. How to set that code using the environment variable when the app is started on GitLab Pages.

Reading the documentation on Google, it looks like we might only need steps 1-3 and as it is at the domain level, we will not need to add anything to the site. This makes it a lot simpler as I am already managing the DNS records with Terraform and I am not sure how env vars get added to Pages.

Step 1 - Add the domain to Google

A search brought up this other blog post describing part of the process, but it was missing some of the other configuration for Google itself. It did point at the Terraform provider that can do the adding to Google so was a good starting point.

Since the blog post there is a newer version, 0.4.5, so we will start by adding that to the providers.tf file.

terraform {
  required_providers {

    # Other providers omitted

    # Google is going to be used in the code that the blog provided so we need to add it explicitly.
    google = {
      source  = "hashicorp/google"
      version = "5.10.0"
    }

    googlesiteverification = {
      source  = "hectorj/googlesiteverification"
      version = "0.4.5"
    }
  }
}

I created a new file to add the Google stuff to called google_search_resources.tf. Following from the blog post, I copied the code block over and got to editing it to my needs.

Firstly, in my version of Terraform you don’t need the = in the variable block and I don’t need the variables anyway, so remove those lines.

The blog has the idea of adding a service account to be able to manage this for us, which I like the idea of, so that gets kept.

# Create new service account
resource "google_service_account" "siteverifier" {
  account_id   = "google-site-verifier"
  display_name = "Google Site verification account"
}

That will need a key to authenticate with Google, so that needs creating next.

# Generate service account key
resource "google_service_account_key" "siteverifier" {
  service_account_id = google_service_account.siteverifier.name
}

Now we have the component parts, we can use them to authenticate the provider, so we need the provider block.

# Initialise provider with service account key
provider "googlesiteverification" {
  credentials = base64decode(google_service_account_key.siteverifier.private_key)
}

Now we need to switch the API on to allow us to send it the domain we want to authenticate.

# Enable site verification api
resource "google_project_service" "siteverification" {
  service = "siteverification.googleapis.com"
}

Now it feels like we are getting somewhere, right? I have a variable block in my variables.tf file for the sites domain name so I can reuse that to pass it through to the Google verification API. This has a depends_on meta tag for the enabling of the API.

variable "domain_name" {
  type = string
}

# Request for DNS token from site verification API
data "googlesiteverification_dns_token" "run_domain" {
  domain = var.domain_name

  depends_on = [google_project_service.siteverification]
}

From here we can go to the cloudflare_resources.tf file to add our record to CloudFlare with the verification code. We are depending on the data block above as we are going to pull the variables from the returned values. I think this will always be a txt record, but this gives us the option to use the returned type and not run into any mismatch issues.

resource "cloudflare_record" "google_search_verification_txt" {
  name    = data.googlesiteverification_dns_token.run_domain.record_name
  proxied = false
  ttl     = 60
  type    = data.googlesiteverification_dns_token.run_domain.record_type
  value   = data.googlesiteverification_dns_token.run_domain.record_value
  zone_id = var.cloudflare_zone_id

  depends_on = [data.googlesiteverification_dns_token.run_domain]
}

Finally, we are able to ask Google to verify that we have the right record on our DNS to grant us ownership.

# Request google to verify the newly added verification record
resource "googlesiteverification_dns" "run_domain" {
  domain = var.domain_name
  token  = data.googlesiteverification_dns_token.run_domain.record_value

  depends_on = [cloudflare_record.google_search_verification_txt]
}

Step 1.5 - Adding the missing provider

From here I ran the terraform plan command and was greeted with a wall of red, the above code relies on the Google terraform provider (remember that from earlier?) so I added that in as you saw.

Now, how to authenticate with that new provider? I am wanting to run this on the GitLab CI so I don’t really want to be logging into the gcloud cli each time. Some further reading led me back to service accounts.

This one needs to be created manually (or I guess you could authenticate by the cli and add it in as a block to this or another terraform directory). I headed over to the Google Cloud Console and signed in (you might need to sign up, there is a free tier) and created a new project to house these accounts.

SIDE NOTE: I should have made a ‘global’ type project to house the terraform service account but that can be something for another time.

Once you have created the project, navigate to the service account menu and create a new one. I called mine ‘Terraform’ and gave it ‘Editor’ permissions from the list of options shown.

For the service account, generate the key file in JSON format and keep it somewhere safe, this has the ability to access your Google Cloud. I created a directory in my project called .secrets and the added it to the .gitignore file.

We are now ready to configure the Google provider. It will need the project key of the one that you created to hold the service accounts and the credentials taking a file to the key file.

provider "google" {
  project     = "smale-codes"
  credentials = file(".secrets/smale-codes-8d38205a8c4b.json")
}

Running terraform plan now gets us a little further but not the whole way. We are depending on the Google provider and its work to give us the key for the service account key to configure the googlesiteverification provider, and that is apparently not a thing. You cannot add a depends_on in the provider.

Back to Google and I found this StackOverflow Post that uses the -target flag to run part of the change and then we can run the rest after we have the data.

terraform plan -target="google_service_account_key.siteverifier" gives us something that looks like it will provide us with the keys to the kingdom, so lets apply it! terraform apply -target="google_service_account_key.siteverifier"

More Red

We are once again seeing red! This time we are seeing it as we do not have the IAM API enabled. This is something we can do via terraform, so I added the block below. I have also added a depends_on in the service account creation for good measure.

# Enable IAM api
resource "google_project_service" "iam_api" {
  service = "iam.googleapis.com"
}

# Create new service account
resource "google_service_account" "siteverifier" {
  account_id   = "google-site-verifier"
  display_name = "Google Site verification account"

+  depends_on = [google_project_service.iam_api]
}

And more red! This time we are missing the cloudresourcemanager API as it is not enabled, this seems fundamental for us to do anything via teraform so I opted to do it manually. I think there was a link in the error message, it was very late, I was annoyed at it not working and my notes got sloppy.

After getting that enabled I ran the targetted plan again (terraform apply -target="google_service_account_key.siteverifier") and…

Green, but that is only 1/2 of the config, time to run the apply without the stabelisers. Time to run terraform apply.

Green again! Or is it?

So at this point this post is reading like a story and not a guide. There was a good few hours after this green trying to work out why I was not able to see the site in my Google Search console. Some lessons learned here:

1. You cannot destroy the resources of googlesiteverification, it just hung.
2. You cannot remove the resources and run an apply to try and get rid of them.
3. If you authenticate with a service account, the service account owns the verification, not you…

This left me with two choices, sack it in and do it manually, or take the repo and add the ability to update the ownership with your email address.

Down the Terraform Provider Reabbit Hole we go

So our story and adventures continue. We are going to see if we can use the existing code to cobble together a new resource to hit the Google update API

Forking Hell

And we are off, the repo is forked, cloned locally and opened in VSCode.

Go, pardon?

I think I must be mad. I am using terraform (new to me) to provision resources for my Astro-based blog (new to me) in Google Search Console (new to me). I am now creating a provider (new to me) writen in Go (new to me). Notice a pattern…

What is one more extension

VSCode has kindly told me that there is a Go extension, so I have isntalled that too. Then that was missing something that I could click install for, thats now done too! 10 points to the author of the installed for the success output message.

All tools successfully installed. You are ready to Go. :)

Nope, that was a disaster

This is now going to be the abbriged version. I got it to run, I had it create the things once, but you could never run an apply again…

I opened a WIP merge request on the original incase there is anyone in the future that wants to get involved.

Closing Notes

In the end I manually generated the code and added it to my terraform to be added to my DNS.