A free k8s cluster on Oracle Cloud via Terraform and Ansible

Building on the theme of recent posts, I want to use Terraform and Ansible to automate the provisioning and setup of a free k8s cluster on Oracle Cloud. But, it did not work out. The guide I followed was too far out of date for the parts to work together. I have left the below (which was as far I as I written) for a guide to get set up, and the code is here.

Open Table of contents

Background
Step 1 - Create an account
Step 2 - Creating some Instances
Resources Used

Background

While I was looking for resources to learn k8s the biggest blocker I always seemed to face was where can you run the cluster in a production-like manner, to really get an understanding of it, without breaking the bank. Usually I get a couple of hours a week if I am lucky to play with code so the ‘X free days’ trials are usually over before I have finished getting setup. In my research I came across a blog post by Loi Cheng on how to Setup a Free Kubernetes Cluster on Oracle Cloud. Perfect, it will be there for when I have time and not costing me a penny.

The blog author notes in the introduction that “This free route definitely requires more manual work to get it up and running, compared to the pre-packaged cloud provider kubernetes. However, we do gain much more knowledge and understanding of kubernetes through the manual setup process.” I want to take this a step further, or back depending on your view point, by writing the terraform and ansible to get this all set up. More learning for me, less hassle for the next person (or future me).

Step 1 - Create an account

Nothing to automate here, just do it.

Step 2 - Creating some Instances

Warm up over, lets dive right in!

I wish there was a way to scaffold the terraform project, luckily I have the repo that manages the domain etc for the blog to act as a donor.

Repo setup

This is the initial directory structure that I am going to use for the terraform:

.
├── README.md
├── backend.tf
├── cloudflare.auto.tfvars
├── cloudflare_resources.tf
├── oracle.auto.tfvars
├── oracle_resources.tf
├── providers.tf
└── variables.tf

I think it will need to be moved to its own directory in due course.

I am going to use GitLab as the backend and the CI once I have it setup to then run any changes we might need down the road.

Cloudflare is going to be the DNS for the domain I am going to use.

GitLab as a Backend

First things first, define the shell variables we are going to need for the terraform init command:

export PROJECT_ID="<PROJECT_ID>" # Found in Settings -> General
export TF_USERNAME="<USERNAME>"
export TF_PASSWORD="<ACCESS_TOKEN>" # a personal access token
export TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/tf_state"

Paste the block into your terminal ready for later, I would keep a copy of the block safe locally in case you close the terminal and need to redo.

In our backend.tf file we need to let terraform know we plan to use a http backend:

terraform {

  backend "http" {

  }
  required_version = ">= 1.1.2"
}

Provider Time

Then in our providers.tf configure the cloudflare and oracle providers. We need to add the name and version to the terraform block.

terraform {
  required_providers {
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.0"
    }
    oci = {
      source  = "oracle/oci"
      version = "5.23.0"
    }
  }
}

In the same file we need to add the provider blocks, one for each. With Cloudflare we are going to use an api token that will need setting up for your profile. It will need the DNS permissions (there was a quick link to it as you start to configure the key). For Oracle Cloud it is a little more involved. I am going to use the API Key auth type again and pass the key in rather than the filepath as I am not sure how well a file would translate to GitLab CI.

provider "cloudflare" {
  api_token = var.cloudflare_api_token
}

provider "oci" {
  tenancy_ocid = var.oracle_tenancy_ocid
  user_ocid    = var.oracle_user_ocid
  fingerprint  = var.oracle_fingerprint
  private_key  = var.oracle_private_key
  region       = var.oracle_region
}

If you are using VSCode and have the terraform plugin like me, your IDE is probably complaining that the variables are not defined.

Defining Variables

In the file called variables.tf, define the variable blocks for configuring the providers. I have them all set as strings.

variable "cloudflare_api_token" {
  type = string
}

variable "oracle_tenancy_ocid" {
  type = string
}

variable "oracle_user_ocid" {
  type = string
}

variable "oracle_fingerprint" {
  type = string
}

variable "oracle_private_key" {
  type = string
}

variable "oracle_region" {
  type = string
}

Danger Zone: Proceed with Caution

In the next section we will be setting our variables, some of which are sensitive. Before setting any variables, add a .gitignore file with the following to prevent leaking the secrets in the git repo. I got this from a GitHub repo of git ignore templates.

# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version
# control as they are data points which are potentially sensitive and subject
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc
.secrets

*.auto.tfvars

oracle.auto.tfvars

Now we are ready to populate the *.auto.tfvars files.

For the provider configuration values, I have split them logically into 2 files, one for each provider. When it comes to things like domain names it gets a little bury, it might be a good idea to have a common set too.

The tenancy and user id can be found from your dashboard and start with the string below, the fingerprint can be displayed using one of these commands, the private key can be found by opening the .pem file with your IDE and you can get the list of region codes from here.

oracle_tenancy_ocid = "ocid1.tenancy.oc1.XXX"
oracle_user_ocid    = "ocid1.user.oc1.XXX"
oracle_fingerprint  = "12:f8:7e:78:61:b4:bf:e2:de:24:15:96:4e:d4:72:53"
oracle_private_key  = "XXXXXXX"
oracle_region       = "uk-london-1"

cloudflare.auto.tfvars

The cloudflare api token will be generated on your profile and will need to have access to edit the DNS.

cloudflare_api_token = "XXXXX"

Init

Now we should have enough defined to init our terraform directory. You can do this by running the following command.

terraform init \
  -backend-config=address=${TF_ADDRESS} \
  -backend-config=lock_address=${TF_ADDRESS}/lock \
  -backend-config=unlock_address=${TF_ADDRESS}/lock \
  -backend-config=username=${TF_USERNAME} \
  -backend-config=password=${TF_PASSWORD} \
  -backend-config=lock_method=POST \
  -backend-config=unlock_method=DELETE \
  -backend-config=retry_wait_min=5

That will install the provider packages and set up terraform to use GitLab as the backend to store our state.

Resources Used

https://github.com/stealthybox/tf-oci-arm/blob/main/compute.tf https://faun.pub/free-ha-multi-architecture-kubernetes-cluster-from-oracle-c66b8ce7cc37 https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/core_instance#attributes-reference https://github.com/oracle/terraform-provider-oci/blob/master/examples/compute/instance/instance.tf#L55 https://github.com/oracle-terraform-modules/terraform-oci-compute-instance/blob/v2.4.1/variables.tf https://gist.github.com/roib20/27fde10af195cee1c1f8ac5f68be7e9b https://github.com/tjtharrison/kubernetes-deploy/tree/main

A free k8s cluster on Oracle Cloud via Terraform and Ansible - Part 1, The Failure

Table of contents