Receive GKE Update Notifications From Google Cloud Project

Today, our task will be to set up Google Kubernetes Engine update notifications in GCP. It’s important to understand that one of the GKE features is mandatory updates. All we can do with it is to plan them in advance and do them when we are fully prepared. And even if the cloud decided to update cluster for us, we whant to see a notification about what happened and then.

diagram

Documentation

There are three articles:

Cluster notifications - general description of events and messages
Receive cluster notifications - step by step guide how to tune a cluster in cloud console or by gcloud
Configure cluster notifications for third-party services - subscribe to notifications in slack with javascript code sample

In general it looks pretty simple - create a pubsub topic and subscription, run some code to pull events from it and send notifications

But there are a couple of things to keep in mind:

we want to stick to IaaC and describe the configuration using terraform
it’s unclear to me why we need to run the code in cloudrun when we already have an up and running kubernetes cluster

Terraform

So, to make the cluster send update events, we must create a topic and refer to it in our cluster.

resource "google_pubsub_topic" "gke_updates" {
  name    = "gke-updates"
  project = var.project_id
}

resource "google_container_cluster" "gke" {
  // ...
  // ... cluster configuration ...
  // ...
  notification_config {
    pubsub {
      enabled = true
      topic   = google_pubsub_topic.updates.id
    }
  }
}

Our next steps depends on how and where we whant to receive notifications.. I personally did not find any suitable project on github for this purpose, so I wrote the code myself - gcp-notifications.

Example of running this tool in gke via terraform:

# create subscription
resource "google_pubsub_subscription" "gke_updates" {
  project = var.project_id
  topic   = "gke-updates"
  name    = "gcp-notifications"
}

# create service account and allow recieve notifications from pubsub
resource "google_service_account" "gcp_notifications" {
  account_id   = "gcp-notifications"
  display_name = "Google Cloud Project monitoring utility"
}
resource "google_project_iam_member" "gcp_notifications_pubsub_subscriber" {
  project = var.project_id
  role    = "roles/pubsub.subscriber"
  member  = "serviceAccount:${google_service_account.gcp_notifications.email}"
}
resource "google_service_account_key" "gcp_notifications" {
  service_account_id = google_service_account.gcp_notifications.name
}

# create kubernetes secret with slack webhook and service account key
resource "kubernetes_secret" "default_gcp-notifications" {
  metadata {
    name      = "gcp-notifications"
    namespace = "default"
  }

  data = {
    SLACK_WEBHOOK_URL = var.gcp-notifications_slack-webhook
    "gauth.json"      = base64decode(google_service_account_key.gcp_notifications.private_key)
  }

  type = "Opaque"
}
# gcp-notification deployment
resource "kubernetes_deployment" "default_gcp-notifications" {
  metadata {
    name      = "gcp-notifications"
    namespace = "default"
  }

  spec {
    replicas = 1
    selector {
      match_labels {
        name = "gcp-notifications"
      }
    }
    template {
      metadata {
        labels {
          name = "gcp-notifications"
        }
      }
      spec {
        container = [{
          image = "ghcr.io/mnacharov/gcp-notifications:latest"
          name  = "app"

          resources {
            limits {
              cpu    = "10m"
              memory = "32Mi"
            }
            requests {
              memory = "16Mi"
            }
          }
          env = [{
            name  = "GCP_PROJECT_ID"
            value = var.project_id
          }, {
            name  = "GCP_SUBSCRIPTION_ID"
            value = "gcp-notifications"
          }, {
            name  = "GOOGLE_APPLICATION_CREDENTIALS"
            value = "/mnt/gauth.json"
          }]
          envFrom = [{
            secretRef {
              name = "gcp-notifications"
            }
          }]
          volumeMounts = [{
            name      = "slack-gcp"
            mountPath = "/mnt/gauth.json"
            subPath   = "gauth.json"
          }]
          securityContext {
            readOnlyRootFilesystem = true
            runAsNonRoot           = true
            runAsUser              = 1000
          }
        }]
      }
      volumes = [{
        name = "slack-gcp"
        secret {
          secretName = "gcp-notifications"
        }
      }]
    }
  }
}

You can check the tool by scheduling a cluster or node-pool upgrade, or by manually posting to the topic with an example from the documentation.

slack