Event-driven Cloud Run: Hono publishes, Python processes, GCS stores

2026-09-22 clouddevopscybersecurity

▶ Run the companion lab

Send 101 to an HTTP endpoint. A Python service eventually writes 101 into a text file. Send 100, and it deliberately writes nothing. The business rule is small enough that we can concentrate on the boundaries: when a request is accepted, who can invoke the processor, and what happens if delivery repeats.

The companion lab implements a public Hono TypeScript service, a Pub/Sub topic and authenticated push subscription, a private Python processor, and a private Cloud Storage bucket. Terraform provisions the resources; Make orchestrates deployment, verification, and cleanup.

Verification status: the lab records passing application tests, local container checks, shell lifecycle tests, and Terraform mock tests. Real GCP deployment, authenticated push delivery, live IAM enforcement, and cloud teardown have not been run. The walkthrough below describes the implementation and the checks you can run in your own project; it is not a report of a successful cloud deployment.

From labs/lab-cloud-run-pubsub-number-processor/README.md:

flowchart LR
  Client -->|POST JSON number| Server[Public Node 22 server]
  Server -->|publish as server identity| Topic[Pub/Sub topic]
  Topic -->|OIDC push as push identity| Processor[Private Python 3.12 processor]
  Processor -->|create-only write as processor identity| Bucket[Private results bucket]

Acceptance is a receipt

The Hono service accepts POST / with a finite JSON number. It rejects malformed JSON, strings, booleans, and nonfinite numeric values with 400, before calling the publisher. The publisher serializes the number into JSON bytes and awaits Pub/Sub’s publication result.

The HTTP response follows that result:

From labs/lab-cloud-run-pubsub-number-processor/server/src/http/app.ts:

    try {
      const messageId = await publish(number);
      return context.json({ messageId }, 202);
    } catch {
      return context.json({ error: "Message publication failed" }, 503);
    }

A 202 response carries a messageId. It means Pub/Sub accepted the message. Processing can still be pending, can fail and retry, or can finish by skipping the number. The client needs that message ID to correlate a later completion log.

This is the first useful property of the design: the public request does not wait for storage, while still refusing to claim acceptance before publication succeeds.

The subscription delivers the work

The server never calls Python. Terraform connects the topic to the processor through a push subscription:

From labs/lab-cloud-run-pubsub-number-processor/terraform/services/06_subscription.tf:

resource "google_pubsub_subscription" "processor" {
  name                       = "${var.lab_name}-push"
  topic                      = local.topic
  ack_deadline_seconds       = 60
  message_retention_duration = "86400s"
  expiration_policy { ttl = "" }
  retry_policy {
    minimum_backoff = "10s"
    maximum_backoff = "60s"
  }
  push_config {
    push_endpoint = google_cloud_run_v2_service.processor.uri
    oidc_token {
      service_account_email = local.push_sa
      audience              = google_cloud_run_v2_service.processor.uri
    }
  }
  depends_on = [google_cloud_run_v2_service_iam_member.processor_invoker]
}

The push endpoint and OIDC audience both use the processor URI. Pub/Sub sends a wrapped message whose message.data contains base64-encoded JSON. Base64 supplies a transport representation, not authentication or encryption.

The Python route decodes that envelope, validates the message ID and finite number, then applies the threshold. Cloud Run handles the caller’s authentication before forwarding the request to the container; Python does not poll a subscription or call a separate acknowledgment API.

The route deliberately acknowledges malformed deliveries with 204 after logging the reason. That policy prevents a poison message from consuming retries indefinitely, at the cost of discarding it. A number at or below 100 also produces 204, with a correlated processed log whose outcome is skipped.

For a qualifying value, the route selects a new UUID name and attempts storage:

From labs/lab-cloud-run-pubsub-number-processor/processor/src/app/http.py:

        object_name = f"{uuid_factory()}.txt"
        try:
            store.write(object_name=object_name, text=number_text(delivery.number))
        except StorageError:
            log.exception(
                "Processing delivery failed.",
                message_id=delivery.message_id,
                number=delivery.number,
                object_name=object_name,
            )
            return "", 503

After a successful write, the route logs outcome="stored", including the message ID and object name, and returns 204. A storage error returns 503 so delivery can be retried. The storage adapter uses a create-only precondition:

From labs/lab-cloud-run-pubsub-number-processor/processor/src/app/storage.py:

            blob = self._bucket.blob(object_name)
            blob.upload_from_string(
                text,
                content_type="text/plain",
                if_generation_match=0,
            )

For 101, the object contains the plain text 101. The generation precondition protects an existing object from overwrite, but it does not deduplicate messages. A write can succeed and its acknowledgment can be lost. On redelivery, the processor generates another UUID and can create a second object for the same message.

The configured subscription retains messages for 24 hours, sets a 60-second acknowledgment deadline, and configures retry backoff from 10 to 60 seconds. These settings support retries; they do not establish exactly-once processing.

Four kinds of identity

The caller arriving at a service and the account running its container have different jobs. The lab makes those jobs explicit:

The processor’s bucket role is roles/storage.objectCreator. Its intended direct grant supports creating results without reading, listing, deleting, or overwriting them. It needs no Pub/Sub subscriber role because the work arrives over HTTP. The server has topic-scoped roles/pubsub.publisher and no bucket grant.

The push account has roles/run.invoker on the processor. Pub/Sub’s service agent obtains a token representing that account; the Python container then runs as the separate processor runtime account. Attaching the push account to the subscription also requires deployment permission to act as that account. None of these paths needs a downloaded service-account key.

The optional LEGACY_TOKEN_CREATOR=true bootstrap setting is for a project where the Pub/Sub service agent lacks the normal token-minting permission. It adds Token Creator on the push account, rather than giving runtime identities a broad project grant. The README documents when to set it and how saved configuration constrains subsequent runs.

All these grants are additive. A narrow bucket binding does not cancel access inherited from a project, folder, or organization. The verifier therefore examines readable ancestor policies as well as testing specific operations.

Private means ingress and identity

The processor configuration sets internal-only ingress:

From labs/lab-cloud-run-pubsub-number-processor/terraform/services/05_processor.tf:

resource "google_cloud_run_v2_service" "processor" {
  name                = "${var.lab_name}-processor"
  location            = var.region
  deletion_protection = false
  ingress             = "INGRESS_TRAFFIC_INTERNAL_ONLY"

This excerpt is the opening of the resource; its remaining configuration sets the runtime identity, image, environment, and instance limits. The invoker binding is a separate resource:

From labs/lab-cloud-run-pubsub-number-processor/terraform/services/05_processor.tf:

resource "google_cloud_run_v2_service_iam_member" "processor_invoker" {
  project  = var.project_id
  location = var.region
  name     = google_cloud_run_v2_service.processor.name
  role     = "roles/run.invoker"
  member   = "serviceAccount:${local.push_sa}"
}

The delivery design uses a same-project Pub/Sub subscription and the default run.app URL, an internal path described in the lab README. It needs no VPC connector, private IP, custom domain, or Eventarc trigger. Cloud Run applies both ingress restrictions and authentication before the Python route runs.

“Only Pub/Sub can invoke” would be too broad a claim. Internal ingress admits eligible internal sources, and IAM authorizes an identity rather than one particular subscription. Someone who can impersonate the push account and reach an allowed internal path could also invoke the processor. A subscription field in JSON supplies no proof of origin. Keep the push account dedicated and account for inherited access.

The verifier is intended to run from a laptop or another host outside an allowed internal path. It expects 404 for external requests both without credentials and with a valid push-account token. That demonstrates the ingress boundary; it is not evidence of an IAM denial. Successful authenticated delivery is checked through the public server, correlated completion logs, and the resulting object.

Follow the seven-step reader flow

The lab README contains the complete setup and commands. Start with make check-local if you want to inspect and test the implementation without Google Cloud credentials. It still needs package/provider registry access, Docker with Buildx, Bash 5 or newer, Make, jq, ShellCheck, uv, and Terraform 1.14.8.

1. Prepare the project and identities. Use your own billing-enabled project and a supported region; the default is europe-west1. Authenticate both the gcloud CLI and Application Default Credentials as the intended principal: gcloud and Terraform use different credential paths. The README separates deployer privileges from reader/verifier privileges and lists the required policy-read and impersonation access.

The public server requires an organization policy that permits its allUsers invoker binding. Both services scale from zero to at most two instances, but public requests can still cause work and cost. Cloud Run, Pub/Sub, registry storage, object storage, and logging are cost-bearing resources.

2. Bootstrap foundation, images, then services. After setting the project, region, and lab name as documented, run:

From labs/lab-cloud-run-pubsub-number-processor/README.md:

make bootstrap PROJECT_ID="$PROJECT_ID" REGION="$REGION" LAB_NAME="$LAB_NAME"

Two Terraform roots solve the initial image dependency. terraform/foundation creates APIs, accounts, registries, messaging, storage, and scoped runtime bindings. The build step pushes both linux/amd64 images and resolves their immutable digests. terraform/services then deploys Cloud Run and the subscription using those digest references.

Each root collects its outputs in 99_outputs.tf. Ignored .lab/config.json and .lab/images.json preserve deployment inputs and image digests. Keep those files and the local Terraform states for recovery and cleanup; subsequent commands reject conflicting inputs rather than silently targeting another project.

3. Publish a number and retain the receipt. The README gets the actual URL from Terraform and checks for 202:

From labs/lab-cloud-run-pubsub-number-processor/README.md:

SERVER_URL="$(terraform -chdir=terraform/services output -raw server_url)"
response_file="$(mktemp)"
http_code="$(curl --silent --show-error --output "$response_file" \
  --write-out '%{http_code}' --request POST \
  --header 'Content-Type: application/json' \
  --data '{"number":101}' "$SERVER_URL/")"
cat "$response_file"
printf '\nHTTP %s\n' "$http_code"
[[ "$http_code" == 202 ]]
MESSAGE_ID="$(jq -er '.messageId | select(test("^[0-9]+$"))' "$response_file")"
rm -f "$response_file"

4. Correlate completion before reading the object. With the reader identity, query processor logs for event="processed", the receipt’s message_id, and outcome="stored". Use the resulting object_name to read the file from the bucket. Delivery and log ingestion are asynchronous, so an initially empty log query is not proof of failure. The README supplies the exact log and storage commands.

5. Inspect the threshold and retries. Publish 100 and a value below it. A skipped completion is positive evidence that the processor handled the message; simply waiting and finding no object is weaker evidence. The verifier checks 100 and 99, then observes each message for an unexpected stored event during a bounded window. That observation cannot establish that an object will never appear. Storage failures and repeated delivery must be interpreted alongside the UUID-per-delivery behavior.

6. Run the cloud and IAM verifier. An authorized administrator temporarily grants the verifier Token Creator on exactly the server, processor, and push accounts. After those documented setup steps, run make e2e from outside the internal ingress path. Remove the three temporary grants afterward, including after failures.

The checks pair allowed and denied operations: the server publishes but cannot perform the tested Storage operations; the processor creates an object but cannot publish, list, read, delete, or overwrite. A unique qualifying number must reach both a correlated stored log and an exact object. Invalid public payloads must return 400; local injected-publisher tests establish the stronger guarantee that validation does not publish.

Default completion polling is 180 seconds, with a 3-second interval and a 30-second observation window after each skip. Unreadable or unresolved policy evidence causes failure, rather than a least-privilege claim. These checks cover defined operations and readable allow policies, not every permission, group membership, deny policy, or future policy change.

7. Tear down while keeping recovery state. Preview the destroy, then remove the deployment:

From labs/lab-cloud-run-pubsub-number-processor/README.md:

./scripts/deploy_cloud.sh --dry-run --destroy
make teardown

Teardown destroys services before foundation resources and stops on failure. It removes resources tracked by the two local states, including the bucket contents and registry images, while retaining the existing project, unrelated resources, and enabled APIs. A successful cleanup removes saved lab configuration and image metadata but retains Terraform state and initialization directories. On failure, preserve the saved files, fix the cause, and rerun teardown.

This pattern is useful when HTTP clients need an acceptance receipt while a separate service owns background processing. The lab keeps that work intentionally small so the delivery, permission, and acknowledgment boundaries stay visible. Before adapting it to a workload where duplicate effects are unacceptable, design durable deduplication around an appropriate event identifier: random filenames alone cannot provide it.