~/tech-with-ugur

Catch It Before Prod: A Layered OSS Scanning Gauntlet for Kubernetes Apps

2026-08-19 cybersecurity

Run the companion lab

In the companion exploit lab, a deliberately vulnerable Node.js app got taken apart on a live Kubernetes cluster: a SQL injection became a stolen SSN, a shelled-out ping became a root shell, a committed key got read straight off a debug endpoint, and one request OOM-killed the pod. Every one of those was caught the most expensive way there is — by attacking a running target in a cluster.

This lab flips the timeline. It takes the same app and catches each flaw at the earliest lifecycle stage it could have been caught: secrets and injection sinks in the source, a vulnerable dependency and a root container at build time, a missing security context in the manifest, and finally a Kubernetes admission gate that simply refuses to let the bad workload start. Then it runs the whole thing again against a hardened twin and watches every finding go green. One command:

./run.sh

The point isn’t any single scanner — it’s the layering. No one tool catches everything, and each stage is the earliest place its class of bug becomes visible. Every code block below is copied verbatim from the lab.

Two apps, side by side

The lab ships the vulnerable app under vulnerable/ and a fixed twin under hardened/, so a reader can diff them line for line. Both go through the same three static stages, then the admission gate, and the results get consolidated into one before/after table. Everything runs locally against throwaway targets and a disposable kind cluster; the safety story is the same canary key and fake data as the exploit lab.

The whole gauntlet is deterministic because it asserts on specific finding IDs, never on counts. That distinction matters more than it sounds: OS and base-image CVE counts drift every time a vulnerability database updates, so a test that keys on “12 findings” is broken by design. This lab keys on custom rule IDs, one permanent CVE, and policy-based misconfig IDs — the things that stay put across a pinned tool version.

labs/lab-k8s-appsec-scanning/expected/findings.json:

{
  "code": {
    "gitleaks": ["vuln-lab-hardcoded-api-key"],
    "semgrep": ["scanners.semgrep.vuln-lab-sqli", "scanners.semgrep.vuln-lab-command-injection"]
  },
  "build": {
    "trivy-sca": ["CVE-2021-23337"],
    "trivy-image": ["CVE-2021-23337"],
    "trivy-dockerfile": ["DS-0002"]
  },
  "deploy": {
    "trivy-k8s": ["KSV-0012", "KSV-0014", "KSV-0011"]
  },
  "admission": {
    "kyverno": ["require-run-as-non-root"]
  }
}

That file is the whole contract. ./run.sh e2e asserts, id by id, that every one of those shows up on the vulnerable app and is gone on the hardened one.

The code stage: catch it in the diff

The cheapest place to catch a bug is before an image is ever built — while it’s still just text in a pull request. Two tools cover this stage.

Gitleaks hunts for secrets. The exploit lab’s key is committed straight into source, so a one-rule Gitleaks config finds it:

labs/lab-k8s-appsec-scanning/scanners/gitleaks/.gitleaks.toml:

title = "scanning-lab gitleaks rules"

[[rules]]
id = "vuln-lab-hardcoded-api-key"
description = "Hardcoded internal API key committed to source"
regex = '''sk-vuln-lab-[A-Za-z0-9-]+'''
keywords = ["sk-vuln-lab-"]

That rule fires on exactly the line the exploit lab handed to any unauthenticated caller off /api/debug/config:

labs/lab-k8s-appsec-scanning/vulnerable/app/src/config.ts:

// DELIBERATELY VULNERABLE: a hardcoded credential committed to source control
// and baked into the container image. Real code must never do this — the whole
// point of the secret-leak exploit is that this value is trivially recoverable.
export const INTERNAL_API_KEY = "sk-vuln-lab-DO-NOT-USE-0000-canary"; // example-only fake key; real secrets must never be committed

Semgrep does SAST — it matches source patterns, not secrets. The lab ships two custom rules, one per injection sink:

labs/lab-k8s-appsec-scanning/scanners/semgrep/rules.yaml:

rules:
  - id: vuln-lab-sqli
    languages: [typescript]
    severity: ERROR
    message: >-
      SQL is built with string interpolation of user input. Use a parameterized
      query ($1 placeholders passed to the driver) instead.
    patterns:
      - pattern-regex: '(SELECT|INSERT|UPDATE|DELETE)[^`;]*\$\{'
  - id: vuln-lab-command-injection
    languages: [typescript]
    severity: ERROR
    message: >-
      A shell command is built with string interpolation of user input and run
      through a shell. Use execFile with an argument array and validate input.
    patterns:
      - pattern-either:
          - pattern-regex: 'exec(Async)?\(`[^`]*\$\{'
          - pattern-regex: '`(ping|sh|bash|curl|wget)\b[^`]*\$\{'

Those two rules flag the two sinks the exploit lab turned into a UNION payload and a root shell — the interpolated SQL in src/db/customers.ts and the shell string in src/system/lookup.ts. Both are visible in the diff, before anyone runs anything.

One small determinism detail worth knowing if you go to assert on Semgrep output: Semgrep namespaces custom rule IDs by the config file path, so the bare vuln-lab-sqli comes back as scanners.semgrep.vuln-lab-sqli. That’s why the expected/findings.json above lists the namespaced form — it’s what the tool actually emits, learned by reading real output rather than guessing.

The code stage runs fully offline against those two lab-authored rulesets:

labs/lab-k8s-appsec-scanning/scripts/scan-code.sh:

log "Code stage: Gitleaks (secrets) on ${variant}/app"
# gitleaks exits non-zero when it finds leaks (expected on the vulnerable app);
# --exit-code 0 makes it always exit 0 while still writing the JSON report.
gitleaks dir "${variant}/app" --no-banner --exit-code 0 \
  --config scanners/gitleaks/.gitleaks.toml \
  --report-format json --report-path "${out}/gitleaks.json"

log "Code stage: Semgrep (SAST) on ${variant}/app"
semgrep --config scanners/semgrep/rules.yaml --metrics=off --quiet --json \
  --output "${out}/semgrep.json" "${variant}/app/src"

The build stage: what the source doesn’t show

Some problems aren’t in your source at all — they’re in what you pull in and what you package. Trivy does three different jobs here, and the split between them is the whole lesson.

labs/lab-k8s-appsec-scanning/scripts/scan-build.sh:

log "Build stage: Trivy SCA (dependencies) on ${variant}/app"
trivy fs --scanners vuln --quiet --format json \
  --output "${out}/trivy-sca.json" "${variant}/app"

log "Build stage: Trivy Dockerfile misconfig on ${variant}/app"
trivy config --quiet --format json \
  --output "${out}/trivy-dockerfile.json" "${variant}/app/Dockerfile"

log "Build stage: docker build ${image}"
docker build -q -t "${image}" "${variant}/app" >/dev/null

log "Build stage: Trivy image scan on ${image}"
trivy image --scanners vuln --quiet --format json \
  --output "${out}/trivy-image.json" "${image}"

Trivy’s SCA scan reads package-lock.json and reports the deliberately outdated dependency — lodash@4.17.20, carrying CVE-2021-23337. That’s the one CVE the lab asserts on, because it’s permanent: 4.17.20 will always carry it, so the assertion never drifts.

Trivy’s image scan then runs against the container the script just built and reports the same CVE baked into the layers. That pairing is the point: a dependency scan only proves the vulnerable version is declared in a manifest. The image scan proves it actually shipped — that the package made it through npm ci and into the layers a cluster would pull. Declared and shipped are different claims, and only one of them can hurt you.

Trivy’s Dockerfile scan (trivy config) flags the container running as root. The vulnerable Dockerfile has no USER directive at all:

labs/lab-k8s-appsec-scanning/vulnerable/app/Dockerfile:

FROM node:22.23.2-alpine

WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci --omit=dev

COPY src ./src

EXPOSE 3000

CMD ["npm", "start"]

No USER, so the container runs as root — Trivy reports it as DS-0002. A detail that’ll bite you if you assert on Trivy output: at the pinned version (0.71.1), the misconfiguration ID in the JSON is the bare DS-0002, not the AVD-DS-0002 form you’ll see in the docs and the finding URL. Assert on what the tool actually emits, which is why expected/findings.json lists the bare IDs.

The deploy stage: a scan, then a gate

The manifest is the last static artifact before the cluster, and it carries its own class of problem — not bad code, but missing hardening. Trivy’s config scan reads the Kubernetes YAML and flags what’s absent:

labs/lab-k8s-appsec-scanning/vulnerable/k8s/deployment.yaml:

    spec:
      # DELIBERATELY VULNERABLE: no securityContext, no cpu limit, no liveness probe
      containers:
        - name: vuln-app
          image: vuln-app-scanning:v1

No runAsNonRoot (KSV-0012), no readOnlyRootFilesystem (KSV-0014), no CPU limit (KSV-0011) — three misconfig IDs, all reported before the manifest is ever applied. But a static scan is still just a report. Someone has to read it and act. The deploy stage’s second half closes that gap with enforcement that doesn’t depend on anyone reading anything.

Kyverno runs as a validating admission controller inside the cluster. Its policy requires the exact things the vulnerable manifest is missing:

labs/lab-k8s-appsec-scanning/policy/kyverno/require-hardened.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-hardened-workloads
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: require-run-as-non-root
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [appsec-scan]
      validate:
        message: "Pod spec.securityContext.runAsNonRoot must be set to true"
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true

With that policy enforcing, the cluster itself refuses the vulnerable Deployment — the require-run-as-non-root rule denies admission — and admits the hardened one. The lab tests this with a server-side dry-run, which fires the webhook without actually scheduling a pod:

labs/lab-k8s-appsec-scanning/scripts/admission.sh:

  log "Admission: vulnerable manifest (expect DENY)"
  for attempt in $(seq 1 6); do
    if kubectl --context "${KCTX}" apply --dry-run=server \
      -f vulnerable/k8s/deployment.yaml >"${out}/vulnerable.txt" 2>&1; then
      echo "UNEXPECTED: vulnerable manifest was admitted" >&2
      cat "${out}/vulnerable.txt"
      return 1
    fi

This is the one stage that catches nothing a static scan didn’t already flag — and that’s exactly why it matters. Trivy reported the missing security context; Kyverno stops the workload from starting anyway. The difference between a report and a gate is whether a bug can still reach production after someone ignores the report. Everything upstream produces findings a human has to triage. Kyverno is the one control that fails closed.

Getting that gate to run reliably in a fresh cluster turned out to be the fiddliest part of the whole lab. kubectl rollout status returns as soon as Kyverno’s Deployment is ready — but its admission webhook isn’t serving TLS yet at that moment, so a cold run would race and fail on the policy apply. The fix is to retry the policy apply until the webhook answers, and to retry the first dry-run past any transient webhook error, which is why the loop above exists.

The fixes: what makes each finding disappear

The hardened twin isn’t a different app — it’s the same app with the smallest fix that clears each finding. Diffing the two is the most useful thing in the lab, because it shows the secure pattern right next to the insecure one.

The SQL injection becomes a bound parameter:

labs/lab-k8s-appsec-scanning/hardened/app/src/db/customers.ts:

// Parameterized: user input travels as a bound value ($1), never as SQL text.
export function buildSearchSql(q: string): { text: string; values: string[] } {
  return {
    text: "SELECT full_name, email FROM customers WHERE full_name ILIKE $1 ORDER BY id",
    values: [`%${q}%`],
  };
}

The shelled-out command becomes execFile with an argument array and an input allowlist — no shell means no metacharacter can be interpreted:

labs/lab-k8s-appsec-scanning/hardened/app/src/system/lookup.ts:

// Allowlist: hostnames and IPv4/IPv6 literals only — no shell metacharacters.
const HOST_RE = /^[A-Za-z0-9](?:[A-Za-z0-9._:-]{0,252})$/;

export function isValidHost(host: string): boolean {
  return HOST_RE.test(host);
}

export async function runLookup(host: string): Promise<string> {
  if (!isValidHost(host)) {
    return "invalid host";
  }
  try {
    // execFile with an argument array: `host` is a single argv element passed
    // to ping directly — never parsed by a shell.
    const { stdout, stderr } = await execFileAsync("ping", [
      "-c",
      "1",
      "-W",
      "1",
      host,
    ]);
    return `${stdout}${stderr}`;
  } catch (err) {
    const e = err as { stdout?: string; stderr?: string };
    return `${e.stdout ?? ""}${e.stderr ?? ""}`;
  }
}

The hardcoded key becomes a runtime read from the environment, and the debug endpoint that dumped it is deleted entirely:

labs/lab-k8s-appsec-scanning/hardened/app/src/config.ts:

export function loadInternalApiKey(
  env: NodeJS.ProcessEnv = process.env,
): string {
  return env.INTERNAL_API_KEY ?? "";
}

The outdated dependency is a one-character bump — 4.17.20 to 4.17.21 — which clears CVE-2021-23337 from both the SCA and the image scan. The root container gets a multi-stage build and a USER node:

labs/lab-k8s-appsec-scanning/hardened/app/Dockerfile:

FROM node:22.23.2-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev

FROM node:22.23.2-alpine AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY package.json ./
COPY src ./src
USER node
EXPOSE 3000
HEALTHCHECK CMD wget -qO- http://127.0.0.1:3000/health || exit 1
CMD ["npm", "start"]

And the manifest gets the full security context the policy demands — pod-level runAsNonRoot, container-level dropped capabilities and a read-only root filesystem, CPU and memory limits, and both probes:

labs/lab-k8s-appsec-scanning/hardened/k8s/deployment.yaml:

      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: hardened-app
          image: hardened-app-scanning:v1
          imagePullPolicy: IfNotPresent
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

Run the gauntlet against this twin and every asserted ID is gone, and Kyverno admits the workload.

Why layers win

Six controls across three stages, and the reason it’s six and not one is that each catches something none of the others would:

That last distinction is the one to hold onto. Everything before Kyverno produces a finding — a claim a human has to read, believe, and act on. Kyverno is a gate: it fails closed whether or not anyone read the report. A mature pipeline wants both, because reports catch things early and cheaply while a gate is the backstop for everything the reports missed or nobody triaged.

And the mapping of tool to stage isn’t arbitrary — each one lives at the earliest point in the lifecycle where its class of bug becomes visible. The secret and the sinks are visible in the diff, so catch them in the diff. The vulnerable dependency isn’t visible until you resolve the lockfile, and whether it shipped isn’t visible until you build the image. The missing security context isn’t visible until there’s a manifest. Push each check as far left as it’ll go, layer them so nothing slips between the seams, and put a gate at the end that doesn’t trust anyone to have read the earlier ones.

The exploit lab proved all four bugs are real by turning each into stolen data, a shell, a leaked key, and a dead pod. This lab proves every one of them was catchable for free, at rest, before any of it ran — with tools you can brew install this afternoon.

The full lab — both app variants, all six checks, the Kyverno admission stage on a throwaway kind cluster, and the automated end-to-end proof — is in the lab README. Clone it, run ./run.sh, and watch the before/after table go all green.