Runtime Secrets with Sealed Secrets and External Secrets Operator

The Failure

The team stored Kubernetes Secrets as plaintext YAML in the infra repo. They added the files to .gitignore to prevent committing them. This created two problems: the secrets were not version-controlled (no audit trail for changes), and new team members could not deploy because they did not have the secret files. The team resorted to sharing secrets via Slack DMs.

GitOps requires everything in Git. Secrets are part of everything. The solution is not to exclude secrets from Git but to encrypt them before committing.

The Mechanism

Two Approaches

Feature	Sealed Secrets	External Secrets Operator
Secret source	Encrypted in Git	External store (Vault, AWS SM, GCP SM)
Rotation	Manual (re-seal + commit)	Automatic (sync interval)
GitOps compatible	Yes (encrypted YAML in repo)	Partially (ExternalSecret manifest in repo, actual secret synced)
Complexity	Low	Medium
Best for	Static secrets, API keys, TLS certs	Dynamic secrets, database passwords, rotated credentials

Decision Rule

• If the secret changes less than once a quarter → Sealed Secrets
• If the secret must be rotated automatically → External Secrets Operator
• If the secret is generated dynamically (e.g., database credentials) → External Secrets Operator

The Implementation

Sealed Secrets

# Install kubeseal CLI
brew install kubeseal

# Create a regular Kubernetes Secret
kubectl create secret generic checkout-secrets \
  --namespace=production \
  --from-literal=PAYMENT_API_KEY=sk_live_xxx \
  --from-literal=STRIPE_WEBHOOK_SECRET=whsec_xxx \
  --dry-run=client -o yaml > secret.yaml

# Seal it (encrypt with the cluster's public key)
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml

# The sealed secret is safe to commit to Git
rm secret.yaml  # Delete the plaintext version

# HARDENED: Sealed Secret — safe to store in Git
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: checkout-secrets
  namespace: production
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "false"
spec:
  encryptedData:
    PAYMENT_API_KEY: AgBx8z...encrypted...base64==
    STRIPE_WEBHOOK_SECRET: AgCy9w...encrypted...base64==
  template:
    metadata:
      name: checkout-secrets
      namespace: production
    type: Opaque

The Sealed Secrets controller in the cluster decrypts the SealedSecret and creates a regular Kubernetes Secret that pods can mount.

External Secrets Operator with Vault

# HARDENED: SecretStore pointing to Vault
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
  namespace: production
spec:
  provider:
    vault:
      server: https://vault.acme.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: checkout-service
          serviceAccountRef:
            name: checkout-service
---
# HARDENED: ExternalSecret synced from Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db-credentials
  namespace: production
spec:
  refreshInterval: 5m # Sync every 5 minutes
  secretStoreRef:
    name: vault-store
    kind: SecretStore
  target:
    name: checkout-db-credentials
    creationPolicy: Owner
    template:
      type: Opaque
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@db.production.acme.com:5432/checkout"
  data:
    - secretKey: username
      remoteRef:
        key: database/creds/checkout-readwrite
        property: username
    - secretKey: password
      remoteRef:
        key: database/creds/checkout-readwrite
        property: password

Pod Configuration

# HARDENED: Pod mounts both Sealed Secret and External Secret
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-service
  namespace: production
spec:
  template:
    spec:
      serviceAccountName: checkout-service
      containers:
        - name: checkout
          image: ghcr.io/acme/checkout-service:abc123
          envFrom:
            # Static secrets (API keys, webhook secrets)
            - secretRef:
                name: checkout-secrets
            # Dynamic secrets (database credentials, rotated automatically)
            - secretRef:
                name: checkout-db-credentials
          volumeMounts:
            - name: tls-cert
              mountPath: /etc/tls
              readOnly: true
      volumes:
        - name: tls-cert
          secret:
            secretName: checkout-tls

Secret Rotation Workflow

# Rotation script for Sealed Secrets
# Run when a secret value changes
#!/bin/bash
set -euo pipefail

SECRET_NAME=$1
NAMESPACE=$2
KEY=$3
NEW_VALUE=$4

# Create new secret with updated value
kubectl create secret generic "$SECRET_NAME" \
  --namespace="$NAMESPACE" \
  --from-literal="$KEY=$NEW_VALUE" \
  --dry-run=client -o yaml | \
  kubeseal --format=yaml --merge-into \
  "apps/checkout-service/overlays/$NAMESPACE/sealed-secrets.yaml"

echo "Sealed secret updated. Commit and push to trigger ArgoCD sync."

The Gate

ArgoCD syncs the SealedSecret or ExternalSecret manifests from Git. The Sealed Secrets controller or External Secrets Operator creates the actual Kubernetes Secret in the cluster.

If a SealedSecret fails to decrypt (wrong encryption key, corrupted data), the controller logs an error and the Secret is not created. The pod that depends on it will fail to start, and the deployment will not become healthy.

External Secrets Operator reports sync status via the ExternalSecret’s .status field. ArgoCD health checks monitor this status. If the sync fails, ArgoCD marks the application as degraded.

The Recovery

Sealed Secrets controller key is lost: All SealedSecrets become undecryptable. Re-seal all secrets with the new controller key. This is why you back up the Sealed Secrets controller key (stored as a Kubernetes Secret in the kube-system namespace).

External Secrets Operator cannot reach Vault: Existing secrets in the cluster remain valid until they expire. The Kubernetes Secret already exists and pods continue to use it. But new credentials will not be synced. Fix the Vault connectivity. If Vault is down for longer than the credential TTL, pods will fail when the database credential expires.

Secret value is wrong in production: For Sealed Secrets, update the value, re-seal, commit, push. ArgoCD syncs the new SealedSecret, the controller decrypts it, and the pod picks up the new value on restart. For External Secrets, update the value in Vault. The operator syncs within refreshInterval.