Migrating Millions in Healthcare Revenue: A Zero-Downtime ECS to EKS Strategy

Zero-Downtime ECS EKS Migration: Orchestrating a 6-Team Production Cutover at Scale

Healthcare revenue cycle services handling millions in transactions migrated from AWS ECS to EKS without dropping a single request. The transition reduced P99 latency by 28% and cut autoscaling response times from 185 seconds to just 22 seconds.

Why This Matters

In high-stakes environments like healthcare finance, technical limitations in ECS—such as 3-5 minute autoscaling lags and resource bin-packing inefficiencies—pose direct risks to financial stability and patient care. While ideal models suggest seamless scaling, the reality of month-end traffic spikes requires event-driven autoscaling via KEDA and granular pod-level security through IRSA to maintain performance under pressure.

Key Insights

• ECS service autoscaling relied on CloudWatch metrics with a 3-5 minute delay, causing 85%+ CPU spikes and 45-second P99 latencies during peak windows.
• KEDA (Kubernetes Event-driven Autoscaling) enabled pod-level scaling based on SQS queue depth, reducing scale-out trigger times from 185 seconds to 15 seconds.
• IAM Roles for Service Accounts (IRSA) replaced instance-wide permissions, providing pods with precise OIDC-based authentication to S3 and RDS.
• ExternalSecrets Operator synced with HashiCorp Vault to automate secret rotation every 30 days, eliminating manual task restarts.
• Target group-level blue-green deployment at the Application Load Balancer (ALB) allowed for 15-second traffic shifts and instantaneous rollbacks.

Working Examples

ServiceAccount with IAM role annotation for IRSA

apiVersion: v1
kind: ServiceAccount
metadata:
  name: remittance-processor-sa
  namespace: finance
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/RemittanceProcessorRole

Terraform IAM role with OIDC trust for EKS

resource "aws_iam_role" "remittance_processor" {
  name = "RemittanceProcessorRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.eks.arn }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub": "system:serviceaccount:finance:remittance-processor-sa"
        }
      }
    }]
  })
}

KEDA ScaledObject for event-driven SQS scaling

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: remittance-processor-scaler
  namespace: finance
spec:
  scaleTargetRef:
    name: remittance-processor
  minReplicaCount: 5
  maxReplicaCount: 50
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789/remittance-queue
        queueLength: "10"
        awsRegion: us-east-1

Practical Applications

• Use Case: Real-time remittance processing systems can leverage KEDA to scale from 5 to 42 pods in under 2 minutes during 5,000 msg/min spikes.
• Pitfall: Setting short ExternalSecrets refresh intervals (e.g., 5m) can trigger Vault rate limiting (429 errors); use longer intervals (1h) with manual sync annotations instead.
• Use Case: SRE teams can use Harness CD canary stages (10% to 100%) with automated rollbacks based on P99 latency thresholds exceeding 10s.
• Pitfall: Aggressive KEDA cooldown periods (e.g., 30s) cause cluster thrashing; implement a stabilization window of at least 300 seconds for scale-down events.

References:

• https://dev.to/krishnakanth_eswaran_6000/zero-downtime-ecs-eks-migration-orchestrating-a-6-team-production-cutover-at-scale-1pe6