awaylab-aws

Pulumi-based AWS platform repo for provisioning EKS clusters and bootstrapping GitOps addons with Argo CD, including Karpenter-based node provisioning.

Overview

This repo provisions, per stack:

• EKS cluster
• Bootstrap managed node group
• IAM roles for cluster, nodes, and Karpenter controller (IRSA)
• OIDC provider for IRSA
• Karpenter interruption SQS queue
• Argo CD namespace and Helm release
• Argo CD Application pointing to the stack overlay in this repo

The Pulumi program lives in __main__.py and consumes stack config from Pulumi.<stack>.yaml files.

Current Stack Configuration

• Stacks: dev, prod
• Region: us-east-2 (all current stacks)
• Kubernetes version:
  • dev: 1.36
  • prod: 1.35
• Runtime toolchain: uv + local .venv

Repository Layout

.
├── __main__.py
├── Pulumi.yaml
├── Pulumi.dev.yaml
├── Pulumi.prod.yaml
├── pyproject.toml
├── apps/
│   └── karpenter/
│       ├── base/
│       └── overlays/{dev,prod}/
└── argocd/
  └── overlays/{dev,prod}/
        └── applications/
            ├── karpenter.yaml
            ├── karpenter-crds.yaml
            └── karpenter-controller.yaml

Prerequisites

• AWS account with permissions for EKS, IAM, EC2, SQS, and related resources
• Pulumi CLI configured and logged in
• Python >=3.14
• uv installed
• kubectl and AWS CLI installed for operations/troubleshooting

Setup and Deploy

1. Install dependencies and create the virtual environment:

uv sync

2. Select a stack:

pulumi stack select dev

3. Preview changes:

pulumi preview --stack dev

4. Apply changes:

pulumi up --stack dev

GitOps and Karpenter Notes

• Argo CD is installed by Pulumi, then a cluster-addons Application is created.
• That Application points to argocd/overlays/<stack> in this same repo.
• Karpenter controller role ARN is injected via the overlay values in argocd/overlays/<stack>/applications/karpenter-controller.yaml.
• Karpenter interruption queue is created in AWS with name equal to the EKS cluster name and should match the controller value settings.interruptionQueue.

Useful Outputs

Check key values after deploy:

pulumi stack output --stack dev cluster_name
pulumi stack output --stack dev karpenter_controller_role_arn
pulumi stack output --stack dev karpenter_interruption_queue_name
pulumi stack output --stack dev karpenter_interruption_queue_arn
pulumi stack output --stack dev gitops_overlay_path

Troubleshooting

1) Verify AWS auth/session first

aws sts get-caller-identity

If expired:

aws login

2) Verify controller role has expected IAM permissions

Example: check EC2 Describe permissions on the Karpenter controller role.

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<account-id>:role/<karpenter-controller-role> \
  --action-names ec2:DescribeInstanceTypes ec2:DescribeSubnets ec2:DescribeSecurityGroups

Example: check SQS permissions against the interruption queue ARN.

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::<account-id>:role/<karpenter-controller-role> \
  --action-names sqs:GetQueueUrl sqs:GetQueueAttributes sqs:ReceiveMessage sqs:DeleteMessage sqs:ChangeMessageVisibility \
  --resource-arns arn:aws:sqs:us-east-2:<account-id>:<queue-name>

Expected result for each action: EvalDecision: allowed.

3) Verify cluster context and Karpenter runtime wiring

aws eks update-kubeconfig --name "$(pulumi stack output --stack dev cluster_name)" --region us-east-2

Check ServiceAccount role annotation:

kubectl -n karpenter get sa karpenter -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}' && echo

Check deployment env/config (note deployment name):

kubectl -n karpenter get deploy karpenter-controller -o yaml | rg -n "INTERRUPTION_QUEUE|interruption|queue|role-arn|KARPENTER"

4) Check controller logs for common errors

kubectl -n karpenter logs deploy/karpenter-controller --tail=200 | rg -i "AccessDenied|UnauthorizedOperation|NonExistentQueue|GetQueueUrl|DescribeInstanceTypes|AWS.SimpleQueueService|interruption"

Common patterns:

• AccessDenied on EC2/IAM/SQS: usually missing IAM action or bad condition scope
• AWS.SimpleQueueService.NonExistentQueue: queue exists but controller still has stale state or wrong queue name
• No logs from expected deployment: verify you are using deploy/karpenter-controller (not deploy/karpenter)

5) Roll the controller after IAM or queue updates

kubectl -n karpenter rollout restart deploy/karpenter-controller
kubectl -n karpenter rollout status deploy/karpenter-controller --timeout=180s

Then re-check logs.

6) Single-node dev cluster scheduling issues

If Karpenter controller pod is Pending with anti-affinity complaints, keep controller replicas at 1 in dev overlay values.

Common Commands

# Preview/apply
pulumi preview --stack dev
pulumi up --stack dev

# Inspect outputs
pulumi stack output --stack dev

# Export stack state for deep inspection
pulumi stack export --stack dev > /tmp/dev-stack.json

Cleanup

pulumi destroy --stack dev