LLM 스터디 5주차 - Terraform으로 AWS EKS에 vLLM Production Stack 배포하기 — Qwen3-8B와 LMCache CPU 오프로딩까지

$  aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA --region us-east-2"Value": 0.0   # README 기본 리전 - GPU 쿼터 없음$  aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA --region us-east-1"Value": 8.0   # 이미 열려있음$  aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA --region us-west-2"Value": 8.0   # 이것도 열려있음$  aws service-quotas get-service-quota --service-code ec2 --quota-code L-DB2E81BA --region ap-northeast-2"Value": 0.0   # 계정 기본 리전(서울)도 0

$  aws eks describe-cluster-versions --region us-east-1clusterVersion: "1.36"   defaultVersion: true   status: STANDARD_SUPPORTclusterVersion: "1.35" ......$  aws ssm get-parameter \    --name /aws/service/eks/optimized-ami/1.36/amazon-linux-2023/x86_64/nvidia/recommended/image_id \    --region us-east-1{ "Parameter": { "Value": "ami-0a40b6166e4b6ea59", ... } }

$  cp env-vars.template env-vars

$  terraform init...Error: Incompatible provider versionProvider registry.terraform.io/hashicorp/template v2.2.0 does not have apackage available for your current platform, darwin_arm64.

$  terraform init...Terraform has been created a lock file .terraform.lock.hclTerraform has been successfully initialized!

$  terraform plan...Plan: 100 to add, 0 to change, 0 to destroy.

$  aws pricing get-products --service-code AmazonEC2 --region us-east-1 \    --filters "Type=TERM_MATCH,Field=instanceType,Value=g6.2xlarge" | grep "Demand Linux g6.2xlarge Instance Hour"$0.9776 per On Demand Linux g6.2xlarge Instance Hour

$  terraform apply -auto-approve...module.eks.aws_eks_cluster.this[0]: Creation complete after 10m46smodule.eks.module.eks_managed_node_group["cpu_pool"].aws_eks_node_group.this[0]: Creation complete after 1m49smodule.eks.module.eks_managed_node_group["gpu_pool"].aws_eks_node_group.this[0]: Creation complete after 1m50smodule.eks_addons.module.aws_load_balancer_controller.helm_release.this[0]: Creation complete after 21smodule.eks_addons.module.cert_manager.helm_release.this[0]: Creation complete after 59s

$  aws eks update-kubeconfig --name vllm-eks-prod --region us-east-1 --kubeconfig ./kubeconfig$  export KUBECONFIG=./kubeconfig$  kubectl get pods -n vllmNAME                                                     READY   STATUS                  RESTARTSvllm-gpu-qwen3-8b-gpu-deployment-vllm-7d79dc5559-hhjqh   0/1     Init:CrashLoopBackOff   6

$  kubectl logs -n vllm vllm-gpu-qwen3-8b-gpu-deployment-vllm-7d79dc5559-hhjqh -c downloader...Successfully installed ... huggingface_hub-1.30.0 ...Warning: `huggingface-cli` is deprecated and no longer works. Use `hf` instead.Hint: `hf` is already installed! Use it directly.

$  kubectl logs -n vllm <qwen-pod> -c downloaderFetching 15 files: 100%|██████████| 15/15 [02:11<00:00,  8.78s/it]✓ Downloaded  path: /data/models/qwen3-8b$  kubectl logs -n vllm <qwen-pod>INFO vLLM API server version 0.8.5.post1Loading safetensors checkpoint shards: 100% Completed | 5/5

$  kubectl get pods -n vllmNAME                                     READY   STATUSvllm-gpu-qwen3-8b-gpu-deployment-vllm    1/1     Running        # ✅ vLLM 엔진 정상vllm-gpu-deployment-router               0/1     CrashLoopBackOff  # ❌ router만 계속 재시작

$  kubectl logs -n vllm <pod> -c downloaderWARNING: Retrying ... after connection broken by  'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443):  Read timed out. (read timeout=300.0)")'

$  kubectl exec -n vllm <pod> -c downloader -- python3 -c "import socketprint(socket.getaddrinfo('pypi.org', 443))"# IPv4 주소(151.101.x.223)도 정상 반환됨 → DNS 문제 아님$  kubectl exec -n vllm <pod> -c downloader -- python3 -c "import socket, times = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.settimeout(5)s.connect(('151.101.64.223', 443))print('OK')"# 151.101.64.223:443 OK in 0.00s → TCP 3-way handshake는 즉시 성공$  kubectl exec -n vllm <pod> -c downloader -- python3 -c "import socket, ssl, timectx = ssl.create_default_context()s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.settimeout(10)s.connect(('151.101.64.223', 443))ss = ctx.wrap_socket(s, server_hostname='pypi.org')"# FAILED at 10.01s: TimeoutError: handshake operation timed out

$  kubectl exec -n vllm <pod> -c downloader -- cat /sys/class/net/eth0/mtu9001

$  terraform apply -auto-approve -target=helm_release.calicohelm_release.calico: Modifications complete after 21s$  kubectl rollout status daemonset/calico-node -n calico-systemdaemon set "calico-node" successfully rolled out$  kubectl delete pod -n vllm <stuck-pod>$  kubectl logs -n vllm <new-pod> -c downloaderFetching 15 files: 100%|██████████| 15/15 [00:XX<00:00]✓ Downloaded

$  terraform apply -auto-approve...Apply complete! Resources: 3 added, 0 changed, 1 destroyed.Outputs:vllm_api_url = "http://k8s-vllm-vllmgpui-c4bdadcd08-54353942.us-east-1.elb.amazonaws.com/v1"

$  curl http://k8s-vllm-vllmgpui-....elb.amazonaws.com/v1/models{"object":"list","data":[{"id":"/data/models/qwen3-8b", ...}]}$  curl http://k8s-vllm-vllmgpui-....elb.amazonaws.com/v1/chat/completions \  -H "Content-Type: application/json" \  -d '{"model":"/data/models/qwen3-8b","messages":[{"role":"user","content":"한국어로 짧게 자기소개 해줘."}],"max_tokens":100}'{"choices":[{"message":{"content":"<think>\nOkay, the user asked for a shortself-introduction in Korean. Let me start by greeting them...","role":"assistant"},"finish_reason":"length"}], ...}

$  aws eks update-kubeconfig --name vllm-eks-prod --region us-east-1Updated context arn:aws:eks:us-east-1:872515291231:cluster/vllm-eks-prod in /Users/hyeonjaelee/.kube/config$  kubectl config use-context arn:aws:eks:us-east-1:872515291231:cluster/vllm-eks-prodSwitched to context "arn:aws:eks:us-east-1:872515291231:cluster/vllm-eks-prod".$  kubectl get nodesNAME                          STATUS   ROLES    AGE   VERSIONip-10-20-1-123.ec2.internal   Ready    <none>   3h    v1.36.3-eks-cb19647ip-10-20-1-13.ec2.internal    Ready    <none>   3h    v1.36.3-eks-cb19647ip-10-20-2-212.ec2.internal   Ready    <none>   3h    v1.36.3-eks-cb19647

$  kubectl port-forward svc/vllm-gpu-router-service 30080:80 -n vllm &$  export vllm_api_url=http://localhost:30080/v1$  curl -s ${vllm_api_url}/models | jq .{  "object": "list",  "data": [    {      "id": "/data/models/qwen3-8b",      "object": "model",      "created": 1788589890,      "owned_by": "vllm",      "root": "/data/models/qwen3-8b",      "parent": null,      "max_model_len": 16384,      "permission": [        {          "id": "modelperm-e84ca9c3aa7e436e8adca6e93cd00eed",          "object": "model_permission",          "created": 1788589890,          "allow_create_engine": false,          "allow_sampling": true,          "allow_logprobs": true,          "allow_search_indices": false,          "allow_view": true,          "allow_fine_tuning": false,          "organization": "*",          "group": null,          "is_blocking": false        }      ]    }  ]}

$  kubectl -n vllm get svcNAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)vllm-gpu-qwen3-8b-gpu-engine-service   ClusterIP   172.20.91.183    <none>        80/TCP,55555/TCP,9999/TCPvllm-gpu-router-service                ClusterIP   172.20.183.144   <none>        80/TCP,9000/TCP

$  curl -s ${vllm_api_url}/completions \  -H "Content-Type: application/json" \  -d '{    "model": "/data/models/qwen3-8b",    "prompt": "Toronto is a",    "max_tokens": 20,    "temperature": 0  }' | jq '.choices[].text'" city in Canada, located in the province of Ontario. It is the largest city in Canada and the"

$  kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n kube-prometheus-stackForwarding from 127.0.0.1:3000 -> 3000Forwarding from [::1]:3000 -> 3000

$  kubectl get secret -n kube-prometheus-stack kube-prometheus-stack-grafana \    -o jsonpath="{.data.admin-password}" | base64 --decodeprom-operator

$  kubectl get configmap vllm-dashboard -n kube-prometheus-stack -o jsonpath='{.data}' \    | python3 -c "import json,sys; d=json.load(sys.stdin)for k,v in d.items():    dash=json.loads(v)    print('Dashboard title:', dash.get('title'))    for p in dash.get('panels', []): print(' -', p.get('title'))"

$  kubectl get configmap vllm-dashboard -n kube-prometheus-stack -o jsonpath='{.data.vllm-dashboard\.json}' \    | python3 -c "import json,sysd=json.load(sys.stdin)for p in d.get('panels', []):    if 'GPU KV' in p.get('title',''):        print(p['title'])        for t in p.get('targets', []): print(' ', t.get('expr'))"GPU KV Usage Percentage  vllm:gpu_cache_usage_percGPU KV Cache Hit Rate  vllm:gpu_prefix_cache_hits_total{endpoint="service-port"} / vllm:gpu_prefix_cache_queries_total

$  curl -s .../metrics | grep "gpu_prefix_cache\|gpu_cache_usage_perc"# (결과 없음 — 둘 다 없음)$  curl -s .../metrics | grep "prefix_cache\|kv_cache_usage_perc"vllm:prefix_cache_queries_total           3736.0vllm:prefix_cache_hits_total              2096.0vllm:external_prefix_cache_queries_total  1640.0vllm:external_prefix_cache_hits_total     0.0vllm:kv_cache_usage_perc                  0.0

$  kubectl get configmap vllm-dashboard -n kube-prometheus-stack -o jsonpath='{.data.vllm-dashboard\.json}' > dashboard.json$  sed -i '' \    -e 's/vllm:gpu_prefix_cache_hits_total/vllm:prefix_cache_hits_total/' \    -e 's/vllm:gpu_prefix_cache_queries_total/vllm:prefix_cache_queries_total/' \    -e 's/vllm:gpu_cache_usage_perc/vllm:kv_cache_usage_perc/g' \    dashboard.json$  kubectl patch configmap vllm-dashboard -n kube-prometheus-stack \    --type merge --patch-file=<(python3 -c "import json; print(json.dumps({'data':{'vllm-dashboard.json': open('dashboard.json').read()}}))")configmap/vllm-dashboard patched

$  kubectl logs -n kube-prometheus-stack -l app.kubernetes.io/name=grafana -c grafana-sc-dashboard --tail=5{"msg": "Writing /tmp/dashboards/vllm-dashboard.json (ascii)", "level": "INFO"}

$  curl -s --data-urlencode 'query=vllm:prefix_cache_hits_total{endpoint="service-port"} / vllm:prefix_cache_queries_total' \    http://localhost:19090/api/v1/query | jq '.data.result[0].value'[1788597948.265, "0.5610278372591007"]$  curl -s --data-urlencode 'query=vllm:kv_cache_usage_perc' \    http://localhost:19090/api/v1/query | jq '.data.result[0].value'[1788597948.807, "0"]

$  kubectl get configmap vllm-dashboard -n kube-prometheus-stack -o jsonpath='{.data}' \    | grep -io "lmcache[a-z_]*\|cpu_offload[a-z_]*"# (결과 없음)

$  grep -rln "lmcache" /Users/hyeonjaelee/production-stack --include="*.md" -i | grep -i grafanahelm/README.md   # "LMCache Dashboard" 섹션 — Average TTFT, Cache hit rate, Retrieve speed,                 # Local CPU cache usage, 총 요청 토큰 수, 총 히트 토큰 수 6개 패널$  find /Users/hyeonjaelee/production-stack -iname "*lmcache-dashboard*"helm/dashboards/lmcache-dashboard.json

$  curl -s "https://hub.docker.com/v2/repositories/lmcache/vllm-openai/tags?page_size=100" \    | python3 -c "import json,sys; d=json.load(sys.stdin)for r in d['results']:    if 'rc' not in r['name'] and 'nightly' not in r['name']: print(r['name'], r['last_updated'])"latest        2026-08-24T18:57:22Zv0.5.4-cu129  2026-08-24T19:28:49Zv0.5.4        2026-08-24T18:57:24Z   # ← latest와 동일 시각/크기, 이걸 고정해서 사용v0.5.3        2026-08-05T22:35:30Z

$  kubectl logs -n vllm <pod> | grep LMCache | head -5LMCache INFO: Creating LMCacheEngine instance vllm-instanceLMCache INFO: Creating LMCacheEngine with config: {'chunk_size': 256, 'local_cpu': True,  'max_local_cpu_size': 12.0, ...}LMCache INFO: Created backend: LocalCPUBackend (LocalCPUBackend)LMCache INFO: LMCache initialized for role KVConnectorRole.WORKER with version 0.5.4-g3e11b8ed,  vllm version 0.27.1, ...

$  kubectl logs -n vllm <pod> | grep "LMCache hit tokens"LMCache INFO: Reqid: cmpl-...-0, Total tokens 13, Inference Engine computed tokens: 0, LMCache hit tokens: 0, need to load: 0LMCache INFO: Reqid: cmpl-...-1, Total tokens 13, Inference Engine computed tokens: 0, LMCache hit tokens: 0, need to load: 0

$  kubectl port-forward svc/vllm-gpu-qwen3-8b-gpu-engine-service 18000:80 -n vllm &$  curl -s http://localhost:18000/metrics | grep "lmcache:num_stored_tokens_total"lmcache:num_stored_tokens_total{...} 0.0   # 28토큰짜리 prompt로는 저장 자체가 안 됨

$  curl -s ${vllm_api_url}/completions \  -H "Content-Type: application/json" \  -d '{"model":"/data/models/qwen3-8b","prompt":"<439-token prompt>","max_tokens":10}'$  curl -s http://localhost:18000/metrics | grep num_stored_tokens_totallmcache:num_stored_tokens_total{...} 256.0   # 정확히 chunk_size(256) 하나가 저장됨

$  kubectl logs -n vllm <pod> | grep "LMCache hit tokens"Reqid: ...-b0aefc6a, Total tokens 439, Inference Engine computed tokens: 432, LMCache hit tokens: 256, need to load: 0