☸️ Understanding Kubernetes Service Account Tokens in EKS

Understanding Kubernetes Service Account Tokens in EKS

When a Kubernetes pod starts, it usually needs a way to securely communicate with the Kubernetes API Server. To make this possible, Kubernetes automatically creates and mounts a temporary authentication token into the pod.

What Is kube-api-access?

Inside every pod, Kubernetes creates a special volume like: 

kube-api-access-xxxxx

This volume contains:

  • Service Account Token (JWT)
  • Cluster CA Certificate
  • Namespace Information

These files are mounted inside the container at: 

/var/run/secrets/kubernetes.io/serviceaccount/

How Pod Authentication Works 

Pod Starts
   ↓
Kubelet prepares pod
   ↓
Kubelet requests token from API Server
   ↓
API Server signs JWT token
   ↓
Token mounted into pod
   ↓
Container starts successfully

What Is the Token?

The token is a JWT (JSON Web Token) used for Kubernetes authentication. It identifies the pod and its service account.

Example: 

{
  "namespace": "default",
  "serviceAccount": "my-app",
  "pod": "example-pod"
}

This allows Kubernetes to verify: 

"This request came from this specific pod."

Why Pods Need This Token

Many Kubernetes applications communicate with the API Server:

  • Argo CD
  • Datadog Agents
  • ExternalDNS
  • AWS Load Balancer Controller
  • metrics-server
  • Custom Applications

Without the token, the pod cannot authenticate securely.

Understanding the Error 

MountVolume.SetUp failed for volume "kube-api-access"
failed to fetch token
failed to generate token
while signing jwt
grpc: the client connection is closing

What This Means

During pod startup:

  • Kubelet requested a token
  • The API Server started generating the JWT
  • The connection was interrupted before completion

As a result, Kubernetes could not mount the token volume, so the pod startup failed temporarily.

Common Causes

  • Temporary API Server connectivity issue
  • Node resource pressure
  • Kubelet restart
  • EKS control plane hiccup
  • Networking/CNI issue
  • Large rollout or mass pod restart

Does It Fix Automatically?

In many cases, yes. Kubernetes continuously retries the token request process. 

Token request fails
   ↓
Kubelet retries
   ↓
API Server responds normally
   ↓
Token mounted successfully
   ↓
Pod starts

When Should You Investigate Further?

If the issue happens continuously:

  • Multiple pods stuck in ContainerCreating
  • Nodes become NotReady
  • kube-system pods unstable
  • Repeated token mount failures
  • Frequent kubelet reconnects

Simple Real-World Analogy 

New employee joins office
   ↓
Reception creates badge
   ↓
Badge printer connection breaks
   ↓
Employee waits temporarily
   ↓
Retry succeeds
   ↓
Employee enters building

That is very similar to how Kubernetes service account token generation works during pod startup.

Popular posts from this blog

☁️ AWS Global Accelerator (GA) + Route 53

🟧☁️ AWS + 🌍 Global Accelerator (GA) + 🌐 Route 53 what they are and how they work together (especially for ALB/EKS). ☁️ AWS = Cloud platform 🌍 GA = Fast global routing + static IPs 🌐 Route 53 = DNS ☁️ What is AWS? AWS (Amazon Web Services) is Amazon’s cloud platform where you run servers, containers, databases, networking, DNS, monitoring, and more. EC2 (servers), EKS (Kubernetes), RDS (databases) ALB/NLB (load balancers), S3 (storage), Route 53 (DNS) What is AWS Global Accelerator? Global Accelerator (GA) improves availability and performance by routing users through AWS’s private global network to the nearest healthy endpoint . Key idea: Users hit GA (static Anycast IPs). GA picks the best Region/endpoint automatically. Static Anycast IPs (2 fixed IP addresse...
Read more

🐳 Docker Filesystem Internals (AdvancEd)

Docker Filesystem Internals (Advanced) This post is for readers who already understand Docker basics and want to learn how Docker manages files internally. If you are new to Docker, read the beginner guide first. Containers Do NOT Have Their Own Disk A common misunderstanding is that Docker containers have their own disk. In reality: Containers do not have a separate physical filesystem. Docker creates a filesystem view using layers stored on the host machine. This layered filesystem is implemented using a Union Filesystem , most commonly OverlayFS on Linux. Docker Images Are Read-Only Layers A Docker image is made up of multiple read-only layers. Each instruction in a Dockerfile creates a new layer. FROM python:3.11 RUN pip install flask COPY app.py . This results in layers lik...
Read more

Understanding RabbitMQ Classic Mirrored Queues and Quorum Queues

Image
💚💚 💚 Understanding RabbitMQ Classic Mirrored Queues and Quorum  Queues 💚💚 💚   I understood RabbitMQ only after I understood why Classic Mirrored Queues were replaced . When I first started learning RabbitMQ, I kept hearing: Classic Queues Classic Mirrored Queues Quorum Queues Raft Consensus Everyone said:  "Use Quorum Queues." But I couldn't understand why. To understand Quorum Queues , I first had to understand how Classic Mirrored Queues worked.
Read more

🐳 Docker Tutorial for Beginners: Step-by-Step with a Simple Example

Docker Explained Using a Very Simple App (Beginner Friendly) Goal: Build a tiny Python app, run it normally, then run the same app using Docker, and understand why Docker helps. Step 1: Create a Very Small Python App Create a file named app.py : print("Hello I am running inside Docker!") Step 2: Run the App WITHOUT Docker Run this on your machine: python app.py Common real-life problems: Python not installed, wrong Python version, missing dependencies, different OS behavior this is where the “works on my machine” issue starts. Step 3: Introduce Docker Instead of relying on your system, Docker lets you package the runtime (Python) together with your app. We’ll do that using a Dockerfile . Step 4: Create a Dockerfile Create a file named Dockerfile (no extension): FROM python:3.11 WORKDIR /app COPY app.py . CMD ["py...
Read more

☸️What’s Inside EKS? A Beginner’s Guide to Its Core Components

Image
 EKS (Elastic Kubernetes Service) EKS is a managed service by AWS that runs Kubernetes for you. It takes care of setting up and managing the Kubernetes control plane so you can focus on running your applications. Kubernetes  Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications. It helps run apps reliably across a cluster of machines, automatically handling scheduling, scaling, updates, and recovery. Pod A Pod is the smallest unit in Kubernetes. It can hold one or more containers that share the same network and storage. All containers in a pod are scheduled and managed together. Container A Container is a lightweight, standalone package that includes everything an app needs to run — code, runtime, libraries, and dependencies. It ensures the app runs the same no matter where it’s deployed. 👀1 pod can have multiple containers but 1 container cannot belongs to multiple pods.
Read more

☸️ Kubernetes Taints and Tolerations(with Node Affinity)

Kubernetes Taints and Tolerations(with Node Affinity + EKS Upgrade Context) Kubernetes scheduling is the process of deciding which node runs a given Pod . This page explains how Kubernetes keeps Pods away from “inappropriate” nodes using taints (node-side restrictions) and tolerations (pod-side permissions), and how that relates to node affinity (pod-side attraction). 1) Core idea: attraction vs repulsion Node affinity (attraction) Node affinity is a property of Pods that attracts them to a set of nodes. You can express it as: Hard requirement (must match), or Preference (nice to have). Taints (repulsion) Taints are the opposite of node affinity. They allow a node to repel a set of Pods. Tolerations (permission) Tolerations are applied to Pods. A toleration allows the scheduler to schedule Pods onto nodes with matching taints. Important...
Read more

AWS Load Balancer Controller Upgrade Guide: v2.x to v3.3

Image
AWS Load Balancer Controller Upgrade Guide: v2.x to v3.3 What is AWS Load Balancer Controller? AWS Load Balancer Controller is a Kubernetes controller that integrates Amazon EKS with AWS Elastic Load Balancing services. It watches Kubernetes resources and automatically creates or updates AWS load balancing resources based on the desired state defined inside the cluster. Instead of manually creating load balancers, listeners, target groups, and security group rules in AWS, teams can define Kubernetes resources such as Ingresses and Services. The controller then reconciles those resources with AWS. What Does It Manage? AWS Load Balancer Controller commonly manages: Application Load Balancers (ALB) Network Load Balancers (NLB) Listeners and listener rules Target groups Target registration and deregistration Security group rules Ingress resources TargetGroupBinding resources Beca...
Read more

🐳 Build a Tiny Flask Web App in Docker (with Ports)

docker run -p 8080:5000 flask-hello Then open in your browser: http://localhost:8080 Step 6: Stop the container Press Ctrl + C in the terminal. Quick Troubleshooting Browser says “This site can’t be reached”: Make sure the container is running and you used the -p port mapping. Flask runs but not accessible: Confirm Flask is running with host="0.0.0.0" inside Docker (not 127.0.0.1 ). Port already in use: Try using a different host port, for example: docker run -p 9090:5000 flask-hello Next Post Ideas Docker image layers and caching (explained simply) Docker volumes : keeping data after a container stops Running the same container in Kubernetes (Pod basics)
Read more

☁️ Amazon S3 Explained: More Than Just Object Storage

Amazon S3 Explained: More Than Just Object Storage When people hear Amazon S3 , they usually think it is only used to store files like images, logs, backups, or documents. That is true, but S3 has grown into much more than simple object storage. Today, S3 can be used for object storage, file-like access, immutable backups, lifecycle management, static websites, event-driven workflows, analytics, and data lake storage. In this post, let us understand what S3 is and how it is used in real-world AWS environments. 1. What is Amazon S3? Amazon S3 stands for Simple Storage Service . It is mainly an object storage service . In S3, data is stored as objects inside buckets. Bucket → Object Each object usually contains: The actual data Metadata A unique object key Example: Bucket name: my-app-logs Object key: logs/2026/app.log Even though this looks like a folder path, S3 does not work like a traditional filesystem internally. The folder-like str...
Read more

☸️ CoreDNS and AWS VPC CNI in EKS

CoreDNS vs AWS VPC CNI in EKS: What Each Component Actually Does When working with Amazon EKS , two important components help your Kubernetes cluster communicate properly: CoreDNS and AWS VPC CNI . Both are part of the networking story, but they do very different jobs. CoreDNS helps pods find services by name, while AWS VPC CNI gives pods network connectivity using IP addresses from your AWS VPC. In simple words: CoreDNS answers: “What IP address belongs to this service name?” AWS VPC CNI answers: “How does traffic reach that pod over the network?” 1. What is CoreDNS? CoreDNS is the DNS server used inside Kubernetes. Its main job is to resolve Kubernetes service names into IP addresses. For example, if one pod wants to call another service using a name like: user-service.default.svc.cluster.local CoreDNS resolves that service name into the correct Kubernetes service IP. What CoreDNS Does Resolves Kubernetes service names Supports pod-to...
Read more