☸️ Kubernetes Taints and Tolerations(with Node Affinity)

Kubernetes Taints and Tolerations(with Node Affinity + EKS Upgrade Context)

Kubernetes scheduling is the process of deciding which node runs a given Pod. This page explains how Kubernetes keeps Pods away from “inappropriate” nodes using taints (node-side restrictions) and tolerations (pod-side permissions), and how that relates to node affinity (pod-side attraction).

1) Core idea: attraction vs repulsion

Node affinity (attraction)

Node affinity is a property of Pods that attracts them to a set of nodes. You can express it as:

  • Hard requirement (must match), or
  • Preference (nice to have).

Taints (repulsion)

Taints are the opposite of node affinity. They allow a node to repel a set of Pods.

Tolerations (permission)

Tolerations are applied to Pods. A toleration allows the scheduler to schedule Pods onto nodes with matching taints.

Important: Tolerations allow scheduling but don’t guarantee scheduling. The scheduler still evaluates other parameters as part of its function (resources, nodeSelector / nodeAffinity, topology rules, etc.).

Put simply:
Taints repel Pods.
Tolerations allow Pods to “pass” that repelling rule.
Node affinity attracts Pods to particular nodes (required or preferred).

2) Why taints and tolerations exist

Taints and tolerations work together to ensure that Pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node, marking that node as:

“This node should not accept Pods that do not tolerate these taints.”

3) Adding and removing a taint (node-side rule)

You add a taint to a node using kubectl taint. For example:

kubectl taint nodes node1 key1=value1:NoSchedule

This places a taint on node node1. The taint has:

  • key: key1
  • value: value1
  • effect: NoSchedule

Meaning:
No Pod will be able to schedule onto node1 unless it has a matching toleration.

To remove that taint, you can run:

kubectl taint nodes node1 key1=value1:NoSchedule-

4) Defining tolerations (pod-side permission)

You specify tolerations in the PodSpec. The following two tolerations both “match” the taint from the example above, and therefore a Pod with either toleration can schedule onto node1.

Example A: operator Equal

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoSchedule"

Example B: operator Exists

tolerations:
- key: "key1"
  operator: "Exists"
  effect: "NoSchedule"

The doc notes that the default value for operator is Equal.

5) Matching logic: how Kubernetes decides a toleration matches a taint

A toleration “matches” a taint if:

  • Keys are the same, and
  • Effects are the same, and
  • One of the following is true:
    • The operator is Exists (in which case no value should be specified), or
    • The operator is Equal and the values are equal.

Two special cases

  • Empty key: If the key is empty, the operator must be Exists. This matches all keys and values, but the effect still needs to match.
  • Empty effect: An empty effect in the toleration matches all effects (but the key still needs to match).

6) A key warning: manually setting .spec.nodeName bypasses the scheduler

Normally, the default Kubernetes scheduler considers taints and tolerations when selecting a node. However, if you manually specify .spec.nodeName for a Pod:

  • You bypass the scheduler, and
  • The Pod is bound to the node you assigned, even if that node has NoSchedule taints.

But there is an important follow-up:

  • If the node has a NoExecute taint, the kubelet will eject the Pod unless the Pod has an appropriate toleration.

7) Example Pod with toleration

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  tolerations:
  - key: "example-key"
    operator: "Exists"
    effect: "NoSchedule"

8) Taint effects: NoSchedule, PreferNoSchedule, NoExecute

NoSchedule

  • No new Pods will be scheduled on the tainted node unless they have a matching toleration.
  • Pods currently running on the node are not evicted.

PreferNoSchedule

  • A “soft” version of NoSchedule.
  • The control plane tries to avoid placing a Pod that does not tolerate the taint on the node, but it is not guaranteed.

NoExecute

This affects Pods that are already running on the node:

  • Pods that do not tolerate the taint are evicted immediately.
  • Pods that tolerate the taint without specifying tolerationSeconds remain bound forever.
  • Pods that tolerate the taint with tolerationSeconds remain bound for that amount of time, then are evicted.

NoExecute with tolerationSeconds

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoExecute"
  tolerationSeconds: 3600

Meaning:
If this Pod is running and a matching NoExecute taint is added to the node, it stays for 3600 seconds, then is evicted (unless the taint is removed before time is up).

9) Multiple taints and multiple tolerations (filter behavior)

Nodes can have multiple taints and Pods can have multiple tolerations. Kubernetes processes multiple taints and tolerations like a filter:

  1. Start with all taints on the node.
  2. Ignore taints for which the Pod has a matching toleration.
  3. Any remaining (un-ignored) taints apply their effects.

In particular:

  • If there is at least one remaining taint with effect NoSchedule, Kubernetes will not schedule the Pod onto that node.
  • If there is no remaining NoSchedule taint, but at least one remaining PreferNoSchedule taint, Kubernetes will try not to schedule the Pod onto that node.
  • If there is at least one remaining taint with effect NoExecute, the Pod will be evicted (if already running there) and will not schedule there (if not running).

Multi-taint example from the doc

Taint a node like this:

kubectl taint nodes node1 key1=value1:NoSchedule
kubectl taint nodes node1 key1=value1:NoExecute
kubectl taint nodes node1 key2=value2:NoSchedule

And a Pod has these tolerations:

tolerations:
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoSchedule"
- key: "key1"
  operator: "Equal"
  value: "value1"
  effect: "NoExecute"

Result:
The Pod still cannot schedule onto the node because there is no toleration matching the third taint (key2=value2:NoSchedule).
But if the Pod was already running on that node and the taints were added later, it can continue running because the only untolerated taint is NoSchedule (which does not evict existing Pods).

10) Numeric comparison operators (alpha feature: Kubernetes v1.35)

In addition to Equal and Exists, Kubernetes introduced numeric comparison operators: Gt and Lt to match taints with integer values. This is useful for threshold-based scheduling (for example, SLA tiers).

  • Gt matches when the toleration value is greater than the taint value.
  • Lt matches when the toleration value is less than the taint value.

For numeric operators:

  • Both toleration and taint values must be valid integers for a match.
  • If either value cannot be parsed as an integer, the toleration does not match.
  • The API server validates toleration values (Pods), but node taint values are not validated at registration time.

Numeric example (SLA)

Node taint:

kubectl taint nodes node1 servicelevel.organization.example/agreed-service-level=950:NoSchedule

Pod toleration (Gt):

tolerations:
- key: "servicelevel.organization.example/agreed-service-level"
  operator: "Gt"
  value: "900"
  effect: "NoSchedule"

This matches because 950 > 900 (for the Gt operator).

Pod toleration (Lt) example:

tolerations:
- key: "servicelevel.organization.example/agreed-service-level"
  operator: "Lt"
  value: "1000"
  effect: "NoSchedule"

Warnings (feature gate considerations)

Before disabling the related feature gate, you should find and update workloads that use Gt/Lt to avoid validation errors or controller loops, and delete any pending Pods using those operators.

11) Example use cases

Dedicated nodes

If you dedicate a set of nodes for exclusive use, you can:

  • Apply a taint to those nodes (repel everyone else), and
  • Add a matching toleration to the Pods that are allowed to use them.

If you want to ensure those Pods run only on the dedicated nodes, you typically combine this with:

  • A matching node label on those nodes, and
  • A node affinity requirement so those Pods must choose that labeled set.

Nodes with special hardware (e.g., GPUs)

You can taint GPU nodes so general workloads avoid them, and only GPU workloads (with tolerations) can schedule there.

Kubernetes can also automate tolerations for these hardware cases using extended resources and specific admission controllers (so you don’t manually add tolerations everywhere).

Taint-based evictions

Taints can also drive eviction behavior when nodes become unhealthy, using the NoExecute effect.

12) Taint-based evictions (built-in taints)

The control plane can automatically taint nodes when certain conditions are true. These are built-in taints such as:

  • node.kubernetes.io/not-ready (Ready condition is False)
  • node.kubernetes.io/unreachable (Ready condition is Unknown)
  • node.kubernetes.io/memory-pressure
  • node.kubernetes.io/disk-pressure
  • node.kubernetes.io/pid-pressure
  • node.kubernetes.io/network-unavailable
  • node.kubernetes.io/unschedulable
  • node.cloudprovider.kubernetes.io/uninitialized (for external cloud providers until initialized)

When a node is drained, the node controller or kubelet can add relevant taints with NoExecute, and remove them when the fault condition returns to normal.

Important note about unreachable nodes

If a node is unreachable, the API server may not be able to communicate eviction decisions to the kubelet. In that case, some Pods might continue to run on the partitioned node until communication is restored.

Rate limiting

The control plane limits the rate of adding new taints to nodes to avoid triggering too many evictions at once (for example, during a large network disruption).

13) Default tolerations and why they matter (especially during upgrades)

Kubernetes automatically adds tolerations for:

  • node.kubernetes.io/not-ready
  • node.kubernetes.io/unreachable

Typically with tolerationSeconds=300 (5 minutes), unless you explicitly set them. This means many Pods remain bound to nodes for about 5 minutes after a problem is detected.

This is one big reason you hear tolerations mentioned during EKS upgrades: nodes transition through states (draining / replacement / readiness changes), and these tolerations help avoid immediate mass eviction and instability.

DaemonSet behavior

DaemonSet Pods are created with NoExecute tolerations for not-ready and unreachable with no tolerationSeconds, ensuring they are not evicted due to those problems.

14) Controller behavior changes (note about Kubernetes 1.29+)

Historically, the node controller handled taint-based evictions. After Kubernetes 1.29, taint-based eviction moved into a separate component called taint-eviction-controller.

Users can optionally disable taint-based eviction by configuring the controller manager to exclude it.

15) Taint nodes by condition (why scheduler checks taints, not conditions)

Kubernetes converts certain node conditions into taints with a NoSchedule effect. The scheduler checks taints (not raw node conditions) when it makes scheduling decisions.

Example: If DiskPressure is active, Kubernetes adds a node.kubernetes.io/disk-pressure taint, and new Pods are not scheduled onto that node.

You can ignore these node condition taints for newly created Pods by adding corresponding tolerations, but this must be done carefully.

QoS note (memory-pressure)

Kubernetes adds node.kubernetes.io/memory-pressure toleration on Pods that have a QoS class other than BestEffort. BestEffort Pods are treated more strictly under pressure.

DaemonSet controller auto-tolerations

The DaemonSet controller automatically adds NoSchedule tolerations to prevent DaemonSets from breaking, including:

  • node.kubernetes.io/memory-pressure
  • node.kubernetes.io/disk-pressure
  • node.kubernetes.io/pid-pressure (1.14+)
  • node.kubernetes.io/unschedulable (1.10+)
  • node.kubernetes.io/network-unavailable (host network only)

16) Final summary (connect everything: nodeSelector, required/preferred affinity, taints/tolerations)

Node selection and attraction (Pod-driven)

  • nodeSelector: simplest hard constraint using exact labels
  • nodeAffinity (required): hard constraint with richer matching rules
  • nodeAffinity (preferred): soft preference used to “score” nodes (nice to have)

Protection and permission (Node-driven + Pod permission)

  • Taints (node): repel Pods (block, discourage, or evict)
  • Tolerations (pod): allow a Pod to ignore specific taints

Key sentence: Node affinity attracts Pods to nodes. Taints repel Pods from nodes. Tolerations let Pods pass the repelling rule, but they never force placement. The scheduler still checks other constraints and scoring rules.

Popular posts from this blog

☁️ AWS Global Accelerator (GA) + Route 53

🟧☁️ AWS + 🌍 Global Accelerator (GA) + 🌐 Route 53 what they are and how they work together (especially for ALB/EKS). ☁️ AWS = Cloud platform 🌍 GA = Fast global routing + static IPs 🌐 Route 53 = DNS ☁️ What is AWS? AWS (Amazon Web Services) is Amazon’s cloud platform where you run servers, containers, databases, networking, DNS, monitoring, and more. EC2 (servers), EKS (Kubernetes), RDS (databases) ALB/NLB (load balancers), S3 (storage), Route 53 (DNS) What is AWS Global Accelerator? Global Accelerator (GA) improves availability and performance by routing users through AWS’s private global network to the nearest healthy endpoint . Key idea: Users hit GA (static Anycast IPs). GA picks the best Region/endpoint automatically. Static Anycast IPs (2 fixed IP addresse...
Read more

🐳 Docker Filesystem Internals (AdvancEd)

Docker Filesystem Internals (Advanced) This post is for readers who already understand Docker basics and want to learn how Docker manages files internally. If you are new to Docker, read the beginner guide first. Containers Do NOT Have Their Own Disk A common misunderstanding is that Docker containers have their own disk. In reality: Containers do not have a separate physical filesystem. Docker creates a filesystem view using layers stored on the host machine. This layered filesystem is implemented using a Union Filesystem , most commonly OverlayFS on Linux. Docker Images Are Read-Only Layers A Docker image is made up of multiple read-only layers. Each instruction in a Dockerfile creates a new layer. FROM python:3.11 RUN pip install flask COPY app.py . This results in layers lik...
Read more

Understanding RabbitMQ Classic Mirrored Queues and Quorum Queues

Image
💚💚 💚 Understanding RabbitMQ Classic Mirrored Queues and Quorum  Queues 💚💚 💚   I understood RabbitMQ only after I understood why Classic Mirrored Queues were replaced . When I first started learning RabbitMQ, I kept hearing: Classic Queues Classic Mirrored Queues Quorum Queues Raft Consensus Everyone said:  "Use Quorum Queues." But I couldn't understand why. To understand Quorum Queues , I first had to understand how Classic Mirrored Queues worked.
Read more

🐳 Docker Tutorial for Beginners: Step-by-Step with a Simple Example

Docker Explained Using a Very Simple App (Beginner Friendly) Goal: Build a tiny Python app, run it normally, then run the same app using Docker, and understand why Docker helps. Step 1: Create a Very Small Python App Create a file named app.py : print("Hello I am running inside Docker!") Step 2: Run the App WITHOUT Docker Run this on your machine: python app.py Common real-life problems: Python not installed, wrong Python version, missing dependencies, different OS behavior this is where the “works on my machine” issue starts. Step 3: Introduce Docker Instead of relying on your system, Docker lets you package the runtime (Python) together with your app. We’ll do that using a Dockerfile . Step 4: Create a Dockerfile Create a file named Dockerfile (no extension): FROM python:3.11 WORKDIR /app COPY app.py . CMD ["py...
Read more

☸️What’s Inside EKS? A Beginner’s Guide to Its Core Components

Image
 EKS (Elastic Kubernetes Service) EKS is a managed service by AWS that runs Kubernetes for you. It takes care of setting up and managing the Kubernetes control plane so you can focus on running your applications. Kubernetes  Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications. It helps run apps reliably across a cluster of machines, automatically handling scheduling, scaling, updates, and recovery. Pod A Pod is the smallest unit in Kubernetes. It can hold one or more containers that share the same network and storage. All containers in a pod are scheduled and managed together. Container A Container is a lightweight, standalone package that includes everything an app needs to run — code, runtime, libraries, and dependencies. It ensures the app runs the same no matter where it’s deployed. 👀1 pod can have multiple containers but 1 container cannot belongs to multiple pods.
Read more

☸️ Kubernetes Taints and Tolerations(with Node Affinity)

Kubernetes Taints and Tolerations(with Node Affinity + EKS Upgrade Context) Kubernetes scheduling is the process of deciding which node runs a given Pod . This page explains how Kubernetes keeps Pods away from “inappropriate” nodes using taints (node-side restrictions) and tolerations (pod-side permissions), and how that relates to node affinity (pod-side attraction). 1) Core idea: attraction vs repulsion Node affinity (attraction) Node affinity is a property of Pods that attracts them to a set of nodes. You can express it as: Hard requirement (must match), or Preference (nice to have). Taints (repulsion) Taints are the opposite of node affinity. They allow a node to repel a set of Pods. Tolerations (permission) Tolerations are applied to Pods. A toleration allows the scheduler to schedule Pods onto nodes with matching taints. Important...
Read more

AWS Load Balancer Controller Upgrade Guide: v2.x to v3.3

Image
AWS Load Balancer Controller Upgrade Guide: v2.x to v3.3 What is AWS Load Balancer Controller? AWS Load Balancer Controller is a Kubernetes controller that integrates Amazon EKS with AWS Elastic Load Balancing services. It watches Kubernetes resources and automatically creates or updates AWS load balancing resources based on the desired state defined inside the cluster. Instead of manually creating load balancers, listeners, target groups, and security group rules in AWS, teams can define Kubernetes resources such as Ingresses and Services. The controller then reconciles those resources with AWS. What Does It Manage? AWS Load Balancer Controller commonly manages: Application Load Balancers (ALB) Network Load Balancers (NLB) Listeners and listener rules Target groups Target registration and deregistration Security group rules Ingress resources TargetGroupBinding resources Beca...
Read more

🐳 Build a Tiny Flask Web App in Docker (with Ports)

docker run -p 8080:5000 flask-hello Then open in your browser: http://localhost:8080 Step 6: Stop the container Press Ctrl + C in the terminal. Quick Troubleshooting Browser says “This site can’t be reached”: Make sure the container is running and you used the -p port mapping. Flask runs but not accessible: Confirm Flask is running with host="0.0.0.0" inside Docker (not 127.0.0.1 ). Port already in use: Try using a different host port, for example: docker run -p 9090:5000 flask-hello Next Post Ideas Docker image layers and caching (explained simply) Docker volumes : keeping data after a container stops Running the same container in Kubernetes (Pod basics)
Read more

☁️ Amazon S3 Explained: More Than Just Object Storage

Amazon S3 Explained: More Than Just Object Storage When people hear Amazon S3 , they usually think it is only used to store files like images, logs, backups, or documents. That is true, but S3 has grown into much more than simple object storage. Today, S3 can be used for object storage, file-like access, immutable backups, lifecycle management, static websites, event-driven workflows, analytics, and data lake storage. In this post, let us understand what S3 is and how it is used in real-world AWS environments. 1. What is Amazon S3? Amazon S3 stands for Simple Storage Service . It is mainly an object storage service . In S3, data is stored as objects inside buckets. Bucket → Object Each object usually contains: The actual data Metadata A unique object key Example: Bucket name: my-app-logs Object key: logs/2026/app.log Even though this looks like a folder path, S3 does not work like a traditional filesystem internally. The folder-like str...
Read more

☸️ CoreDNS and AWS VPC CNI in EKS

CoreDNS vs AWS VPC CNI in EKS: What Each Component Actually Does When working with Amazon EKS , two important components help your Kubernetes cluster communicate properly: CoreDNS and AWS VPC CNI . Both are part of the networking story, but they do very different jobs. CoreDNS helps pods find services by name, while AWS VPC CNI gives pods network connectivity using IP addresses from your AWS VPC. In simple words: CoreDNS answers: “What IP address belongs to this service name?” AWS VPC CNI answers: “How does traffic reach that pod over the network?” 1. What is CoreDNS? CoreDNS is the DNS server used inside Kubernetes. Its main job is to resolve Kubernetes service names into IP addresses. For example, if one pod wants to call another service using a name like: user-service.default.svc.cluster.local CoreDNS resolves that service name into the correct Kubernetes service IP. What CoreDNS Does Resolves Kubernetes service names Supports pod-to...
Read more