Troubleshooting Failed Health Checks for Load Balancers in Amazon EKS: A Comprehensive Guide

Updated at: 28/06/2025

Amazon EKS load balancers are crucial for application availability. Failed health checks can cripple your services. This comprehensive guide provides a step-by-step approach to diagnosing and resolving these issues. We'll cover common causes, from misconfigured probes to underlying application problems, offering practical solutions to get your EKS cluster back online swiftly and reliably. Let's dive in!

Step-by-Step Instructions

  1. Check Pod and Container Status

    • Check the status of the application container in the pod. If the container isn't running, the load balancer health check won't be answered and will fail.
    Check the status of the application container in the pod. If the container isn't running, the load balancer health check won't be answered and will fail.
    Check Pod and Container Status

  2. Verify Service and Pod Labels

    • Check the pod and service label selectors. Ensure your Kubernetes service is using the correct port and labels match between the service and pods.
    Check the pod and service label selectors. Ensure your Kubernetes service is using the correct port and labels match between the service and pods.
    Verify Service and Pod Labels

  3. Inspect Endpoints for Missing Entries

    • Check for missing endpoints. The Kubernetes controller continuously scans for pods matching its selector and updates endpoints. Incorrect labels lead to missing endpoints.
    Check for missing endpoints. The Kubernetes controller continuously scans for pods matching its selector and updates endpoints. Incorrect labels lead to missing endpoints.
    Inspect Endpoints for Missing Entries

  4. Examine Service Traffic Policy and Security Groups

    • Check the service traffic policy and cluster security groups. Ensure the `spec.externalTrafficPolicy` is set to `Cluster` (not `Local`), and that security groups allow traffic between node groups.
    Check the service traffic policy and cluster security groups. Ensure the `spec.externalTrafficPolicy` is set to `Cluster` (not `Local`), and that security groups allow traffic between node groups.
    Examine Service Traffic Policy and Security Groups

  5. Confirm Target Port Configuration

    • Verify that your service is configured for the correct target port. The target port must match the container port.
    Verify that your service is configured for the correct target port. The target port must match the container port.
    Confirm Target Port Configuration

  6. Validate AWS Load Balancer Controller Permissions

    • Verify that your AWS load balancer controller has the correct permissions to update security groups and allow traffic.
    Verify that your AWS load balancer controller has the correct permissions to update security groups and allow traffic.
    Validate AWS Load Balancer Controller Permissions

  7. Review Ingress and Service Annotations

    • Check the ingress annotations and Kubernetes service annotations for issues specific to your application load balancer or network load balancer.
    Check the ingress annotations and Kubernetes service annotations for issues specific to your application load balancer or network load balancer.
    Review Ingress and Service Annotations

  8. Perform Manual Health Check

    • Manually test a health check. Use a test pod to check the health check path and HTTP response status code. For TCP checks, use `netcat`.
    Manually test a health check. Use a test pod to check the health check path and HTTP response status code. For TCP checks, use `netcat`.
    Perform Manual Health Check

  9. Investigate Network Connectivity

    • Check the networking. Verify communication between node groups, network ACLs, route tables, and that the kube-proxy is running correctly.
    Check the networking. Verify communication between node groups, network ACLs, route tables, and that the kube-proxy is running correctly.
    Investigate Network Connectivity
[RelatedPost]

Tips

  • If HTTP response status code is not 200, adjust your health check path or use ingress annotations to modify the expected response code range.
  • Restart the kube-proxy to force it to recheck and update iptables rules if there are issues.

Common Mistakes to Avoid

1. Incorrect Health Check Configuration

Reason: The health check path, protocol, or port specified in the load balancer configuration doesn't match the application's actual endpoint or availability.
Solution: Verify the health check settings against the application's actual listening port and endpoint, ensuring they precisely align.

2. Application Not Ready During Health Check

Reason: The application might be slow to start or experience delays in initializing, causing it to fail the initial health checks before it's fully operational.
Solution: Implement a readiness probe in your application's deployment that ensures the application is fully initialized and ready to handle requests before marking it healthy.

FAQs

My load balancer health checks keep failing, but my application seems to be running fine inside the pods. What could be wrong?
This often points to a mismatch between your health check configuration and your application's readiness. Ensure your health check probe (HTTP, TCP, or EXEC) targets the correct port and path, and that your application responds appropriately within the probe's timeout. Check your application logs for errors that might only be visible during the health check.
I've updated my application, and now my load balancer health checks are failing. What steps should I take?
After an application update, verify the updated containers are correctly responding to the health check probes. Check your deployment rollout strategy (e.g., rolling update) to ensure a smooth transition. If the issue persists, roll back the deployment to your previous stable version while debugging the health check failures in the new version.