Health Checks

A health check evaluates a component against an explicit definition of acceptable operation. A TCP connection can establish that a port accepts connections, while an HTTP request can verify that an application responds with an expected status. A functional check might exercise a representative transaction. These checks provide different levels of confidence: a running process or successful connection does not prove that an application can serve useful requests.

Health signals should match the decision they control. A liveness check asks whether restarting a component could restore operation. A readiness check asks whether it should receive new work. A startup check allows initialization to finish before other probes enforce their normal limits. For example, an application loading a large cache might be alive but not ready. Restarting it repeatedly during initialization prevents recovery rather than helping it.

Load balancers use health checks to identify backends eligible for traffic. Engineers configure the protocol, port, request path, timeout, interval, and success criteria, along with failure and recovery thresholds. When a backend becomes unhealthy, routing can shift to healthy capacity while another controller repairs or replaces the failed component. Removing a backend from routing does not itself restart it, create replacement capacity, or preserve in-flight requests. Graceful draining, retries, and sufficient spare capacity remain necessary.

Do not assume that an unhealthy status always blocks traffic. Some load balancers deliberately fail open when every backend is unhealthy. For example, an Application Load Balancer routes to its unhealthy targets when a target group contains only unhealthy registered targets across its enabled Availability Zones. Understand the implementation's all-unhealthy behavior and maintain redundancy across failure domains rather than treating health checks as a substitute for available capacity.

Automated replacement requires an explicit connection between the health signal and the recovery controller. An EC2 instance can pass infrastructure status checks while its application fails. An Auto Scaling group must be configured to use Elastic Load Balancing health results if those failures should trigger replacement. Its health check grace period protects newly launched instances from premature health-based replacement; it does not make an unready application eligible for load-balancer traffic.

Container platforms also expose distinct failure signals. In Kubernetes, readiness failures remove Pods from normal Service traffic, while repeated liveness failures can restart containers. Startup probes defer liveness and readiness probing until startup succeeds. In Amazon ECS, inspect task-definition health checks, essential-container exits, load-balancer target health, service events, and the service health check grace period. A service repeatedly replacing tasks may be reacting correctly to a bad application build, or incorrectly to an unsuitable probe.

During deployment troubleshooting, compare failing and working versions, then verify the probe path, listening address, container port mappings, security groups, network ACLs, response codes, and initialization time. An HTTP redirect, authentication requirement, or virtual-host mismatch can fail a probe even when ordinary user requests succeed. Blue/green and rolling deployments should require new capacity to become healthy before removing old capacity. For serverless deployments, use validation invocations, synthetic requests, and error or latency alarms rather than assuming there is a conventional instance health endpoint.

Detection speed trades off against stability. Short intervals and low failure thresholds detect faults quickly but increase probe traffic and sensitivity to transient delays. Deep checks can expose broken dependencies, but making every instance fail liveness during a shared database outage can create a restart storm. Keep checks inexpensive, bound dependency calls with timeouts, and choose whether dependency failure should affect readiness, liveness, or only an alarm.

Finally, measure end-to-end recovery rather than only probe latency. Detection, routing changes, provisioning, initialization, and data recovery all contribute to the recovery time objective. Health checks do not establish the recovery point objective or guarantee data durability; replication, backups, and recovery procedures determine acceptable data loss. Correlate probe results with application logs, platform events, metrics, and recent changes before selecting a recovery action.