Connection Pool Timeouts and HTTP Client Configuration

The SimpleClientHttpRequestFactory used in the previous section creates a new TCP connection for every HTTP request. In production, you use a connection pool. Connection pools add a third timeout surface that catches teams by surprise: the connection acquisition timeout.

The Connection Pool Timeout Surface

With a pooled HTTP client, an outgoing request goes through four phases, each with its own timeout:

1. Connection acquisition. Wait for an idle connection in the pool. If all connections are in use, wait for one to become available. Timeout: connectionRequestTimeout.

2. Connection establishment. If no idle connection exists and the pool has capacity, create a new TCP connection. Timeout: connectTimeout.

3. TLS handshake. If HTTPS, negotiate the TLS session. Included in the connection timeout.

4. Data transfer. Send the request, wait for the response. Timeout: socketTimeout (read timeout).

// PRODUCTION - Apache HttpClient 5 with Spring Boot RestClient
@Configuration
public class HttpClientConfig {

    @Bean
    public RestClient fraudDetectionRestClient() {
        PoolingHttpClientConnectionManager connectionManager =
                PoolingHttpClientConnectionManagerBuilder.create()
                        .setMaxConnPerRoute(20)  // Max connections to fraud-detection host
                        .setMaxConnTotal(50)     // Max connections across all hosts
                        .build();

        CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(connectionManager)
                .setDefaultRequestConfig(RequestConfig.custom()
                        .setConnectionRequestTimeout(Timeout.ofMilliseconds(500))
                        // How long to wait for a connection from the pool.
                        // If the pool is exhausted and all connections are in use,
                        // fail fast. Do not queue indefinitely.

                        .setConnectTimeout(Timeout.ofSeconds(1))
                        // How long to wait for the TCP handshake.
                        // If fraud-detection is unreachable, know quickly.

                        .setResponseTimeout(Timeout.ofSeconds(2))
                        // How long to wait for the response after sending the request.
                        // This is the timeout that protects against slow responses.
                        .build())
                .evictIdleConnections(TimeValue.ofSeconds(30))
                // Close connections that have been idle for 30 seconds.
                // Prevents sending requests on stale connections that the
                // server has already closed.
                .build();

        return RestClient.builder()
                .baseUrl("http://fraud-detection:8080")
                .requestFactory(new HttpComponentsClientHttpRequestFactory(httpClient))
                .build();
    }
}

Connection Pool Exhaustion

Connection pool exhaustion is a distinct failure mode from thread pool exhaustion, and it has the same effect.

If the fraud detection service responds slowly, connections are held longer. If maxConnPerRoute is 20 and each connection is held for 5 seconds instead of 50 milliseconds, the pool supports 4 requests per second instead of 400. At 100 requests per second, the pool is exhausted in 200 milliseconds.

Without connectionRequestTimeout, the threads waiting for a connection from the pool will block indefinitely. With it set to 500 milliseconds, they get a ConnectionRequestTimeoutException after half a second and can return an error to the caller. The thread is freed. The thread pool is protected.

// PRODUCTION - Handling connection pool exhaustion in the client
@Component
public class FraudDetectionClient {

    private final RestClient restClient;
    private final MeterRegistry registry;

    public FraudDetectionClient(
            @Qualifier("fraudDetectionRestClient") RestClient restClient,
            MeterRegistry registry) {
        this.restClient = restClient;
        this.registry = registry;
    }

    public FraudScore score(PaymentRequest request) {
        try {
            return restClient.post()
                    .uri("/api/fraud/score")
                    .body(request)
                    .retrieve()
                    .body(FraudScore.class);
        } catch (ResourceAccessException e) {
            if (e.getCause() instanceof ConnectionRequestTimeoutException) {
                // Connection pool exhausted - fraud service may be degraded
                registry.counter("fraud.client.pool.exhausted").increment();
                throw new FraudServiceUnavailableException(
                        "Connection pool exhausted for fraud-detection", e);
            }
            throw e;
        }
    }
}

The fraud.client.pool.exhausted counter is a leading indicator. When this metric starts climbing, the fraud detection service is degrading, and your connection pool is bearing the cost. This metric should trigger the same alert as elevated fraud detection latency.

Sizing Connection Pools

Use Little’s Law again:

connections_needed = request_rate * average_response_time

For fraud detection under normal conditions:

• Request rate: 100/second
• Average response time: 50ms (0.05 seconds)
• Connections needed: 100 * 0.05 = 5

Setting maxConnPerRoute to 20 gives a 4x safety margin. Under degraded conditions with 500ms response times, you need 50 connections. The pool at 20 will be exhausted, and the connectionRequestTimeout will reject excess requests fast.

The pool size is a lever. Making it larger absorbs more degradation before rejecting requests. Making it smaller rejects sooner, which protects your thread pool but fails more requests. The correct size depends on how much degradation you want to absorb silently versus how quickly you want to signal a problem. For the transaction platform, absorbing mild degradation (2-3x normal latency) while failing fast on severe degradation (10x or more) is the right tradeoff.