HTTP Connection Limits

The default connection limits configured by the S3 client are set to the highest possible value (999 per host and 999 total active connections). This is to avoid any possible bottleneck due to client-imposed connection limits.

Changing Apache HTTP Client Connection Limits

The default HTTP implementation used by the S3 client is the Apache HTTP Client. The S3 client sets the maximum connection limits to the highest allowed by the Apache client (999). You can set these limits to a lower value with the following configuration:

// configure apache connection manager
PoolingClientConnectionManager connectionManager = new org.apache.http.impl.conn.PoolingClientConnectionManager();

// limit active connections per destination host for this client instance
// if using the smart-client, this is the limit per node, otherwise this is the max connections allowed to ECS
connectionManager.setDefaultMaxPerRoute(connectionLimitPerHost);

// if using the smart-client, this is the max total connections to ECS,
// otherwise this is semantically the same as (and should match) the previous setting
connectionManager.setMaxTotal(connectionLimitTotal);

// set connection manager property in S3Config (this will get passed down to the Apache client)
s3Config.setProperty(ApacheHttpClient4Config.PROPERTY_CONNECTION_MANAGER, connectionManager);

Lowering the limits is useful if you have a distributed application with many processes and many threads, and want to avoid impacting other applications connecting to the same ECS cluster by limiting the active connections your application consumes.

Changing HttpURLConnection Limits

Note that if you specify the HttpUrlConnection client, the above configuration does not apply because connection limits are set in system properties. The default connection limit is 5 connections per destination, and this setting is not altered by the S3 client. If you use the HttpURLConnection handler, it is highly recommended that you increase this value or you may see bottlenecks in multithreaded applications that share a client instance.