create together.

Recently, when viewing Apache logs for an application deployed in Elastic Beanstalk (EB) behind an Elastic Load Balancer (ELB), I noticed a frequently recurring 408 error being triggered by an internal connection.

The 408 error was caused because of a connection draining setting being enabled which in turn caused EB to think that requests were returning a 4xx error, thus triggering numerous alarms and actions.

Here’s a sample of what the logs looked like

172.30.2.169 (-) - - [03/May/2016:22:08:51 +0000] "GET /status HTTP/1.1" 200 7 "-" "ELB-HealthChecker/1.0"
172.30.0.244 (-) - - [03/May/2016:22:08:52 +0000] "GET /status HTTP/1.1" 200 7 "-" "ELB-HealthChecker/1.0"
172.30.2.169 (-) - - [03/May/2016:22:09:13 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:13 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"

Here is what I did to resolve the issue.

Create a new file in your Elastic Beanstalk .ebextensions directory called 05-408fix.config with the following contents:

files:
 "/etc/httpd/conf.d/mod_reqtimeout.conf" :
   mode: "000644"
   owner: root
   group: root
   content : |
      <IfModule reqtimeout_module>
        RequestReadTimeout header=XXXX,MinRate=500 body=XXXX,MinRate=500
      </IfModule>
      TimeOut XXXX

In the above configuration, replace XXXX with the value of your ELB Connection Draining Timeout plus 2 seconds.

For instance, if my ELB Connection Draining Timeout value is set to 60, I would set the value of XXXX in the above configuration to 62.