How to fix Elastic Load Balancer 408 Errors in Elastic Beanstalk
Recently, when viewing Apache logs for an application deployed in Elastic Beanstalk (EB) behind an Elastic Load Balancer (ELB), I noticed a frequently recurring 408 error being triggered by an internal connection.
The 408 error was caused because of a connection draining setting being enabled which in turn caused EB to think that requests were returning a 4xx error, thus triggering numerous alarms and actions.
Here’s a sample of what the logs looked like
172.30.2.169 (-) - - [03/May/2016:22:08:51 +0000] "GET /status HTTP/1.1" 200 7 "-" "ELB-HealthChecker/1.0"
172.30.0.244 (-) - - [03/May/2016:22:08:52 +0000] "GET /status HTTP/1.1" 200 7 "-" "ELB-HealthChecker/1.0"
172.30.2.169 (-) - - [03/May/2016:22:09:13 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:13 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
172.30.2.169 (-) - - [03/May/2016:22:09:17 +0000] "-" 408 - "-" "-"
Here is what I did to resolve the issue.
Create a new file in your Elastic Beanstalk .ebextensions
directory called 05-408fix.config
with the following contents:
files:
"/etc/httpd/conf.d/mod_reqtimeout.conf" :
mode: "000644"
owner: root
group: root
content : |
<IfModule reqtimeout_module>
RequestReadTimeout header=XXXX,MinRate=500 body=XXXX,MinRate=500
</IfModule>
TimeOut XXXX
In the above configuration, replace XXXX
with the value of your ELB Connection Draining Timeout plus 2 seconds.
For instance, if my ELB Connection Draining Timeout value is set to 60
, I would set the value of XXXX
in the above configuration to 62
.