Help Center

Connections randomly failing on Hardened Ubuntu instances

Problem:

Services hosted on a Hardened Ubuntu instance intermittently report "connection refused" messages to clients.


Symptoms:

On a web server, this can exhibit as an intermittent "Connection Refused" error.  On the server itself, you will see dmesg logs similar to the following:

[354569.897944] [UFW LIMIT BLOCK] IN=eth0 OUT= MAC=06:29:84:XX:b2:XX:06:5c:3c:e1:XX:28:08:00
SRC=4.3.2.1 DST=1.2.3.4 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=TCP SPT=56201
DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0


Cause:

By default, the UFW process limits clients to 3 connections per minute on average.  This is fine (and is intended) for most low-bandwidth tasks, such as administering the server with SSH, but for more advanced web applications this will quickly be exhausted, leading to UFW LIMIT BLOCK events as exhibited above.


Solution:

To see what the current setting is, run the following command:

$ sudo ufw show raw | grep LIMIT
  0   0 LOG   all  ...  limit: avg 3/min burst 5 LOG ... "[UFW LIMIT BLOCK] "
  0   0 LOG   all  ... limit: avg 3/min burst 5 LOG ... "[UFW LIMIT BLOCK] "

To discover which services are being rate limited:
$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] Anywhere on eth0           LIMIT IN    Anywhere
[ 2] 22/tcp                     ALLOW IN    Anywhere
[ 3] 443/tcp                    LIMIT IN    Anywhere
[ 4] 25/tcp                     ALLOW IN    Anywhere
[ 5] Anywhere (v6) on eth0      LIMIT IN    Anywhere (v6)
[ 6] 22/tcp (v6)                ALLOW IN    Anywhere (v6)
[ 7] 443/tcp (v6)               LIMIT IN    Anywhere (v6)
[ 8] 25/tcp (v6)                ALLOW IN    Anywhere (v6)

To remove rate limiting for a particular service:

$ sudo ufw allow https
Rule updated
Rule updated (v6)
$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] Anywhere on eth0           LIMIT IN    Anywhere
[ 2] 22/tcp                     ALLOW IN    Anywhere
[ 3] 443/tcp                    ALLOW IN    Anywhere
[ 4] 25/tcp                     ALLOW IN    Anywhere
[ 5] Anywhere (v6) on eth0      LIMIT IN    Anywhere (v6)
[ 6] 22/tcp (v6)                ALLOW IN    Anywhere (v6)
[ 7] 443/tcp (v6)               ALLOW IN    Anywhere (v6)
[ 8] 25/tcp (v6)                ALLOW IN    Anywhere (v6)


To enable rate limiting on a particular service:

$ sudo ufw limit ssh
Rule updated
Rule updated (v6)
$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] Anywhere on eth0           LIMIT IN    Anywhere
[ 2] 22/tcp                     LIMIT IN    Anywhere
[ 3] 443/tcp                    ALLOW IN    Anywhere
[ 4] 25/tcp                     ALLOW IN    Anywhere
[ 5] Anywhere (v6) on eth0      LIMIT IN    Anywhere (v6)
[ 6] 22/tcp (v6)                LIMIT IN    Anywhere (v6)
[ 7] 443/tcp (v6)               ALLOW IN    Anywhere (v6)
[ 8] 25/tcp (v6)                ALLOW IN    Anywhere (v6)


The limits themselves can be adjusted by editing the /etc/ufw/user.rules file, at the bottom of the file:

$ sudo vi /etc/ufw/user.rules
...
### RATE LIMITING ###
-A ufw-user-limit -m limit --limit 3/minute -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT
-A ufw-user-limit-accept -j ACCEPT
### END RATE LIMITING ###
COMMIT

In our future Ubuntu AMI releases, we may relax these policies somewhat to provide you with a more flexible base image.  We will be sure to do so consistent with the STIG guidelines.

Did this solve your problem?