Hello! This is a mystery where your goal is to solve a debugging problem! You'll collect clues, interpret evidence, and ultimately solve the Case of the Connection Timeout!

In the sidebar, you'll notice a "What you know" list. As you collect more clues, the list will update with what you've learned.

[[Start->Landing]]

You need to access a service on a weird port, so you type in machine.corp.yourcompany.com:1234 in your browser. You're very sure that you have the right address and port number.

It says that it's loading for about 60 seconds (so long!!!), and then you get this error message:

[[Click "Try again"->Reload page]] [[Try using curl->Try curl]] [[Try another website->Try another website]] [[Try tcpdump->Try tcpdump]]

The exact same thing happens. You wait for 60 seconds, and get the same error message. You try again one more time just for good measure. Same result. <>

[[Go back->Landing]]

You go check your email to make sure it's working. You get distracted briefly by an email from your boss and spend ten minutes replying to it, but your internet is definitely working. <>

[[Go back->Landing]]

You run:

 $ curl machine.corp.yourcompany.com:1234

After waiting for 2 minutes, you finally get this error message:

curl: (28) Failed to connect to machine.corp.yourcompany.com port
1234: Connection timed out

Interesting, that's the same message as before.

[[Go back->Landing]]

You run: sudo tcpdump -i any port 1234

This is the output you see:

 $ sudo tcpdump -i any port 1234 

tcpdump: data link type LINUX_SLL2 
 tcpdump: verbose output
suppressed, use -v[v]... for full protocol decode 
 listening on
any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144
bytes

[[Go back->Landing]] [[Reload the page in your browser->Try tcpdump 2]]

Here's the tcpdump output you see:

 17:33:01.563903 lo In IP
192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win
65495, options [mss 65495,sackOK,TS val 4247063724 ecr 0,nop,wscale
7], length 0 
 17:33:02.582918 lo In IP 192.168.1.121.38002 >
12.3.2.5.1234: Flags [S], seq 596753197, win 65495, options [mss
65495,sackOK,TS val 4247064743 ecr 0,nop,wscale 7], length 0 

17:33:04.662878 lo In IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags
[S], seq 596753197, win 65495, options [mss 65495,sackOK,TS val
4247066823 ecr 0,nop,wscale 7], length 0 
 17:33:08.719754 lo In
IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win
65495, options [mss 65495,sackOK,TS val 4247070880 ecr 0,nop,wscale
7], length 0

These packets are all SYN packets. A SYN packet is the first packet sent in any TCP connection. The way you can tell they're SYN packets is from the Flags [S] in this tcpdump line:

17:33:01.563903 lo In IP 192.168.1.121.38002 > 12.3.2.5.1234: Flags [S], seq 596753197, win 65495

So now we know that there are a bunch of SYN packets being sent, but not any other packets.

The weird thing about this is that usually there's just 1 SYN packet sent at the beginning of each TCP connection, but here there are 4, being sent over and over again.

<> <>

SYN packets being retried usually happens because something (usually a firewall) is blocking the SYN packets from being sent, so there's no reply.

But what could it be?

[[Think about what could be blocking the packets->Packet blocking]]

Your server is hosted in AWS. A couple of words you remember hearing related to firewalls are "security groups" and "iptables".

[[SSH into the machine->SSH]] [[Go to the AWS console-> AWS]]

You have SSH access to the machine, so you run

 ssh machine.corp.yourcompany.com

And you're in. What do you want to do next?

[[Try using curl->Curl remote]] [[Look at the iptables rules->iptables]] [[Go back->Packet blocking]]

You go to the AWS console, and look at the security groups. You find the security groups rule definition for that instance, and it looks like this. The port number you were using was 1234.

Do you think this is the cause of your problem?

[[Yes->AWS 2]] [[No->AWS 2]]

These rules are saying that only traffic on ports 22, 80, and 443 is allowed. So your request on port 1234 isn't allowed! It looks like AWS is the problem!

<> <> <>

You double check with your security team to make sure it's safe to allow traffic on that port, and you add a new rule to allow traffic on port 1234.

[[Try the webpage again->AWS final]]

It works! Everything is perfect.

[[Epilogue->Epilogue]]

Connection timeouts aren't always caused by firewall problems, but it's a very common cause! Anytime I see a very very long wait followed by a connection timeout, I check if I'm having a firewall problem by using tcpdump to look for the pattern of SYN packets being retried over and over and over and over again.

To see if this problem is caused by the iptables firewall, you need to print out all iptables rules.

You look up how to print out all iptables rules, and you see that you can do it by running sudo iptables-save.

Here's what you see:

 $ sudo iptables-save 
 # Generated by iptables-save v1.8.7
on Wed Apr 14 17:57:11 2021 
 *filter 
 :INPUT ACCEPT
[33501:125953140] 
 :FORWARD DROP [0:0] 
 :OUTPUT ACCEPT
[23862:4282122] 
 :DOCKER - [0:0] 
 :DOCKER-ISOLATION-STAGE-1 -
[0:0] 
 :DOCKER-ISOLATION-STAGE-2 - [0:0] 
 :DOCKER-USER -
[0:0] 
 -A FORWARD -j DOCKER-USER 
 -A FORWARD -j
DOCKER-ISOLATION-STAGE-1 
 -A FORWARD -o docker0 -m conntrack
--ctstate RELATED,ESTABLISHED -j ACCEPT 
 -A FORWARD -o docker0 -j
DOCKER 
 -A FORWARD -i docker0 ! -o docker0 -j ACCEPT 
 -A
FORWARD -i docker0 -o docker0 -j ACCEPT 
 -A
DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j
DOCKER-ISOLATION-STAGE-2 
 -A DOCKER-ISOLATION-STAGE-1 -j RETURN

 -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP 
 -A
DOCKER-ISOLATION-STAGE-2 -j RETURN 
 -A DOCKER-USER -j RETURN 

COMMIT 
 # Completed on Wed Apr 14 17:57:11 2021 
 # Generated
by iptables-save v1.8.7 on Wed Apr 14 17:57:11 2021 
 *nat 

:PREROUTING ACCEPT [17006:1104893] 
 :INPUT ACCEPT [16958:1088669]

 :OUTPUT ACCEPT [145378:14161750] 
 :POSTROUTING ACCEPT
[144916:14119410] 
 :DOCKER - [0:0] 
 -A PREROUTING -m addrtype
--dst-type LOCAL -j DOCKER 
 -A OUTPUT ! -d 127.0.0.0/8 -m
addrtype --dst-type LOCAL -j DOCKER 
 -A POSTROUTING -s 172.17.0.0
/16 ! -o docker0 -j MASQUERADE 
 -A DOCKER -i docker0 -j RETURN

 COMMIT 
 # Completed on Wed Apr 14 17:57:11 2021

Do you want to keep investigating iptables, or go back?

[[Keep investigating ->Iptables 2]] [[Go back->SSH]]

That was a lot of iptables rules. How can we figure out if they're what's causing our problem? <><>

Here's what we're going to do. iptables has a way to check exactly which individual iptables rules are being triggered called counters, so we're going to ask iptables!

 $ sudo
iptables-save -c 
 # Generated by iptables-save v1.8.7 on Thu Apr
15 12:04:21 2021 
 *filter 
 :INPUT ACCEPT [452:2223662] 

:FORWARD DROP [0:0] 
 :OUTPUT ACCEPT [373:80464] 
 :DOCKER -
[0:0] 
 :DOCKER-ISOLATION-STAGE-1 - [0:0] 

:DOCKER-ISOLATION-STAGE-2 - [0:0] 
 :DOCKER-USER - [0:0] 

[0:0] -A FORWARD -j DOCKER-USER 
 [0:0] -A FORWARD -j
DOCKER-ISOLATION-STAGE-1 
 [0:0] -A FORWARD -o docker0 -m
conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
 [0:0] -A
FORWARD -o docker0 -j DOCKER 
 [0:0] -A FORWARD -i docker0 ! -o
docker0 -j ACCEPT 
 [0:0] -A FORWARD -i docker0 -o docker0 -j
ACCEPT 
 [0:0] -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0
-j DOCKER-ISOLATION-STAGE-2 
 [0:0] -A DOCKER-ISOLATION-STAGE-1 -j
RETURN 
 [0:0] -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP 

[0:0] -A DOCKER-ISOLATION-STAGE-2 -j RETURN 
 [0:0] -A DOCKER-USER
-j RETURN 
 COMMIT 
 # Completed on Thu Apr 15 12:04:21 2021

 # Generated by iptables-save v1.8.7 on Thu Apr 15 12:04:21 2021

 *nat 
 :PREROUTING ACCEPT [17826:1154724] 
 :INPUT ACCEPT
[17776:1137824] 
 :OUTPUT ACCEPT [151426:14790277] 

:POSTROUTING ACCEPT [150964:14747937] 
 :DOCKER - [0:0] 

[21:1566] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER 

[119:5540] -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j
DOCKER 
 [8400:613089] -A POSTROUTING -s 172.17.0.0/16 ! -o
docker0 -j MASQUERADE 
 [0:0] -A DOCKER -i docker0 -j RETURN 

COMMIT

What do you think this means? Have any of these rules ever been used?

[[Yes->Iptables 4]] [[No->Iptables 4]]

Some of these rules have been used! iptables' output is pretty confusing at first, so let's look at this line from the output and break it down if you haven't looked at it before:

[21:1566] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

  The [21:1566] is the count of how many
times that rule has been triggered. 
 <> But we still don't know how we
could tell if these iptables rules are actually causing our problem.
iptables isn't just used for blocking traffic, it's used for lots of
things.
 <><>  Here's what we're going to do. The idea
is that if we reset all of iptables' counters, it'll be easier to
tell if they've been triggered by something we did, or by something
unrelated. 
  Reset all the iptables counters 
  Run
our failed request again 
  See if any of the rules get
triggered by that request 
 
 
  Let's get started on
that:  [[Reset the counters->Iptables 6]]You look up how to reset
iptables' counters, and you find this article. It
says to run sudo iptables -Z. So you do that, and then
you look at the counters again.  $ sudo iptables -Z 
 $ sudo
iptables-save -c # Generated by iptables-save v1.8.7 on Thu Apr 15
12:08:38 2021 
 *filter 
 :INPUT ACCEPT [452:2223662] 

:FORWARD DROP [0:0] 
 :OUTPUT ACCEPT [373:80464] 
 :DOCKER -
[0:0] 
 :DOCKER-ISOLATION-STAGE-1 - [0:0] 

:DOCKER-ISOLATION-STAGE-2 - [0:0] 
 :DOCKER-USER - [0:0] 

[0:0] -A FORWARD -j DOCKER-USER 
 [0:0] -A FORWARD -j
DOCKER-ISOLATION-STAGE-1 
 [0:0] -A FORWARD -o docker0 -m
conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
 [0:0] -A
FORWARD -o docker0 -j DOCKER 
 [0:0] -A FORWARD -i docker0 ! -o
docker0 -j ACCEPT 
 [0:0] -A FORWARD -i docker0 -o docker0 -j
ACCEPT 
 [0:0] -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0
-j DOCKER-ISOLATION-STAGE-2 
 [0:0] -A DOCKER-ISOLATION-STAGE-1 -j
RETURN 
 [0:0] -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP 

[0:0] -A DOCKER-ISOLATION-STAGE-2 -j RETURN 
 [0:0] -A DOCKER-USER
-j RETURN 
 COMMIT 
 # Completed on Thu Apr 15 12:04:21 2021

 # Generated by iptables-save v1.8.7 on Thu Apr 15 12:04:21 2021

 *nat 
 :PREROUTING ACCEPT [17826:1154724] 
 :INPUT ACCEPT
[17776:1137824] 
 :OUTPUT ACCEPT [151426:14790277] 

:POSTROUTING ACCEPT [150964:14747937] 
 :DOCKER - [0:0] 

[21:1566] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER 

[119:5540] -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j
DOCKER 
 [8400:613089] -A POSTROUTING -s 172.17.0.0/16 ! -o
docker0 -j MASQUERADE 
 [0:0] -A DOCKER -i docker0 -j RETURN 

COMMIT 
  Wait... nothing has changed! Some of the counters
still haven't been cleared. What's going on? <>
<>  What's going on is that iptables has
multiple "tables", and sudo iptables -Z actually only
clears the counters for the default table (the filter
table). This one isn't obvious at all -- I've gotten tripped up by
this many times and I always forget about it. 
  You look at
the output again, and see that the counters that aren't being cleared
are in the nat table 
 You run sudo iptables -t
nat -Z to clear the nat table.  $ sudo
iptables -t nat -Z 
 $ sudo iptables-save -c 
 # Generated by
iptables-save v1.8.7 on Thu Apr 15 12:26:28 2021 
 *filter 

:INPUT ACCEPT [10509:49406485] 
 :FORWARD DROP [0:0] 
 :OUTPUT
ACCEPT [8539:1872558] 
 :DOCKER - [0:0] 

:DOCKER-ISOLATION-STAGE-1 - [0:0] 
 :DOCKER-ISOLATION-STAGE-2 -
[0:0] 
 :DOCKER-USER - [0:0] 
 [0:0] -A FORWARD -j DOCKER-USER

 [0:0] -A FORWARD -j DOCKER-ISOLATION-STAGE-1 
 [0:0] -A
FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j
ACCEPT 
 [0:0] -A FORWARD -o docker0 -j DOCKER 
 [0:0] -A
FORWARD -i docker0 ! -o docker0 -j ACCEPT 
 [0:0] -A FORWARD -i
docker0 -o docker0 -j ACCEPT 
 [0:0] -A DOCKER-ISOLATION-STAGE-1
-i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 
 [0:0] -A
DOCKER-ISOLATION-STAGE-1 -j RETURN 
 [0:0] -A
DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP 
 [0:0] -A
DOCKER-ISOLATION-STAGE-2 -j RETURN 
 [0:0] -A DOCKER-USER -j
RETURN 
 COMMIT 
 # Completed on Thu Apr 15 12:26:28 2021 

# Generated by iptables-save v1.8.7 on Thu Apr 15 12:26:28 2021 

*nat 
 :PREROUTING ACCEPT [0:0] 
 :INPUT ACCEPT [0:0] 

:OUTPUT ACCEPT [0:0] 
 :POSTROUTING ACCEPT [0:0] 
 :DOCKER -
[0:0] 
 [0:0] -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

 [0:0] -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j
DOCKER 
 [0:0] -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j
MASQUERADE 
 [0:0] -A DOCKER -i docker0 -j RETURN 
 COMMIT 

# Completed on Thu Apr 15 12:26:28 2021  Great. Everything is
cleared. Now let's make a request to the service and see if it
triggers any of those counters. [[Make a request->Iptables 8]]You go
to machine.corp.yourcompany.com:1234 in your browser (it
fails again, just like last time), and then you go back to your SSH
session. You look at the counters one last time. Here's what you see.
 $ sudo iptables-save -c # Generated by iptables-save v1.8.7 on
Thu Apr 15 12:30:42 2021 
 *filter 
 :INPUT ACCEPT
[12550:58251661] 
 :FORWARD DROP [0:0] 
 :OUTPUT ACCEPT
[10155:2222169] 
 :DOCKER - [0:0] 
 :DOCKER-ISOLATION-STAGE-1 -
[0:0] 
 :DOCKER-ISOLATION-STAGE-2 - [0:0] 
 :DOCKER-USER -
[0:0] 
 [0:0] -A FORWARD -j DOCKER-USER 
 [0:0] -A FORWARD -j
DOCKER-ISOLATION-STAGE-1 
 [0:0] -A FORWARD -o docker0 -m
conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
 [0:0] -A
FORWARD -o docker0 -j DOCKER 
 [0:0] -A FORWARD -i docker0 ! -o
docker0 -j ACCEPT 
 [0:0] -A FORWARD -i docker0 -o docker0 -j
ACCEPT 
 [0:0] -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0
-j DOCKER-ISOLATION-STAGE-2 
 [0:0] -A DOCKER-ISOLATION-STAGE-1 -j
RETURN 
 [0:0] -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP 

[0:0] -A DOCKER-ISOLATION-STAGE-2 -j RETURN 
 [0:0] -A DOCKER-USER
-j RETURN 
 COMMIT 
 # Completed on Thu Apr 15 12:30:42 2021

 # Generated by iptables-save v1.8.7 on Thu Apr 15 12:30:42 2021

 *nat 
 :PREROUTING ACCEPT [13:741] 
 :INPUT ACCEPT
[13:741] 
 :OUTPUT ACCEPT [73:8565] 
 :POSTROUTING ACCEPT
[73:8565] 
 :DOCKER - [0:0] 
 [0:0] -A PREROUTING -m addrtype
--dst-type LOCAL -j DOCKER 
 [0:0] -A OUTPUT ! -d 127.0.0.0/8 -m
addrtype --dst-type LOCAL -j DOCKER 
 [0:0] -A POSTROUTING -s
172.17.0.0/16 ! -o docker0 -j MASQUERADE 
 [0:0] -A DOCKER -i
docker0 -j RETURN 
 COMMIT 
 # Completed on Thu Apr 15 12:30:42
2021 
  What does this mean? Is the problem caused by
iptables? [[Yes->Iptables final]] [[No->Iptables final]] All of
those counters are still zero, which means that none of the rules
were triggered. So it looks like the problem wasn't iptables at all.
It took a lot of work just to figure that out, but that's what
debugging's like sometimes! 
 <>  This procedure of clearing the counters, making the
failed request, and then checking the counters again can be really
helpful when debugging firewall issues and I've used it to find
problems with my iptables rules in the past. 
 Onto the next
potential culprit...
  [[Go back->Packet
blocking]] 
You run:  $ curl localhost:1234/status
{"status": "ok"} 
  You get a reply instantly and everything
looks good. 
 <>
 [[Go back->SSH]] 
<> <>   you've found
new information!! 
 <>  Adding
"<>" to your "What you know" list.
 
 <> <> <> <> <> <> <> 
<>  <> &
nbsp; <>  <><> <><> 
 
 <>  <>  <>
<> "<>".<> 
<> What you know: <> nothing yet! <>  <>  _name
 <> 
 <>

you've found new information!!