This article describes the TCP 3-way handshake and builds upon this knowledge to explain technically how port scanning works.
Transmission Control protocol (TCP) is a connection oriented protocol. It begins with a handshake and ends with a termination session. The first part of this article will give an overview of the handshake after which we'll dive deeper and see what the network traffic looks like. The last part describes what happens when a 3-way handshake does not complete in a normal fashion and how we can use this to perform port scanning on a system.
For quick reference, these are the contents of this article:
3-way handshake overview
Normal TCP traffic begins with a 3-way handshake. In this handshake, a client initiates the conversation by requesting to have a communication session with the server:
- The client first sends a Synchronization packet (SYN).
- If the server accepts, it responds with a Synchronization Acknowledgment (SYN-ACK) to let the client know it's open and ready for communication.
- The client responds with an Acknowledgment (ACK). Now the session begins and the socket is created.
Note that I use the terms client and server to clearly distinguish the party that initiates the handshake (client) from the one responding to that (server).
No data is sent during the 3-way handshake. Only after the connection has been established, data transfer takes place.
Sequence numbers
One important feature of this handshake is the synchronization of the TCP sequence numbers. In TCP each byte has a sequence number that is increased for every byte sent. This way both parties can keep track of the amount of bytes that were sent, allowing for reliable transfer on top of the unreliable IP-protocol. The sequence number is a topic by itself, but this simplified explanation is enough for the purpose of explaining the TCP 3-way handshake.
TCP packets
Let's dive deeper into the 3-way handshake and see what the network traffic looks like when this handshake takes place. I used Wireshark to capture the traffic when browsing to my own website, which is the basis for this explanation.
In this packet capture, we can clearly see the SYN, SYN-ACK and ACK packets involved in the 3-way handshake:
Note that the length of these packets is 0, since there is no actual data transmitted (it's only a handshake after all).
The source (Client) is my computer with IP address 192.168.1.202 and port 18987 and the destination (Server) is a Cloudflare server on 104.18.60.128 on port 443. I have Cloudflare in front of my website, so this makes sense. Now, let's take a closer look at what these individual packets look like.
SYN
Selecting the SYN-packet (the top one in the picture above) will display its details in Wireshark. We can see the Client is the Source with IP address 192.168.1.202 (me) and the Server is the destination with IP address 104.18.60.128 (Cloudflare). Expanding the Transmission Control Protocol header details we can see the SYN-flag is set:
This initial SYN-packet also attempts to synchronize sequence numbers. In the SYN-packet details, we can see on my side the Sequence number is 0.
SYN-ACK
Coming back from the server, we can see the Cloudflare server at 104.18.60.128 responds to me, the client at 192.168.1.202 with a SYN-ACK. Expanding the TCP header details we can see that both the SYN-flag and the Acknowledgment-flag are set:
Since the handshake attempts to synchronize sequence numbers, we see the Sequence number is set to 0 as expected. Note that the sequence number is marked as relative sequence number. Wireshark does this so it's easier to read. See this page in the Wireshark manual for more information.
ACK
The last packet in the 3-way handshake shows the client (192.168.1.202) sending the final acknowledgment to the server (104.18.60.128). In the TCP header we see that the Acknowledgment flag is set.
We can also see that the Sequence number is set to 1 and the data transfer can start.
Port scanning
The 3-way handshake as explained in the previous chapter, is based on a normal connection scenario. In that normal scenario, the server had a port exposed (i.e. port 443) and allowed connections to be made to that port.
Whenever the server does not respond or does not allow connections to be made to a port (because of a firewall for example), the 3-way handshake is not completed in the normal fashion. Analyzing how the ports of a server respond to certain TCP header flags is the basis for port scanning.
In my lab setup I have two virtual machines:
- Kali Linux on IP 192.168.1.128 for scanning
- Windows 7 on IP 192.168.1.130 as a server.
Nmap
I use Nmap for scanning, which comes pre-installed on Kali Linux. Nmap is a free and open source utility for network discover and security auditing. This awesome tool is jam-packed with features, but for this purpose I use its port scanning functionality. More specifically, I'll use it to perform a TCP SYN-scan.
SYN-Scan
SYN-scan is the default for Nmap port scans and is often referred to as half-open scanning, because you don't open a full TCP connection. You send a SYN packet, as if you are going to open a real connection and then wait for a response.
Three possible outcomes for a port can be expected and in my lab environment I configured the Windows 7 server to respond in each of these ways. Let's check what the Nmap output looked like:
The -sS parameter performs a SYN-scan and using the -p parameter, you can specify the ports to be scanned. In my lab, I had ports 22, 80 and 139 setup specially, so those are the ones I scanned.
Inspecting the result in the image above, we can see that port 22 was closed, port 80 was open and port 139 was filtered. Let's see what this means and what the handshake and network traffic look like for each of these states.
Open
Port 80 was found open. This means the server would be able to complete the full 3-way handshake for this port and allow incoming connections had we actually completed it. From a server-side that means it would reply with a SYN-ACK. If we wanted to make a connection, we should have replied with an ACK. However, doing a port scan, we're not aiming to make the full 3-way handshake so we reply with a RESET instead:
The network traffic for this situation looks as follows:
As we can see, the SYN and SYN-ACK are followed by a RST to the server. The port is also considered open if a SYN packet (without the ACK flag) is received in response. This can be due to an extremely rare TCP feature known as a simultaneous open or split handshake connection.
Closed
Nmap discovered that port 22 was closed. In this case, the SYN packet sent by the client was returned with a packet where both RESET and ACK flags are set, indicating the server was not listening on port 22 :
Also in the network traffic we can clearly see the SYN was followed by a RST from the server:
Filtered
Nmap will attempt to resend the SYN-packet if no response is received. If after several retransmissions no response is received, the port is marked as filtered.
This can also be seen when inspecting the network traffic for this situation:
Ports marked as filtered are typically behind a firewall. Even closed ports behind a firewall may be marked as filtered. Another situation in which a port is marked as filtered, is when an ICMP unreachable error (type 3, code 0, 1, 2, 3, 9, 10, or 13) is received in response.
Final thoughts
Having at least some basic knowledge about the TCP protocol will benefit software developers, security professionals and system administrators in their daily jobs. Capturing network traffic and analyzing a server's response can really help you debug nasty connectivity or configuration issues, or provide useful when scanning a network. Knowing then where the 3-way handshake goes awry will be valuable.
