TCP Handshake Process and Queues




As shown above, there are two queues: a SYN queue (or incomplete connection queue) and an accept queue (or complete connection queue).

In the three-way handshake, after receiving a SYN packet from the client, the server places the connection information in the SYN queue and sends a SYN-ACK packet back to the client.

The server then receives an ACK packet from the client. If the accept queue isn’t full, you should either remove the information from the SYN queue and put it into the accept queue, or execute as tcp_abort_on_overflow instructed.

At this point, if the accept queue is full and tcp_abort_on_overflow is 0, the server sends a SYN-ACK packet to the client again after a certain period of time (in other words, it repeats the second step of the handshake). If the client experiences even a short timeout, it is easy to encounter a client exception.


ss command






Here, the Send-Q value in the second column is 50, indicating that the accept queue on the listen port (the third column) is 50 at most. The first column, Recv-Q, indicates the amount of the accept queue that is currently being used.

The size of the accept queue depends on min(backlog, somaxconn). The backlog is passed in when the socket is created, and somaxconn is an OS-level system parameter.

The size of the SYN queue depends on max(64, /proc/sys/net/ipv4/tcp_max_syn_backlog) and the OSs of different versions may be different.

netstat command


It is important to note that the Recv-Q data shown by netstat -tn has nothing to do with the accept queue or SYN queue. This must be emphasized here so as not to confuse it with the Recv-Q data shown by ss -lnt.

In netstat, if the connection is not in Listen state, Recv-Q means that the received data is still in a cache and has not been read by the process. This value represents the bytes that have not been read by the process. Send is the number of bytes in the send queue that have not been acknowledged by the remote host.

TCP/IP stack has two options to implement the backlog queue for a socket in the LISTEN state.

Door #1: A Single Queue

A single queue can contain connections in two different states:

  • SYN RECEIVED
  • ESTABLISHED

And only connections in ESTABLISHED state can be returned to the application by calling the accept() syscall.

Therefore, the length of this queue is determined by the backlog argument of the listen() syscall.

Door #2: A SYN Queue & An Accept Queue

In this case, connections in state SYN RECEIVED are added to the SYN queue, and later moved to the accept queue when their state changes to ESTABLISHED.

So, the accept queue stores only connections ready for the accept() syscall.

Unlike the previous case, the backlog argument of the listen() syscall determines the size of the accept queue.

BSD’s Choice

BSD chooses the option 1 as its behavior implementation (though it internally indeed uses two queues).

When the queue is full, instead of sending back a SYN/ACK packet in response to received SYN packet, it simply drops the SYN packet, and let the client to retry.

Modern Linux’s Choice

Since Linux 2.2, Linux uses the 2nd option and make

  • backlog argument for the maximum length of the accept queue
  • /proc/sys/net/ipv4/tcp_max_syn_backlog for the maximum length of SYN queue; and newer kernel uses /proc/sys/net/core/somaxconn (identified via net.core.somaxconn)

By the way, from the point of the view of the client, a connection will be in state ESTABLISHED after reception of the SYN/ACK packet.

0x01 When SYN Queue is Full

When received a SYN packet but SYN queue is full, then:

  • If net.ipv4.tcp_syncookies=0, then the SYN packet is simply discarded
  • If net.ipv4.tcp_syncookies=1, then
    a. if the accept queue is full AND req_young_len > 1, then the SYN packet is discarded
    req_young_len is the number of connections that haven’t been retransmitted SYN/ACK packet in SYN queue
    b. otherwise, generate syncookies for this SYN packet; This packet is presumed a malicous packet.

0x02 When Accept Queue is Full

When received an ACK packet, but the accept queue is full:

  • If tcp_abort_on_overflow=1, then the TCP/IP stack replies back a RST packet, then the connection is removed from the SYN queue.
  • If tcp_abort_on_overflow=0, then the stack marks the connection as ACKED (i.e. ignores this ACK packet) and leaves the connection in SYN queue; soon, the timer would then go off and resend a SYN/ACK packet back, the client can resend an ACK packet again.
    Or if the maximum retry limit is reached, a RST packet is still replied and the connection is removed from the SYN queue

The maximum retry number is configured by net.ipv4.tcp_synack_retries.

However, if the accept queue is full, the kernel will also impose a limit on the rate at which SYN packets are accepted: If too many SYN packets are received, some of them will be dropped.

In this case, it is up to the client to retry sending the SYN packet.

0x03 Disadvantages of Using a Single Queue

There are two major causes of the single queue that might be full.

(1) Application calling accept() is not fast enough, making completed (ESTABLISHED) connections fill the queue
(2) RTT between the server and the client is so big, that incomplete connections (in state SYN RECEIVED) fill the queue.

By using one queue, it assumes that an application is expected to tune the backlog

  • not only taking into account how it intents to process newly established incoming connections,
  • but also in function of traffic characteristics such as the round-trip time.

If we use two dedicated queues, the SYN queue then effectively implies ACK packets in flight; and the accept queue implies how application level handles the completed connections.

Comments

Post a Comment

Popular posts from this blog

TCP Keepalives & its sequence numbering

  https://www.freesoft.org/CIE/RFC/1122/114.htm TCP Keep-Alives Implementors MAY include "keep-alives" in their TCP implementations, although this practice is not universally accepted. If keep-alives are included, the application MUST be able to turn them on or off for each TCP connection, and they MUST default to off. Keep-alive packets MUST only be sent when no data or acknowledgement packets have been received for the connection within an interval. This interval MUST be configurable and MUST default to no less than two hours. It is extremely important to remember that ACK segments that contain no data are not reliably transmitted by TCP. Consequently, if a keep-alive mechanism is implemented it MUST NOT interpret failure to respond to any specific probe as a dead connection. An implementation SHOULD send a keep-alive segment with no data; however, it MAY be configurable to send a keep-alive segment containing one garbage octet, for compatibility with erroneous TCP implemen...
Read more

Why is the TCP Stack Dropping SYN Packets?

 https://medium.com/dataseries/why-are-linux-kernel-protocol-stacks-dropping-syn-packets-5ee5cab351a6 The Per-host PAWS Check Results in the Drop of SYN Packets Symptom This is the most common problem in the actual production environment: For servers with both the net.ipv4.tcp_tw_recycle and the net.ipv4.tcp_timestamps enabled, the probability of this problem is very high when the server has NAT client access. From the client side, the symptom of this problem is that the new connection is unstable. Sometimes it can be connected and sometimes it cannot. Per-Host PAWS Principle For background information, PAWS is short for Protect Against Wrapped Sequences, which is a means for preventing sequence numbers from being wrapped. Next, Per-host checks the IP address of the peer host rather than the quad-tuples of the IP port. The way per-host PAWS checks is as follows: For the quintuple peer host IP of the TIME_WAIT socket that is quickly recycled, which helps to prevent the interference ...
Read more

TCP FSM

Image
  TCP FSM TCP Finite State Machine (FSM) States, Events and Transitions State State Description Event and Transition CLOSED This is the default state that each connection starts in before the process of establishing it begins. The state is called “fictional” in the standard. The reason is that this state represents the situation where there is no connection between devices—it either hasn't been created yet, or has just been destroyed. If that makes sense. J Passive Open:  A server begins the process of connection setup by doing a passive open on a TCP port. At the same time, it sets up the data structure ( transmission control block or TCB ) needed to manage the connection. It then transitions to the  LISTEN  state. Active Open, Send  SYN :  A client begins connection setup by sending a  SYN  message, and also sets up a TCB for this connection. It then transitions to the  SYN-SENT  state. LISTEN A device (normally a server) is waiti...
Read more