Introduction

SYN Cookies were originally proposed to solve SYN Flood attacks. In high-concurrency network scanning, how to distinguish between TCP packets belonging to the scanner and normal traffic is a critical issue. Scanners (such as Masscan, ZMap) borrow the principle of Syncookies to verify responses without maintaining massive connection states, thereby improving scanning efficiency and reliability. This article will explain the application of Syncookies technology in scanning, combined with the implementation of Masscan.

In a normal TCP three-way handshake:

  1. Client sends SYN
  2. Server replies SYN-ACK and allocates connection resources in memory
  3. Client replies ACK, handshake completes

If an attacker forges a large number of source IPs to send SYN packets without replying the final ACK, the server’s half-open connection queue (SYN_RECV state) will be exhausted, preventing legitimate users from establishing connections. This is a SYN Flood attack.

In Linux, you can view the current number of connections in the half-open queue with the following command:

netstat -an | grep :80 | grep SYN_RECV | wc -l

The size of the half-open queue is controlled by net.ipv4.tcp_max_syn_backlog.

Mechanism of SYN Cookie The core idea of SYN Cookie is: do not allocate resources immediately upon receiving a SYN packet, but generate a verifiable Initial Sequence Number (ISN) through an algorithm.

  1. Receive SYN

    • Do not occupy the half-open queue
    • Generate ISN: 
      ISN = f(srcIP, srcPort, dstIP, dstPort, secret, timestamp)
    • f() is usually a cryptographic hash function (like SipHash)
    • secret and timestamp prevent forgery

  2. Reply SYN-ACK

    • Set the packet sequence number to ISN

  3. Receive ACK

    • Client returns ack = ISN + 1
    • Server recalculates ISN and verifies
      • Match → Establish connection
      • Mismatch → Discard

In this way, the server can resist large-scale SYN Flood attacks in a stateless manner.

SYN Cookie in Linux Kernel

Linux has supported SYN Cookies since 2.1.x/2.2.x. Enable method:

sysctl -w net.ipv4.tcp_syncookies=1

0 → Off, 1 → On

When the half-open queue is full, the kernel will activate the SYN Cookie mechanism to verify connection legitimacy using sequence numbers. The disadvantage is that it cannot support some TCP options (such as window scaling, timestamps), which may lead to misjudgment under high load.

Syncookies Application in Masscan

Masscan does not rely on the kernel to maintain connection states during scanning, but uses raw sockets or libpcap to capture all TCP packets. Advantages of doing so:

  1. Efficient Capture: Directly acquire TCP packets passing through the network card.
  2. Flexible Processing: Parse fields like sequence number and acknowledgment number as needed.

But the problem it brings: How to distinguish scanning traffic belonging to Masscan from normal traffic of other programs (such as a browser)?

Solution: Borrow the idea of Syncookies.

Masscan’s Implementation Approach Masscan assigns a unique sequence number (Seq) to each scan request, generated via the SipHash algorithm.

Input fields: srcIP, dstIP, srcPort, dstPort

Output value: Take the lower 32 bits as the sequence number.

Since the TCP protocol allows the client SYN packet’s sequence number to be any value, Masscan can use this to achieve stateless verification.

Workflow

Sending Phase:

  1. hash = SipHash(srcIP, dstIP, srcPort, dstPort)
  2. seq = hash & 0xffffffff
  3. Send SYN packet, Seq = seq

Receiving Phase:

  1. Capture TCP packet (raw socket or libpcap)
  2. Recalculate hash = SipHash(srcIP, dstIP, srcPort, dstPort)
  3. Judge:
    • If (hash & 0xffffffff) == Seq in TCP packet → Belongs to Masscan scanning traffic
    • Otherwise → Belongs to other program traffic, discard
flowchart TD A[Start Scanning] --> B["Calculate SipHash(srcIP, dstIP, srcPort, dstPort)"] B --> C["Take lower 32 bits as Sequence Number Seq"] C --> D["Send SYN packet, Seq=hash result"] D --> E["Capture TCP packet (raw socket/libpcap)"] E --> F["Recalculate SipHash(srcIP, dstIP, srcPort, dstPort)"] F --> G["Take lower 32 bits as verification value"] G --> H{"Verify: Does Seq match?"} H -->|Yes| I["Belongs to Masscan traffic → Continue processing"] H -->|No| J["Belongs to other traffic → Discard"]

Technical Advantages

Uniqueness: For the same IP/Port pair, the sequence number is fixed in one scan, avoiding confusion.

Unpredictability: SipHash is difficult for attackers to forge.

Stateless Verification: No need to maintain a large-scale connection table, only one hash operation is needed for verification.

Summary

SYN Cookie technology was originally proposed to defend against SYN Floods, but its core idea: using verifiable sequence numbers to achieve stateless verification, has been borrowed by Masscan and applied to high-performance port scanning. This method allows scanners to remain efficient and reliable even under extremely high concurrency, while avoiding misjudgment of external traffic. It can be said that SYN Cookie has evolved from a defense mechanism into an important performance optimization means in cybersecurity tools.