by jeffmcjunkin

Often when doing penetration tests, clients will ask me to scan their external network presence[1]. For smaller companies, I can often use nmap from start to finish for all my scanning needs. However, for the sake of larger network ranges let’s separate out some of our scanning needs:

1. Network sweeping: Determining which IPv4 addresses have any listening services (finding “live” hosts)

2. Port scanning: Determining listening TCP and UDP ports on target systems

3. Version scanning: Determining the version of services and protocols spoken by open TCP and UDP ports

If the external IP range is roughly ten thousand hosts or fewer, nmap will work just fine for each of these needs. Often, though, larger companies can own tens or even hundreds of thousands of IPv4 addresses. How can we determine in a few hours which of these IPv4 addresses have a listening host? nmap’s default behavior only sends a few probe requests — if all of those probe requests fail, the host is marked offline and no further probes are sent. We can skip the network sweeping with the -Pn option, but then nmap will scan every single configured port for every single IP address. Since the large majority of external IPv4 ranges won’t have listening services, for large network ranges this could take weeks, months, or even years! What we need is some way to efficiently do a network sweep (find which IPv4 addresses have listening services) before handing that smaller list to nmap for further port and version scanning.

Why does nmap have a hard time with such huge network ranges? Fundamentally, nmap is a synchronous tool — that is, it tracks the connection requests and waits for replies. If a TCP connection request (a SYN) doesn’t get any reply, nmap will eventually timeout and declare that service filtered. nmap certainly runs many probe requests in parallel, but filtered services (and unassigned IPv4 addresses) can really slow it down.

In contrast to synchronous tools like nmap, there are several tools that don’t track connections — also known as asynchronous scanners. Examples include scanrand, ZMap, and my personal favorite masscan.

masscan is my favorite of the asynchronous scanning tools for several reasons. First and foremost, it uses the same syntax as nmap whenever possible, which makes it easier to pick up. Second, even amongst asynchronous scanning tools it’s really, really fast. Effectively, with proper network interfaces and drivers it’s limited only by your bandwidth. With two Intel 10 gigabit ethernet adapters it can scan the entire IPv4 internet in six minutes, transmitting over 10 million packets per second. If nmap is light speed, ZMap and scanrand are ridiculous speed, and masscan is ludicrous speed.

First, let’s look at masscan’s basic syntax for scanning the well-known TCP ports of a large network, such as Apple’s ~16 million IPv4 addresses:

masscan -p0-1023

Scanning speed

By default, masscan will only send 100 packets per second. Counting 18 bytes for the Ethernet header, 20 bytes for a TCP header, and 20 more for the IPv4 header, that’s only 5,800 bytes per second, or ~46 kilobits per second. Because masscan scans ports and hosts evenly (that is, randomly), the scanning bandwidth you use will be evenly distributed across the hosts and ports you scan. Unintentional Denial-of-Service can be a concern with high-bandwidth scans on smaller network ranges, but a 1-10 megabits per second (or --rate 20000, twenty thousand packets per second) should be pretty safe. Virtual machines can safely go up to --rate 200000, which is 93 megabits per second of outgoing scanning traffic — but check with your client if you need to use these higher speeds.

Doing a network sweep

How can we determine if a given IPv4 address has any listening TCP services? Well, we could scan 65,536 (ports zero through 65,535) ports, but for larger network ranges that’ll make for long scanning times, even with a high --rate. More commonly, I’ll select nmap’s top 100 or 1,000 ports by popularity. If any IPv4 address responds to any SYN packet (whether it’s closed with a RST or open with a SYN-ACK), we’ll save out that host and scan it using more specialized tools such as nmap or even a vulnerability scanner like Nessus.

Let’s use a small trick to get nmap to tell us that list of ports. We’ll scan our own system and output XML format to STDOUT. The XML format of nmap shows the exact parameters used for a scan, but crucially it’ll also translate between --top-ports X and the actual list of ports in a concise fashion. Here I’ll choose to display the top hundred ports, but you could just as easily choose the top ten or the top thousand.

$ nmap localhost --top-ports 100 -oX - | grep services
<scaninfo type="connect" protocol="tcp" numservices="100" services="7,9,13,21-23,25-26,37,53,79-81,88,106,110-111,113,119,135,139,143-144,179,199,389,427,443-445,465,513-515,543-544,548,554,587,631,646,873,990,993,995,1025-1029,1110,1433,1720,1723,1755,1900,2000-2001,2049,2121,2717,3000,3128,3306,3389,3986,4899,5000,5009,5051,5060,5101,5190,5357,5432,5631,5666,5800,5900,6000-6001,6646,7070,8000,8008-8009,8080-8081,8443,8888,9100,9999-10000,32768,49152-49157"/>

Now we can copy that list of ports into masscan and scan our target range. We’ll keep using Apple as our victim example network. At 100,000 packets per second, this will use around 32 megabits per second of traffic.

$ sudo masscan -oG apple-masscan.gnmap -p 7,9,13,21-23,25-26,37,53,79-81,88,106,110-111,113,119,135,139,143-144,179,199,389,427,443-445,465,513-515,543-544,548,554,587,631,646,873,990,993,995,1025-1029,1110,1433,1720,1723,1755,1900,2000-2001,2049,2121,2717,3000,3128,3306,3389,3986,4899,5000,5009,5051,5060,5101,5190,5357,5432,5631,5666,5800,5900,6000-6001,6646,7070,8000,8008-8009,8080-8081,8443,8888,9100,9999-10000,32768,49152-49157 --rate 100000

Note that masscan supports the same -oG filename.gnmap option as nmap does. We’ll read through that output list (the so-called “greppable” format) to find the list of hosts that are alive. Given 16 million target IPv4 addresses and 100 TCP ports each, this scan will take around five hours to complete — which is well within what I’d consider a “reasonable” timeframe. Let’s look at the first few lines of the resulting file:

# Masscan 1.0.3 scan initiated Thu Jul 20 22:24:40 2017
# Ports scanned: TCP(1;7-7,) UDP(0;) SCTP(0;) PROTOCOLS(0;)
Host: ()  Ports: 443/open/tcp////
Host: ()   Ports: 179/open/tcp////
Host: () Ports: 8081/open/tcp////
Host: () Ports: 8081/open/tcp////

We only need the IPv4 address, so we’ll use egrep to search for lines beginning with “Host: ” and cut to take the second field. We’ll also sort and make the lines unique with uniq, just in case masscan writes the same IPv4 address twice.

$ egrep '^Host: ' apple-masscan.gnmap | cut -d" " -f2 | sort | uniq > apple-alive

Now we have a much smaller list of IPv4 addresses to work with, one address per line. As a parting example, we can use this as an input list with nmap to do a more thorough scan:

# nmap -PN -n -A -iL apple-alive -oA apple-nmap-advanced-scan

Using our masscan-generated file, nmap will now be able to do its job much more quickly!

Please let me know in the comments if you find this useful for your workflow. I love nmap, but sometimes larger tasks call for more specialized tools.

Thanks for reading! – Jeff McJunkin

[1] Hopefully the client is risk-aware enough to consider a “assume breach” mentality and give penetration testers internal network access in addition to the external network scan, but that’s a separate story.

Spare: In fact, nmap is absolutely my favorite tool for service scanning (that is, differentiating between Apache 2.2 and IIS 8.0 on the listening port 80).