Dr. Dobb's Journal June 2000
O ver the years, different network tools have been developed for diagnostic and measurement purposes. There are a large number of available measurement tools, which were developed by individual researchers or research organizations. The Cooperative Association for Internet Data Analysis Group (CAIDA; http://www.caida.org/) has generated a comprehensive taxonomy that lists and organizes a broad collection of network measurement tools (see CAIDA Measurement Tool Taxonomy; http://www.caida.org/tools/taxonomy/).
In this article, I'll present an overview of a number of useful Internet tools that I categorize into the following groups:
Packet-collection software is used to collect traffic measurements on a packet-by-packet basis. Examples include Tcpdump (http://ee.lbl.gov/; also see "traceroute," by V. Jacobson, ftp://ftp.ee.lbl.gov/traceroute.tar.Z), which is a UNIX and Linux portable packet collector that captures information from the packet traffic on a network. Tcpdump is a passive tool, in that it monitors the network traffic and does not inject traffic into the network. Other tools are active (they do inject traffic into the network). When tcpdump is used for traffic traces, the host computer is put in promiscuous mode and all packets (whether or not they were addressed to the host computer) are pulled into the host computer. A number of filtering options are offered by tcpdump. The tcpdump software can collect either the packet header or the whole packet and can capture all or some of the packet data (TCP, UDP, and so on). For example, filtering allows for the capture of only TCP traffic, only port 80 http traffic, and the like. Tcpdump works in conjunction with libpcap (http://ee.lbl.gov/), which is a packet-dumping program. A sample libpcap application is pcapture (http://ee.lbl .gov/), which captures all the packet data to disk.
As network speeds increase, the amount of data also increases, and the ability to take in and store the desired information without overwhelming the host computer becomes an important issue. The BSD Packet Filter (see "The BSD Packet Filter: A New Architecture for User-level Packet Capture," by Steven McCanne and Van Jacobson, USENIX Conference, 1993, available via anonymous ftp://ftp.ee.lbl.gov/ bpf.tar.Z) does preprocessing below the application level to allow the host computer to take tcpdump traces on high-speed networks that carry large amounts of data. The idea is to discard all unnecessary data (packets) as soon as possible (at a low level) in order to save CPU cycles. Processing packets at a lower level saves the CPU cycles that would be needed to move any discarded packets to the higher application level for processing.
Packetman (http://www.cs.curtin.edu .au/~netman/etherman.html), which runs on UNIX and DOS, is a LAN-oriented packet dump program. IPTraf (http://cebu .mozcom.com/riker/iptraf/index.html) is an IP LAN monitor that creates network statistics and provides postprocessing Perl scripts. Intended for general-purpose packet analysis, Argus (ftp:// ftp.sei.cmu .edu/pub/argus-1.5/) provides a packet storage and analysis environment.
Bundled packet-collection software collects packet traces and provides packet stream analysis and statistics. Some bundled packet-collection software comes with the operating system. For example, Sun Solaris contains a packet collector named "snoop." Likewise, Iptrace is the IBM AIX packet-collection program. There are also commercial software packet analyzers which are not included in the OS and are not free. One example is EtherPeek (http://www.aggroup.com/prodinfo/ products.html), a Windows- and Apple-based Ethernet packet analyzer.
Internet measurement tools use "probing packets," which are injected into the network to gather various information including network performance measures. Ping (ftp://ftp.arl.mil/pub/ping.shar/) is probably the best known and most commonly used Internet measurement tool. It can determine if a machine is reachable, and may provide information on delay. Within the class of Internet measurement tools, the Hop-by-Hop characterization tools (also based on probing packets) determine Hop-by-Hop delay on a path from a source to a destination. An important example is Traceroute (ftp://ftp.ee.lbl .gov/traceroute.tar.Z), an Internet utility that sends UDP packets to a selected destination. The destination sends back an ICMP message. Traceroute sends out the first UDP packet with a hop count to destination of one and successively increases this count by one. Each increase allows the UDP packet to travel one more hop to the destination (Figure 1). An intermediate gateway will send an ICMP packet if the maximum hop count for the packet has been reached. In essence, each gateway on the way to a destination will send back an ICMP message. These messages give the hop count and the delay to each gateway on the path to the destination. Traceroute identifies each intermediate node on a route and the round-trip delay to each node. Nikhef traceroute (ftp:// ftp .nikhef .nl/pub/network/traceroute/) is a traceroute variant that allows a number of options. Users may skip the first few hops, or request the min/ave/max round-trip times for each hop. Pathchar (ftp://ftp .ee.lbl.gov/pathchar/) sends a number of packet sizes to each hop in order to estimate performance characteristics for each hop as a function of packet length on a path from a source to destination.
A number of sites around the world are part of the Surveyor project (http://io .advanced.org/csg-ippm/). This tool has the ability to monitor one-way delays by using timestamps and Global Positioning Satellite (GPS) technology to obtain accurate transit interval measurement by virtue of time synchronization (via GPS). Surveyor machines periodically send ping packets to each other (full mesh coverage). Two key network performance metrics are measured; packet loss and one-way delay between the 59+ different Surveyor sites. Surveyor can also conduct passive tests. NLANR has an active monitor program (http://amp .nlanr.net/AMP/) that provides round-trip information (min, mean, max, and so on) and packet loss percentages from the 90+ active monitor sites.
Internet throughput tools give bandwidth information for an end-to-end path. TReno (http://www.psc.edu/networking/treno/) creates a simplified, user-level implementation of a TCP-like transport protocol. This allows for the measurement of throughput independent of the actual TCP implementation used in the Host. It is a good reference platform for prototyping TCP changes and determining the bandwidth a process would get if it were running over the most recent TCP version. Another interesting end-to-end tool is bing (http://spengler.econ.duke.edu/ ~ferizs/). While keeping the amount of extra traffic low, bing compares round-trip time for different packet lengths and determines link bandwidth. (B|C) probe (http://www.cs.bu.edu/students/grads/carter/ tools/Tools.html) is really two programs. Bprobe can be used to check the capacity of a bottleneck link. Cprobe sends out a small stream of packets, measures the packet separation upon return, and from this determines the amount of bandwidth available taking into account the bandwidth demands of competing traffic. Other tools widely used in the Internet include:
ISP measurements tools are used to measure the performance of the various Internet Service Providers along an end-to-end Internet path. ClearInk Weather Report (http://www.internetweather.com/) periodically pings different sites to help determine in which Internet provider problems lie. It attempts to measure performance on the Internet. Inverse Internet Measurement Service (http:// www.inversenet.com/products/index .html) creates ISP performance profiles. This allows for the comparison of performance for different ISPs. National Internet Measurement Infrastructure (NIMI) (http://www.psc.edu/ networking/nimi/) is based on Vern Paxson's Network Probe Daemon. It is designed to measure the global Internet.
Used by Vern Paxson to characterize end-to-end Internet routing, the Network Probe Daemon (http://nic .merit.edu/ipma/npd/) is a prototype of Internet measurement infrastructure. NetNow (http://nic.merit.edu/ ipma/netnow/) measures packet loss and latency across components of the Internet. IPMA (http:// www.merit.edu/ ipma/) has a number of monitors in over 10 different countries. It collects statistics on latency, packet loss, and routing.
In the high-performance measurement tool category, you find tools typically used to measure throughput and other key performance parameters in high performance networks. Netperf (http://www.netper.org/netperf/NetperfPage.html) measures LAN-based network performance such as latency, throughput, and TCP transaction speed. This is done on a point-to-point basis. Ttcp (ftp://ftp.arl.mil/pub/ttcp/) can also be included in this class. It can be used as a throughput benchmark as well as a load generator. NetSpec (http:// www.ittc.ukans.edu/Projects/AAI/products/netspec/) is a scripting language for writing throughput benchmarks with complex communication and workload patterns.
Network trace data is typically processed and analyzed using a variety of traffic analysis tools to extract various types of views.
Sugih Jamin applied the idea of creating a TCP network traffic generator (tcplib) that is based on the characterization of the well-known port applications from empirical data. Network traffic traces were used to identify the core applications that have the largest percentage of the total packets sent. Each of these port applications was then analyzed and modeled. The output of tcplib is the combined output of each modeled application. HTTP traffic was less than 1 percent of the total traffic at the time and so it was not included in the tcplib traffic generator (for more information, see "tcplib: A Library of TCP internetwork Traffic Characteristics," by P. Danzig and S. Jamin, Report CS-Sys-91-01, Computer Science Department, University of Southern California, 1991).
The Internet tools identified here are by no means all of the available resources for Internet developers. For more detailed information, you can start with any of the URLs listed here, or for a more expanded listing, go to http://www.caida.org/tools/taxonomy/.
DDJ