Article Figure 1 Listing 1 Listing 2 mar2006.tar

Getting to Know Your Network -- Part II

Luis Enrique Muñoz

In the first article of this series, I showed how to build a script, aconfig, which allowed us to quickly execute commands on network devices (http://www.sysadminmag.com/documents/s=9950/sam0602h/). It also allowed us to use Perl to generate commands on the fly and process the result of those commands. To tell aconfig what to do, I wrote some small snippets of configuration commands, called ascripts. Those ascripts are used to gather version and configuration information from the network devices. In this article, I'll take the process one step further and build all this into a neat network inventory. (The complete listings for this article are available from the Sys Admin Web site at: http://www.sysadminmag.com.)

A network inventory, for our purposes, is not much more than a database of network devices, allowing us to store information regarding those devices. So let's pick up just where we left off last month. We had two files for each network device. The one with a ".version" extension contained the results of the show version command. The other, with a ".config" extension, contained the output of the show run command. This is valuable because now tools like find and grep could help us to locate interesting information, but this approach is clearly not the best we can do.

For easier manipulation, we should now look at ways to transfer the information on these files to a database. This database would let us quickly find information about our network using SQL queries. As I'll show later, this information can easily include rather nice network diagrams. But first, let's work on getting at the juicy bits of information.

The database schema we will be using through this article is shown in Figure 1. For now, we are only concerned with tables "device", "interface", "address", and "subnet". The rest will be used later when we use additional scripts to learn about other devices plugged into our network.

For this article, I'll be using SQLite, a small database manager that runs on a wide variety of systems. In fact, this could even be used for your production setup. The SQL commands to produce the schema depicted in Listing 1 (myconfigCDBI.pm) are as follows: 

CREATE TABLE device
(
  device TEXT PRIMARY KEY,
  location TEXT,
  contact TEXT,
  serial TEXT,
  hardware TEXT,
  software TEXT,
  class TEXT
);

CREATE TABLE interface
(
  interface TEXT,
  device TEXT,
  description TEXT,
  PRIMARY KEY (interface, device),
  FOREIGN KEY (device) REFERENCES device (device)
);

CREATE TABLE address
(
  ip INTEGER,
  interface TEXT,
  device TEXT,
  cidr TEXT,
  type TEXT,
  PRIMARY KEY (ip, interface, device),
  FOREIGN KEY (cidr) REFERENCES subnet (cidr),
  FOREIGN KEY (interface) REFERENCES interface (interface),
  FOREIGN KEY (device) REFERENCES device (device)
);

CREATE TABLE subnet
(
  cidr TEXT PRIMARY KEY,
  first INTEGER,
  last INTEGER
);

CREATE TABLE endpoint
(
  endpoint TEXT PRIMARY KEY,
  os TEXT,
  time INTEGER,
  vendor TEXT
);

CREATE TABLE sighting
(
  endpoint TEXT,
  time INTEGER,
  interface TEXT,
  device TEXT,
  PRIMARY KEY (endpoint, time, interface, device),
  FOREIGN KEY (endpoint) REFERENCES endpoint (endpoint),
  FOREIGN KEY (device) REFERENCES device (device),
  FOREIGN KEY (device, interface) REFERENCES interface (device, interface)
);

CREATE TABLE assignment
(
  ip INTEGER,
  time TEXT,
  endpoint TEXT,
  cidr TEXT,
  PRIMARY KEY (ip, time),
  FOREIGN KEY (endpoint) REFERENCES endpoint (endpoint),
  FOREIGN KEY (cidr) REFERENCES subnet (cidr)
);
Provided that we place these SQL commands in the file "config.sql", the database can easily be created with this command: 

$ sqlite3 ~/config.db < config.sql
There are a few choices of what to use to access the database from the Perl scripts used to extract information from the files in our output directory. Personal taste seems to be a significant criterion here. In this article, I'll use Class::DBI, which provides a nice object-oriented layer above the database. With Class::DBI, we will be spared the mix of Perl and SQL in the same scripts, unless we really want that.

The examples presented in this article perform many tasks on the database client that could be done at the server with more complex SQL. For small networks, infrequent use, or demonstration purposes, this is perfectly acceptable. However, when higher performance is required, you will want to pay attention to this important detail.

I've placed all the class definitions in a single file that I called "MyConfigCDBI.pm" (see Listing 1), so I can invoke them from the scripts that need access to the database. I am also including Class::DBI::AbstractSearch so that more complex searches can be performed by the Perl scripts. We will use this later.

The code shown previously takes care of lots of things that happen conveniently out of the way, such as providing simple accessors to columns and relationships, building the SQL required to fetch the requested objects, caching and efficiently managing database resources, and updating the object when required. With this, the next script, which is responsible for extracting device data and loading it into the database, becomes much simpler because it does not even have to include SQL in the code. Let's look at the important parts of the script, "config2db" (Listing 2). 

 1: #!/usr/bin/perl
 2:
 3: use strict;
 4: use warnings;
 5: use lib 'scripts';
 6: use IO::File;
 7: use NetAddr::IP;
 8: use Cisco::Reconfig;
 9: use MyConfigCDBI;
10: $|++;
Lines 1 to 10 are our usual script invocation. I've added the use lib, which allows us to conveniently tell Perl where to look for the modules we invoke, in addition to the standard places. This lets us "use MyConfigDBI" to bring in all the Class::DBI related definitions presented earlier. Cisco::Reconfig will be of help to parse the configuration files and get to what we want. NetAddr::IP will help with parsing the IP addresses and subnets we find within the config and converting them to the format we need in the database. 

14: sub get_version
15: {
16:     my $fh = new IO::File $_[0];
17:     my $version;                # The output of show version
18:     my $class;
19:
20:     unless ($fh)
21:     {
22:         warn "Failed to open version file $_[0]: $!\n";
23:         return (map { 'Could not read version file' } 1 .. 3);
24:     }
25:
26:     do { local $/ = undef; $version = <$fh>; };
27:     close $fh;
28:
29:     my $sw = ($version =~ m/^(.+Software .+ Version .+)$/m)[0]
30:         || 'Cannot find software version';
31:     my $hw = ($version =~
32:               m/^(cisco .+ (processor .+|bytes of \
                    memory\.?))$/mi)[0]
33:               || 'Cannot find hardware version';
34:     my $sn = ($version =~ m/processor board ID \
          (\w+( \([^\)]+\))?)/mi)[0]
35:         || 'Cannot find serial number';
36:
37:     if ($version =~ m/cisco\s(Cat6k-|WS-C(29|3[57]|40))/)
38:     {        $class = 'switch' }
39:     elsif ($version =~ /cisco\s(2[56]\d\d|18\d\d
40: |WS-C65\d\d|17\d\d|CISCO76\d\d)\s/ix)
41:     {        $class = 'router' }
42:     elsif ($version =~ /cisco\sAIR-AP/)
43:     {        $class = 'access-point' }
44:
45:     return ($sw, $hw, $sn, $class);
46: }
Lines 14 to 46 define the get_version function, which is responsible for parsing the output of the show version command stored previously, and they return various pieces of useful information for our inventory -- namely, which version of software is in the device, what class of hardware it is, serial or identification number, and a generic class that we will use later to produce some nice diagrams.

Essentially, this code slurps in the file with the version information and applies a number of regular expressions to it for extraction of the relevant information. I've left here the regexes that I think are most robust or general. Depending on which network devices you happen to have, you may want to look at their version information and adjust the regular expressions to fit your needs. 

48: sub get_base_info
49: {
50:     my %k = map { $_ => $_[0]->get($_) }
51:     ('hostname', 'snmp-server contact', 'snmp-server location');
52:
53:     return map { $k{$_} =~ s/^$_\s+//; chomp($k{$_}); $k{$_} } \
          sort keys %k;
54: }
In our local environment, the network staff is pretty obsessive about the SNMP configuration, which contains very useful details, such as the location of the device and who should be contacted in case of failure of said device. This, together with the device's hostname, is extracted by the get_base_info function on lines 48 to 54. 

56: sub get_interfaces
57: {
58:     my $config = shift;
59:     my @ifs = ();
60:     for my $if ($config->get('interface')->all())
61:     {
62:         my @container = (($if->text =~ m/interface (\S+)/)[0]);
63:         next unless $container[0];
64:         my $desc = ($if->get('description')->text =~ \
              m/description (.+)$/)[0];
65:         push @container, $desc;
66:         for my $addr ($if->get('ip address'))
67:         {
68:             next if $addr->text =~ m/no ip address/;
69:             next unless $addr->text;
70:             my ($ip, $mask, $class) =
71:                 $addr->text =~ m/ip address (\S+) \
                      (\S+)\s*(secondary)?/;
72:             $class ||= 'primary';
73:             my $i = NetAddr::IP->new($ip, $mask);
74:             next unless $i;
75:             push @container, [ $i, $class ];
76:         }
77:         push @ifs, \@container;
78:     }
79:     return @ifs;
80: }
Lines 56 to 80 define the get_interfaces function, which goes through the configuration file and parses the interface information, extracting the name, description, and possibly the list of IP addresses. This code is made simpler with the help of Cisco::Reconfig, which handles the subtle details of parsing. 

82: MyConfig::CDBI->connection('dbi:SQLite:dbname=config.db');
With all those definitions in place, it's time to begin processing our config files. Line 82 creates the connection to the database. Thanks to the inheritance used when defining the Class::DBI classes, this line is all that's needed. Note that I am not specifying the full path of the database, so I have to run my scripts in the directory in which the database resides. 

 84: for my $c (@ARGV)
 85: {
 86:     my $config = readconfig($c);
 87:     my ($dev, $loc, $ctc) = get_base_info($config);

 89:     if (not defined $dev)
 90:     {
 91:         warn "$c does not define a device host name\n";
 92:         next;
 93:     }
   ...
138: }
Lines 84 to 138 simply loop over the specified configuration files passed in the command line. In line 86, the configuration file is read by Cisco::Reconfig's readconfig. In line 87, we start getting basic information with the get_base_info function introduced previously. Lines 89-93 are a simple sanity check to make sure we parsed useful information from the given files. Otherwise, we let our user know and skip to the next file. 

95:     my $vc = version_file($c);
96:     my ($sw, $hw, $sn, $cl) = get_version($vc);
In lines 95-96, we invoke the get_version function to extract information from the show version output matching the configuration file we're reading. 

100:     my $device = MyConfig::CDBI::Device->find_or_create(
101:         device => $dev,
102:         location => $loc,
103:         contact => $ctc,
104:         serial => $sn,
105:         hardware => $hw,
106:         software => $sw,
107:         class => $cl
108:     );
Finally, at lines 100 through 108, we start using the services of Class::DBI. Here, we attempt to look up a device in the database with the given characteristics. If we're unable to find it, it gets created automatically. 

112:     for my $if (get_interfaces($config))
113:     {
114:         my $ifname = lc shift @$if;
115:         my $desc = shift @$if;
116:         print "$c $device $ifname\n";
117:         my $interface = MyConfig::CDBI::Interface->find_or_create(
118:             interface => $ifname,
119:             device => $dev,
120:             description => $desc
121:         );
122:         for my $ip (@$if)
123:         {
124:             my $subnet = MyConfig::CDBI::Subnet->find_or_create(
125:                 cidr => $ip->[0]->network->cidr,
126:                 first => scalar $ip->[0]->network->numeric,
127:                 last => scalar $ip->[0]->broadcast->numeric,
128:             );
129:             my $address = MyConfig::CDBI::Address->find_or_create(
130:                 ip => scalar $ip->[0]->numeric,
131:                 interface => lc $interface->interface,
132:                 device => $device->device,
133:                 cidr => $subnet->cidr,
134:                 type => $ip->[1],
135:             );
136:         }
137:     }
At lines 112 to 137, we use the same technique to add interfaces and subnets, assigning the IP addresses of the interfaces when defined. Now, it's only a matter of using find to invoke the script, which in my case is done like this: 

$ find ./output -name '*.config' -type f | xargs scripts/config2db
Thanks to line 116, in my case, I see many, many lines like the following: 

./output/10.64.106.129.config CT-TC-ZUL loopback0
./output/10.64.106.129.config CT-TC-ZUL fastethernet0
./output/10.64.106.129.config CT-TC-ZUL serial0
./output/10.64.141.1.config CORE02-P1-BLDG1 loopback0
./output/10.64.141.1.config CORE02-P1-BLDG1 multilink1
./output/10.64.141.1.config CORE02-P1-BLDG1 multilink2
./output/10.64.141.1.config CORE02-P1-BLDG1 port-channel1
./output/10.64.141.1.config CORE02-P1-BLDG1 port-channel2
./output/10.64.141.1.config CORE02-P1-BLDG1 serial1/0/0:0
./output/10.64.141.1.config CORE02-P1-BLDG1 serial1/0/1:0
./output/10.64.141.1.config CORE02-P1-BLDG1 serial1/0/2:0
./output/10.64.141.1.config CORE02-P1-BLDG1 serial1/0/3:0
...
At this point, our database should be nicely populated with device inventory. Let's do a simple test to show to put this information to some use. Say, for example, that a new vulnerability exists, affecting devices running Cisco IOS 12.0(5)WC7. We could very quickly find out how serious this hypothetical threat would be for us: 

$ sqlite3 config.db
sqlite> select count(*) from device where software like '%12.0(5)WC7%';
30
So, according to our recent inventory, 30 devices in our corporate network would be vulnerable. We can also get an idea about where the problem areas would lie in our network: 

sqlite> select count(*),location from device where software
   ...> like '%12.0(5)WC7%' group by location;
   10|Floor 1 Bldg1
   10|Floor 2 Bldg1
    9|Floor 3 Bldg1
    1|Basement Bldg1
This threat could affect operations in one of our buildings. We can get the list of IP addresses required to reach those devices and perform the upgrade. In this last case, since the database is storing the integer representation of the IP address we want, we will combine the query with a one-liner Perl script to convert the data on the fly: 

$ echo "select a.ip from device d, address a where d.software like
  '%12.0(5)WC7%' and d.device = a.device;" | sqlite3 config.db | \
  perl -MNetAddr::IP -pe '$_ = NetAddr::IP->new($_) . "\n"'
10.64.73.10/32
10.64.73.11/32
...
With the tools introduced so far, you could also write an ascript that goes out to those devices and determines whether they are vulnerable in a more fine-grained way. Many vulnerabilities require a certain configuration to be present in the device, so you could check this. Better yet, it could apply suitable workarounds or perform an automatic upgrade. I did this with a few thousand DSL modems a few years ago.

In the next article, we'll gather some more information about our network, including what's plugged into it, and also produce some pretty diagrams.

Luis has been working in various areas of computer science since the late 1980s. Some people blame him for conspiring to bring the Internet into his home country, where currently he spends most of his time teaching others about Perl and taking care of network security at the largest ISP there as its CISO. He also believes that being a sys admin is supposed to be fun.