Embedded Systems

Debugging under GNU/Linux

Randy Zack

Programmers from fields as diverse as enterprise computing to embedded systems are venturing into Linux. If you are one of them, here’s a tool that should be at the top of your download list.

Introduction

If you are writing C/C++ programs for GNU/Linux machines, you will need to do some debugging as well. And unless you have purchased a C/C++ compiler other than gcc or g++, or are really enamored of debugging using printf, you will need to know how to use the GNU symbolic debugger, gdb. This article is intended to be a quick introduction to gdb for programmers who are used to debugging in a graphical environment, with an emphasis on features of gdb that you may not be used to in a graphical environment.

What Is gdb?

gdb is the GNU symbolic debugger, used to interactively debug executable programs. It lets you do all the things you can do in the debuggers you’re used to, including:

gdb also has some features that are particular to Unix systems, and other features that I have not seen in other debuggers. These features will be the focus of this article; they include:

There are also some tricks that I’ve learned, one of which I will include at the end of the article.

Where to Find gdb

gdb comes with most GNU/Linux installations. It will be part of the development tools, which include the GNU C/C++ compiler (gcc/g++), GNU assembler (gas), GNU linker (gld), GNU make (gmake), etc. (Note that on GNU/Linux machines, these tools are named cc/c++, as, ld, and make, respectively. I will refer to these tools by these standard names, not the original names.) If you are using a different Unix variant, you can get the sources for these development tools from the GNU web site (www.gnu.org) or various mirrors.

Of course, building a C/C++ compiler written in C is a bit of a chicken/egg problem. But the Free Software Foundation also has these tools available for some of the more common Unix variants, such as SCO Unix, Solaris, HP/UX and others. If you have a single-user C/C++ compiler from your Unix vendor, and are feeling adventurous, you can build an unlimited-user version of the GNU development tools. That process is a bit beyond the scope of this article, however. One problem with doing this is that different Unix vendors have different methods of including debugging information in executables. Because of this, gcc and gdb may not work well together on your Unix machine. Or gdb may not work well with programs compiled with the vendor’s C/C++ compiler, or the vendor’s debugger may not work well with programs compiled with gcc. However, if you are using GNU/Linux, these problems are irrelevant. (And if you are not, then solving these problems involves modifications to gcc and/or gdb, which is beyond the scope of this article. In either case, we will assume that the C/C++ compiler you have works well with gdb, and leave it at that.)

One advantage of getting the source code is that it comes with a quick reference card that you can print out. This is the file gdb/refcard.ps in the source distribution. This reference card will give you the basic information you need to debug programs. (Note to the GNU/Linux distributors: it would be really helpful to have this file included in the development packages.)

GNU/Linux has a good on-line help system called "info." You can get detailed information about any of the features discussed here, along with all the features not discussed, by executing info gdb from a command line.

Finally, gdb itself has internal help. You can get a quick list of commands by typing help, and more information on specific commands by typing help <command>. For example, type help set or help set print to get information about the options you can set, and the print options you can set (respectively).

Preparing Your Programs

There are a few different ways of preparing your programs for use with gdb. The most basic is to compile with any flags, and then link without using -s (strip symbols.) This allows you to get stack information while debugging, but not much else. As such, it can be useful when examining a core file (more later) but is not really useful for interactive debugging.

The most useful way to prepare your programs is to compile and link with -g. This includes all the symbolic information in the final executable, and allows full symbolic debugging, which includes seeing the values of variables, setting variables, seeing stack information, setting breakpoints, etc. Of course, your executable file grows accordingly.

You may find one quirk with gdb when you compile with some optimization flags. This is a perfectly valid thing to do, but may cause the current line number to jump around erratically during execution. This is because optimization causes the program to be rewritten a bit during compilation. I think you can see the use of this, though. How often have you had a bug that only shows itself when the program is compiled fully optimized? At least now you can still do useful debugging.

The first thing you’ll notice about gdb when you start it is that it is a command-line driven program. There are graphical front ends to gdb that can simplify your debugging experience. See "GUI Debugging Notes" at the end of this article for some (incomplete) information.

A Sample Program

The sample program I will use to illustrate the points in this article appears in Listing 1. This is a stupid little program which has a major bug (referencing a null pointer.) It also handles some signals, and has a lot of functions that match a particular regular expression. All of these features will show some useful feature of gdb, in due time.

There are three main ways of starting gdb. You can statically examine a core file generated by a program crash. You can run a program on disk. Or you can attach to a currently executing program.

Core files

Statically examining core files is the one method not available under Windows (as far as I know.) It is also the least useful in general. This is because the program is no longer executing, so all you can do is collect evidence for why the program crashed. But it can sometimes be used to determine why a program crashed at a user’s site, for example, just by having the user send the core file generated.

Consider the sample program. If you compile and run it, a core file will be generated. Compile using cc -c -g test01.c and link using cc -g -o test01 test01.o, and then run it. You should see the output listed. You can now start gdb using gdb test01 core to examine the core file, and you will see the current stack frame where the program died:

#0  0x40051861 in strlen ()

You can see the current stack by typing backtrace or bt, and see:

#0  0x40051861 in strlen ()
#1  0x804854c in func4 (ptr=0x0) at test01.c:7
#2  0x80485f6 in func3 (i=0) at test01.c:22
#3  0x804863a in main () at test01.c:33

You can examine the previous stack frame by typing up, and see:

#1  0x804854c in func4 (ptr=0x0) at test01.c:7
7           printf( "strlen( %s ) = %d\n", ptr, strlen( ptr ));

You can print the current value of ptr by typing print ptr or p ptr to see:

$1 = 0x0

And now we know exactly why this program died: we passed a null pointer into strlen. In this particular case, we didn’t learn anything we couldn’t have learned by executing the program itself from within gdb, and letting the program crash. But there are times when getting this much information from a user can really help in the debugging process.

Executing a Program on Disk

This feature is so like any debugger you have ever worked with, it deserves only a little discussion. You start gdb by executing gdb test01 at which time you see the gdb prompt. You then run your program by executing run [arglist] where [arglist] is the list of command-line arguments you want to give your program.

Attaching to Executing Programs

You can attach to a currently running process with gdb. This is useful in a couple of situations that I run into all the time. The first is when you are trying to debug the user interface of your program. Since gdb is a command-line tool, when you run your program under the control of gdb, both gdb and your program are using the same terminal. Interacting with gdb therefore destroys the screen formatting that you are trying to debug. The second situation is when you are trying to debug a daemon process that forks itself as it is starting (putting itself in the background). gdb will follow the parent process, so you aren’t able to debug the child process. Starting the daemon, and then attaching to the child allows you to debug the daemon.

The command-line invocation for attaching to a process is simply gdb program pid, where pid is the process id of the process you want to debug. The process is now stopped, and is under the control of gdb. At this point, you control the process just as if you were debugging any other program.

Handling Signals

Signals are a feature of Unix programming not available under Windows (unless you are using one of the many Unix-on-Windows tools). Signals allow a program to try to do something reasonable when something unreasonable happens. For example, you can tell your program to try to close all its files cleanly when the user presses the INTERRUPT key (usually Ctrl-C.) A more thorough discussion of signals is beyond the scope of this article. But the point is that gdb by default detects signals thrown by your program, and stops execution at that point. This means that it is difficult to debug your signal-handling code.

There are two ways to debug your signal-handling code with gdb. You can send a signal to your program through gdb, or you can tell gdb to change its default handling of signals.

Consider the sample program in Listing 1 again. This program handles the SIGHUP signal(which is sent when a user logs off of the computer while the program is running in the background, for instance) and SIGINT (which is sent when the user presses the INTERRUPT key, usually Ctrl-C.) It would be hard to send this particular program a SIGHUP or SIGINT signal, since this program executes so quickly. However, you can easily send it a SIGHUP signal from within gdb by executing signal 1 at the gdb prompt. (Note that SIGHUP is #defined as 1 in /usr/include/signal.h, or one of the files it includes.)

Sometimes, though, you really need to have run your program through a real life situation that causes the signal to be sent, and you want to run under gdb. In this case, the best thing to do is to change the default handling for the signal in question. You do this by using gdb’s handle command. (You can see the default action by executing info signals.) For each signal, gdb can either stop or not stop, announce or not announce, and pass or not pass the signal to the program. Each of these three actions is independent of each other. To change the default behavior, you can execute signal 1 pass to cause gdb to pass the signal to the program, for example.

Calling functions

One of the features I really miss when I’m debugging using Visual C/C++ is the ability to call functions easily. Sure, I can set the program counter to the top of the function, execute the function, and then set the program counter back to where I was. At least, conceptually I can do that, though I have never tried. With gdb, I can execute any function that is part of the executing program just by typing call func(params). For example, again considering the trivial program in Listing 1, I can execute call func3(20) or call func4("this is a test"). And this can be done without any special preparation apart from the usual debugging preparation, and without modifying source code in any way.

One reason I commonly use this feature is when I have a routine that prints a collection of interesting information to the screen. I can call this routine from any point in the program without actually modifying the source code to call it.

I may also use this feature to debug a newly written function that I have not actually called from any location in my program yet, because I am still developing the feature. I want to test the function, but I don’t want to change too much code that will need to be changed back after I am done.

Advanced Breakpoints

Sometimes you are debugging a program, and want to set a large number of breakpoints. If you are lucky, the functions you want to break at are named similarly (because you are testing all the signal-handling code, for instance). You can set multiple breakpoints based on a regular expression, using the rbreak command. For example, to set breakpoints at both sigfunc1 and sigfunc2 with a single command, execute rbreak sigfun. Since both sigfunc1 and sigfunc2 are matched by the regular expression "sigfun", breakpoints are set at those two functions. Once set, these breakpoints are treated just like any other breakpoints, and can be disabled, have conditions set on them, etc. You need to be a bit careful when using this feature. If you execute rbreak func in the sample program, you will get breakpoints at func3, func4, sigfunc1 and sigfunc2. And if you execute rbreak sig in the sample program, you will get breakpoints at sigfunc1, sigfunc2, and signal (for which you have no source!)

You can also make gdb execute a collection of commands every time it stops at a particular breakpoint. The obvious use for this is to print the values of certain variables. Another use could be to set variables to known (or suspected) correct values, having determined that they were being set incorrectly in the program, and you want to test the new settings. Any collection of commands can be executed, just by executing:

commands n
(command-list)
end

where n is the breakpoint number, and (command-list) is the list of commands to execute when breakpoint n is hit.

Initialization and Scripting

You can create new commands in gdb by using the define command. Any function you define can take up to ten arguments, referenced in the defined command as $arg0 ... $arg9. For example:

define px
  print /x $arg0
end

You now have a command for printing in hexadecimal instead of the default print radix (which is different for different data types.)

This feature is only moderately useful unless you can save the new commands in a file and execute that file every time you start gdb. And of course you can. The source command takes a single argument, which is the name of a file containing gdb commands. gdb reads this file and executes any commands, including any define commands.

When you start gdb, it automatically sources two different files (if they exist.) First, it reads the file $HOME/.gdbinit, and then it reads the file ./.gdbinit. So $HOME/.gdbinit should contain commands you want to always execute for all programs you debug. And the .gdbinit file in the program directory should contain commands that are specific to the current program. For example, in my $HOME/.gdbinit I have set print pretty on so that structures are printed nicely. And then I have a .gdbinit file in certain program directories for executing initialization commands for that particular program.

Variable Printing Trick

One trick I have learned when dealing with arrays involves a few of the neat features of gdb. The relevant features are:

So, set a local variable, named $i (since this is an invalid C name, it should not interfere with your program,) to 0. Then print a single element of the array, indexed by $i++, which auto-increments the $i variable. And then press return for as many elements of the array you want to see. For example, again looking at the program in Listing 1, set a breakpoint at func4, run the program, and then enter:

(gdb) set $i=0
(gdb) p ptr[$i++]
$1 = 48 '0'
(gdb)
$2 = 49 '1'
(gdb)
$3 = 50 '2'
(gdb)
$4 = 51 '3'
(gdb)
$5 = 52 '4'
(gdb)
$6 = 53 '5'
(gdb)
$7 = 54 '6'
(gdb)
$8 = 55 '7'
(gdb)
$9 = 56 '8'
(gdb)

Note that the (gdb) prompt with nothing after it is where I just pressed return. For a character array like this, it would have been much simpler to just p ptr to see the whole thing. But arrays of structures can be printed just as easily.

Conclusion

gdb is a powerful program for debugging purposes. It has all the features you expect to find in a debugger, such as stepping through the program line by line, setting breakpoints, seeing and setting variable values, etc. It also has features specific to programming and debugging in a Unix environment, which make it particularly appealing for this kind of work. If you are programming on GNU/Linux machines, you need this tool. And the more you familiarize yourself with it, the more useful it becomes.

GUI Debugging Notes

1) xxgdb is a basic X-window graphical front end for gdb. You can install this program by installing the "xxgdb" package.

2) ddd is a motif graphical front end for gdb (and other debuggers), available from the Free Software Foundation (www.gnu.org).

3) gdb can be executed from within GNU/Emacs. You must still control the debugger from a command line. The advantage is that source code is shown in an emacs window automatically.

Randy Zack has been programming computers for 25 years and working with DOS/Windows and Unix variants since 1989, and with Linux in particular since 1994. He can be reached at rlzack@juno.com.