Bob Whitten is a senior software engineer for X O Technologies, Inc., Valencia, CA, a manufacturer of turbine flow meters, transmitters for flow meters, and flow totalizers and controllers. A programmer for 10 years, Bob has been involved in many embedded systems, especially "system" code. He can be reached at (805) 257-5542.
Though my favorite debugging environment is Turbo Debug, most of my projects are not MS-DOS-based, running instead on a microcontroller tied directly and intimately to the surrounding hardware. Usually, as soon as the prototype hardware is available, an effort is made to get something running on it, to see that the hardware works and to give us and management a good feeling that the project is going well. After all, if the hardware works, then the project is half-done, right? (This is where the software team lets out a loud groan...) When that first something is running, does it work? Does it do everything expected? More importantly, can you test the incremental software builds as they are produced?
Sometimes the hardware doesn't work, or not as specified. Or the software team interpreted the spec one way, and the hardware team went the other way. (Of course, the argument goes that this should have all come out during the technical walk-throughs, but since everybody thought they understood it, nobody mentioned it.) Usually, an LSI interface chip is involved and the documentation on it seemed clear, but later it turns out not to work the way everyone thought.
Other times the software doesn't work, usually because the software team isn't talking together enough (or in the case of a one-person job, the software engineer isn't talking to himself enough). Embedded programs can be tricky to write and even trickier to debug. I've been writing (and debugging) programs of this sort for a while now, and in this article I'd like to share what I've learned of how to use "external" tools in debugging. Three main tools — an Oscilloscope (and its kid brother, the logic probe), a Logic Analyzer, and an In-Circuit Emulator provide very different levels of help, and each has its place.
Using An Oscilloscope
An oscilloscope displays, or "traces", electrical signals from one or more input channels on a cathode-ray tube, showing how these signals change during a given time interval. 'Scopes have lots of knobs and switches, so they are a practical tool if you already know how to use them or have a good working relationship with someone who does. Attaching 'scope probes to hardware, especially prototype hardware, can threaten your job security, so I usually try to find someone else to do it. The oscilloscope can be useful mostly because it has a "trigger" circuit, which can be set to initiate a trace either when a signal goes high or when it goes low. The trigger once or repeatedly. Repeatedly is the normal setting since the image traced on the screen fades quickly; a signal that is repeated often will appear brighter. Sometimes the challenge of using a 'scope is making the program cycle on a regular enough basis to get a readable trace.The 'scope's screen is calibrated in centimeters, with voltage measurements on the y-axis, and time on the x-axis. You can select both the voltage range and the timing range. For digital circuits, the voltage range should be set to conveniently display zero to five volts.
Since the information displayed on the 'scope is limited to a couple of channels (two bits), it seems almost useless. It's amazing what a simple tool can do in the hands of skilled person, however, and the 'scope is no exception. I've been fortunate to work with people that seem to make the 'scope sing a ballad.
For example, if you're programming a microcontroller, sometimes it's enough to know whether or not the code reached a certain point. Since there is usually some output bit somewhere that is not used or does not cause any problems if set (like a Light-Emitting Diode), the code can include "milestones", where these outputs are set to indicate that the code got there.
for (sum = 0, i=0; i < 2048; i++) sum += *(PROM + i) if (sum != 0) while (1) ; /* hang forever */ outbyte( LED_PORT, 0x01); /* turn on the OK light */Now, arguably, the 'scope isn't needed here since the light will either go on or not. But what if the light goes on, but gets reset so rapidly that it never appears to light? What if the LED is inserted backwards, so it doesn't light? Just putting the 'scope probe on the output pin and looking for a change will begin to diagnose the problem.In addition to simple "does it get there" debugging, the 'scope is a great way to perform timing measurements. For example, a task that should complete within 30 ms could set an I/O port at its beginning and clear the I/O bit at completion. This will generate a pulse that can be traced on the 'scope. The time to execute the task and the time between executions, can then be easily read off the 'scope, based on the graduations on the CRT face.
Using two channels, the turn-around time for communications message processing can be easily measured by attaching the transmit line to one channel and the receive line to the other channel. You can even decode the message from the 'scope trace if you know the communications protocol well enough. (This is lots of fun with NRZI standards like HDLC.)
A good 'scope is considered a minimum requirement in most shops where hardware is being designed, but a 'scope can be overkill for other tasks. If you just need to do some "did the signal go high" testing, a logic probe might be adequate. The logic probe senses digital logic levels and has an LED for a signal high, another for signal low, another for a "pulsing" signal (slowed to human speeds), and yet another, labeled "memory", to show that a signal went high and then low (a single pulse). These are cheap (less than $50), simple, small, and don't have a lot of knobs.
Using A Logic Analyzer
A logic analyzer (LA) is like a collection of logic probes, in that it looks at logic levels at many locations, either high or low, but unlike a logic probe, an LA also allows those levels to be "clocked" into memory, usually based on the microcontroller clock signal. In most applications the LA must have at least as many input lines as there are address lines on the processor — more is better. The state of the lines is remembered on the basis of the clock input, which can be set to clock on either the high-going or the low-going edge. A careful study of the handbook for the particular processor is often needed to set clocks and clock edges correctly, and sometimes just experimenting till it "works" is the only way.Since microcontrollers may go through a million or so instructions in a second, just saving every instruction in the LA's limited memory is not feasible. To focus on an area of interest, the LA has its own kind of trigger mechanism, which can be as simple as waiting for some or all of the input lines to match user-set values, e.g., a given address. The analyzer may be set to start collecting frames into memory after the trigger is hit, or it may collect frames until the trigger, known as a "pre-trigger". In pre-trigger mode, if you set the "trigger" to the PANIC code, the LA will capture the addresses of the instructions executed immediately before the PANIC. Most LAs provide additional, very complex trigger schemes, to allow the user to catch a bug that occurs only in unusual circumstances.
While using an LA is a definite improvement over a 'scope, it has its own challenges. For starters, the LA doesn't understand C. It reports what it sees in machine language (i.e., ones and zeros, converted to hexadecimal), unless you've paid extra for a "personality module" that can display these codes in assembly language. Thus, unless you were born with sixteen fingers, you'd better have a hex calculator close at hand.
To use an LA to debug your C code, make your compiler produce listings with intermixed assembly language, and learn enough assembly language to understand what the compiler produced. Be sure you've turned off all optimizations — otherwise you'll find your lines of code moved around or folded together. If you use a linker, as you usually must, you will have to add the link map offsets to the addresses in each module's listing to produce the addresses seen by the LA. Sometimes, you can force the linker to align modules on 256 (100 hex) byte boundaries, making the hex arithmetic easier to figure in your head.
Logic analyzers are not ICEs (in-circuit emulators); an LA can only "see" the electrical signals on the microcontroller's bus. The LA can't "see" the activity of important circuits (communications, Analog-Digital conversion, timers, DMA), located within the microcontroller. Also, the LA doesn't allow you to stop and examine things and then continue. You can mitigate this limitation somewhat, at least during debug, by having your program copy important internal state information to an external memory location (causing the internal register data to appear on the external data bus).
The Art Of Debugging With A Logic Analyzer
As I remarked earlier, LAs typically have complex triggering mechanisms. Usually, simply triggering on a given address is sufficient, but when the really tough, once-every-hour bug comes along, the fancy triggering capability is invaluable. This is because the bug happens long before it is detected. If the trigger can be set to the place where the error is detected (for example, the hardware is set to a "fail-safe" state), sometimes there is enough "pre-trigger" memory to find out why the code got to this place.When that is not enough, the trickery has to start. Some analyzers will allow selective collection into memory, effectively expanding the memory by excluding un-interesting sections of code. Or, if there are only two paths that can bring the code to this one point, you can configure the trigger on an "OR" case: "trigger if either of these addresses is seen." Sometimes a certain section of code will execute correctly three times and fail consistently the fourth time. The trigger can sometimes be set to trigger on the "Nth" occurrence of an address.
As an aside, the LA can help find bugs that a traditional debugger like Turbo Debug cannot, because the LA is non-invasive. The timing of the code and the contents of memory are not affected by the logic analyzer — both are changed when a debugging program is loaded. Though it's usually easier to follow unoptimized code, in some cases you may be forced to debug the optimized version. When a bug is reported from the field, it may not manifest itself the same way unless the exact code from the field is used, loaded at exactly the same address.
Multi-level triggering is required when the suspicious code works most of the time. For example, a trigger might be set to trigger if the following sequence occurs: State 1 is reached, then State 2 is reached; if State 3 occurs before State 4, then trigger, otherwise, start over looking for State 1. This retiggering feature finds bugs of the type where the execution thread wanders off into code that is run commonly, but is not correct in a certain context.
An LA can also trigger on data accesses. It can be triggered on either a read or write, and even on the data at a certain address being accessed. This can help in those maddening situations where a data structure is getting "bashed" somewhere, but you haven't the foggiest where in the inch-thick listing that might be. The fancy triggering can come in quite handy in these cases. Let's say that the structure is legitimately changed in only one piece of code. The retriggering mechanism works well here. The Set-up would be something like this (this set-up is based on the Nicolet analyzer that I'm most familiar with):
S1 — a write request to the given data address
S2 — the start of the code that is allowed to change this address
S3 — the end of the code allowed to change this.
1. Collect frames until S1 occurs, then done. If S2 occurs first, go to step 2.
2. Wait for an S3 to occur, then go to step 1.
The analyzer can be set to trigger on "sequence done" or "memory full". "Sequence done" would be a good choice here (the pre-trigger memory will have the addresses of code leading up to the fault). This sequence should only be "done" if some other code writes to address S1. If S1 is written to during initialization, a step before these may be in order:
0. Wait for the address of the end of the initialization code, S4.
When a problem seems impossible to trigger on, I always get out the instruction book for the analyzer again, and hope to find something I missed the first dozen times through. I also do a "reality-check" if I've hooked the analyzer up and strange results appear. A reality-check is just a trace that triggers on the "trace memory full" condition after a restart. This trace should show the addresses and data from the first few instructions, and gives confidence that the analyzer is clocking correctly. It may also show how much switch bounce is in the reset button, by starting over and over again several times.
The most important part about using an analyzer is that setting "good" trigger sequences is an art that will be acquired over time.
Using An ICE
An In-Circuit Emulator (ICE) is different from a logic analyzer in that it replaces the microcontroller and allows the user a high degree of control over the execution of the processor. Because of this control, it is much more like the "Turbo Debug" environment. The user can single-step through the program, set breakpoints (much like in a regular debug program), and set watchpoints (executing until a variable is changed) that are checked in real time (not in slow-motion like the debug programs). At a break, you can examine the internal registers and the memory locations and the I/O ports.Many ICEs also include a trace option that allows the emulator to do the functions of an LA. This includes collecting a trace of where the execution has been, and fancy multi-level triggering. The ICE also supplies a few digital input lines for the user to connect as he pleases, to monitor the prototype hardware.
The ICE manufacturer also may have made a deal with the C compiler companies to allow source level debugging of the code, including single-line stepping, and setting a break based on a line number, and examination of variables by name. This makes the C programmer even more at home, and reduces the learning time significantly.
Since the ICE allows the execution to be stopped, the hardware's "watch-dog" timer must be disabled, if one exists. Also, disable code checksum tests during testing, since the ICE can change the contents of the program. Also, bear in mind that the C source single-step mode is line-oriented, so keep each line simple.
ICE is also good for patching "dumb mistakes" on the fly. For example, what if you wrote: "if ( a = = b )" but you meant "if (a > b)"? Making that one small change could mean 20 minutes work if you have to go back to your desk, edit, recompile, relink, reload, etc. With the ICE, you can "patch" the code and continue.
Conclusion
The tools described in this article can be useful in various circumstances. A 'scope is sometimes my first line of attack because it does certain tasks better than the others (like measuring timing). An emulator with integrated logic analyzer seems like the most powerful tool, but sometimes a logic analyzer has more triggering levels, or more input lines, or something that is required for a particular problem. Also, while a logic analyzer isn't tied to any one microcontroller, an emulator generally is (though you can purchase "personality modules" for other processors).Be flexible. Yet don't tell your boss that you can do it all with just a 'scope, either. I believe that software schedules get off track worst during the debugging phase. Nobody wants to plan to make mistakes. Don't forget to allow time to learn any new tools or methods that you'll have to learn.
In all of debugging, try to become "wholistic". Accept information through whatever means it comes, not just by staring blankly into the screen on the 'scope, logic analyzer, or emulator. If your product has LEDs, make sure they blink in the ways you expect them to. Listen to the clicks and clacks of external hardware, or to the change in tone of the power supply when the load changes. If you feel heat radiating from the hardware and you don't think it should, check it out — just do so carefully; I've burned my fingers more than once removing PROMs that I installed backwards. If you smell smoke, make sure it's not your hardware.
The choice of what tools to use can be very difficult. While the ICE seems the best choice, it can also be the most expensive, since you may need to buy a different one for the next project. Maybe the project is so simple that the code can be checked out on a PC, with minimal testing on actual hardware. Good debugging tools will never make up for bad programming, and many projects were completed without any fancy tools. The best tool to use is the best tool available for the task at hand. But who hasn't used a screwdriver handle to tap something into place, or a table knife to remove a screw? The craftsman can take what tools he has, and make them do his bidding.