Dr. Dobb's Journal May 2002
I bought that amount in sepia, went to another shop and bought same amount — winning roll at first shop, losing at second — came out even.
The Moon is a Harsh Mistress, — Robert Heinlein
After my father returned from World War II, he and his brother Robert bought a restaurant on the square in Hummelstown, Pennsylvania. A cash register made in the 1920s by the National Cash Register Company (long before conglomerates became acronyms) was part of the deal.
He married my mother in 1947, they added a Gift Shop, and for the next 17 years that cash register rang up everything they sold. A bypass highway sucked the financial oxygen from Hummelstown's business district, but the gift shop survived as Mom became the town's tax collector and a Notary Public. She closed the shop in late 2000.
That sturdy old cash register served for the better part of a century with rare oilings, no cleaning, no periodic maintenance, and no repairs. Its 21 keys included "No Sale," 1 through 9, 10 through 90, $1, and $2. You engaged three fingers with the appropriate dollars, dimes, and pennies keys, then pushed down hard to ring up a sale. Three tabs appeared behind the glass window to indicate the sale.
In the 1920s, nearly all the transactions one might expect in a retail establishment were under $3.00, so anything over that amount required repeated pokes on the $1 or $2 keys. Although an odometer dial with a reset lever totalized the daily sales under the flip-up lid, the cash register did not add up separate items or print a receipt. Just before the auction, I found a yellowed paper tape roll for the register's audit printer with its original factory seal still intact; we had never used that feature.
Making a sale with more than one item, figuring sales tax, and returning change required mental arithmetic. I grew up using that thing and it certainly helped me learn my numbers, although I'm positive I was less useful than I recall.
Even the smallest of stores now have point-of-sale terminals that look up prices, compute totals, process credit cards, maintain inventory, and even present slide shows on otherwise idle screens. It seems the clerks cannot, for reasons we noted here last month, function without cybernetic augmentation.
Let's take a look at how some of these new registers fare in the real world. We'll see that even a clever and well-designed bit of hardware may not work quite as its designers might have expected. Our story begins one morning as our family set out on what was to have been a quick shopping trip.
We were waiting in the checkout line at a craft store when all the lights went out. I noticed that the "Open" lights above the registers remained on and figured that the store manager had the good sense to run the registers from an uninterruptible power source or was lucky enough to have them on another phase of the electrical service.
Many events can cause brief power interruptions. In our area, gray squirrels scamper along utility cables above roads and parking lots to keep from being mashed by cars. A few unfortunate ones immolate themselves across the ceramic insulators at the end of the line, popping the circuit interrupter on the pole or in a nearby substation. It's evolution in action: off the road, but up in smoke.
Utility circuit interrupters include automatic reclosers that, unlike home circuit breakers or fuses, attempt to restore power three times before locking open. Squirrels generally burn out during the first or second restrike, producing a series of pulses and low-voltage sags on the affected circuit.
Engineers designing industrial embedded systems generally include sturdy power supplies with long hold times to ride out brief interruptions, but the two-second cycle time of utility reclosers far exceeds the capabilities of any reasonable supply. Justifying additional product cost for items such as huge capacitors or internal batteries that won't improve normal operation requires more effort than most organizations can muster. At the low end, product cost counts for everything.
As a result, power supplies for PC-class computers became optimized for low cost at the expense of energy storage, to the extent that they cannot withstand more than a half-cycle or two of missing power. Losing power for a dozen milliseconds will crash a typical PC and, with no warning of an impending shutdown, the CPU cannot properly terminate its work in progress. Cross-dressing a commodity PC in point-of-sale cabinetry doesn't improve its tolerance for power-supply irregularities, either.
All computer systems go through a boot sequence when their power goes on. Booting a microprocessor-based embedded system may be as simple as reading a few bytes from ROM, verifying the integrity of battery-backed RAM, and continuing from the last checkpoint. More complex systems require a lengthy filesystem integrity check before resuming normal operation. Interrupting any recovery process with an additional power blip can have catastrophic effects because few people plan for a recovery-recovery operation.
As proof, consider what happens when the power goes off as you're reflashing the BIOS ROM in your PC. In the old days, you could wait for a FedExed replacement chip. Nowadays, the replacement part consists of the whole system board with its soldered-in-place Flash ROM chip. The expense of a socket can't withstand the pressure of "it won't happen that often" logic. Perhaps the cost of an on-site service call renders the parts cost irrelevant, but one wonders if a better design might eliminate the problem entirely.
Flash ROMs for PC BIOSes typically include a separate,write-protected boot block that should remain unchanged while reloading the rest of the ROM. That works perfectly well very nearly all the time. Sometimes it doesn't, for reasons Murphy can best explain, and a busted boot block means a new board.
Turning off the power during a filesystem rebuild gives you a good indication of how much importance the designers put on that part of the specifications. Error-handling code typically doesn't get enough attention and recovery code gets even less, so that's a tough test to pass.
Although the craft shop's lights came on within a few seconds, we didn't start moving through the registers. Being that sort of bear, I sidled into the unoccupied aisle to our right, hitched an eyeball over the barrier to get an employee's-eye view of the register, and saw a Windows 95 boot screen with its familiar scandisk thermometer chugging across the bottom. Yup, at least some cash registers do Windows!
It transpired that none of the cashiers knew the cold-booting procedure, which required the store manager's personal ministration on each keyboard after the scandisk finished its lengthy exam. To his credit, he tried to shuffle the cashiers around to get each register in action as quickly as possible, but the back-end system (presumably running from a UPS) assumed they were still signed on to the registers they'd been using before the crash and rejected their new logons.
After the cashiers logged on, they discovered that, although the screens displayed a bitmap image of the receipt, the printers didn't work. After the usual cycle-the-printer-power trick didn't work, the manager made the rounds again, applying a Vulcan Nerve Pinch to the printers that finally brought things back to full operation.
Elapsed time from the power blink until we were outside: 23 minutes. Loss of customer confidence: complete.
Inadequate power conditioning, an inappropriate OS, insufficient operator training, undocumented procedures...the list goes on. How many of those do you include in the specs for your embedded systems? They're common at the high end, but smaller systems now have the same failings and need the same attention during design.
We proceeded to a grocery store for their loss-leader frozen turkey. The freezer sported several "49¢ per pound" signs, so we added a hefty boulder to the cart. Everything went smoothly until we reached the door, at which point Mary, who had been scrutinizing the receipt, said "Hey, we paid a buck a pound for that bird!" Indeed, the label stuck to the bird's wrapper plainly read "Price per pound: $0.99." I returned to the freezer and found about half a dozen of the remaining boulders were priced at 99¢/lb and, inevitably, we'd picked one from that set. We adjourned to the Customer Service Counter, where the simplest way to solve the problem was get a refund for Bird A, pick out a properly marked Bird B, and buy that one.
In this day and age of barcoding and databases, we figured they would simply update their UPC database to reflect the new price and add a sign saying "Sale price will ring up at register." The manager said that, nope, they spent the night removing yesterday's stickers and affixing stickers with today's UPC barcode and prices. Evidently they missed a few birds, sorry about that, nothing personal, silly mistake.
It was about then that I noticed the "Sell before" dates on the stickers were a week in the past. The manager reassured us by saying those dates didn't mean anything, they're just arbitrary numbers, not to worry. Because the "Use before" date on the wrapper of the bird was a month away, we cut our losses and continued on our way.
The store's registers scan the barcode, look up the price, update the inventory, and charge the customer's credit card automatically. If any link in that chain breaks, the process fails. In our case, the fault lay in a simple sticker.
Which is obviously not an embedded systems issue. Right?
Next on our agenda was the health-food store where Mary had ordered a bag of soybeans earlier in the week. The clerk on the phone hadn't known the exact size of the bag, so Mary asked them to get whatever their standard bag was, which the clerk guessed at 20 pounds.
When they hauled it out of the stockroom, we discovered we'd signed up for 50 pounds of soybeans, which will last us for quite a while. You could pretty much hear the manager thinking "Whew!" when we agreed to absorb 4 dB more beans than we expected.
The cashier wrestled the bag onto the conveyor and tapped the item number on what was obviously another PC in POS drag. The automatic scale weighed it, the register computed the total price, knocked off a 10 percent bulk discount, and we were on our way. After I'd maneuvered the bag through the van's pod bay door, Mary handed me the receipt and asked if I saw anything odd.
Turned out that we'd bought 22.78 pounds of soybeans, not the 50 pounds clearly marked on the bag. No, it wasn't a metric-to-American conversion problem, either. I wasn't about to undock the bag and pilot it back uphill to the store, so we returned to see what they could do without the evidence.
After some huffing and puffing, the manager started a new transaction, typed in the missing weight, and let the register compute a new sale based on a 27.22 pound bag.
You've probably figured out what happened: A 50-pound bag of soybeans is just about twice as long as the scale's sensor plate. With only half the bag on the plate, the scale did the best it could.
Which is not an embedded systems issue either. Right?
You can see our adventure from at least two angles — either we had a really bad morning or this sort of thing happens to everyone all the time. We normally examine our receipts, but obviously we must start doing so before moving very far from the register lane. In both cases the errors would have been less obvious had we filled the carts with other items.
High-reliability system engineering already includes fault tolerance analysis, operator training requirements, and a myriad of other topics that are completely foreign to the low-end embedded design process. As those low-end systems become more complex, more interconnected, and more essential, we're beginning to see more failures that affect more people in much nastier ways. Seemingly simple systems must begin to shoulder the cost of doing thorough designs and careful implementations, which means vendors must demonstrate that the up-front costs actually provide some benefit.
Judging from my experiences and what I hear from Usually Reliable Sources, operator training (or the lack thereof) has much to do with many systems failures. It's easy to blame the employees and their lack of education, but they may not be the real problem.
Comprehensive manuals and well-defined procedures simply can't withstand high employee turnover; you may find that nobody in the building participated in the first power-on and that the oral traditions passed down from generation to generation cover only the most common procedures. Cold-booting a PC-oid register, for example, lies well outside the envelope.
We geeks tend to overestimate everybody else's technical interests. Because embedded systems are, by definition, invisible, we cannot expect users to understand the details of how a device works: They simply expect it to get the job done.
That old cash register may never ring up another sale, but it remains Y2K compliant, immune to power outages, easy to understand, and dead simple to operate. Can we say the same of our current designs?
Rolling the dice for a purchase, a basic part of Heinlein's Lunar society, lets you pick your own odds and payoff. Under those rules, we would have come out about even for the day — lost the turkey, won the soybeans. I'm not sure whose dice one should trust, though, as pseudo-random numbers from a cash register scare me.
Although we'll never know what caused that particular power blink, interruptions occur frequently enough that my half-dozen Windows and Linux boxes, plus network hubs and modems and related doodads, all run from UPS power. I'm surprised that more businesses don't follow suit.
If you don't already subscribe to comp.risks, a low traffic, moderated digest Usenet newsgroup, you should. Trawl the Risks archives at ftp://ftp.sri.com/risks or peruse the summaries at http://www.csl.sri.com/illustrative.html.
And, yes, that shopping trip actually happened just as you read it here!
DDJ