Dear DDJ,
This letter is in response to the article "Implementing the Rhealstone Real-Time Benchmark," by Rabindra P. Kar, which appeared in the April 1990 issue of Dr. Dobb's Journal. There are several areas of the benchmark that could be improved, which we discuss below:
The interrupt disable time is the worst-case time in the kernel that interrupts are disabled (at task level), in order for the kernel to manipulate critical data structures. The processor cannot respond to interrupts at all during this time. Unfortunately, it is extremely difficult to measure this value, since there is little or no external indication from the processor that interrupts are disabled. You can measure it statistically using sophisticated hardware (emulators, oscilloscopes, and so on), but this measurement may not yield the true worst-case time. The only real way to determine it is by identifying the instruction stream in the kernel that disables interrupts for the longest period, and then calculating the timings of those instructions using worst-case timing figures for the processor. This may include cache misses, pre-fetch queue fills, memory wait states, etc.
Interrupt latency is the time it takes from the point an interrupt is acknowledged by the processor, up to the first application instruction in the interrupt service routine. The article describes one way to measure this value. For some kernels, this value is simply the time taken by the processor to respond to the interrupt, since they require no preamble or vectoring code through the kernel, but instead let the interrupt vector point directly to the interrupt service routine.
Interrupt service time is the time it takes, at interrupt level, to do whatever work is required by the kernel and application before returning to task level. Since nested interrupts at the same priority generally are not allowed during this period (although in some systems they can be), this timing can significantly affect interrupt response to bursts. This timing can be measured, given the appropriate test case. It should be recognized that different kernels have different mechanisms for servicing interrupts, and some have more alternatives than others. We suppose a particular kernel should be able to put forward its best case for this measurement.
Any benchmark attempting to define interrupt response characteristics should address all three of these factors. Rhealstone currently measures only interrupt latency.
There are other techniques available for avoiding priority inversion in these systems through explicit control of the application. The simplest way, if the application allows for it, is to avoid having unequal priority tasks compete for the same resources. If this is not possible, the following method can be used: Let's assume the classic example of three tasks H, M, and L, with respective priorities high, medium, and low. Tasks H and L share a common resource. Task M is unrelated to H or L and does not compete for the resource. When L wants to acquire a resource, it should explicitly raise its priority to that of the highest-priority task that also uses that resource (H), before taking the resource. This requires a few extra system calls, but achieves the purpose. It behaves a little differently than the iRMX/iRMK style, which lets M run briefly until H waits on the resource. However, the former behavior may be more efficient in that respect, as it avoids two context switches into and out of M. It does block M temporarily, even if H doesn't really want the resource at all during the time L has it.
To summarize, we feel that these issues should be addressed to improve the validity of the proposed Rhealstone benchmark. However, even with these corrections, the accuracy in performance prediction of a synthetic benchmark, such as Rhealstone, can never come close to that of an actual real-time program. We feel that benchmarks for real-time should describe or incorporate actual real-time programs. These programs will presumably call some proprietary real-time operating system services, so the operating system calls should be converted to a standard, portable, operating system interface such as POSIX, with its Real-Time Extensions. Only then can meaningful performance measurements be obtained.
Glenn Kasten, Ready Systems David Howard, Ready Systems Bob Walsh, Ready Systems
Robin responds: The authors of this letter raise some interesting and valid issues about the Rhealstone benchmark. Responding to each issue in-depth would make this almost a full-length article, so I will address some of the more important objections that they have raised.
One major criticism of Rhealstones is that it specifies the measurement of average times rather than worst-case time. This criticism stems from the notion that real-time systems (hardware + real-time OS or kernel) are benchmarked primarily to validate their critical response-time capability. The expectation here is that the Rhealstone number achieved by the system should indicate if it will meet interrupt response or other deadlines with 100 percent certainty. The reality is that benchmarks are used, by the computer industry, to evaluate and compare average, long-run performance of "typical system operations," after determining that the system can meet the job's minimum requirements. There is little doubt that any widely used real-time benchmark will be put to similar use. Consequently, Rhealstones measure average performance of "typical real-time operations." Running Rhealstones will not lift the burden of determining worst-case response times from the application designer's shoulders; it was never intended for that purpose.
Section 4 in the letter states that interrupt disable time, interrupt latency, and interrupt servicing together affect interrupt response; and further claims that Rhealstones only measures latency. My article defines interrupt latency as the delay between the CPU's receipt of an interrupt request and the execution of the first application-specific instruction. Their letter defines latency from the point that an interrupt is acknowledged by the CPU. In effect, Rhealstones measure both interrupt disable time as well as "latency," as defined in the letter. It does not measure interrupt service time, because that is entirely a function of the application, NOT the system.
Section 5 points out that real-time system performance may be impacted if the system is running a large number of concurrent tasks. It suggests that the benchmark should measure task-switch time, for example, with a hundred active tasks. Their point is well taken but, I believe, it is inappropriate for the benchmark to specify what the "background load" should be. Why a hundred tasks? Why not five tasks or five hundred? And what should each task be doing? Background loading is a very application-specific issue. The only way to obtain a generic benchmark number is to use an unloaded system (no background application tasks).
The most important objection raised in this letter is that we should not be trying to devise or use synthetic benchmarks like Rhealstones (or Whetstones or Dhrystones) at all. A book on computer architectures is cited to support the assertion that "a mixture of real programs" can be a better vehicle for performance evaluation than a synthetic benchmark. We could have a long and vigorous debate on that point. But even if it were true, Kasten, Howard, and Walsh seem to have overlooked some major drawbacks of using a suite of actual real-time programs as a benchmark.
Dear DDJ,
Facing the start of a Windows development, I recently reread the article which Tim Paterson and Steve Flenniken contributed to the March 1990 issue of DDJ, "Managing Multiple Data Segments Under Microsoft Windows." It was a good article. But I have a few questions about its applicability to Windows 3.0.
The obvious first question is does Windows 3.0 support the undocumented call they use to register the local segment table? Since recent Microsoft apps that ran under Win 2.0 and that they say use these techniques run unchanged under Win 3.0, I would guess that the answer is "Yes, the world does look a bit different, though, in Win 3.0 since the segment table is no longer handling segment numbers but rather protected-mode segment descriptors." Again, I would guess that this shouldn't make too much difference.
In the second article in the series, Paterson and Flenniken spend time discussing how the segment table technique fits in the EMS. Since Win 3.0 manages all memory on a machine, I would think that one need no longer worry about EMS but should write an application as if one had much more global memory available for allocation. One might still be memory constrained on a machine with 1 Meg of RAM running in Win 3.0 real mode, but now it seems that the solutions are to write to use less RAM (which might include roll-your-own virtual memory) or to require more RAM installed as extended memory and use Win 3.0's protected memory mode.
I actually find it a bit disturbing that Microsoft uses these techniques in their products but does not document them. It seems to give the lie to their assertion that Microsoft apps writers have no secret information about Windows that gives them a competitive advantage.
In addition to commenting on the above, I would appreciate any other comments Tim Paterson might have with regard to using memory in Win 3.0.
Steve Williams
3Com Corporation
Santa Clara, California
Tim responds: Steve asked some very good questions about the relationship between Windows 3.0 and the segment table I described in my February/March article. Fortunately, the answers are surprisingly simple, once you know a little bit about the new Windows.
Windows 3.0 has three operating modes, selected when it starts up. Real mode is identical to Windows 2.x, so it can run all Windows 2.x applications unchanged. Real mode even works on 8088/8086 processors, just as Windows 2.x did. Real mode also supports the segment table exactly as I described in my article. In fact, the new Windows Software Development Kit (SDK) includes a description of the Define Handle Table() function that kicks in the segment table, although the one-paragraph description is a little thin to relate a full understanding of its application.
The other modes for Windows 3.0 are Standard mode (or 286 mode) and 386 Enhanced mode. From the standpoint of a Windows application, these modes are the same; their main difference is in how they handle non-Windows programs. These modes run the processor in protected mode, where the values loaded into the segment registers are "selectors," not paragraph addresses. Even when the Windows memory manager moves or discards a segment in protected mode, the selector never changes; as far as the application is concerned, there is no memory movement. The concept of the segment table is meaningless, and the DefineHandleTable() function is ignored.
In other words, if you're willing to limit your applications to 286 or better processors running Windows 3.0, ignore my article. Not only will you get to pass far pointers around willy-nilly with no concern for memory movement, you will also get transparent access to extended memory. But if you want to run on the older machines (8088/8086) or with Windows 2.x, Windows 3.0 Real mode changes nothing. It appears that most companies, including Microsoft, will be taking the first approach.
I, too, found it disturbing that Microsoft applications were using a technique that was not documented -- that's why Steve Flenniken and I wrote the article. However, people at Microsoft left the impression that this was more of an oversight (or maybe just too much trouble to document) than intentionally holding back. I find myself working for Microsoft once again, and no one has complained to me that I told a secret.
Dear DDJ,
Regarding your June 1990 hypertext issue: How do I usually read DDJ? I usually read it on the train or as bedside reading. I also usually mark up the listings. Hypertext has its place, but not in the quiet contemplation that must accompany learning.
John O. Goyo
Port Credit, Ontario, Canada
Dear DDJ,
Lacking Barr Bauer's special qualifications and insights into the patent ethos, I would nonetheless like to reply to his letter which appeared in the July 1990 edition of DDJ. Mr. Bauer makes a convincing case that an inventor or investor should be encouraged to advance the cause of technological innovation. Whether patent laws do this or not is not clear.
It appears that patent laws can benefit large, well-financed organizations, but this benefit comes more from the ability to defend themselves in court than from any innate protection afforded by the laws themselves. Many innovations of the twentieth century (button-release socket wrenches, intermittent-control windshield wipers, FM radio, television, to name only a few) have been appropriated from their original inventors and exploited by large organizations. And, ironically, a significant part of the success story of American industry in the late 1800s-early-1900s and beyond is the story of industrial espionage and the infringement of some key European patents.
Whether or not investors and inventors benefit in the long run from patent laws, it seems to me that a larger question when the laws are applied to software is whether the craft or society itself benefits. As Mr. Bauer points out, changes in software come so rapidly that patents may outlive their usefulness well before they expire. If this is so, how can developers ever hope to build on existing software to further the state of the art?
I think patenting will tend to further fragment the development of software, adding unnecessary costs and delaying true innovation. Isaac Newton said "I have stood on the shoulders of giants," and by this he meant: No programmer is ever going to get anywhere if he has to go back to square one every time he boots his system.
Phil Wettersten
Chillicothe, Ohio
We welcome your comments (and suggestions). Mail your letters (include disk if your letter is lengthy or contains code) to DDJ, 501 Galveston Dr., Redwood City, CA 94063, or send them electronically to CompuServe 76704,50 or via MCI Mail, c/o DDJ. Please include your name, city, and state. We reserve the right to edit letters.