In this installment I am going to investigate the use of my XDR_Stream class. In the time since I wrote the previous column, I have been experimenting with XDR_Stream on a fairly regular basis. Most of this has been related to data communications, but in the back of my mind, I have been thinking about using XDR as an object persistence mechanism.
In my experimentation, I have been reminded of the adage that "code isn't reusable until it has been reused at least once." I might go even further and declare that it isn't even usable until it has been reused at least once. (We have all encountered libraries that seemed to have been designed in such a way that only the original developer could have possibly found them easy to use.) One of the things I like to do when I design something is let it sit for a while and then go back and try to use it myself. I may not approach it with the total ignorance of a new user, but even a few days away from daily working with the internals of a library can shed light on how well the externals are thought out. In this column, I am going to take advantage of the fact that "The (B)Leading Edge" is now web based and go into some detail about the considerations that I have dealt with in trying to make my XDR_Stream into something useful. This column could easily be subtitled "How a Design Evolves." Here I will look at some improvements that I made to XDR_Stream to make it easier to use in its primary role as a means for encoding/decoding XDR streams for use as a communication protocol. Next column, I will go into some extensions that make XDR_Stream more usable as a data persistence mechanism.
One thing I did was take a complete look at the set of member functions that I provided and described in the previous column [1]. As I used the XDR_Stream, I asked myself "Does this make sense?", "Would something else be easier to use or more intuitive?", and finally, "Should this be a member function, a non-member function, or is it something I do not really need at all?"
I will mention a couple of the simple changes that I made (the code is available for download from the CUJ website). For example, I originally had the following member functions:
oXDR_Stream& vput (const std::vector<unsigned char>&); size_t vget (std::vector<unsigned char>&); template<typename T> oXDR_Stream& vput_arr (const std::vector<T>&); template<typename T> size_t vget_arr (std::vector<T>&);The first two were for XDR's variable length opaque data type, and the second two were for variable length array encoding/decoding. I quickly realized that I did not need any of them. What I needed were insertion and extraction operators for the appropriate objects. I removed the above and added the following non-member functions:
oXDR_Stream& operator<<(oXDR_Stream&, const std::basic_string<unsigned char>&); iXDR_Stream& operator>>(iXDR_Stream&, std::basic_string<unsigned char>&); template<typename T> oXDR_Stream& operator<<(oXDR_Stream&, const std::vector<T>&); template<typename T> iXDR_Stream& operator>>(iXDR_Stream&, std::vector<T>&);You will note that I switched from a vector<unsigned char> to a basic_string<unsigned char> as the object to read/write opaque data to/from. I discovered that the string interface was much more useful for manipulating a chunk of opaque data than the vector interface.
I also changed some of the member function names. Now, instead of:
put vput get vgetto encode/decode opaque data, I have:
put_opaque vput_opaque get_opaque vget_opaqueI did this primarily for safety and clarity. In ordinary IOStreams, the put/get functions operate on arrays of the underlying character type. For an XDR_Stream, this would correspond to an array of XDR_Chars. The opaque data functions read and write from raw character buffers. I decided that it might be confusing at some point to have the opaque data functions have the same names as what might be expected for XDR_Char strings.
Finally, I also made a first pass at adding error handling. It turned out that the original version of XDR_Stream actually worked fairly well without error handling. The binary nature of the XDR protocol eliminates most possible failure modes that ordinary IOStreams have to check. Nevertheless, some error handling is obviously necessary. One of the primary reasons that I derived XDR_Stream from std::basic_ios was to have access to the iostate information that basic_ios maintains. The XDR_Stream code now incorporates basic error checking and sets the iostate accordingly.
Using XDR_Stream
One reader wrote to me and asked how using an XDR_Stream was simpler than using the xdr_xxx function libraries. On the one hand, I guess this is just a matter of opinion — it usually feels easier to use what you already know than to learn something new. On the other hand, I know that I started looking seriously at XDR as a protocol for some high performance data transmission stuff that I needed to do primarily because I now had the library available. The underlying xdr_xxx libraries were always available, but the XDR_Stream class just seemed to make the protocol much more approachable — if that makes any sense.
Let me make a (hopefully brief) side note. In my experience, I have found that when it comes to being accessible, functionality falls into several categories. Obviously, the most accessible functions are built into the programming language itself. These functions fall into the first category. This can range from something a simple as having string be a built-in type, to the facilities for multitasking. The second category contains functions that are part of the standard library of the language in use. Personally, I tend to lump these two categories together. Is "new" (or malloc) a library function or a built-in operator? As a result, I am firmly in the "put it in the library, if you have a choice" camp. Not everyone feels the same way. Clearly, most programmers are less familiar with the details of their standard library than they are with their programming language, so you can make the case that if you really want something used, you have to put it into the language. These first two categories are overwhelmingly the most important. I think it is safe to say that there are an awful lot of professional programmers out there that do not know much beyond these two categories.
The third category is those libraries that are distributed with the development environment itself. Examples of these are Microsoft's Foundation Classes and the Rogue Wave Library. A lot of people apparently think these are actually part of the standard library.
The fourth category is standard APIs that are widely supported on different platforms. Certain POSIX APIs, such as Pthreads, and the X-Window APIs are good examples of this. I would say that the XDR API falls into this category.
While there are probably a lot of people who know the xdr_xxx functions, it is also safe to say that most programmers do not eagerly set out to learn APIs in the fourth category or greater. No matter what the subject area, such APIs always have some quirks. First you have to learn the naming convention (e.g., xdr_xxx). Does the library have to be "initialized" before it can be used? What do the return values mean (is zero success or failure)? Even if the API is well designed and consistent, there is always some learning curve. Ordinarily, something like my XDR_Stream library would fall even lower down the hierarchy as a local utility library. But because the library is modeled on the IOStream library that is part of the standard library, I feel that moves the accessibility way up the scale. There is also the fact that the library is now a C++ class library instead of a C API. I feel that these two factors work together to make learning and using the XDR_Stream library much easier than the xdr_xxx functions.
In any case, I will hopefully show what I mean in some of the examples that follow.
A Concrete XDR_Stream
First, just to make things realistic, let me present a set of concrete XDR_Stream classes (Listing 1). These classes each take a pointer to an existing streambuf as an argument. In this way, I can effectively "wrap" an XDR_Stream around another stream. The existing streambuf can come from a standard stream such as a file stream, a string stream, or some user-defined streambuf such as a socket stream. Obviously, the latter is very useful when using an XDR_Stream as a communication protocol.
For most of the discussion that follows, I will be using a simple example: a little address database. The primary record in this collection is a Contact. (Please note that for expediency I do not show the qualifications for namespace.)
struct Contact { string name; Address addr; string phone; };where the Address portion is:
struct Address { string street1; string street2; string city; char state[2]; char zip[5]; };String issues
Now suppose that I want to write the XDR inserter and extractor for a Contact. First cut might look like this:
oXDR_Stream& operator<< (oXDR_Stream& oxs, const Contact& obj) { oxs << obj.name; oxs << obj.addr; oxs << obj.phone; return oxs; }and
iXDR_Stream& operator>> (iXDR_Stream& ixs, Contact& obj) { ixs >> obj.name; ixs >> obj.addr; ixs >> obj.phone; return ixs; }For unrepentant APL programmers, these can be reduced to:
return oxs << obj.name << obj.addr << obj.phone;and
return ixs >> obj.name >> obj.addr >> obj.phone;You cannot get much simpler than that. If we describe Contact in XDR's notational language, it looks almost exactly like it is coded:
struct Contact { string name<>; Address addr; string phone<>; );The corresponding functions for Address are only a little more complicated. Before we can write the functions for Address however, we have to decide how we want to encode the two fixed-length character arrays. As I mentioned in my previous column, XDR has no concept of a char as a fundamental type — a character is considered to be a very short integer. XDR also has no concept of a fixed-length string. So if we want to encode the arrays as arrays in XDR, they have to be encoded as an array of integers:
int state[2]; int zip[5];Obviously, since each character is going to be encoded as four bytes, this is pretty wasteful. Alternatively, the data can be represented as a fixed-size chunk of opaque data:
opaque state[2]; opaque zip[5];This will encode the data in the most optimal way for storage. Unfortunately, it now places the burden of doing any necessary data conversion on the application program. If you only work in an ASCII world, this might be acceptable to you, but I work almost daily with platforms that still use EBCDIC. One of the advantages of using a protocol like XDR is that it is responsible for handling such conversions when they are necessary. As a result, I choose to encode the fixed-sized character arrays as XDR strings. The XDR description of Address is:
struct Address { string street1<>; string street2<>; string city<>; string state<2>; string zip<5>; };This shows that state can be only two characters in length, and zip a maximum of five. With this decided, writing the corresponding XDR_Stream functions becomes straightforward:
oXDR_Stream& operator<< (oXDR_Stream& oxs, const Address& obj) { oxs << obj.street1 << obj.street2 << obj.city; oxs << string(obj.state, sizeof(obj.state)); oxs << string(obj.zip, sizeof(obj.zip)); return oxs; }The extractor is only slightly more complicated:
iXDR_Stream& operator>>(iXDR_Stream& ixs, Address& obj) { ixs >> obj.street1 >> obj.street2 >> obj.city; string str; ixs >> str; str.copy(obj.state, sizeof(obj.state)); ixs >> str; str.copy(obj.zip, sizeof(obj.zip)); return ixs; }Please note I am not considering errors here. The XDR_Stream itself, like any IOStream, will (or should) reject attempts to insert or extract items when the stream is not in a good state. I will admit that a production version of the above should, at the very least, construct and test a sentry object before attempting any operations. Furthermore, the Address extractor might want to verify that exactly two characters were read for the state, and five for the zip code.
Now, at about this point, I realized that just because XDR does not have the concept of a fixed-size string, there is no reason that my XDR_Stream should not support an interface for such. With this in mind, I added the following two member functions:
// output function oXDR_Stream& vput_string (const char* s, size_t n); // input function size_t vget_string (char* s, size_t n);I named the functions vput/vget to be consistent with the naming convention I had already used for variable-length arrays and opaque data.
I briefly considered giving my functions the semantics of the C library strncpy function. In other words, on output, the vput_string function would copy characters into the stream until either n characters had been copied, or until the next character in s was a null. On input, my thought was that if the actual length of the string in the XDR stream was less than the size of the buffer (as specified by n), then vget_string could terminate the resulting buffer with a null character. After some thought, I decided that this was not a good idea. (I warned you that I was going to give you an in-depth look at my thinking as I evolved the library interface.)
In the example above, I used the string member function copy. This function takes a pointer to a character buffer, the size of the buffer, and an optional starting position within the string (defaults to zero). It copies characters from the string into the buffer until it either runs out of characters or it runs out of buffer. Simple and safe and the appropriate way to set the values of state and zip in the extraction function. (I have to note that copy is probably the most underutilized string member function. Even people that seem to use string a lot often do not even know that it exists.)
The problem with copy is that it does not null terminate the buffer after copying the characters. Since copy returns the actual number of characters it copies, writing the code to null terminate the buffer is trivial if it is necessary, but it is just one more thing that has to be remembered and done. Many times I have used copy and been a little annoyed that it did not null terminate the buffer for me. In fact, on more than one occasion I have chosen to write something like:
strncpy(s, str.c_str(), sizeof(s));instead of
s[ str.copy(s, sizeof(s)-1) ] = '\0';because the former seems like a simpler way of getting a null-terminated string into a character array. The problem with the former is that it is not guaranteed to null terminate the string. It only works if the array is larger than the number of characters in str. Otherwise, strncpy will simply fill up the array with n characters. So, while I might find the behavior of string's copy function annoying because I have to do the null termination myself, it is guaranteed that I would find the alternative even more annoying.
In order for copy to guarantee that the array will always be null terminated, it would have to only copy up to n-1 characters. This is the way the basic_istream functions that extract into character arrays work. This is well-documented (and logical) behavior, but it always drives me nuts. If I want to extract four characters from a stream (for example a time in military format) into an array of type char[4], then I have to remember to do:
is.read(tmstr, 4);I cannot do:
is >> setw(4) >> tmstr;nor
is.get(tmstr, 4);Either of the latter only read three characters into tmstr. But they do guarantee that those three characters are null terminated.
The string copy member function does not worry about null terminating the resulting buffer. And I decided that my vget_string function would not either.
With the addition of these two functions to the XDR_Stream interface, the inserter and extractor for Address becomes:
oXDR_Stream& operator<< (oXDR_Stream& oxs, const Address& obj) { oxs << obj.street1 << obj.street2 << obj.city; oxs.vput_string(obj.state, sizeof(obj.state)); oxs.vput_string(obj.zip, sizeof(obj.zip)); return oxs; }and
iXDR_Stream& operator>> (iXDR_Stream& ixs, Address& obj) { ixs >> obj.street1 >> obj.street2 >> obj.city; long len = ixs.vget_string(obj.state, sizeof(obj.state)); assert(len == sizeof(obj.state)); len = ixs.vget_string(obj.zip, sizeof(obj.zip)); assert(len == sizeof(obj.zip)); return ixs; }You will note that the XDR description of Address did not change. I just use a different interface for encoding the two strings: state and zip. Again, the error handling is probably not acceptable for true production code.
Pointers
With a slightly improved string interface, I turned my attention to what I could/should do to make XDR_Stream more useful as an object persistence mechanism. I have to explain that my attitude towards simple object persistence in one of mutual assistance. The persistence mechanism provides some facilities, and the client then uses these facilities to actually do the persistence. Stated another way, I expect that the client will have to write some code, and maybe change some classes, in order to save and restore the objects. On the one hand, this gives the client more control, but it does mean that persistence has to be designed into the class and cannot just be added later.
When it comes to extra facilities needed in XDR_Stream to assist in saving and restoring objects, I am basically talking about dealing with pointers. Up to this point, XDR_Stream has the same philosophy toward pointers as other IOStreams classes: in other words it treats them as synonyms for arrays, but otherwise nothing special. This is fine if pointers are nothing but synonyms for arrays, but that is not all they are used for. More often than not, a pointer represents something in dynamically allocated memory. So far, XDR_Stream has taken the same approach to dynamic memory as IOStreams: allocating client objects is the client's responsibility. I decided that this was not sufficient if I wanted to build an object persistence mechanism around an XDR_Stream.
Trying to look at things systematically, I identified four different reasons for pointers to be used in a design:
- As a reference to a array whose size is known only at run time
- For something that was optional
- When reference semantics are actually required
- For a polymorphic object
Each of these uses has some impact on the evolution of XDR_Streams.
Dynamically Sized Arrays
XDR has the concept of both fixed- and variable-sized arrays of arbitrary data types. It also has the concept of fixed- and variable-sized opaque data, and of variable-sized character strings. In C/C++, variable-sized data is often represented by a pointer to an array allocated from the free store. It certainly seems unacceptable to have to allocate an array of the maximum possible size in order to read in variable-length data from an XDR_Stream.
The simplest solution to this problem is to use the standard library containers instead of pointers. In fact, in the previous column, I noted that since an iXDR_Stream did not have any provision to pass in a null pointer and have the function allocate an array of the correct size, using the standard library containers was the only way to avoid having to allocate an array of the maximum size.
The following extraction functions work with standard library containers.
operator>> (iXDR_Stream&, std::vector<T>&); operator>> (iXDR_Stream&, std::basic_string<unsigned char>&); operator>> (iXDR_Stream& ixs, std::string&);The first deals with variable-length arrays, the second with variable-length opaque data, and the third with variable-length strings.
Unfortunately, I decided that I couldn't tolerate my own arrogance in this case. I could not justify supporting only the standard library containers for this functionality. In effect, I was dictating one design solution.
To provide a concrete example, suppose that our Address struct is changed to:
struct Address { char* street1; char* street2; char* city; char state[2]; char zip[5]; };For variable-length arrays and opaque data, there is a simple workaround to the absence of functionality for having an XDR_Stream allocate the array or buffer for the client. As I mentioned last column, the client can read the length field as an integer, allocate the appropriate-sized array, and then read the data using the API for fixed-sized data. This means it is possible to easily create non-member functions that provided the capability. Since XDR does not have the concept of fixed-sized strings, creating a function to handle this case requires reading the string as opaque data and then doing any necessary character conversion (note that the vget_string function I added above still expects to extract the length field of the string as part of its operation). After using XDR_Stream for a little while, I decided that this functionality was useful enough to be made a part of XDR_Stream's regular interface.
First, I thought seriously about just enhancing the existing vget functions so that instead of taking a pointer and a length, they would take reference to a pointer and an optional length. If the reference was to a null pointer, then the function would allocate the appropriate buffer on behalf of the client. This has the advantage that a minimum of new signatures are added to iXDR_Stream interface, but in the end I rejected this approach. It seemed to be mixing functionality a little too much.
What I mean is that, on the one hand, a lot of variable-length data types do not specify a maximum length. Therefore I wanted my API to reflect this. On the other hand, if a client buffer is used, I always want the size of the buffer to be specified. So,
vget_array(buf);should be an acceptable call when buf is a null pointer and no maximum size is specified, but
vget_array(buf, 1024);should be required when buf is not a null pointer.
Since I cannot have both of the above, I decided to add three signatures to iXDR_Stream. Note that these are non-member functions:
pair<unsigned char*, size_t> vget_opaque (iXDR_Stream&, size_t n = -1); template<class T> pair<T*, ptrdiff_t> vget_array (iXDR_Stream&, ptrdiff_t n = -1); pair<char*, size_t> vget_string (iXDR_Stream&, size_t n = -1);For the return value, I borrowed an example in the standard library (get_temporary_buffer in header <memory>). The returned value is a pair with the first value the pointer to the allocated array, and the second value the number of elements in the array. The optional argument to each function specifies the maximum size of the array to allocate. If this argument is -1 (default), there is no maximum limit and the function attempts to allocate an array of whatever size is specified by the data in the stream. Upon return, the client becomes responsible for ownership of the array. The array must be freed by calling delete[] on the returned pointer.
Note that the problem of C-style strings being null terminated rears its ugly head yet again. Whereas I decided that the vget_string member function would never null terminate the string copied into the array, the vget_string non-member function works just the opposite, it always null terminates the resulting string. I do not like this kind of inconsistency, but the alternative is worse — at least for me.
Note that even though the resulting string returned by this version of vget_string is null terminated, I decided to keep the return value the same as for the other signatures. This allows for the possibility that a string might have embedded null characters, and it provides a little added jog to the memory that freeing the resulting array is now the responsibility of the client.
With these changes, the encode and decode operators for Address now look like:
oXDR_Stream& operator<< (oXDR_Stream& oxs, const Address& obj) { oxs << obj.street1 << obj.street2 << obj.city; oxs.vput_string(obj.state, sizeof(obj.state)); oxs.vput_string(obj.zip, sizeof(obj.zip)); return oxs; }and
iXDR_Stream& operator>> (iXDR_Stream& ixs, Address& obj) { delete[] obj.street1; obj.street1 = vget_string(ixs).first; delete[] obj.street2; obj.street2 = vget_string(ixs).first; delete[] obj.city; obj.city = vget_string(ixs).first; long len = ixs.vget_string(obj.state, sizeof(obj.state)); assert(len == sizeof(obj.state)); len = ixs.vget_string(obj.zip, sizeof(obj.zip)); assert(len == sizeof(obj.zip)); return ixs; }You will note that the encode function did not change, only the decoder.
Optional Data
Another common use for pointers is when something is optional. In this case, the pointer acts as both a flag to indicate the absence or presence of the data, as well as providing the reference to the data when it is present.
It turns out that the XDR notation contains the concept of optional data, and it even uses the C pointer syntax to describe it. In our example, let us suppose that the address portion of a Contact is optional:
struct Contact { std::string name; Address* addr; std::string phone; };The XDR description for this looks almost exactly like the code.
struct Contact { string name<>; Address *addr; string phone<>; );When optional data is encoded by XDR, it is treated like a variable-length array with a maximum possible length of one. This is equivalent to:
Address addr<1>;In the stream, the length field is either a zero, and no data follows the length, or it is one and one data element is encoded in the stream.
I thought about just leaving this up to the user to test the pointer and do the appropriate encoding. In the above case, we could have the statement:
oxs.vput_array(addr, (addr ? 1 : 0));Unfortunately, I found two potential problems with this. First, I thought that it was too easy to forget that the null case must be encoded also and write something like:
if (addr) oxs.vput_array(addr, 1);The more subtle problem is how to decode the data. It might be assumed that:
addr = vget_array(ixs, 1).first;would handle the situation, and indeed it will — almost. Since C++ does not support zero-length arrays, the vget_array function will have to return a null pointer if the array length in the stream is zero. Fine, so far, but if the data is present in the stream, then vget_array will allocate an array to hold the data by doing:
T* t = new T[1];Since the pointer was allocated by new[], it has to be deallocated by calling delete[]. This is probably not what the destructor for Contact is designed to do. A lot of programmers might figure that this is not really an issue. After all, the array form of delete is primarily to tell the compiler that it has to invoke the destructor more than once, and in this case, once is enough. Unfortunately, the Standard is clear that if you do not use the same form for new and delete, you have undefined behavior. This type of undefined behavior can be a source of very subtle bugs in the future, so it is always better to avoid it. For that reason, I decided to add two more non-member functions to the interface.
template<class T> oXDR_Stream& put_optional (oXDR_Stream&, const T* x); template<class T> T* get_optional (iXDR_Stream&)Using these on my example yields:
oXDR_Stream& operator<< (oXDR_Stream& oxs, const Contact& obj) { oxs << obj.name; put_optional(oxs, obj.addr); oxs << obj.phone; return oxs; };and
iXDR_Stream& operator>> (iXDR_Stream& ixs, Contact& obj) { ixs >> obj.name; obj.addr = get_optional<Address>(ixs); ixs >> obj.phone; return ixs; }With these changes, I felt that XDR_Streams were really starting to be truly useful. Up to this point, all the changes were helpful whether the XDR_Stream was being used for communications or anything else. In the next column, I will talk about some enhancements that I added to support reference semantics and finally support for polymorphic object encoding/decoding. As you will see, these enhancements fall outside the XDR protocol specification.
Reference
[1] Jack W. Reeves. " The (B)Leading Edge: Creating a Whole New Stream Class, " C/C++ User's Journal C++ Experts Forum, July 2001, <http://www.cuj.com/experts/1907/reeves.htm>.
Jack W. Reeves is an engineer and consultant specializing in object-oriented software design and implementation. His background includes Space Shuttle simulators, military CCCI systems, medical imaging systems, financial data systems, and numerous middleware and low-level libraries. He currently is living and working in Europe and can be contacted via jack_reeves@bleading-edge.com.