José is vice president of engineering at Mainsoft Corp. He can be reached via Internet mail at jluu@mainsoft.com.
Sooner or later, all developers writing cross-platform applications to run on 80x86/Windows and RISC/UNIX platforms have to grapple with binary-file portability. Binary files saved by a PC usually have a data representation natural to the 80x86 architecture. If the application program is ported to the UNIX environment, the data representations in the new system are likely to have a different byte order, alignment, and size. To minimize programming efforts, you want the file read/write routines to work in both 80x86 and RISC environments and produce compatible data files, all without an elaborate rewrite of the source code.
Programmers choose to write binary data because it's easy. Usually, the format of the binary file is the same as the data image stored in memory. The code necessary to write binary data to a file is often as simple as a few instructions. There is no need to convert the data into some complex format. The raw data is just sent to the disk. It is simple, fast, and concise, but it also tends to be nonportable. One of the most popular forms of binary data, for instance, is the binary graphic image (bitmap). The graphic image consists of a header structure indicating how the data is arranged, followed by the data itself.
With MainWin for Workstations, a tool that provides the Windows API on UNIX workstations, one of our goals was to create a truly portable way to handle the differences in binary-data representation in all environments, while retaining a single source-code base and without having to manually add a lot of elaborate data-translation code. We knew programmers cross-developing PC-based Windows applications to run on these workstations would often need to deal with binary-file portability.
Our solution to the binary-data conundrum was to create an underlying technology we call "DDR compiler." While DDR is short for "DOS data representation," it could more precisely be referred to as the "Intel 16-bit data representation."
Our scheme works best when programmers simply write the contents of a structure to disk; see Example 1. Using DDR-based tools, you can automatically generate file read/write macros that handle the differences in binary-data formats between the 80x86 and a number of RISC workstation platforms (IBM RS/6000, HP700, Sun, SGI, and the like). You can also extend the tool to translate the data representation to or from practically any architecture. This is because the tool is built from a simple script that uses a common UNIX network utility found on most workstations, rpcgen.
The DDR compiler takes for its input the data typedefs and structures from your application's header files. Its output is a new header file and a C source file that implements custom read/write functions. This file can be easily incorporated into the application, making binary data available across all supported platforms. Because the new read/write macros are conditionally defined, they're expanded back into their original forms when compiled on the PC, allowing a single code base to be used for both the PC and RISC platforms.
This article details the basic DDR architecture. The complete source code for the DDR compiler described here is available free of charge at ftp.mainsoft.com. Although Mainsoft retains all rights to the source code, you can download a copy, modify it, use it for your own purposes (commercial or otherwise), and include the object code in your library. We do ask that you don't upload the software to a BBS or pass it along to others.
Of course, movement of data between systems with different data representations isn't new. It has been a perennial problem, and there are many schemes for managing such operations. The one we employed is based on that used by the Sun Remote Procedure Call (RPC) protocol.
The idea behind RPC is to allow programs to easily access compute resources across a network. A program running in one machine makes a call to a procedure in another machine. To make this work, a scheme had to be devised to standardize the data representations of the procedures' calling arguments and the results. To execute a procedure in another system, calling parameters are converted to a binary neutral format understood by the other system. The return values from the call are also provided in the binary neutral format. Each system provides the logic to translate data to and from the common binary format. This process, which is illustrated in Figure 1, is referred to as "data marshaling." There are a number of formats of this kind, each corresponding to a given protocol, including Courier data representation of the Xerox Network Protocol, Sun's XDR, and the X409 ISO standard.
To accomplish data translation to and from the standard neutral format, you have to write the necessary translation routines. Inventors of these protocols were quick to realize that this was a mechanical task that could easily be automated. That's how they came up with structure compilers that would write the translation procedures for them. The most widely available compiler is the rpcgen tool, so we chose it to develop the DDR compiler. Rpcgen is the structure compiler used for developing RPC procedures compliant to the Sun RPC protocol, so it is potentially available on many platforms.
Rpcgen's basic functionality is exactly what's needed to translate the 80x86's common data format into a form usable on RISC workstations. Rpcgen generates the high-level translation routines for the complex structures. However, nothing within rpcgen's domain allows for direct translation between particular data representations. For that, we developed our own low-level DOS/UNIX translation library routines. Figure 2 illustrates how we use rpcgen for DDR functionality.
The DDR tool consists of the ddrgen program and associated library and header files. Ddrgen is simply a script that wraps around a call to rpcgen and subsequently modifies the files generated by rpcgen so that calls are made to the ddr library instead of the xdr library.
Ddrgen takes as input the structure definition of the binary data, defined with approximately the same syntax as a C header file. I say "approximately" because the ddrgen syntaxer (the rpcgen syntaxer) does not allow complex structure declarations.
You might need to simplify some of the data-structure definitions. For instance, the rpcgen front end does not support the declaration of nested structures. In such situations, the nested structure can be declared at the top level and its name used inside other structures in order to achieve the same results.
The structure definitions drive ddrgen. As a result, ddrgen outputs three files that can be incorporated into your application source code:
The DDR compiler outputs a C file with the ddr_MYSTRUCTURE function call, which contains a series of function calls that encode or decode each data element. Since the input structure contains three data elements, the resulting function has three translation calls, arranged in the same order as the structure definition. Thus, the output file would look like Example 2(b).
Ddrgen creates the translation calls by constructing the function-call names from the data types called out in the input structure definition. The low-level functions for integers, shorts, and the like are provided in the translation library. To call the ddrgen-generated functions, you replace your binary file fread and fwrite function calls with new DDR calls that know about the translation code. (This assumes that the program is written in the way I described earlier, simply reading and writing data structures directly to a file. If the program does otherwise, some code would have to be rewritten to employ this scheme.)
Four new macros replace the standard functions.
To see how all the elements of the DDR system work together, I'll examine how a single data translation takes place, starting at a ddr_fread function call and tracing the process all the way through. I'll assume that you ran the DDR compiler and did the edits to your source files.
The ddr.h file, which is included by filename_ddr.c, contains many of the core definitions of the DDR system, including the conditional statements (#ifdefs) that select between the PC-compatible read/write routines and the new translation routines. The code is set up so that if it is compiled in a PC environment, the original fread and fwrite routines are used. If compiled for a supported RISC machine, the DDR routines are used. In this example, assume that I'm cross-developing to a 32-bit RISC machine.
In this case, the ddr_fread macro is expanded, as in Example 4(a). ddr_fread creates a DDR handle (ddrs) using the ddrstdio_create function. ddrstdio_create will initialize the ddr handle for reading from the file stream and set up compatible data-translation function calls in the DDR structure. In addition, it knows this will be a DDR_DECODE operation, and it stores the DDR_DECODE operation code in the DDR handle to use later to select a read routine.
The next line constructs a call to the user's structure-translation function. In the example, the user's structure is called MYSTRUCTURE; this parameter is substituted for ##Name to produce Example 4(b).
Recall that this function was created by ddrgen (in the filename_ddr.c). It sequentially executes a translation of each element in the structure. For this example, focus on just the first element since they all act approximately the same; see Example 4(c).
The function ddr_int in Example 4(c) is a call to the DDR int translation function. (The function ddr_struc1 is a call to a function that has also been constructed by the ddr compiler when it has processed the previously defined struc1. This is how we handle nested data structures.)
The translation functions are constructed using a few primitives keyed to the 80x86 architecture: Read routines read 80x86 data, and write routines write 80x86 data. These primitives include read/write routines for byte (8-bits), short (16-bits), and long (32-bits).
ddr_int uses the DDR_DECODE opcode to select the DDR_GETSHORT macro. Example 5(a) shows the code that does it in ddr.c. ddr_int creates a short storage space (sValue) to receive the int from the selected macro. Notice that if the opcode was DDR_ENCODE, the DDR_PUTSHORT routine would have been selected. DDR_GETSHORT is a macro that expands Example 5(b). The macro DDR_GETSHORT is expanded to a call to the function stored in the getshort field of the operations_vector of ddrs.
Recall that I chose to use ddr_fread for example, and that the macro expanded to open our DDR handle with the function call ddrstdio_create(). As a part of its process, ddrstdio initialized ddrs->operations_vector to a whole list of low-level function calls designed specifically for the fread translation. This gives the flexibility to have other getshort operations read data from different sources. For instance, creating the DDR handle with ddrmem_create would allow translating data to or from a memory buffer instead of a file.
You can see in Listing One (page 88) that operations_vector -> getshort is a pointer to a function. It was initialized by the create function to call ddrstdio_getshort. This routine fetches the next int from the input stream and does the conversion, storing the result in the target structure. Example 6(a) shows this step.
The routine first freads the next short into a little 2-byte buffer, sBuffer. (Notice that this is the fread that you replaced with the ddr_fread macro. The DDR compiler has built all this superstructure above it and a little code below it also.)
Finally, MoveShort is called; see Listing Two (page 88). MoveShort has been selected by #ifdef architecture to be the translation macro for the selected target architecture. For this example, I'm assuming a Sparc or other Big-endian RISC machine. The actual byte-by-byte translation now takes place; see Example 6(b). (This implementation happens to be a macro, but it could also be implemented as a function.) This little gem takes bytes from pSrc (source) and stores them in pDest (destination), swapping the two bytes, as required by the RISC machine. pDest points to the short sValue allocated in the ddr_int routine. ddr_int will eventually get control back and put the short translated value (sValue) at the location pointed by ip (an integer pointer--4 bytes) in the application data structure. This implements the 2- to 4-byte extension required for integers.
To recapitulate, I substituted ddr_fread for fread. I then went from ddr_fread to ddrstdio_create and ddr_MYSTRUCTURE. I then chose to follow the first ddr translation, an int, which took us to ddr_int, and then to DDR_GETSHORT. DDR_GETSHORT expanded to ddrs->operations_vector>getshort in the DDR structure, which was initialized earlier (by ddrstdio_create) to ddrstdio_getshort. It read a 2-byte element from the input stream, translated it with a version of MoveShort (selected by an #ifdef architecture), and returned.
For all its complication internally, the DDR compiler turns out to be simple to use. It is a very useful tool for creating platform-independent read/write routines for binary data. The overall scheme is here. If you want to investigate it further, you can download the complete DDR compiler source code via ftp at ftp.mainsoft.com.
struct MYSTRUCTURE {
int mydata1;
struc1 mydata2;
long mydata3;
} mydata;
nStatus = fwrite (&mydata, sizeof(MYSTRUCTURE), 1, stream);
(a)
struct MYSTRUCTURE { int mydata1;
struc1 mydata2;
long mydata3;
};
(b)
bool_t ddr_MYSTRUCTURE (ddrs, objp)
DDR *ddrs;
MYSTRUCTURE *objp;
{
if (!ddr_int(ddrs,objp->mydata1))return (FALSE);
if (!ddr_struc1(ddrs,objp->mydata2))return (FALSE);
if (!ddr_long(ddrs,objp->mydata3))return (FALSE);
return (TRUE);
}
nStatus = write (fd, &mydata, sizeof(mydata)); ddr_write (fd, mydata, sizeof(mydata), MYSTRUCTURE, &nStatus); nStatus = fwrite (&mydata, sizeof(mydata), 1, stream); ddr_fwrite (&mydata, sizeof(mydata), 1, stream, MYSTRUCTURE, &nStatus); nStatus = read (fd, &mydata, sizeof(mydata)); ddr_read (fd, &mydata, sizeof(mydata), MYSTRUCTURE, &nStatus); nStatus = fread (&mydata, sizeof(mydata), 1, stream); ddr_fread (&mydata, sizeof(mydata), 1, stream, MYSTRUCTURE, &nStatus);
(b) substituting MYSTRUCTURE for ##Name; (c) the ddr_MYSTRUCTURE routine sequentially translates each structure element.
(a)
ddr_fread (pData, nSize, nNumber, stream, Name, pnStatus) { DDR ddrs;
ddrstdio_create (&ddrs, stream, DDR_DECODE);
if (ddr_##Name(&ddrs,(void*)(pData)))
*(pnStatus) = ddrs.nCount;
else *pnStatus = -1;
ddrstdio_destroy (&ddrs); /* kill the handle when done */
}
(b)
if (ddr_MYSTRUCTURE(&ddrs,(void*)(pData))) ...
(c)
bool_t ddr_MYSTRUCTURE (ddrs, objp)
DDR *ddrs;
MYSTRUCTURE *objp;
{
if (!ddr_int(ddrs, objp->mydata1)) return (FALSE);
if (!ddr_struc1(ddrs, objp->mydata1)) return (FALSE);
/* etc... */
return (TRUE);
}
(a)
bool_t ddr_int(DDR * ddrs, int *ip) { short sValue;
switch (ddrs->operation) {
case DDR_ENCODE:
return (DDR_PUTSHORT(ddrs, ((short *) ip)));
case DDR_DECODE:
*ip = 0; /* clear whole int because the reading will
only affect the lower part */
if (!DDR_GETSHORT(ddrs, &sValue)) {
return (FALSE);
}
*ip = sValue; /* this is how the translated int is passed
back up */
return (TRUE);
(b)
#define DDR_GETSHORT (ddrs, shortp) \
(*(ddrs)->operations_vector->getshort)(ddrs, shortp)
(a)
boo_t ddrstdio_getshort(DDR * ddrs, short *sp) { short sBuffer;
if (fread((caddr_t) & sBuffer, sizeof(short), 1, (FILE *)
ddrs->ddr_private) != 1)
return (FALSE); /* If you cant get the int, return a fail
code */
MoveShort(&sBuffer, sp);
ddrs->nCount += sizeof(short);
return (TRUE);
}
(b)
#define MoveShort(pSrc,pDest) (*(char*)(pDest) = \
*((char *)(pSrc)+1), \
*((char*)(pDest)+1) = *(char *)(pSrc))
typedef struct {
enum ddr_op operation;
struct ddr_ops *operations_vector;
void * ddr_public; /* for application usage */
char * ddr_private; /* for internal use 1 */
char * ddr_base; /* for internal use 2 */
int nCount; /* for internal use 3 */
} DDR;
struct ddr_ops {
BOOL (*getlong)();
BOOL (*putlong)();
BOOL (*getshort)();
BOOL (*putshort)();
BOOL (*getbytes)();
BOOL (*putbytes)();
void (*destroy)();
} ;
enum ddr_op {
DDR_ENCODE = 0,
DDR_DECODE = 1,
} ;
#ifdef unix /* used in Ddr.h (via makeheaders) */
#if defined(sparc) || defined(rs6000) || defined (hp700) || defined(m88k)
#define BIG_ENDIAN
#define ALIGNED
#elif defined(i86) || defined(vax)
#define LITTLE_ENDIAN
#elif defined(mips)
#define LITTLE_ENDIAN
#define ALIGNED
#endif
/* Move<type>(from,to) */
/*
* Byte moving, with byte-swapping.
* For use on 68000, sparc and most of the riscs
*/
#if defined(BIG_ENDIAN)
#define MoveByte(pSrc,pDest) (*(char*)(pDest) = *((char*)(pSrc)),1)
#define MoveShort(pSrc,pDest) (*(char*)(pDest) = *((char*)(pSrc)+1),\
*((char*)(pDest)+1) = *(char*)(pSrc),2)
#define MoveLong(pSrc,pDest) (*(char*)(pDest) = *((char*)(pSrc)+3),\
*((char*)(pDest)+1) = *((char*)(pSrc)+2),\
*((char*)(pDest)+2) = *((char*)(pSrc)+1),\
#elif defined(LITTLE_ENDIAN)
#ifndef ALIGNED
/* Low-level byte moving, without byte-swapping. Use these definitions for 386
* and other low-enders ciscs that dont have alignment constraints. (vax) */
#define MoveByte(pSrc,pDest) (*(char *)(pDest) = *(char*)(pSrc),1)
#define MoveShort(pSrc,pDest) (*(short *)(pDest) = *(short*)(pSrc),2)
#define MoveLong(pSrc,pDest) (*(long *)(pDest) = *(long*)(pSrc),4)
#else
/* use this for mips(ultrix) and may be other riscs (alpha) */
#define MoveByte(pSrc,pDest) (*(char *)(pDest) = *(char *)(pSrc),1)
#define MoveShort(pSrc,pDest) (*(char*)(pDest) = *((char*)(pSrc)),\
*((char*)(pDest)+1) = *(char*)(pSrc)+1,2)
#define MoveLong(pSrc,pDest) (*(char*)(pDest) = *((char*)(pSrc)),\
*((char*)(pDest)+1) = *((char*)(pSrc)+1),\
*((char*)(pDest)+2) = *((char*)(pSrc)+2),\
*((char*)(pDest)+3) = *(char*)(pSrc)+3, 4)
#endif
#else
#error "unknown machine architecture"
#endif
Copyright © 1994, Dr. Dobb's Journal