Jim is a software engineer on OS/2 and IBM Workplace technologies in Boca Raton, Florida. He can be reached through the DDJ offices.
Endian is a processor-addressing model that affects the byte ordering of data and instructions stored in computer memory, and the data's representation provided by a programming language. Endian concepts can be confusing since there are different Endian types, different ways to represent these types, and intertwined considerations for both code and data portability between opposite-endian hardware platforms. Historically, the term "Endian" comes from Gulliver's Travels, by Jonathan Swift:
It is computed that eleven Thousand Persons have, at several Times, suffered Death rather than submit to break their Eggs at the smaller End.
In the first installment of this two-part article, I will lay the groundwork by examining what Endian means from the programmer's perspective. In next month's article, I'll discuss how you can write portable software by applying Endian-neutral design and programming principles.
The most common addressing models are Big-endian, derived from the left-to-right order of writing in western-culture languages, and Little-endian, stemming from the right-to-left order of arithmetic operations in hardware processors. As Figure 1 illustrates, the Big-endian (BE) addressing model assigns or maps the lowest address to the highest-order (that is, the most significant or leftmost) data byte of a multibyte-scalar data item. The Little-endian (LE) addressing model assigns or maps the lowest address to the lowest-order (least significant or rightmost) data byte of a multibyte-scalar data item.
The "Endianness" of a multibyte-scalar data type such as an integer halfword or word is BE or LE. When compiled for a LE processor, its byte order is the reverse of the byte order compiled for a BE processor. The simplest way to think about Endian is that a LE scalar data item is equivalent to a byte-reversed BE scalar data item. Such a scalar should be treated as a single, indivisible data item although it has more than one byte and is composed of smaller addressable units of storage. Aggregate data such as files, data structures, and arrays are composed of multiple data elements; each element that is a multibyte scalar has Endianness. Byte values or single-byte character data do not have Endianness because the smallest addressable unit of memory is one byte; consequently, byte order is not an issue.
Some processors are Little-endian (Intel x86), others are Big-endian (IBM AS/400, System/370, Macintosh), and some are bi-endian (PowerPC) and can run in either BE or LE mode. In turn, the Endianness of software (code and data) is determined by the processor for which it is written.
The data structure in Figure 2 shows how Endianness can affect addressability and byte order. When a data structure containing different data types is compiled for a BE processor and again separately for a LE processor, note the following about the compiled data structure:
An Endian model maps addresses to the bytes of a multibyte scalar. There are different ways to illustrate Endian maps and forms of data for human viewing. The byte addresses of a LE data item are shown in either left-to-right or right-to-left order, with byte values appearing in the opposite order. For a BE data item, both addresses and bytes are shown in the same left-to-right order. The relationship between BE and LE mappings and their forms of representation are shown in Figure 3. Figure 4 is based on the sample data structure in Figure 2 but illustrated in the alternate left-to-right addressing form for LE. A disadvantage of this form is that the scalar data items do not appear in the more readable (to western cultures) left-to-right order.
In addition to BE and LE, other related Endian maps and forms may exist as part of a processor's addressing architecture or its implementation. Some special forms may be internal to a processor and transparent to software; they should not be confused with BE and LE, which are visible to software. BE and LE are most common, but you should not categorically assume that they are the only addressing models in existence and that all data in the world is only BE or LE.
Finally, it is interesting to compare how halfword, word, and doubleword integers can appear as members of a data structure in BE and LE form.
The data structure in Figure 5(a) has its BE/LE byte-address mappings shown next to it. Figure 5(b) shows a different mapping for LE than before. Finally, Figure 5(c) shows yet a different byte-address mapping for LE. For BE, the byte address of each byte value is the same in (a), (b), and (c) of Figure 5; for LE, the byte addresses are all different for the same byte value.
Multibyte-scalar data should be treated by software as a single, indivisible entity, such as an integer, pointer, or float. You can write code that treats a scalar as aggregate data by addressing a specific byte location or byte subfield internal to the scalar. This practice results in code that is not readily portable between Endians. In Figure 5, the short-integer s3.k data item is at address 04 for both Endian types, but its two component bytes are at different addresses! A program accessing data at location (char*) &s3.k+1 would find 0x16 when running in BE mode and 0x15 in LE mode. In short, when twiddling with the internal bits and bytes of scalar data, do not assume they are stored at a particular address; otherwise, such a program may break when ported to a different Endian. Bits can be more portably selected in BE or LE with bitwise operations such as n & 0x03FC0000 and be independent of byte address. The important principle is not to rely on those bits being stored at a particular byte address.
The classification of a processor, program, or data according to the addressing model it is based on (usually BE or LE) is its Endian type. A processor or program is said to execute in BE or LE mode. Furthermore, Endianness means being of a certain Endian type or mode. More generally, Endianness means the technical considerations for executing in different Endian modes and porting program code and data between BE and LE platforms. Endianness is not limited to any particular component of a system but can occur wherever data is addressed, retrieved, stored, processed, or transmitted.
A single-endian processor is architected as either Big- or Little-endian; most Intel processors, for example, are LE. Some processors are bi-endian, such as the PowerPC, which has the ability to run in either BE mode, LE mode, or both under software control. Bi-endian capability makes it possible to migrate existing operating systems, their applications, and data from both BE and LE platforms to a common bi-endian processor such as the PowerPC. (For more details, see the accompanying text box entitled, "PowerPC Bi-endian Capabilities.") The operating system is responsible for handling Endian-specific controls, registers, and interrupts that a processor may provide.
A processor has Endianness as a characteristic of its architecture. Therefore, hardware units that have embedded processors, such as video displays, printers, or communications adapters, take on the processor's Endianness, as does any software supporting the hardware unit. The Endianness of input/output data and commands between devices and adapters attached to a system of opposite Endian must be taken into consideration. Typically, all related system and attached hardware from a given manufacturer has the same Endian type. The situation is even more complicated in distributed computing environments, as described in the accompanying text box, "Distributed Environments and Endian."
The user of a stand-alone, single-endian system with all of its data being of the same Endian does not encounter Endian-related problems; however, if data of another Endian type is imported by LAN, communications, diskette, or other media, then software must handle conversion to the correct Endian. Endian conversion of data requires knowing the data structure, data type, and Endian type. A cross-platform application that runs on different Endian types of platforms needs Endian-conversion capability for data interchange with itself across different platforms. When applications are different, a conversion utility can be written to convert data files between different applications running on opposite-endian platforms.
The machine-executable instructions of compiled source code are handled as data during compilation into binary code and loading from disk for execution, and while being managed by the operating system. When being handled as data by other software, binary program code is subject to the same effects of Endianness as data and should be treated as multibyte scalar data.
A programming language represents data based on the same addressing model (Endian type) inherent to the processor for which it is compiled; left-to-right for BE and right-to-left for LE. Programming languages may extend data representation and provide data constructs down to the bit level (for example, bit fields in C) even though the processor allows addressing only to the byte level. A bit field, which can be thought of as a "tiny" integer, is a contiguous set of bits, where the most significant bit is on the left end and least significant bit is on the right end. Multiple bit fields can be defined within a word. The programming language, in general practice, extends the Endian type down to the bit-field level; that is, multiple bit fields defined within a word are represented in left-to-right order for BE and right-to-left order for LE.
Endian awareness is needed in today's open, interconnected systems for program portability and data interchange across BE and LE platforms. There are two basic consequences of Endianness:
The PowerPC is a bi-endian RISC processor that supports both Big- and Little-endian addressing models. The bi-endian architecture provides hardware and software developers with the flexibility to choose either mode when migrating operating systems and applications from their current BE or LE platforms to the PowerPC. Figure 6 shows the address mapping of its 32-bit executable instructions when running in BE mode and LE mode. These examples illustrate how program instructions are like multibyte-scalar data and are subject to the byte-order effect of Endian.
Each individual PowerPC machine instruction occupies an aligned word in storage as a 32-bit integer containing that instruction's value. In general, the appearance of instructions in memory is of no concern to the programmer. Program code in memory is inherently either a LE or BE sequence of instructions even if it is an Endian-neutral implementation of an algorithm.
How does the PowerPC handle both LE and BE addressing models? The processor calculates the effective address of data and instructions in the same manner whether in BE mode or LE mode; when in LE mode only, the PowerPC implementation further modifies the effective address to provide the appearance of LE memory to the program for loads and stores.
The operating system is responsible for establishing the Endian mode in which processes execute. Once a mode is selected, all subsequent memory loads and stores will be affected by the memory-addressing model defined for that mode. Byte-alignment and performance issues need to be understood before using an Endian mode for a given application. Alignment interrupts may occur in LE mode for the following load and store instructions:
A very powerful feature of the PowerPC architecture is the set of integer load-and-store instructions with byte reversal that allow applications to interchange or convert data from one Endian type to the other, without performance penalty. These load-and-store instructions are lhbrx/sthbrx, load/store halfword byte-reverse indexed and lwbrx/stwbrx, load/store word byte-reverse indexed. They are ideal for emulation programs that handle LE-type instructions and data, such as the emulation of the Intel instruction set and data. These instructions significantly improve performance in loading and storing LE data while executing PowerPC instructions in BE mode and emulating the Intel instruction behavior; this eliminates the byte-alignment and data-conversion overhead found in architectures that lack byte-reversal instructions. Currently, these instructions can be accessed only through assembly language. Until C compilers provide support to automatically generate the right load and store instructions for this type of data, C programs can rely on masking and concatenating operations or embed the assembly-language byte-reversal instruction.
--J.R.G.
A distributed application running between client desktops, servers, midframes, and mainframes depends on the communications model and its API for resolving Endian differences. In a mixed, distributed environment, applications must be able to compensate for differences in data representation between the systems that participate in the application.
Specific implementations for handling Endian and other conversions exist within applications written to lower-layer communications APIs. Higher-level application-development models like the Remote Procedure Call (RPC) of the Distributed Computing Environment (DCE) provide more general and robust support that isolates applications from these differences.
Most existing distributed software is written directly to a communications API. Typical communication interfaces are TCP/IP with a sockets or streams interface, NetBIOS with its own control block-based interface, or various SNA or ISO OSI interfaces.
Although communications APIs guarantee that data will be transmitted/received between network nodes, they do not understand the data types being transmitted and cannot convert data or data attributes, including Endian type, between clients and servers that have dissimilar data representations. This forces a distributed application to compensate for any differences.
DCE RPC allows an application to be developed as if it were nondistributed. At the same time, it allows any of the application's subroutines to be executed on a remote system. The RPC application-development model divides the local (client) and remote (server) parts of a program along an application's internal procedural interfaces.
Since the remote procedures are application defined, they must be able to support a variety of high-level language data types, including int, char, and struct. RPC hides the fact that data communications take place between client and server subroutines, and one of its functions is to interpret and convert native data-representation differences that may exist between the communicating systems. These differences include the addressing model (Endianness), alignment rules, character-set encoding, floating-point conventions, and numerical data formats.
Unlike writing directly to a communications API, writing to the DCE RPC interface allows you to ignore data representation and Endian conversion. DCE RPC can convert a well-defined, broad set of data types, including most C scalar and vector types as well as some extended types for use in a distributed environment. Examples of the latter include a byte data type to protect data from any conversion and a pipe data type to transfer large blocks of data.
The RPC data marshaling and unmarshaling routines handle the bulk of the data-conversion responsibility. Marshaling converts typed data into an encoded, linear buffer suitable for data communications. Unmarshaling recreates the typed data by interpreting the encoded data in the buffer. The marshaling/unmarshaling process takes, for example, a struct data type, decomposes it into its elements, and writes the data and a description of the struct into a single logical buffer. Unmarshaling rebuilds the struct by reading the data and description contained in the buffer.
A typical client/server call has at least two data transfers: The first is from client to server, and the second is the return flow back from server to client. The RPC subsystem takes the arguments from the procedural interfaces and assembles them into buffers using the Network Data Representation (NDR) encoding rules. The buffers constructed by the RPC marshaling routines include the data itself, as well as descriptors defining the type, size, and relative location of the data and its elements. Additional protocol information includes a field describing the native data representation of the transmitting system.
Embedded in the buffers containing the transmitted data is a variable that classifies the data as Big- or Little-endian. The algorithm used to properly decode or unmarshal the data buffers uses the principle of receiver-makes-right; see Figure 7. The receiver determines from the protocol information whether the transmitter's data representation is the same as its own. If so, no conversion is necessary. If not, a specific, standard conversion routine is called for each data type unmarshaled from the received packet(s). The data can then be presented to the application in the native-machine format.
In summary, a distributed application either compensates for any Endian differences when using lower-layer comunications APIs or uses a higher-level model such as DCE RPC that supports automatic conversions.
--J.R.G.
Figure 2 Typical C data structure and its Endian maps.
Figure 3 Relationship between Big- and Little-endian mappings and their forms; both mappings are 4-byte word examples.
Figure 4 Multibyte-scalar data items are reversed in this representation (as compared to Figure 2).
Figure 5 Comparing halfword, word, and doubleword integers as members of a data structure in Big- and Little-endian form.
Figure 6 The address mapping of PowerPC 32-bit executable instructions when running in BE and LE modes.
Figure 7 Typical DEC RPC call/return sequence.
Copyright © 1994, Dr. Dobb's Journal