My goal this month is to start showing you the other, more extensive group of changes in the works for Standard C. Understand, they are still subject to modification. Member nations of ISO, and other interested parties, get several rounds of comments before we in WG14 freeze the normative addendum. I like to think, however, that this particular material is reasonably stable. It should give you a portent of how Standard C will look in another year or so.

Equally important, my goal is to suggest some ways you might actually use these new capabilities. To some extent, the conversion to using a large character set is simple. Just look for all the places where you manipulate text as type char. Replace them with objects and values of type wchar_t. Whatever the implementation chooses for a large character set, its codes must be representable as that type. But all sorts of subtle changes must occur as well. I'll endeavor to show you some of them.

A display or printer that does the right thing with these multibyte sequences may well show Kanji characters, just the way the user desires. Your program neither knows nor cares what happens to them outside its boundaries. Multibyte is usually the representation of choice outside programs. If you plan to do little or no manipulation of such text, it can also be the representation of choice inside a program as well.

But lets say you have occasion to manipulate small amounts of text from a large character set within your program. You need to distinguish character boundaries so you can rearrange text or insert other characters. In that case, wide characters are the representation of choice inside the program. You want to deal, at least sometimes, in elements and arrays of type wchar_t.

Standard C already provides a minimal set of functions for converting back and forth between multibyte and wide character. You can also write wide-character constants, as L'x', and wide-character strings, as L"abc". Several people have demonstrated that you can do quite a bit of work with just what's already standardized. (See, for example, my book, The Standard C Library, Prentice Hall, 1992.)

One thing we chose not to add to the original C Standard, however, was the ability to read and write wide characters directly. A format string can contain multibyte characters as part of its literal text, but you cannot easily read and write data of type wchar_t. You have to convert wide characters to a multibyte string in a buffer before you print it out. Or you have to read a multibyte string into a buffer before you convert it to wide characters. That can be a nuisance.

%C writes to the output the multibyte sequence corresponding to the wchar_t argument. A shift-dependent encoding begins and ends in the initial shift sequence. That can result in redundant shift codes in the output, but the print functions make no attempt to remove them.

%S writes to the output the multibyte sequence corresponding to the null-terminated wide-character string pointed to by the pointer to wchar_t argument. The same rules apply for shift codes as for %C. You can use a precision, as in %.5S, to limit the number of array elements converted.

In no event will either of these conversion specifiers produce a partial multibyte sequence. They may, however, report an error. A wide-character encoding may consider certain values of type wchar_t invalid. Try to convert one and the print functions will store the value of the macro EILSEQ in errno. That macro is now added to the header <errno.h>.

As usual, the behavior of the scan functions for the same conversion specifiers is similar, but different from that for the print functions:

%C reads from the input a multibyte sequence and converts it to a sequence of wide characters. It stores these characters in the array designated by the pointer to wchar_t argument. You can use the field width to specify a character count other than one, as in %5C. The input sequence must begin and end in the initial shift state (if states matter).

%S does the same job, but with two critical differences. The multibyte sequence to be converted is determined by first skipping leading whitespace, then consuming all input up to but not including the next whitespace. Here, whitespace has its old definition of being any single character for which isspace returns true in the current locale. (Yes, that can cause trouble with some multibyte encodings.) The resultant sequence must begin and end in the initial shift state. The other difference is that the scan functions store a terminating null after the sequence of converted wide characters.

A multibyte encoding may consider certain character sequences invalid. Try to convert one and the scan functions will store the value of the macro EILSEQ in errno. The print functions may generate no incomplete multibyte sequences, but the scan functions aren't nearly as tidy. They can leave the input partway through a multibyte character, or in an uncertain shift state, for all sorts of reasons. Remember, both families of functions are still essentially byte-at-a-time processors.

• eliminating redundant shift states when generating a stream of multibyte characters
• keeping track of shift states across function calls when scanning a stream of multibyte characters
• producing and consuming whole multibyte characters even in the absence of shift codes.

So why did WG14 agree to add machinery which is known to be inadequate? Because, for many applications, it is good enough, thank you. People writing "internationalized" applications have discovered a whole spectrum of needs. Earlier, I described a class of programs that need essensially no special support for large character sets. Others can use what's already in the C Standard. The biggest nuisance that many practicing programmers keep reporting is this omitted ability to read and write wide characters directly. For a small cost in code added to the print and scan functions, you get enough code to help many programmers.

• a character code that can be represented as type unsigned char, or
• the value of the macro EOF, declared in <stdio.h>

It is only natural that the world of wide characters should demand analogous machinery. Thus <wchar.h> defines a macro and a type that support programming with wide meta-characters:

• The type wint_t can represent any value representable as type wchar_t.
• The macro WEOF can be represented as type wint_t and is distinguishable from all valid wide-character codes.

There's one small subtlety I hate to let slide by without notice. The type wint_t is often represented with more bits than the type wchar_t. That gives the implementor lots of choices for the value of WEOF. But that need not be the case. It's perfectly valid for wint_t to be the same size as wchar_t. Then, the value of WEOF must be chosen from among the invalid wide-character codes. (And there better be at least one to choose from.)

So don't fall into the habit of thinking of wchar_t as some unsigned integer type. And don't assume that WEOF is —1, or any other negative value. You may one day be surprised. (If you want to write highly portable code, in fact, prepare for the possibility that types char and int are the same size. But that's a whole 'nother sermon.)

• What a function does each time you call it depends on more than just the value of its arguments. That makes its interface harder to learn and to use properly. strtok, for example, is notoriously hard to master.
• You can't share the use of a function with memory among different tasks at the same time. Or you have to be very careful to save and restore context, as with setlocale and localeconv.
• A stored pointer from one call can be sabotaged by a later call. tmpnam(NULL), for example, returns a pointer to a filename that changes with each call. Even worse, a call to gmtime changes the data pointed at by an earlier return from localtime.

Worse, we perpetuated this practice when we added several functions that manipulate large character sets. As they parse or generate multibyte strings, they may have to keep track of the current shift state. So the functions mblen, mbtowc, and wctomb have private memory. You initialize the memory to the initial shift state with one kind of call. You then progress through a multibyte string with other kinds of calls, and the memory tracks the shift state.

Our major reason for using private memory was to avoid adding yet another argument to each of these functions. That argument also needs a special type definition as well. We didn't want all that semantic detail if people weren't going to parse multibyte strings all that much. After all, we've gotten by with strtok all these years, haven't we?

Well, it turns out we were wrong. Seems lots of people are manipulating multibyte strings these days. And they chafe at the shortcomings I outlined above. Thus the new type defintion mbstate_t, and the new functions that use it. I'll describe them all next month.