My goal this month is to show you many of those new functions. I do so in the context of how you might actually use them, should the need arise. One day, your enterprise might feel moved to sell code into Japan, China, or the Arab world. You might find an advantage in writing code that is truly international in its handling of text. Or you might want to manipulate text in numerous fonts and colors. If so, you'll find no more sophisticated machinery than what's already in Standard C, at least not in any other standardized programming language. The additions described here make for a particularly rich environment for manipulating large character sets.

I remind you that what I'm describing here is part of a "normative addendum" to the ISO C Standard. It is still subject to balloting within SC22, the parent committee of the C standards committee WG14. It may change in response to comments or criticism. But we've been reviewing it within WG14 for about three years now. I think it's mostly stable.

Not all the classes are exactly the same across implementations. You can be sure that isdigit returns true (nonzero) only for the ten decimal digits. You can be sure that ispunct returns true for all the punctuation required to write a C program. ispunct also often returns true for such characters as the at sign @, but it doesn't have to. It might return true for additional characters that you don't expect. In short, Standard C says that the punctuation class is to some extent implementation defined.

That's not the end of it. When we put locales in Standard C, we allowed for additional changes to the character classification functions. A program starts out in the "C" locale, where the behavior is pretty traditional. You can call the function setlocale, however, to change to a new locale. If that locale specifies a new category LC_CTYPE, several of the functions declared in <ctype.h> can begin behaving differently. They can recognize additional punctuation, or lower-case letters, or a few other critters.

The proper style these days for classifying characters is thus to use the <ctype.h> functions religiously. They are more likely to get the right answer regardless of execution character set and regardless of current locale.

• The existing character classes may sensibly include characters not representable as single-byte characters.
• Some implementations may see fit to define whole new classes of characters.
• To address the first consideration, the new header <wchar.h> provides parallels for all the functions delcared in <ctype.h>:

int iswalnum(wint_t wc);
int iswalpha(wint_t wc);
int iswcntrl(wint_t wc);
int iswdigit(wint_t wc);
int iswgraph(wint_t wc);
int iswlower(wint_t wc);
int iswprint(wint_t wc);
int iswpunct(wint_t wc);
int iswspace(wint_t wc);
int iswupper(wint_t wc);
int iswxdigit(wint_t wc);
wint_t towlower(wint_t wc);
wint_t towupper(wint_t wc);

These functions essentially behave the same as their older cousins for characters that are also representable as single-byte characters. (The language in the normative addendum is twisty, but that's what it amounts to.) Some can also return true for additional wide characters, by an extension of the same latitude granted their older cousins.

The parallels break down in two cases, both involving the exclusion of whitespace. The older functions isgraph and ispunct are defined in terms of what other functions accept, minus the space character ' '. The newer functions iswgraph and iswpunct are defined in terms of what the analogous functions accept, minus the characters accepted by iswspace. That certainly includes the space character, but it might also include others, even single-byte white-space characters. We on WG14 couldn't resist the temptation to properly generalize these functions, even at the cost of some backward compatibility.

What this means in practice is really rather simple, despite my nit-picking descriptions. You can change your programs to handle large character sets mostly by:

• replacing char declarations for text objects with wchar_t declarations
• replacing calls to the functions declared in <ctype.h> with their analogs declared in <wchar.h>

So what they proposed was an open-ended set of new classifications. An implementation publishes a list of property names. These are just null-terminated strings like so many other names in C. (They could be multibyte strings, but who knows or cares? All you have to do is read them in printed form and be able to reproduce them as string literals in a program you write.)

To make use of one of these new classifications, you obtain its handle by calling the function wctype, declared in <wchar.h>, as in:

static wctype_t hirigana = wctype("hirigana");

size_t wcsftime(wchar_t *s, size_t maxsize, const wchar_t *format,
   const struct tm *timeptr);

• Its format argument, which controls the formatting of the generated string, is a wide-character string.
• The generated string is also a wide-character string.

WG14 nevertheless elected to add wcsftime. The idea is to eliminate wherever possible the need to represent any multibyte strings within a program. That in turn eliminates the need to convert back and forth between multibyte and wide character forms. And that, in the short run, makes it easier to convert programs to manipulating text represented with a large character set instead of the current single-byte sets. In the long run, that will also keep programs cleaner.

So to convert time strings to wide-character form:

• First convert any calls to asctime or ctime to their equivalents in terms of strftime.
• Replace pointer to char and array of char declarations for string objects with analogous wchar_t declarations.
• Replace calls to strftime with calls to wcsftime, declared in <wchar.h>.

double wcstod(const wchar_t *nptr, wchar_t **endptr);
long int wcstol(const wchar_t *nptr, wchar_t **endptr, int base);
unsigned long int wcstoul //(const wchar_t *nptr, wchar_t **endptr, int base);

On the other hand, several other functions declared in <stdlib.h> have been supplemented considerably. These are the ones that help you walk along multibyte strings, or that convert between multibyte and wide-character forms. We added them because experience to date says we should. Each fills some gap we found in the minimal set of conversion functions in the current C Standard.

The first addition looks simple enough:

int wctob(wint_t wc);

What does wctob buy you? It is certainly convenient, for one thing. It's a nuisance to do the equivalent using existing functions:

• Set up a wide-character string consisting of the argument wc followed by a null wide character.
• Convert it to a multibyte string in yet another buffer, by calling wcstombs.
• Check that the converted string consists of a single (non-null) character, followed by the null terminator.
• Pick up that character as the value of interest, then clean up the mess.

So part of the proposed addition is a set of functions that do the same as existing conversion functions, but ask the caller to provide a pointer to the state memory. (Note the similarity to wcstok, described above.) The functions use the state memory to keep track of the current shift state, for multibyte encodings that are state dependent, that is. You provide a pointer to an object of (nonarray) type mbstate_t, declared as usual in <wchar. h>.

The functions decide how best to use this storage on your behalf. You can initialize the mbstate_t object to zero and be sure that it represents the initial shift state. Or you can call any of several functions (described below) in such a way that they enter the initial shift state. You can also test such an object to see whether it represents the initial shift state, by calling the function:

int sisinit(const mbstate_t *ps);

int mbrlen(const char *s, size_t n, mbstate_t *ps);
int mbrtowc(wchar_t *pwc, const char *s, size_t n, mbstate_t *ps);
int wcrtomb(char *s, wchar_t wc, mbstate_t *ps};
size_t mbsrtowcs(wchar_t *dst, const char **src, size_t len, mbstate_t *ps);
size_t wcsrtombs(char *dst, const wchar_t **src, size_t len, mbstate_t *ps);