I was working on a small piece of middleware last week that needed to handle user nicknames from a global player base. I started off the old-fashioned way, using standard char arrays, assuming UTF-8 encoding would just "work" because that's what most modern terminals use. But as soon as I tried to calculate the length of a name to truncate it for a UI element, everything broke.

The Byte Trap

Let's look at what happened. I tried to process the Japanese greeting "こんにちは" (Konnichiwa). In my head, that's five characters. But look at what C tells me:

#include <stdio.h>
#include <string.h>

int main() {
    char *greeting = "こんにちは"; 
    printf("String: %s\n", greeting);
    printf("Length: %zu\n", strlen(greeting));
    return 0;
}

When I ran this, the output for the length wasn't 5. It was 15. Why? Because strlen doesn't count characters; it counts bytes until it hits a null terminator. In UTF-8, those Japanese characters take up three bytes each. If I had tried to truncate this string at index 5 to "save space," I would have sliced a multi-byte character right down the middle, leaving the terminal to print a weird replacement character (usually a black diamond with a question mark) because the resulting byte sequence was invalid.

Bringing in uchar.h

I realized I needed a way to treat characters as fixed-width units regardless of how they're encoded for transmission. That's where <uchar.h> comes in. It was introduced in C11 to give us types that actually map to Unicode standards: char16_t for UTF-16 and char32_t for UTF-32.

I decided to pivot to char16_t. To do this, I have to use a specific prefix for my string literals. If I just use double quotes, I'm back to the char byte-array mess. I need the u prefix.

#include <stdio.h>
#include <uchar.h>

int main() {
    // The 'u' prefix tells the compiler this is a UTF-16 string
    char16_t *u_greeting = u"こんにちは";
    
    // Since this is an array of char16_t, 
    // the size is now predictable.
    size_t len = sizeof(u_greeting[0]) * 5; // Simplified for this example
    
    printf("Unicode characters are now handled as 16-bit units.\n");
    return 0;
}

Now, each element in u_greeting is a char16_t. I'm no longer guessing how many bytes a character takes; I know each one is exactly 2 bytes. This makes indexing and slicing significantly safer. If I need even more headroom—say, for some obscure ancient script or a massive library of emojis—I can use char32_t with the U prefix.

The Printing Problem

Here is where I hit a wall. I tried to printf("%s", u_greeting) and the compiler started screaming at me. printf expects a char *, and char16_t * is a completely different type. C's standard library is, frankly, pretty stunted when it comes to printing Unicode directly to the console. It doesn't have a printf equivalent for char16_t.

To actually see my work, I had to realize that uchar.h provides the types, but it doesn't provide the I/O. To print these, you typically have to convert them back to UTF-8 (which the terminal understands) or use OS-specific APIs (like WriteConsoleW on Windows). For my debugging, I wrote a quick loop to print the hexadecimal values of the characters just to verify they were stored correctly:

#include <stdio.h>
#include <uchar.h>

int main() {
    char16_t *u_greeting = u"こんにちは";
    
    // Manually iterating through the UTF-16 string
    for (int i = 0; i < 5; i++) {
        printf("Char %d: U+%04X\n", i, (unsigned short)u_greeting[i]);
    }
    return 0;
}

This approach confirms that the characters are stored as their actual Unicode code points. It's a bit of a chore to get them onto the screen, but having char16_t and char32_t means my internal logic—sorting, searching, or truncating—is finally based on actual characters rather than random byte offsets.

📋 Practical Task

Exercise: Unicode Character Code Point Extractor

Build a program that allows a user to define a string containing a mix of standard ASCII, emojis, and non-Latin characters using char32_t. Your program should not attempt to print the string itself (as printf cannot handle char32_t), but instead must iterate through the string and print the hexadecimal Unicode code point for every character present.

• Use the U prefix for a char32_t string literal.
• Include a string that contains at least one emoji (e.g., U"Hello 🌍").
• Use a loop to print each character in the format: Index [X]: U+XXXXX.
• Ensure you correctly identify the end of the string by looking for the null terminator \0.