Skip to Content
Course content

186: Unicode Handling with the unicode Package

Click on the "Edit" button in the top corner of the screen to edit your slide content.

I've seen this happen more times than I can count: a developer writes a validation function that works perfectly during local testing in the US, but the moment the app hits a global market, users from France, Japan, or Germany start reporting that they can't sign up because their names are "invalid."

Here is a snippet of code that looks perfectly reasonable at first glance. It's designed to check if a string consists only of alphanumeric characters.

func IsAlphanumeric(s string) bool {
    for _, r := range s {
        if !((r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9')) {
            return false
        }
    }
    return true
}

// Test cases
fmt.Println(IsAlphanumeric("User123")) // true
fmt.Println(IsAlphanumeric("François")) // false - Wait, what?

The ASCII Trap

The problem here is that we're thinking in ASCII. By checking if a rune falls between 'a' and 'z', we are explicitly telling Go: "If this character isn't part of the English alphabet, it's not a letter."

In Go, a string is a read-only slice of bytes, and when we range over it, we get runes (which are just 32-bit integers representing Unicode code points). The character 'ç' in "François" is a valid Unicode letter, but its integer value is far outside the range of 97 ('a') to 122 ('z'). If you're building software for a global audience, manually defining these ranges is a losing battle. You can't possibly hardcode every alphabet in existence.

Letting the unicode Package Do the Heavy Lifting

Instead of guessing ranges, you should use the unicode package. It contains a massive set of predefined tables that categorize every Unicode character. Whether a character is a Cyrillic letter, a Kanji ideograph, or a Greek symbol, the unicode package already knows about it.

Here is how I would rewrite that validation logic to be truly global:

import "unicode"

func IsAlphanumeric(s string) bool {
    for _, r := range s {
        if !unicode.IsLetter(r) && !unicode.IsDigit(r) {
            return false
        }
    }
    return true
}

// Now these both return true
fmt.Println(IsAlphanumeric("User123")) 
fmt.Println(IsAlphanumeric("François")) 

By using unicode.IsLetter(r), we are asking Go to check if the rune belongs to any Unicode category designated as a letter. It's cleaner, shorter, and it actually works for people whose names don't fit in a 1960s American computer system.

Handling Whitespace and Control Characters

The unicode package isn't just for letters and numbers. When you're cleaning up user input, you'll often find that strings.TrimSpace is great, but sometimes you need more granular control. For instance, you might want to strip out non-printable control characters that can sneak into copy-pasted text and break your database queries.

I usually reach for unicode.IsPrint or unicode.IsSpace for this. IsPrint returns true if the character is printable (meaning it's not a carriage return or a null terminator), while IsSpace identifies any white-space character defined by Unicode, including non-breaking spaces that ' '` wouldn't catch.

One quick tip: if you ever need to check for a specific script (like specifically checking if a string is in Greek), you can look into unicode.Greek. The package provides these "Range Tables" that allow you to be as broad or as specific as your business logic requires.




📋 Practical Task

Build a Global Username Sanitizer

Your task is to create a function called SanitizeUsername(input string) string. This function should take a raw input string and return a "cleaned" version based on the following rules:

  • Keep all Unicode letters (regardless of language).
  • Keep all Unicode digits.
  • Convert all letters to lowercase (use unicode.ToLower).
  • Remove everything else (spaces, punctuation, emojis, and control characters).

Example:
Input: " Jóhn_Doe 123! 🚀 "
Output: "johndoe123"

Implement this using a strings.Builder for efficiency and the unicode package for character classification.

Rating
0 0

There are no comments for now.

to be the first to leave a comment.