I've spent a lot of time writing regexes over the years, and while they're powerful, they can quickly turn into "write-only" code—meaning you can write it, but six months later, you have no idea how to read it. I recently ran into a situation where I had to parse a custom event log format that looked something like this: EVENT:user_signup | ID:402 | AT:2023-10-01 | TAGS:[beta, referral].

My first instinct was to just use split('|'). Let's see how that looks.

log_line = "EVENT:user_signup | ID:402 | AT:2023-10-01 | TAGS:[beta, referral]"
parts = log_line.split('|')
# ["EVENT:user_signup ", " ID:402 ", " AT:2023-10-01 ", " TAGS:[beta, referral]"]


The fragility of splitting
Right away, I'm seeing those annoying leading and trailing spaces. I'd have to strip every single element. But the bigger problem is the data inside. If one of the tags happened to contain a pipe character—which is possible in some log formats—my whole parsing logic collapses. I need a way to move through the string linearly, consuming pieces as I recognize them, rather than hacking the string into chunks and hoping for the best.

This is where StringScanner comes in. It's a standard library tool that acts like a cursor moving across a string. It doesn't change the original string; it just remembers where it is.

require 'strscan'

log_line = "EVENT:user_signup | ID:402 | AT:2023-10-01 | TAGS:[beta, referral]"
scanner = StringScanner.new(log_line)

# Let's try to grab the first key-value pair
if scanner.scan(/(\w+):([^|]+)/)
  puts "Key: #{$1}, Value: #{$2}"
end
# Output: Key: EVENT, Value: user_signup 


Managing the cursor
That worked for the first part, but notice what happened: scan moved the cursor to the end of user_signup . The cursor is now sitting right before that first pipe character. If I try to run that same scan again immediately, it'll fail because the next character is  | , not a word.

I need to account for the "noise" between the data I actually care about. In a real-world parser, you spend as much time skipping garbage as you do capturing data.

require 'strscan'

log_line = "EVENT:user_signup | ID:402 | AT:2023-10-01 | TAGS:[beta, referral]"
scanner = StringScanner.new(log_line)
data = {}

until scanner.eos? # eos? means "End Of String"
  # Skip any whitespace or pipe delimiters
  scanner.scan(/\s*\|\s*/) 
  
  if scanner.scan(/(\w+):([^|]+)/)
    key = $1
    val = $2.strip
    data[key] = val
  else
    # If we hit something we don't recognize, we have to move the cursor
    # otherwise we'll be stuck in an infinite loop.
    scanner.getch 
  end
end

p data 
# {"EVENT"=>"user_signup", "ID"=>"402", "AT"=>"2023-10-01", "TAGS"=>"[beta, referral]"}


Peek and Precision
There's one more thing. Sometimes you need to know what's coming next without actually consuming it. This is the difference between scan (which moves the cursor) and peek (which just looks). 

Suppose I want to handle those TAGS:[...] blocks differently—maybe I want to parse the contents of the brackets into an array. I can use peek to check if the value starts with a bracket before I decide how to process it.

# Inside our loop, instead of a generic value scan:
if scanner.scan(/(\w+):/)
  key = $1
  if scanner.peek(1) == '['
    # We know it's a list!
    scanner.scan(/\[([^\]]+)\]/)
    data[key] = $1.split(',').map(&:strip)
  else
    scanner.scan(/([^|]+)/)
    data[key] = $1.strip
  end
end


By combining scan for consumption and peek for decision-making, I've turned a messy string-splitting exercise into a proper lexical scanner. It's much more robust, and because I'm handling the string linearly, it's generally more performant for very long lines of text.

📋 Practical Task

Build a Custom Command Line Argument Parser

You are building a small tool that accepts custom command strings in the following format: COMMAND:upload FILE:image.jpg VERBOSE:true RETRIES:3. Some values are booleans, some are integers, and some are strings.

Write a Ruby script using StringScanner that parses this string into a Hash. Your parser must:

• Correctly identify the key and the value.
• Convert the value "true" or "false" into actual Ruby true or false (Boolean).
• Convert values that are purely numeric (like "3") into Integer objects.
• Keep all other values as String.

Test input: "COMMAND:sync FILE:data.csv VERBOSE:false RETRIES:5"
Expected output: {"COMMAND"=>"sync", "FILE"=>"data.csv", "VERBOSE"=>false, "RETRIES"=>5}