I was looking over some data processing code recently, and I hit a wall. I had a loop running a simple distance calculation over a few million coordinate pairs. In Python, it worked fine for a small test set, but once I hit the full dataset, the script just crawled. I'm talking "go grab a coffee and maybe a sandwich" levels of slow.

Why is this loop taking forever?

First, let's look at the pure Python approach I started with. I'm calculating the Euclidean distance between two points in 2D space, repeated over a massive array.

import time
import math

def calculate_total_distance(points):
    total = 0.0
    for i in range(len(points) - 1):
        p1 = points[i]
        p2 = points[i+1]
        dist = math.sqrt((p2[0] - p1[0])**2 + (p2[1] - p1[1])**2)
        total += dist
    return total

# 1 million points
points = [(float(i), float(i*2)) for i in range(1000000)]

start = time.time()
calculate_total_distance(points)
print(f"Time taken: {time.time() - start:.4f}s")

Running this takes about 0.4 to 0.6 seconds on my machine. Now, that doesn't sound terrible, but in a production pipeline where this happens thousands of times, it's a killer. I could use NumPy here, and that would be the "correct" way for 90% of cases. But what if the logic is more complex than a simple distance formula? What if I have a custom algorithm that doesn't vectorize easily? That's where I turn to Cython.

Just changing the file extension

The first thing I tried was the "lazy" way. I moved the function into a file called distance.pyx and compiled it using a setup.py script. I didn't change a single line of the logic—I just treated it as Python code that happens to be compiled.

# distance.pyx
import math

def calculate_total_distance(points):
    total = 0.0
    for i in range(len(points) - 1):
        p1 = points[i]
        p2 = points[i+1]
        dist = math.sqrt((p2[0] - p1[0])**2 + (p2[1] - p1[1])**2)
        total += dist
    return total

I compiled it, ran it, and... it was barely faster. Maybe 10-20% improvement. Why? Because even though the code is compiled, every single variable—total, i, p1—is still a Python object. The compiled code is essentially just calling the Python C-API to do the math. We haven't actually escaped the "Python overhead."

Giving the compiler some hints

To actually get C-like speed, I have to stop pretending this is Python and start telling Cython exactly what the data types are. This is where cdef comes in. I'll redefine the loop index and the accumulators as C types.

I'll also change the input from a Python list of tuples to a typed memoryview. This is the real secret sauce; it allows Cython to access the raw memory of an array without checking the Python object wrapper every time.

# distance_fast.pyx
import math
from libc.math cimport sqrt

def calculate_total_distance(double[:, :] points):
    cdef double total = 0.0
    cdef int i
    cdef int n = points.shape[0]
    cdef double p1_x, p1_y, p2_x, p2_y

    for i in range(n - 1):
        p1_x = points[i, 0]
        p1_y = points[i, 1]
        p2_x = points[i+1, 0]
        p2_y = points[i+1, 1]
        
        total += sqrt((p2_x - p1_x)**2 + (p2_y - p1_y)**2)
        
    return total

Notice a few changes: I'm using double[:, :] for the input (a 2D memoryview), cdef int i for the loop, and libc.math.sqrt instead of math.sqrt. The latter is a direct call to the C standard library, skipping the Python module lookup entirely.

When I run this version? It's nearly 50 to 100 times faster than the original Python version. We've gone from "noticeable lag" to "basically instantaneous."

The cost of speed

Now, before you go rewriting your entire codebase in Cython, let's be honest about the trade-offs. I just added a build step to my project. I can't just run python script.py anymore; I have to run python setup.py build_ext --inplace every time I change a line of code in the .pyx file. That kills the "rapid prototyping" feel of Python.

Also, I've lost the flexibility of Python's dynamic typing. If I accidentally pass a list of strings into that double[:, :] memoryview, the program won't just give me a nice TypeError—it might just crash with a segmentation fault if I'm not careful with my boundaries.

My rule of thumb: If NumPy can do it, use NumPy. If you have a complex loop that is the undisputed bottleneck of your app and you've already tried every Pythonic optimization, then reach for Cython. Put the heavy lifting in a .pyx file, keep the high-level logic in .py, and only type the variables that are inside the hottest loops.

📋 Practical Task

Accelerating a Manhattan Distance Summation for 10 Million Points

You have a dataset of 10 million 2D points represented as a NumPy array. Your task is to write a Cython implementation that calculates the sum of the Manhattan distances between consecutive points.

Reminder: Manhattan Distance is calculated as: |x2 - x1| + |y2 - y1|.

Requirements:

• Create a .pyx file.
• Use cdef to type your loop counter and the running total as C types.
• Use a typed memoryview (double[:, :]) for the input array to ensure maximum performance.
• Use the C fabs() function from libc.math for absolute values instead of Python's abs().
• Write a setup.py script to compile the module.

Success Criteria: Your compiled Cython function should process the 10 million points significantly faster (at least 20x) than a pure Python for loop implementation.