Hand tracking in visionOS is where the experience stops feeling like a floating iPad and starts feeling like actual spatial computing. We're moving beyond simple buttons and into the realm of direct manipulation. To get a feel for this, I want us to build a "Gravity Orb"—a simple 3D sphere that reacts when you "pinch" it in mid-air, changing its color and pulsing in size.

Creating the Interactive Orb

First, we need a 3D object to interact with. I'm using a RealityView here because it gives us the direct bridge to RealityKit entities. I'll start by creating a simple sphere with a metallic material. I like the metallic look because it catches the simulated light in the visionOS environment, making it easier to judge depth when you're reaching for it.

import SwiftUI
import RealityKit

struct GravityOrbView: View {
    var body: some View {
        RealityView { content in
            let orb = ModelEntity(
                mesh: .generateSphere(radius: 0.1),
                materials: [SimpleMaterial(color: .systemBlue, isMetallic: true)]
            )
            orb.name = "gravity_orb"
            content.add(orb)
        }
    }
}

Wiring up the Spatial Gesture

Now, we want the orb to react. In visionOS, we use SpatialTapGesture for those pinch-to-select actions. I'll attach the gesture to the RealityView and use the entity parameter to figure out exactly what the user is looking at and pinching.

// Inside the RealityView
.gesture(
    SpatialTapGesture()
        .targetedToAnyEntity()
        .onEnded { value in
            let entity = value.entity
            // I want to change the color to green and scale it up
            if let model = entity as? ModelEntity {
                model.model?.materials = [SimpleMaterial(color: .systemGreen, isMetallic: true)]
                model.scale = [1.2, 1.2, 1.2]
            }
        }
)

Why isn't it reacting?

I just ran this in the simulator, and... nothing. I'm pinching the orb, the system knows I'm tapping, but the onEnded closure never fires. This is a classic "gotcha" in visionOS development that almost every developer hits the first time.

The problem is that a ModelEntity is just visual geometry. For a gesture to "hit" an entity, that entity needs a physical presence in the physics world. If it doesn't have a CollisionComponent, it's essentially a ghost—the gesture passes right through it. I forgot to give the orb a shape for the system to collide with.

Here is the fix. I need to generate a collision shape that matches the sphere's radius:

// Add this inside the RealityView setup, right after creating the orb
orb.generateCollisionShapes(recursive: true)

Now, when I run it again, the pinch is registered immediately. It's a small line of code, but it's the difference between a broken app and a working one.

Adding a bit of polish

Changing the color instantly is a bit jarring. To make it feel more "organic," I'll use a simple animation block. Since RealityKit entities don't use SwiftUI animations, we have to handle the transition of the scale and material carefully. I'll use a basic move(to:relativeTo:duration:) call to make the orb "pulse" when pinched, which feels much more like a physical reaction than a snap.

.onEnded { value in
    let entity = value.entity
    if let model = entity as? ModelEntity {
        model.model?.materials = [SimpleMaterial(color: .systemGreen, isMetallic: true)]
        
        // Pulse effect: Scale up, then we could use a timer or 
        // separate animation to scale it back down.
        var transform = entity.transform
        transform.scale = [1.2, 1.2, 1.2]
        entity.move(to: transform, relativeTo: entity.parent, duration: 0.2, timingFunction: .easeOut)
    }
}

By combining SpatialTapGesture with CollisionComponent and basic Entity transforms, we've created a basic interactive object. The key takeaway here is that in visionOS, "seeing" is not the same as "interacting." Always check your collisions.

📋 Practical Task

Build a "Virtual Doorbell" Interaction

Create a small visionOS project featuring a 3D cube (the "doorbell") using RealityView. Implement the following functionality:

• The cube should start with a red color.
• Ensure the cube has the necessary components to be interactive (remember the lesson's "ghost" mistake!).
• Implement a SpatialTapGesture so that when the user pinches the cube, it changes color to gold and momentarily shifts its position slightly backward (along the Z-axis) to simulate a button being pressed.
• Use the move(to:relativeTo:duration:) method to make the button press feel smooth rather than instantaneous.