A Geometric Approach to Model Steering

Editorial: "Gender biases shall not pass"

The 2026 National Stage featured a problem right at the intersection of linear algebra and AI ethics: "Gender biases shall not pass". To see why this challenge matters, we just need to look at the current economics of AI. Back in 2024, Anthropic's CEO Dario Amodei noted that pre-training a top-tier LLM cost around $100 million. Today, as EpochAI predicted, billion-dollar training runs are the new normal. The reality is simple: once a massive model is trained, retraining it from scratch just to fix an inherent bias is economically impossible.

This brings us to Model Steering. Instead of tweaking weights or starting over, we directly alter the model's internal activations (embeddings) to correct its behavior. For this problem, the goal was to strip gender bias from OpenAI's CLIP embeddings. Here is how we can do it using pure, intuitive geometry.

Central Idea: Isolating and Erasing Concepts

CLIP turns images into 512-dimensional vectors. Since it was trained on the raw internet, this latent space naturally separates images by gender, even when gender is completely irrelevant to the downstream task.

Our objective is straightforward: find the exact direction in this 512-dimensional space that represents "gender", and then project all our data so that this specific direction collapses to zero, leaving the other 511 dimensions untouched.

Geometric Intuition

Imagine the 512-dimensional CLIP space as a standard 3D room. Each image is a floating dot. If we color the "gender 0" dots blue and "gender 1" dots red, we see two separate clouds.

1[ VISUAL 1: ORIGINAL LATENT SPACE ]23         (Gender 1) 4         Red Cloud5            🔴 🔴6           🔴 μ₁ 🔴        μ₁ = Center (Mean) of Gender 17            /8           /  ⟵ d = μ₁ - μ₀ (Gender Axis)9          /10         μ₀ 🔵           μ₀ = Center (Mean) of Gender 011        🔵 🔵12      Blue Cloud13      (Gender 0)

Drawing a line from the center of the blue cloud ( $\mu_0$ ) to the center of the red cloud ( $\mu_1$ ) gives us the gender axis ( $d$ ).

To make the model "blind" to gender, we take all points and project them onto a plane that is perfectly perpendicular to this axis.

1[ VISUAL 2: ORTHOGONAL PROJECTION (Debiasing) ]23                x  (Original Image Embedding)4               /|5              / |6             /  |7            /   |  (xᵀ·d)d ⟵ Gender Component 8           /    |            (How much of 'x' points along 'd')9          /     |10         /      ↓11 ───────o─────────────────────────────────────── Perpendicular Plane12    x_debiased                                   (New, gender-blind space)1314         15        Formula:  x_debiased = x - (xᵀ·d)d

Through this orthogonal projection, we mathematically subtract the gender component from every single image. The points fall flat onto the new plane.

1[ VISUAL 3: FINAL RESULT ]23 ==============================================4 ... 🟣 🟣 🟣 🟣 🟣 🟣 ... (Red & Blue overlap)5 ==============================================6 ^ Debiased Plane

On this new wall, the distance along the red-blue axis is exactly zero. The red and blue clouds overlap entirely (becoming indistinguishable 🟣). The model keeps all the other visual information but loses the ability to separate the images by gender.

Mathematical Formulation

Let $X \in \mathbb{R}^{N \times 512}$ be the embedding matrix and $Y \in {0, 1}^N$ the binary gender labels provided in train_data.pkl.

Finding the bias direction:
We calculate the centroid (mean) for each gender class:

(1)

\mu_0 = \frac{1}{N_0} \sum_{i: Y_i = 0} X_i

(2)

\mu_1 = \frac{1}{N_1} \sum_{i: Y_i = 1} X_i

The direction of the gender bias is the difference between these two means:

(3)

d = \mu_1 - \mu_0

For the mathematical projection to work, we normalize this vector to length 1, obtaining $\hat{d}$ :

(4)

\hat{d} = \frac{d}{||d||_2}

Nullifying the bias:
To remove $\hat{d}$ from any given input vector $x$ , we calculate their dot product: $c = x^T \hat{d}$ .
This scalar $c$ tells us how much of $x$ points along the bias. The actual vector component to remove is $c \cdot \hat{d} = (x^T \hat{d}) \hat{d}$ .

We debias the vector by subtracting this component:

(5)

x_{debias} = x - (x^T \hat{d}) \hat{d}

Proof:
Did we actually erase the gender? Let's project the new vector $x_{debias}$ onto the gender direction $\hat{d}$ and see if we get $0$ :

(6)

(x_{debias})^T \hat{d} = \left( x - (x^T \hat{d}) \hat{d} \right)^T \hat{d}

(7)

= x^T \hat{d} - (x^T \hat{d}) (\hat{d}^T \hat{d})

Since

\hat{d}

is a unit vector,

\hat{d}^T \hat{d} = 1

. Therefore:

(8)

= x^T \hat{d} - x^T \hat{d} = 0

The math checks out. The new latent space is perfectly orthogonal to the concept of gender.

Vectorized Python Implementation

In a competition environment, iterating through 15,000 arrays in a loop is incredibly slow. We need to process the entire matrix at once.

First, we find the magnitude of bias for all images simultaneously using matrix multiplication:

(9)

C_{15000 \times 1} = X_{15000 \times 512} \cdot \hat{d}_{512 \times 1}

Next, we construct the actual bias vectors that need to be subtracted. Mathematically, this is an outer product between the magnitudes ( $C$ ) and the bias direction transposed ( $\hat{d}^T$ ):

(10)

B_{15000 \times 512} = C_{15000 \times 1} \cdot \hat{d}^T_{1 \times 512}

(where $X$ is the training matrix, and $B$ is the resulting Bias Matrix).

Here is how this arithmetic translates to highly optimized NumPy code:

Python
1import numpy as np2​3# 1. Load data matrices4train_image_embeddings = data['train_image_embeddings'] # Shape: (15000, 512)5gender_labels = data['gender_labels']                   # Shape: (15000,)6test_image_embeddings = data['test_image_embeddings']   # Shape: (15000, 512)7​8# 2. Calculate centroids for each class9mean_0 = train_image_embeddings[gender_labels == 0].mean(axis=0)10mean_1 = train_image_embeddings[gender_labels == 1].mean(axis=0)11​12# 3. Find bias direction (d) and normalize it to a unit vector (\hat{d})13direction = mean_1 - mean_014direction = direction / np.linalg.norm(direction) 15​16# 4. Apply Vectorized Debiasing17# 'train_image_embeddings @ direction' computes scalar projections (C)18# 'np.outer' constructs the Bias Matrix (B) by multiplying C with \hat{d}^T19train_debiased = train_image_embeddings - np.outer(train_image_embeddings @ direction, direction)20test_debiased = test_image_embeddings - np.outer(test_image_embeddings @ direction, direction)

Wrapping Up

"Gender biases shall not pass" proves that managing trillion-parameter AI models doesn't always require massive compute budgets. Sometimes, a well-placed orthogonal projection is all you need to strip away an ingrained attribute. As training costs continue to skyrocket, mathematical steering techniques remain the most pragmatic way to align models with human values.