Constellations of the Sleeping Sheriff

Constellations of the Sleeping Sheriff

A Challenge from the Wild West

Note

Pre-trained language models (e.g., BERT, RoBERTa, sentence-transformers) with loaded weights may be explicitly used. GPU acceleration for training and inference is fully permitted. Classical NLP pipelines based on TF-IDF are also permitted and encouraged.

The Story

Somewhere on the dry plains of New Mexico, 1883.

Old Sheriff Clem Dudley had a habit that no deputy ever understood: every evening, no matter how long the day had been, he would place his hat on his horse's saddle, lie down on his blanket, and look straight up at the sky.

"The stars don't lie," he used to say. "Every fugitive, every drifter, every honest man — they all look at the same sky."

One August night, exhausted after three days of chasing the Vega gang through the desert, Clem lay under a sky black as ink, peppered with more stars than he had ever counted. He was trying to fall asleep when the stars began to move — slowly at first, like embers carried by a wind from a dying fire. Then faster and faster. They traced lines between them. Triangles. Spirals. Animals. Digits. Shapes he didn't know the names of.

"Must be the heat," he muttered. But he kept watching.

The shapes became words. Not spoken — felt. Each constellation seemed to carry a meaning, a weight, a story hanging from it like a wanted poster on a sheriff's board. He reached for his pencil to write them down, and then — nothing.

He woke up somewhere else.

The Realm Between the Stars. That's what they called it.

It was a vast and silent territory where the sky was the ground, and the constellations were villages — each a group of stars arranged in a shape, and each shape bearing a name waiting to be matched with its story. The inhabitants of this realm, the Star Scribes, had kept meticulous records for centuries: journals, poems, travel logs, legends — all written by people who had looked at those same star shapes from below, on Earth.

But something had gone wrong. A great celestial storm — the locals called it The Scattering — had ripped every label off every constellation and cast them into the wind. Now thousands of texts floated untethered above the star shapes they once described, and the Star Scribes could no longer tell which story belonged to which sky.

Clem was handed a badge made of starlight and a simple instruction:

"Match the words back to the stars, Sheriff. Before the next storm comes."

The Task

You are provided with a set of star constellations, each described by a sequence of 2D points forming a geometric shape (e.g., a cross, a spiral, a five-pointed figure, digits like 6, 7, 8, combinations of digits, etc.). You are also provided with a large pool of candidate text fragments — journal entries, legends, descriptions — each originally written about one specific constellation.

Your objective is to build a model capable of correctly assigning each text from the pool to its corresponding constellation, based exclusively on the semantic and structural relationship between the coordinate geometry and the text content.

Data Structure

Training Set

train.csv contains 300 labeled constellation-text pairs:

• Each constellation ID appears 40-65 times with slight geometric variations (noise, rotation, scaling).

Test Set

test.csv contains 50 unlabeled constellation sequences (datapointID 1-50):

Each datapointID appears 40-65 times. No text labels are provided.

Candidate Texts

candidates.csv contains 500 text fragments:

• Exactly 50 of these 500 texts are the correct matches for the 50 test constellations.
• The other 450 are distractors.
• The texts are literary in nature — no direct geometric vocabulary appears in them.

Submission Format

The submission file must be in CSV format with the following columns:

Each of the 50 test datapointIDs must appear exactly once.

Example:

1subtaskID,datapointID,answer21,1,4231,2,31741,3,4675...

Evaluation Metric

Accuracy (equivalent to F1-macro)

Each prediction is evaluated as a single-label classification over the 500 candidate text IDs. The prediction is either correct (1) or incorrect (0).

The final metric is the number of correct predictions / 50.

Scoring Function

Raw scores are converted into competition points using a linear function with a minimum threshold:

In short: you must correctly match at least 15 out of 50 constellations (accuracy = 0.30) to earn any points, and correctly matching 43 or more (accuracy >= 0.85) earns the maximum score.

"The stars don't lie. You can trust a star more than a man..." — Sheriff Clem Dudley, 1883