Noisy Oracle Labyrinth

Noisy Oracle Labyrinth

An unknown beacon is hidden behind one of 384 candidate doors. You may ask
one of 144 probes for help, but the receiver answers only with a ternary signal:
red, amber, or green. Receivers are imperfect: their calibration changes
the signal boundaries, and every evaluated response has been frozen in advance.
The same query therefore remains deterministic during evaluation, but a model
that merely memorizes the nominal receiver will not generalize.

Your submission is a declarative decision DAG. At an ask node, the
evaluator reads the hidden response and follows one of three edges. At a guess
node, it tries the listed candidate IDs in order. No submitted code is run.

Public data

• candidates.csv: candidate ID, family and 10 public features;
• probes.csv: probe ID, cost (1, 2 or 3), channel and 8 public features;
• calibration.csv: sparse responses for four familiar receivers. The two
  held-out candidate families have no calibration rows;
• dev_cases.csv: 72 labeled, frozen development cases;
• contract.json: dimensions, response labels and budget;
• baseline.py, strategy_tools.py, validate_submission.py and
  local_score.py: a complete NumPy baseline and local tools.

The public development set is for iteration, not an evaluation split.

Hidden evaluation

The live/partial score contains 96 cases:

• 48 nominal-receiver cases;
• 48 familiar shifted-receiver cases.

The final/complete score contains a disjoint 144 cases:

• 72 targets from candidate families absent from calibration;
• 72 familiar-family targets observed through previously unseen receiver drift.

Targets, complete response rows and hidden receiver parameters are private.

Budget and utility

Each probe costs the integer shown in probes.csv; each attempted final guess
costs 2 units. Every possible root-to-leaf path must use at most 20 units.
The validator checks the most expensive route to every shared DAG node.

For a case solved after spending c units, its utility is

1 - 0.0125 × c.

An unsolved case has zero utility. The displayed score is 100 times mean
utility. The accompanying metric is the identification rate. Larger is better.

This rewards both correct identification and economical, adaptive probing.

Submission format

Upload a ZIP containing exactly one root-level strategy.json, at most 12 MiB.
JSON is strict: duplicate keys, floating-point numbers, NaN, Infinity,
Boolean indices and unknown fields are rejected.

1{2  "schema": "ioai-2026-noisy-oracle-policy-v1",3  "dataset_revision": "noisy-oracle-labyrinth-2026.07.21.1",4  "root": 0,5  "nodes": [6    {"type": "ask", "probe": 12, "red": 1, "amber": 2, "green": 3},7    {"type": "guess", "candidates": [17, 44]},8    {"type": "guess", "candidates": [91]},9    {"type": "guess", "candidates": [203, 8]}10  ]11}

Rules:

• root and every edge are zero-based node indices;
• probe is in [0, 143], candidates are in [0, 383];
• an ask node has exactly type, probe, red, amber, green;
• a guess node has exactly type, candidates, with a non-empty list and no
  duplicates;
• the graph has at most 60,000 nodes, is acyclic, and every node is reachable;
• every possible path ends in a guess and fits the 20-unit budget.

Validate and reproduce the baseline with:

1python3 baseline.py --dataset-dir . --output submission.zip2python3 validate_submission.py submission.zip3python3 local_score.py --dataset-dir . --submission submission.zip

Only Python and NumPy are needed for the supplied baseline. You may train a
different model locally; the submitted artifact must still be the JSON DAG.

Practical directions

Strong approaches may combine a ternary response model, receiver-robust
calibration, information gain divided by probe cost, posterior updates and
careful sharing of equivalent DAG subtrees. The families and channels are
useful structure, not guaranteed labels for the hidden shifts.