The Analytical Language of John Wilkins

🗄️ The Analytical Language of John Wilkins

Imagine a Cabinet of Distinctions whose drawers contain hidden animals. You cannot open a drawer, but you may ask an oracle questions with “yes” or “no” answers and then try to guess the name of the animal.

The goal is to identify each animal using as few calls as possible. This is an interactive identification problem similar to the game “Twenty Questions”: each answer should reduce the set of plausible animals, and the best next question will generally depend on the answers received so far.

Task

Two complete sets are published:

• animals_pool.txt: 1,472 animals that may be proposed;
• questions_pool.txt: 559 questions with “yes” or “no” answers.

For a hidden animal, the original problem exposes the following operations:

Both operations consume one call. The total budget is 15 calls for each hidden animal, including the final guess. A value that does not belong to the corresponding set is rejected without consuming a call.

On the MLCompete platform, the adaptive strategy is expressed as a finite decision graph in policy.json. An ask node branches according to the oracle’s yes/no answer. A guess node ends the round if the proposal is correct; otherwise, the wrong edge is followed. A stop node ends the round without solving it.

This declarative representation is equivalent to a deterministic adaptive solve policy for one animal: every decision may depend on the entire history of answers. For a randomized strategy, fix the random seed before exporting it. No participant-submitted code is executed during evaluation.

Oracle

The oracle is based on Qwen/Qwen2.5-3B-Instruct, pinned to revision aa8e72537993ba99e69dfaafa59ed015b17504d1. For every animal–question pair from the private evaluation required by the evaluator, the organizers use the exact official prompt:

1You are answering a question about one specific animal.2The animal is: {animal}.3Answer with a single word, yes or no.4Question: {question}

The official interactor.py file included in the package is preserved unchanged and does not pin a Hugging Face revision. To reproduce the platform oracle as faithfully as possible, pass the revision above to both AutoTokenizer.from_pretrained and AutoModelForCausalLM.from_pretrained, and use the recorded environment.

Different hardware or numerical libraries may still change a greedy bit near a tie; the platform’s frozen matrix remains authoritative, while local inference is only a prediction of that matrix.

A high-performing approach is to precompute an animal–question table, retain the candidates compatible with the observed answers, and choose the question that splits the remaining candidate set as evenly as possible.

Public and Private Data

The public package contains:

• animals_pool.txt and questions_pool.txt;
• dev.csv, containing 150 publicly labelled animals;
• test1.csv, containing 500 publicly labelled animals;
• the official interactor.py and evaluate.py files for local experiments;
• English and Romanian guides for the policy contract;
• validate_submission.py, a standalone ZIP and policy validator;
• a minimal example and export instructions.

dev.csv and test1.csv are disjoint public sets intended for local evaluation, not secret leaderboard data. The private MLCompete set is drawn only from the 822 animals in animals_pool.txt that do not appear in either of the two public CSV files. It contains 128 unique animals: 32 form the live/partial set, while another 96, disjoint from the first group, form the final/full set. Their exact composition and evaluation order are not disclosed.

The official evaluate.py file runs a Python MySolution implementation in the original format; it does not read or validate the archive containing policy.json. Use validate_submission.py before uploading.

Submission Format

Submit a ZIP archive containing exactly one file at the root of the archive:

1policy.json

Directories, symbolic links, duplicate paths, and additional files are not allowed. policy.json must be a UTF-8 JSON document using the following schema:

1ioai-2026-animal-policy-v1

While the competition is active, the platform also requires a separate source-code attachment. In that field, upload the source file, notebook, or source archive used to construct policy.json. The organizers retain it for verification; the evaluator does not execute it, and it must not be included in the submission ZIP archive.

Identifiers are zero-based line indices in the official normalized sets:

• question: an integer between 0 and 558;
• animal: an integer between 0 and 1471;
• node reference: an integer index in the nodes list;
• null edge: stop without consuming another call.

The exact node formats are:

1{"type":"ask","question":17,"yes":4,"no":5}2{"type":"guess","animal":903,"wrong":8}3{"type":"stop"}

An archive may define at most 32,767 nodes. Every reference must point to an existing node. Unknown keys, duplicate JSON keys, invalid numbers, and incorrect set hashes invalidate the submission.

Minimal Valid Example

1{2  "schema": "ioai-2026-animal-policy-v1",3  "animals_sha256": "87eb93cc5d2a238a38daaeb201c394db935153cc17b03a44a7642f0db9ff35dc",4  "questions_sha256": "d2bd060866382f3dd1329112e73fbd6bf2397352d02469434b7b3d199bd2ca62",5  "root": 0,6  "nodes": [7    {"type": "ask", "question": 0, "yes": 1, "no": 2},8    {"type": "guess", "animal": 0, "wrong": null},9    {"type": "stop"}10  ]11}

The example asks the first published question. After yes, it proposes the first published animal; after no, it stops.

Evaluation

For each private animal, the evaluator starts at root with zero calls and traverses the graph iteratively:

• ask: consumes one call, reads the frozen oracle bit, and then follows yes or no;
• guess: consumes one call; ends successfully if the proposal is correct, otherwise follows wrong;
• stop or null: ends without solving the round and without consuming a call.

Traversal stops after 15 consumed calls, even if the graph contains another outgoing edge. Cycles are safely bounded by this budget.

For one hidden animal:

1round_score = max(0, (1 if the animal was found, otherwise 0)2                     - 0.02 × number_of_calls)

Examples:

• correct guess on call 1: 0.98;
• correct guess on call 5: 0.90;
• correct guess on call 15: 0.70;
• no correct guess: 0.

The two raw metrics are calculated independently:

1partial_metric = mean(round_score for the 32 live animals)2full_metric = mean(round_score for the 96 final animals)3​4partial_score = 100 × partial_metric5full_score = 100 × full_metric

While the competition is active, regular participants see only the partial score. The full score remains hidden and becomes the final leaderboard metric after the competition ends. It is the mean over the 96 full-set rows, not a combined mean over all 128 rows. The maximum possible value for either displayed score is 98.00, because even a correct first guess consumes one call.

Exporting a Policy

Build and test the strategy locally using the official sets and the frozen oracle. Export every decision as an ask, guess, or stop node, replace branches that exceed the 15-call horizon with null or stop, and merge identical subgraphs if necessary to remain within the node limit.

Place only policy.json in the evaluated ZIP archive. Separately upload the source code of the program that constructs the policy, or the corresponding notebook, in the mandatory source-code field. Do not upload model weights or precomputed oracle tables.

Before uploading, run:

1python3 validate_submission.py submission.zip

Attribution

Adapted with permission from “The Analytical Language of John Wilkins” (IOAI 2026 Home Task 3). See the official notebook.

The declarative policy archive, the platform’s private set, and the frozen precomputed oracle are MLCompete adaptations. The task requirements, prompt, allowed value sets, 15-call budget, and evaluation formula follow the official materials.