WobbleTales

Open-200 Encoded narrative reasoning

When language wobbles, can reasoning hold?

WobbleTales asks models to recover meaning from encoded short passages, follow their story logic, and answer precisely—without leaning as heavily on familiar language cues.

Run Open-200 View on GitHub

Whimsy on the surface. Discipline underneath.

Evaluation folio№ 001

OPEN-200-V1

Release
200 encoded items
Default run
10-item sample
Range
1–200 items
Interface
Provider-neutral
RESPONSE CONTRACT { "best_reconstruction": "…",
"answer": "…" }
200encoded challenges
10items by default
3.11+Python
0runtime dependencies

One coupled task

Recover the tale. Understand it. Answer it.

WobbleTales keeps decoding and comprehension together. The model receives one encoded passage in a fresh context and must reconstruct its meaning before giving the single best-supported answer.

  1. 01Encoded narrativeThe tale arrives without familiar surface form.
  2. 02Fresh contextEach item stands alone under the frozen protocol.
  3. 03Precise answerThe model reconstructs and answers in strict JSON.
  4. 04Hosted reportResults return in human- and machine-readable form.

The first public release

Open-200

A fixed collection of 200 encoded narrative-comprehension challenges, packaged with a reproducible runner and a hosted scoring workflow.

Explore the release on GitHub

Reproducible sampling

Choose a seed and rerun the same subset when comparisons matter.

Resumable runs

Long evaluations can continue from saved local state.

Adjustable scope

Run a quick sample or expand all the way to 200 items.

Portable adapters

Use major hosted providers or an OpenAI-compatible endpoint.

Bring your model

Built for frontier-model comparisons.

Native adapters cover widely used APIs. The compatible endpoint adapter extends the same frozen evaluation protocol to other hosted models and gateways.

Quick start

From install to report.

Install directly from GitHub, select a provider and model, then submit the completed run to the scoring service.

01 · Install
python -m pip install \
  "git+https://github.com/stuartjonesstats/wobbletales.git"
02 · Run a reproducible sample
export OPENAI_API_KEY="..."
wobbletales run \
  --adapter openai-responses \
  --model MODEL_NAME \
  --sample-size 10 \
  --sample-seed community-example \
  --run-dir runs/model-name
03 · Submit for scoring
export WOBBLETALES_SCORE_TOKEN="..."
wobbletales submit \
  --run-dir runs/model-name \
  --score-url https://api.wobbletales.com

Provider credentials are read from environment variables and are never written to the run directory. The scoring service returns authenticated HTML and JSON reports.

Read the full run guide

Evaluation workflow

Four moves. One consistent protocol.

  1. 1
    Select

    Choose a provider, model, sample size, and seed.

  2. 2
    Run

    Give every challenge a fresh model context.

  3. 3
    Submit

    Send the completed bundle to hosted scoring.

  4. 4
    Review

    Open the HTML report or use the JSON result.

Good questions

Before you set the tales wobbling.

Still curious? Open a public issue for ordinary questions or use the private security channel for sensitive reports.

What does WobbleTales measure?

It tests whether a model can recover meaning from an encoded narrative and correctly answer a question about that narrative. The value is in the coupled task, not either stage by itself.

Which models can I run?

The public client includes native OpenAI, Anthropic, and Gemini adapters plus an OpenAI-compatible adapter for other providers and gateways.

Must I run all 200 challenges?

No. The default is a reproducible 10-item sample. Choose any sample size from 1 through 200, depending on your budget and evaluation needs.

Are provider credentials stored?

No. Credentials are read from environment variables and are not written to the local run directory.

How do I request scoring access?

Open a scoring-access request on GitHub. Never include API keys, scoring tokens, or other credentials in an issue; access is coordinated privately.

Open-200 is ready

Pick a model. Let the language wobble.

Run the benchmark Open GitHub