conform

A conformance suite for LLM inference engines. vLLM, llama.cpp, Ollama, SGLang and TGI all claim to serve the same models. Feed them identical inputs and they do not always produce identical outputs.

The problem

Browsers had this problem first. Chrome, Firefox and Safari all claim to render the same HTML, mostly agree, and occasionally do not. The web settled it with a shared test suite that pins down who is wrong.

Inference engines are the browsers here and the model file is the HTML. There is no equivalent test suite, so the disagreements get found one confused bug report at a time. They are small, almost never documented, and they silently change your output when you switch engines.

How it works

Send the same request to every engine. Compare against a reference that makes no optimisations, HuggingFace transformers on CPU in float32. Publish the grid.

A test is written once and becomes a row for every configured engine. Most engines need no code at all, because they all speak the same OpenAI-compatible HTTP API and only the URL changes.

A real finding

From the first CI run against Ollama serving qwen2.5:0.5b. This is measured output, not an illustration.

| test                                      | ollama |
|-------------------------------------------|--------|
| test_greedy_is_repeatable                 | pass   |
| test_greedy_ignores_top_p                 | pass   |
| test_stop_string_is_excluded              | pass   |
| test_stop_matches_across_token_boundaries | pass   |
| test_stop_sets_finish_reason              | pass   |
| test_max_tokens_is_respected              | pass   |
| test_truncation_sets_finish_reason_length | pass   |
| test_zero_max_tokens_behaviour            | note   |
    -> returns 'The capital of France is Paris.'
       (finish_reason='stop')

Asked for max_tokens=0, Ollama does not return empty output and does not return a single token. It ignores the limit, generates to the end, and reports finish_reason='stop' as though nothing unusual happened.

Any application computing max_tokens = budget - prompt_tokens will occasionally hit zero, and on this engine a request meaning “generate nothing” becomes an unbounded one. On an engine that returns empty, the same code path is harmless.

It does not declare winners

That row is reported as note, not a failure, because the OpenAI API never defines what max_tokens=0 means. Every check falls into one of three tiers.

A failure tells one team to fix something. A note tells the whole ecosystem that a behaviour needs specifying.

Try it

Python 3.11+. The suite itself needs no GPU and no model.

# install
git clone https://github.com/NiLabs-Org/conform
cd conform && pip install -e ".[reference,dev]"
cp conform.example.toml conform.toml

# point it at something
ollama pull qwen2.5:0.5b && ollama serve

# run
conform run

Results land in results/matrix.json and results/matrix.md. The harness also self-tests against a stub server, including one case that points it at a deliberately broken stop-sequence implementation and asserts the bug is caught. Without that, a green run would be indistinguishable from a run where nothing was reachable.

What gets checked

• Tokenizer — decode(encode(x)) == x across whitespace, unicode, emoji and code
• Determinism — greedy decoding repeatable, and unaffected by top_p
• Stop sequences — excluded from output, matched across token boundaries
• Length — max_tokens excludes the prompt, finish_reason is correct
• Sampling, logprobs, chat templates, structured output — planned