MIT open source · Python 3.10+

Does your local model actually call tools?

caniollama runs a standardized tool-calling test suite against the models in your local Ollama and tells you which ones return real structured tool calls, which fall back to JSON in the text, and which get rejected.

Get started View on GitHub
uvx caniollama check
terminal
$ caniollama check qwen2.5-coder:7b

qwen2.5-coder:7b (quant: Q4_K_M, ollama 0.x, suite 1.x)

Case                     Outcome                Latency
single_tool_basic        structured pass        … ms
multi_tool_5x            structured pass        … ms
no_tools_needed_control  structured pass        … ms
multiturn_tool_result    json fallback          … ms

3/4 cases returned structured tool calls.
Example caniollama check output

Illustrative output. Case names and outcome labels are the real ones from the suite; results depend on your model and Ollama version.

Features

Tests real behavior, not model cards

A model card saying "supports tools" doesn't mean it works in your Ollama setup. caniollama calls your local models and checks what actually comes back.

Standardized suite

Every model gets the same four cases: a basic call, picking one tool out of five, a no-tool control, and a multi-turn follow-up with a tool result.

Clear outcomes

Each case comes back as one of four outcomes: structured pass, JSON fallback, rejected by backend, or error/timeout.

Zero install

Run it once with uvx or pipx run, or install it for good with pip, pipx or uv.

Opt-in, metadata only

Results stay on your machine unless you pass --submit. Submissions carry the model name, Ollama version and pass/fail outcomes. Prompts and model outputs are never sent.

Community registry

A FastAPI backend that collects submissions and serves pass rates per model and per test case.

Self-hostable

Run the registry yourself with the Docker image, then point the CLI at it with --registry-url.

How it works

From your Ollama to a compatibility verdict

The CLI (compat-check) talks to Ollama directly. The registry (compat-registry) is optional.

  1. Discover

    Finds the models pulled into your Ollama (or OLLAMA_HOST) and reads its version.

  2. Run suite

    Sends each test case with tool definitions and records how long the response takes.

  3. Score

    Classifies every response and prints a local report table.

  4. Submit (opt-in)

    With your consent, posts the reduced result to the registry, which aggregates it into per-model pass rates.

The four outcomes

  • structured_pass
    Uses the native tool_calls field.
  • json_fallback
    Tries to call a tool, but writes the JSON into the message content.
  • rejected
    The backend refuses the tools parameter.
  • error
    Timeouts or other failures.

Registry API

Read-only endpoints for tool builders who want to check compatibility before choosing a model:

  • GET /models
  • GET /models/{model_name}
  • GET /stats
  • GET /health · POST /submissions

Quick start

Up and running in a minute

You need Ollama running locally with at least one model pulled. uvx/pipx take care of Python 3.10+ for you.

1. Check every local model

No install needed.

uvx caniollama check
# or
pipx run caniollama check

2. Check one model

Pass the model name as Ollama lists it.

caniollama check qwen2.5-coder:7b

3. Install permanently

Pick your package manager.

pip install caniollama
pipx install caniollama
uv tool install caniollama

4. Share results (optional)

Submits metadata only, after asking for consent.

caniollama check --submit

Query the registry

Live lookup needs a registry backend. Run your own and open this page from localhost to try it:

docker pull ghcr.io/anjalll/caniollama-registry:latest
docker run -p 8000:8000 ghcr.io/anjalll/caniollama-registry:latest
# API docs: http://localhost:8000/docs