nli
3 TagsUpdated MITby Moritz Laurer
Zero-shot classifiers by Moritz Laurer: every option becomes a hypothesis scored for entailment. The most accurate open model on typed decisions in our tests.
ollaya run nli --preset triage "I was charged twice for my subscription this month and want a refund."curl http://localhost:11435/api/decide \
-H "Content-Type: application/json" \
-d '{
"model": "nli",
"state": "I was charged twice for my subscription this month and want a refund.",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {
"billing": "Payments, invoices and refunds",
"technical": "Bugs, errors and outages",
"account": "Login, profile and settings"
}
},
"refund": {
"type": "noul",
"instructions": "Is the customer asking for a refund?"
}
}
}'# Already using a TypeSafe SDK? Set TYPESAFE_BASE_URL=http://localhost:11435 instead.
import requests
response = requests.post(
"http://localhost:11435/api/decide",
json={
"model": "nli",
"state": "I was charged twice for my subscription this month and want a refund.",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {
"billing": "Payments, invoices and refunds",
"technical": "Bugs, errors and outages",
"account": "Login, profile and settings"
}
},
"refund": {
"type": "noul",
"instructions": "Is the customer asking for a refund?"
}
}
},
)
answers = response.json()["answers"]
print(answers["department"]["choice"], answers["refund"]["noul"])const response = await fetch("http://localhost:11435/api/decide", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
model: "nli",
state: "I was charged twice for my subscription this month and want a refund.",
questions: {
department: {
type: "choice",
instructions: "Which team should handle this?",
criteria: {
billing: "Payments, invoices and refunds",
technical: "Bugs, errors and outages",
account: "Login, profile and settings"
}
},
refund: {
type: "noul",
instructions: "Is the customer asking for a refund?"
}
}
}),
});
const { answers } = await response.json();
console.log(answers.department.choice, answers.refund.noul);Models
View all| Name |
|---|
| nlilatest884 MB · 512 ctx · English |
| nli:deberta-v3-large884 MB · 512 ctx · English |
| nli:modernbert-large799 MB · 512 ctx · English |
Each model carries fp16 and fp32 graphs over one weights file, and loads fp16 on a CUDA GPU and fp32 on CPU.
Readme
Zero-shot NLI classifiers by Moritz Laurer. Ollaya turns every option of a question into a hypothesis and asks the model whether the state entails it. Choice and score questions take the most-entailed option, and a yes/no question takes the probability that its statement is entailed.
Models
| Tag | Backbone | Params | License | Typed-decisions accuracy |
|---|---|---|---|---|
nli:latest, nli:deberta-v3-large | DeBERTa-v3-large | 435M | MIT | 0.548 |
nli:modernbert-large | ModernBERT-large | 396M | Apache-2.0 | 0.515 |
Accuracy is the argmax against the majority label on all 400 typed-decisions states. For comparison, laya:en scores 0.361 and gliclass 0.477. The labels have low annotator agreement, so compare the numbers to each other rather than reading them as absolutes.
Usage
ollaya run nli --preset triage "I was charged twice for my subscription this month and want a refund."Point any TypeSafe client at http://localhost:11435 and set the model to nli.
How it works
- One sequence pair per option. Each option is a premise–hypothesis pair (the state, then the option's hypothesis), so the cost grows with the number of options. A question with 20 options runs 20 sequences.
- Batching. All sequences in a request share one batched forward pass.
- Hypothesis templates. They come from each model's recommended phrasing and live in the model's
decisionlayer. - Weights. They are the authors' own
model.safetensors, downloaded from Hugging Face, pinned to a commit and verified by sha256. Ollaya hosts only the ONNX graph (about 5 MB). - Parity. Ollaya's Rust runtime reproduces the Python reference exactly: the same token ids and the same decision on every test question, on CPU and CUDA.
Limits
- Uncalibrated. These are zero-shot entailment models, not trained decision models. Their probabilities are not calibrated, so fit a
CALIBRATIONlayer on your own data before you rely on thresholds. - English. They work best on English text.
- Training data licenses. The model card notes that part of the DeBERTa model's training data carries non-commercial licenses. The author publishes a
-cvariant trained on commercially usable data only; Ollaya does not ship it yet.