Asking through a model

What a host hands a model so it can write a program from a question: the vocabulary as a card, a grammar that keeps its output parseable and to the vocabulary, errors that say where the program is wrong and how to fix it, the loop that retries until a program checks, local models, a domain pack, and scoring it over questions with known answers.

A host that lets someone ask in plain language puts a model in front of jaala: the model writes a program from the question, jaala checks it, and a program that doesn’t check goes back to the model with its error. Package nl holds the parts of that every host would otherwise write alike (#321). What the words mean, the model and the tool that serves it stay with the host. An engine-only host never imports nl.

Each front end is a Go module of its own, so what one depends on reaches only the hosts that import it: go get github.com/panyam/jaala/frontends/nl for this page, frontends/nlmodels for the local model clients below, and frontends/mcp and frontends/cypher for the next two pages. They’re tagged with jaala, frontends/nl/v0.1.N beside v0.1.N. Up to v0.1.71 nl was a package inside jaala’s own module, so a host moving past that tag requires the new module in its place.

The examples use the vocabulary from Host options: the imports and pkg relations, the standard predicates and the deps module.

The card

nl.Card lists every relation and predicate a program can call, one line each: the signature, what a predicate needs bound before it runs, and the first line of the doc. A model reads it in place of the docs. The same vocabulary always gives the same text, so a host can keep it as a cached prompt prefix and pay for it once.

package guide_test

import (
	"context"
	"errors"
	"fmt"
	"strings"

	"github.com/panyam/jaala/datalog"
	"github.com/panyam/jaala/frontends/nl"
)

// Card lists the vocabulary a line per member, for a model's prompt. The
// same vocabulary always gives the same text, so it caches as a prefix.
func Example_card() {
	v, err := vocabulary(source())
	if err != nil {
		panic(err)
	}
	card, err := nl.Card(v)
	if err != nil {
		panic(err)
	}
	fmt.Print(card)
	// Output:
	// # Every relation and predicate a program can call, by module: name(arg: type, ...) # what it holds.
	// # ~type is inferred from the relation's rules, "in" marks an argument the caller must bind, [, arg] is optional.
	// # A predicate says what it needs bound before it runs: "filter" needs every argument.
	// absent(value) filter # reports whether the field carried no value at all, which is different from an empty string and from zero; `not absent(?x)` reads "this field is stated"
	// coalesce(value, fallback, result) needs value+fallback # binds the value, or the fallback when the value is absent
	// imports(from, to)
	// pkg(name, license)
	// graph.closure(relation, from, to, min: number, max: number, hops: number) needs relation+min+max # to is reachable from from in min to max steps of the two-argument relation named in quotes (max 0: no bound), hops the fewest
	// graph.paths(relation, from, to, mode, min: number, max: number, path) needs relation+mode+min+max # each path in min to max steps of the relation named in quotes, mode "trail", "acyclic", "simple", "shortest" or "walk"
	//
	// module deps
	// deps.depends_on(a, b) # Everything a package depends on, directly or through others.
	//
	// module list
	// list.at(list: list, index: number, value) needs list+index # binds the value at an index from 0, a negative one counting from the end (-1 is the last)
	// list.length(list: list, length: number) needs list # binds how many values the list holds
	// list.member(list: list, index: number, value) needs list # binds each value of the list with its index from 0; given a value, binds where it is
	//
	// module num
	// num.add(a: number, b: number, c: number) needs a+b or a+c or b+c # c = a + b, run from any two of the three; units follow the vocabulary's table
	// num.div(a: number, b: number, c: number) needs a+b or a+c or b+c # c = a / b, run from any two of the three; units follow the vocabulary's table
	// num.mul(a: number, b: number, c: number) needs a+b or a+c or b+c # c = a * b, run from any two of the three; units follow the vocabulary's table
	// num.neg(a: number, b: number) needs a or b # b = -a, run from either; the unit carries over
	// num.sub(a: number, b: number, c: number) needs a+b or a+c or b+c # c = a - b, run from any two of the three; units follow the vocabulary's table
	// num.unit(number: number, unit: string) needs number # binds the number's unit ("V", "A"), or absent for a number without one; the number must be bound
	// num.value(number: number, value: number) needs number # binds the number without its unit, so 5 V and 5 A both bind 5; the number must be bound
	//
	// module path
	// path.edge(path: string, index: number, edge) needs path # binds each edge of a path over a relation of three arguments, index from 0
	// path.edges(path: string, edges: list) needs path # binds the path's edges as a list
	// path.length(path: string, steps: number) needs path # binds how many steps the path takes
	// path.node(path: string, index: number, node) needs path # binds each node of the path, index from 0
	// path.nodes(path: string, nodes: list) needs path # binds the path's nodes as a list
	// path.step(path: string, index: number, from, to) needs path # binds each step of the path, index from 0
	//
	// module str
	// str.capitalize(string: string, capitalized: string) needs string # binds the string with its first character in upper case and the rest as written
	// str.concat(head: string, tail: string, text: string) needs head+tail or head+text or tail+text # text is head followed by tail, run from any two of the three; cutting a head or tail off holds for none when text lacks it
	// str.contains(string: string, substring: string) filter # reports whether a string contains a substring
	// str.distance(string: string, other: string, distance: number) needs string+other # binds the edit (Levenshtein) distance between two strings, in characters; both strings must be bound
	// str.glob(string: string, pattern: string) filter # reports whether the whole string matches a SQLite-style glob (`*` any run, `?` one character, `[a-z]` or `[^a-z]` one of a class, `[[]` a literal `[`)
	// str.lower(string: string, lower: string) needs string # binds the string in lower case
	// str.match(string: string, regex: string) filter # reports whether the string matches an unanchored regular expression
	// str.prefix(string: string, prefix: string) filter # reports whether a string starts with a prefix
	// str.replace(string: string, old: string, new: string, result: string) needs string+old+new # binds the string with every occurrence of old replaced by new
	// str.split(string: string, separator: string, index: number, part: string) needs string+separator # binds each part of the string between separators with its index from 0; given an index, binds that part alone, a negative index counting from the end (-1 is the last part)
	// str.suffix(string: string, suffix: string) filter # reports whether a string ends with a suffix
	// str.upper(string: string, upper: string) needs string # binds the string in upper case
}

// Explain turns a program's error into what a model fixes it from: the
// message, the place, and the fix. The CheckError says the same as data.
func Example_explainAnError() {
	v, err := vocabulary(source())
	if err != nil {
		panic(err)
	}
	for _, text := range []string{
		"depends_on(?a, ?b) => ?b",
		"deps.depends_on(?a, ?b),\n  pkg(?b) => ?b",
	} {
		err := datalog.Validate(datalog.MustParse(text), v)
		fmt.Println(nl.Explain(text, err))
		var ce *datalog.CheckError
		if errors.As(err, &ce) {
			fmt.Printf("kind %s, about %s\n\n", ce.Kind, ce.Relation)
		}
	}
	// Output:
	// query: unknown relation "depends_on"; did you mean "deps.depends_on"?
	// line 1, column 1:
	//   depends_on(?a, ?b) => ?b
	//   ^
	// fix: did you mean "deps.depends_on"?
	// kind unknown, about depends_on
	//
	// query: relation "pkg" takes 2 args, got 1
	// line 2, column 3:
	//     pkg(?b) => ?b
	//     ^
	// fix: pkg(name, license)
	// kind arity, about pkg
}

// scriptedModel replies with its drafts in turn. A host's Model would ask a
// real one, a local model through nlmodels or a hosted one.
type scriptedModel struct {
	drafts []string
	n      int
}

func (m *scriptedModel) Name() string { return "scripted" }

func (m *scriptedModel) Complete(_ context.Context, p nl.Prompt) (nl.Completion, error) {
	if m.n > 0 {
		fmt.Printf("told: %s\n\n", p.Turns[len(p.Turns)-1].Text)
	}
	m.n++
	return nl.Completion{Text: m.drafts[m.n-1]}, nil
}

// Translate asks a model for a program, checks it, and hands each error
// back until one checks. With a Base it runs the program too, so an empty
// answer can go to a fallback model.
func Example_translate() {
	v, err := vocabulary(source())
	if err != nil {
		panic(err)
	}
	m := &scriptedModel{drafts: []string{
		`depends_on("app", ?d) => ?d`,
		"```\ndeps.depends_on(\"app\", ?d) => ?d\n```",
	}}
	b := datalog.MustBase(v, source())
	p, tr, err := nl.Translate(context.Background(), "what does app depend on?", v, nil, m, nl.Options{Base: b})
	if err != nil {
		fmt.Println(err)
		return
	}
	for _, t := range tr.Tries {
		fmt.Printf("try by %s: %q\n", t.Model, t.Kind)
	}
	fmt.Println(p.Text)
	for _, r := range p.Rows {
		fmt.Print(r.Bind["d"].S, " ")
	}
	fmt.Println()
	// Output:
	// told: query: unknown relation "depends_on"; did you mean "deps.depends_on"?
	// line 1, column 1:
	//   depends_on("app", ?d) => ?d
	//   ^
	// fix: did you mean "deps.depends_on"?
	//
	// Write the whole program again, fixed.
	//
	// try by scripted: "unknown"
	// try by scripted: ""
	// deps.depends_on("app", ?d) => ?d
	// api log util
}

// Grammar is the program syntax as GBNF, for a decoder to keep a model's
// output to text that parses. Its first rules say what a program is.
func Example_grammar() {
	for _, line := range strings.SplitN(nl.Grammar(), "\n", 4)[:3] {
		fmt.Println(line)
	}
	// Output:
	// root ::= (rule sep)* goal
	// sep ::= "; " | ";\n"
	// rule ::= head " :- " literals
}

// draftsByQuestion answers each question with its own drafts in turn.
type draftsByQuestion map[string][]string

func (d draftsByQuestion) Name() string { return "scripted" }

func (d draftsByQuestion) Complete(_ context.Context, p nl.Prompt) (nl.Completion, error) {
	q := p.Turns[0].Text
	n := len(p.Turns) / 2 // a refused draft adds two turns
	return nl.Completion{Text: d[q][n]}, nil
}

// Eval scores a translator over questions with known answers: a row
// matches by its values, whatever columns the program chose.
func Example_eval() {
	v, err := vocabulary(source())
	if err != nil {
		panic(err)
	}
	b := datalog.MustBase(v, source())
	m := draftsByQuestion{
		"what does app import?":        {`imports("app", ?d) => ?d`},
		"what license does util have?": {`pkg("util") => ?l`, `pkg("util", ?l) => ?l`},
		"which packages import log?":   {`imports(?p, log) => ?p`, `imports(?p, "log") => ?p`},
	}
	cases := []nl.Case{
		{Name: "direct", Question: "what does app import?", Want: [][]string{{"api"}}},
		{Name: "license", Question: "what license does util have?", Want: [][]string{{"Apache-2.0"}}},
		{Name: "importers", Question: "which packages import log?", Want: [][]string{{"api"}}},
	}
	rep := nl.Eval(context.Background(), cases, b, func(ctx context.Context, q string) (nl.Program, nl.Trace, error) {
		return nl.Translate(ctx, q, v, nil, m, nl.Options{})
	})
	fmt.Print(rep)
	// Output:
	// case       pass  tries  refused
	// direct     yes   1
	// license    yes   2      arity
	// importers  yes   2      parse
	// passed 3/3, 1 on the first try; 5 tries, tokens 0 in (0 cached), 0 out
	// refused: arity 1, parse 1; check errors with a fix 1/1
}

A type written ~node is one the engine inferred from a derived relation’s rules rather than one its head declares, and in marks an argument the caller has to bind (#200). filter means every argument has to be bound; a generator lists the arguments each of its modes needs, needs a or b.

The grammar

nl.Grammar() is the program syntax as a GBNF grammar, the form llama.cpp’s grammar field takes (#323). A model decoding under it can only write text that parses, so no retry is spent on a missing bracket or a misplaced =>. The fourth example above prints its first rules; the whole grammar is frontends/nl/testdata/program.gbnf.

It covers jaala’s own spelling, the one these docs use: clauses separated by ; , not for negation, the answer after => with having, order by, after, limit and offset. It leaves out the forms jaala reads only so people and agents can write other Datalogs (comments, a . ending, ?-, !atom, see Syntax), and a type that names another head variable (?k, kind(?o)), which only the parser can check. It allows one space or newline where the spelling has room for one, so a model can’t spend tokens on blank space, and caps a name (a relation’s segment, a variable, a unit) at 32 characters, so a model that wants to write prose can’t run one name on until its token limit (#340).

nl.GrammarFor(v) narrows it to a vocabulary (#324). A call names one of v’s members at an arity it takes, and an argument whose type is a closed domain is a variable or one of the domain’s values, so a model decoding under it can’t write edge(?a), a member that doesn’t exist, or a misspelled kind. A program’s own rules take names nothing at the root of v uses, so a call to any other name is a call to one of them, which the grammar can’t check is defined. nl.GrammarFor(v, nl.GoalOnly()) leaves rules out, so every call is a real member; that’s the strictest form, for a small model asking one question. Both are built once per state of the vocabulary, as the card is.

Parsing, and with GrammarFor the names and counts, is all a grammar promises. Whether every variable is bound, and whether a rule a program calls exists, is still Validate’s, and its errors are the next section’s.

Errors a model can fix from

A program that can’t run as written fails Validate and Eval with a *datalog.CheckError. Its message is the one the error catalogue lists, unchanged, and its fields say the same as data:

  • Kind sorts it: unknown, arity, unbound, type, stratify, definition, aggregate or answer. A host counts failed programs by kind without matching on text.
  • Fix says how to change the program, when the engine can tell: the vocabulary’s suggestion for a name, or the program’s own rule it likely means (str.mine for a rule mine, #341), the signature for a call at the wrong arity, what to bind for an unbound variable. It’s empty for a cycle through negation, where the fix depends on what the program meant.
  • Relation, Args, Var and Rule say what the error is about.

A Query carries no positions: a host can build one in Go, and two parses of one text compare equal. So datalog.Locate(text, err) finds the place in the text instead, the call to Relation with Args arguments (in Rule’s clauses when it names one), else the first ?Var. It returns a ParseError’s own position too, so one call places any error. nl.Explain(text, err) writes all of it the way a model reads it, the second example above: the message, the line with a caret under the place, and the fix.

The loop

nl.Translate puts the card and the errors together (#325). It asks the host’s model for a program, parses and checks it, and when it doesn’t check, sends the draft back with Explain’s text and asks again, up to Options.MaxTries times (3 by default). The third example above is a scripted model getting a name wrong once: the second prompt carries the first draft and its error, and the second draft checks. A reply wrapped in a Markdown code fence is read from inside the fence.

Every prompt carries GrammarFor(v) as Prompt.Grammar, or the host’s own with Options.Grammar (none with Options.NoGrammar), for a model whose decoder takes one. A model that can’t use it ignores it, since every draft is checked anyway; the grammar only saves tries.

A model is the host’s: Translate calls an nl.Model, a name and a Complete taking an nl.Prompt, which is a Prefix holding the card and a few lines of instructions, the same for every question over one vocabulary so a client can send it as a cached prefix, and the conversation so far as Turns. The local model clients below are two such models. A host that can find the names in a question, as Declaire’s name.find turns a short name into ranked ids, implements nl.Grounder, and the candidates go after the question with their values quoted, so the model copies a value rather than spelling it.

Options.Fallback is a second model, usually a bigger one, asked once the first runs out of tries. It starts a fresh conversation with the question, since the first model’s drafts would only cost it tokens. A model that sends back a draft it was already told is wrong has stopped reading the errors, so that try is refused unchecked and the question moves to the fallback, or Translate stops when there is none (#340). With Options.Base, a program that checks is run too (with Options.Eval’s options, such as a Budget): an error is fed back like a check error, and a program answering nothing goes straight to the fallback, told that one did, or back to the same model when no fallback is left. That model is told which literal matched nothing and what it was first called with (Report.Empty, see Host options, #343), so it can change that call rather than guess, and whether the call matches anything by itself (#348), which tells a wrong value from a wrong join. A Complete that fails moves to the fallback as well.

Translate returns the accepted nl.Program (its text, Query, rows when it ran, and the model that wrote it) and an nl.Trace of every try, also when it fails: the model, the text, the error and its Kind, tokens in, out and cached, latency, and, for a draft run over Options.Base, the datalog.Report of that run. An empty answer’s error is an nl.EmptyAnswer, whose Why is the run’s Report.Empty. Kind is a CheckError’s kind, or parse, eval, empty, repeat, answer or model, so a host counts where its retries went without reading messages.

Looking before answering

A model drafting blind guesses at what the data holds, and on Declaire’s questions most of a local model’s failures were guesses that answered nothing: an edge kind, an attribute’s value, an id’s spelling (#393). Options.Actions lets it look first. Each reply may then be an action instead of an answer, and its result comes back as the next turn:

  • look NAME: a relation’s signature and whole doc, or a module’s members.
  • values TARGET: up to Actions.Sample (10) of one argument’s values, as pkg.license or pkg.1, run as pkg(_, ?v) => ?v limit 10.
  • try on its own line, then a draft: it runs with at most Sample rows, and sends back its rows, or why it answered none, as an empty answer is explained.
  • find NAME: the values a name may stand for, from the Grounder, when Translate has one. Grounding the question up front guesses which of its words are names, and this lets the model ask about the one it’s stuck on. Without a Grounder the prompt and grammar don’t offer it.

Any other reply is the answer, judged as a draft always is. An action doesn’t spend a try. Actions.Steps (6) caps them per model, a repeated one is answered with a note that it was asked already, and once they run out the next reply must be the answer, under the answer’s grammar. The grammar sent wraps the draft’s, so a model under GBNF writes only a well-formed action or a draft. Trace.Steps records each action with what went back and what it cost, and Report counts steps and their tokens beside the tries, so the first-try rate still counts answers.

The loop doesn’t read the drafts’ language: a try goes through the front end as an answer does, and a front end that implements nl.Prober answers look and values in its own terms. Cypher’s does, so a model writing Cypher asks values Package.license and the probe runs as MATCH (n:Package) RETURN DISTINCT n.license LIMIT 10.

A draft that guesses a value’s spelling, "mit" where the data holds "MIT", is repaired by Translate itself when the conversation has seen the right one. The values it has seen are those a values, try or find sent back, plus the closed Domain of an argument whose check refused the constant. When a draft answers nothing, or the check refuses one of its constants, each quoted value the data doesn’t hold as written but holds in exactly one other spelling (case and punctuation apart, so "Apache 2.0" meets "Apache-2.0") is rewritten. The new draft is then checked and run again. If it answers, it’s accepted as a Try of its own after the model’s, with Try.Repair saying what changed, and Report.Repairs counts them. If it doesn’t, the model’s own draft is what goes back. The harness never picks between two spellings.

Actions.Compact sends the model one turn in place of the conversation: the question, each action and refused answer as a fact (- values pkg.license and what came back), and the last refusal whole. A small model’s drafts get worse as its context grows, and the facts are what it needs from the turns. A fallback starts afresh, as it does without Compact.

Voting, escalation and a budget

One answer from a small model is one guess. Options.Vote draws more once a model has answered (Vote.Drafts, 3 in all by default, at once with Vote.Parallel for a model safe to call concurrently), judges each as an answer, and keeps the one whose rows most of them answer, the first among equals (#393). Drafts agree when their rows match by value as a multiset, whatever their variables are called, and without a Base when they compile to the same program. A refused draft agrees with nothing. Each draft is asked with Prompt.Draw set, and nlmodels’ clients send it as the seed, but the drafts only differ when the model samples: at temperature 0 a vote agrees with itself every time.

Program.Confidence carries what Translate knows about an answer: how many drafts agreed out of how many, whether it ran and answered rows, whether the pack’s Check passed it, and how many actions came first. Options.Escalate moves an answer whose drafts agree less than that share on to the Fallback, recorded in Trace.Escalations, and when the fallback answers nothing better, the first such answer comes back rather than an error. A host whose fallback is a frontier model or agent pays for it only on the questions the small model was unsure of.

Options.Budget caps the tokens (prompt and reply, every call together) and the time Translate spends, across actions, tries and votes. It stops before the call that would pass it, with a *nl.BudgetSpent. A vote the budget leaves no room for draws fewer drafts.

Try.Class sorts a refused try by what went wrong. grounding is a draft that ran and answered nothing, and construct one that didn’t parse, compile or check. semantics is one the pack’s Check refused, and budget one an evaluation budget stopped. Result.Class does the same for a case Eval scored, with a case that answered the wrong rows as semantics. Report.Failures counts the failed cases by class, and the report’s summary counts votes and escalations.

Front ends

A model needn’t write jaala’s syntax. Options.FrontEnd names a language it writes instead (#354), and Cypher’s is cypher.NewFrontEnd(graph, examples...) (asking in Cypher). The prompt is then the front end’s card, instructions and examples, followed by the pack’s About and data Card but not its jaala examples. The grammar sent is the front end’s. Each draft is compiled before it’s checked and run, and every refusal goes back as the front end explains it, in the draft’s language and at its place in the draft. Everything else is unchanged: repeats, empty answers, the fallback and the pack’s Check. An empty answer goes back at the draft’s place too: cypher.Explain puts the caret under the pattern that wrote the literal Report.Empty ends at (#386).

Each Try and the accepted Program record the Language, and Program.Compiled holds the jaala program the draft became. A compile refusal is the kind compile, or the front end’s own, such as Cypher’s gap for a construct jaala can’t express yet. Report.String names the language, so a host can run one set of questions through both languages on one model and compare the two reports: first-try passes, tries, tokens and what was refused.

A front end is an interface in nl (nl.FrontEnd) that the front end’s own module implements, since nl imports no front end.

Local models

jaala runs no model, but github.com/panyam/jaala/frontends/nlmodels, a module of its own, has clients for local ones already running, so no host takes on HTTP clients it doesn’t use (#330):

  • llamacpp.New(url) asks llama.cpp’s llama-server at its OpenAI-style /v1/chat/completions. It sends the prefix as the system message, the prompt’s grammar as grammar, so the model can only write text that parses, and cache_prompt, so the card is processed once. Its Name is llama.cpp, or llama.cpp: and Model when one is set.
  • ollama.New(url, model) asks an Ollama server for one model, named ollama: and the model. Ollama takes no GBNF grammar (its format is a JSON schema), so the model writes freely and costs more tries; a host wanting the grammar held runs the same weights under llama.cpp.

Both reply with at most MaxTokens (512 by default) at Temperature 0, and take an HTTP client of the host’s. The tokens a try reports are the server’s own counts, with llama.cpp’s cached prompt tokens. Here a stand-in for llama-server answers one program and prints what it was sent:

package guide_test

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/http/httptest"
	"strings"

	"github.com/panyam/jaala/datalog"
	"github.com/panyam/jaala/frontends/nl"
	"github.com/panyam/jaala/frontends/nlmodels/llamacpp"
)

// llamaServer stands in for llama.cpp's llama-server: it answers every chat
// with one program, and says what the client sent it.
func llamaServer(program string) *httptest.Server {
	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		var req struct {
			Messages []struct {
				Role string `json:"role"`
			} `json:"messages"`
			Grammar     string `json:"grammar"`
			CachePrompt bool   `json:"cache_prompt"`
			MaxTokens   int    `json:"max_tokens"`
		}
		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
			http.Error(w, err.Error(), http.StatusBadRequest)
			return
		}
		var roles []string
		for _, m := range req.Messages {
			roles = append(roles, m.Role)
		}
		fmt.Printf("%s %s: messages %s, grammar sent %v, cache_prompt %v, max_tokens %d\n",
			r.Method, r.URL.Path, strings.Join(roles, "+"), req.Grammar != "", req.CachePrompt, req.MaxTokens)
		json.NewEncoder(w).Encode(map[string]any{
			"choices": []any{map[string]any{"message": map[string]string{"role": "assistant", "content": program}}},
			"usage":   map[string]int{"prompt_tokens": 900, "completion_tokens": 12},
		})
	}))
}

// llamacpp.New is an nl.Model over a llama-server already running: it sends
// the card as a cached system prefix and the vocabulary's grammar with every
// question.
func Example_llamaServer() {
	srv := llamaServer(`imports("app", ?d) => ?d`)
	defer srv.Close()
	v, err := vocabulary(source())
	if err != nil {
		panic(err)
	}
	m := llamacpp.New(srv.URL)
	p, tr, err := nl.Translate(context.Background(), "what does app import?", v, nil, m,
		nl.Options{Base: datalog.MustBase(v, source())})
	if err != nil {
		fmt.Println(err)
		return
	}
	t := tr.Tries[0]
	fmt.Printf("%s wrote %s, %d tokens in, %d out\n", t.Model, p.Text, t.TokensIn, t.TokensOut)
	fmt.Println("answer:", p.Rows[0].Bind["d"].S)
	// Output:
	// POST /v1/chat/completions: messages system+user, grammar sent true, cache_prompt true, max_tokens 512
	// llama.cpp wrote imports("app", ?d) => ?d, 900 tokens in, 12 out
	// answer: api
}

Domain packs

The card says what a program can call, but not what the data holds or what a question in this domain usually looks like. A host says that once, in an nl.Pack passed as Options.Pack (#344), so Declaire’s code graph and Agni’s schematics are each a pack over the same Translate:

  • About says what the data is about, in the words its users use. It opens the pack’s part of the prefix, and jaala’s MCP server (#347) writes its instructions from it.
  • Card says what the data holds that the vocabulary can’t, such as which kinds and values a question will name. nl.DataCard(ctx, base) writes the generic half: each relation with how many facts it holds and, per argument, its values with how many facts hold each, or how many there are and the commonest when there are more than twelve. It counts every relation whole, so a host computes it once per data version and adds what only it knows, such as how its ids are spelled.
  • Examples are questions with the programs that answer them, one or two per construct. How to call one relation belongs in its doc instead.
  • Grounder grounds the question when Translate is given none.
  • Check judges an answer that checked and ran: an error refuses the draft as answer and goes back to the same model with its text, as a check error does. It’s for what makes an answer suspect in the domain, such as every value absent, or a hundred rows for a question asking for one.
  • Hint says what your data holds where an empty answer matched nothing. It gets Report.Empty, with the culprit’s Relation and Args, and returns text such as no node kind "entity"; the kinds are package, func, type. Translate adds it to the empty draft’s feedback after jaala’s own explanation, and the MCP server to an empty answer of query, cypher and explain (#430). It runs once per empty answer, so keep it to a few lookups.

The pack’s part goes after the card and the instructions, About and the examples as comments, so the prefix is still the same for every question and stays cached. Translate checks every example against the vocabulary before asking a model, and fails naming the first that doesn’t check, so a pack can’t teach a model a relation the vocabulary lacks. A host’s test can call Pack.Validate and Pack.Prefix itself. Here DataCard writes the guide’s data, and a pack whose Check expects one package refuses a first draft answering three:

package guide_test

import (
	"context"
	"fmt"

	"github.com/panyam/jaala/datalog"
	"github.com/panyam/jaala/frontends/nl"
)

// DataCard writes what the data holds, for a pack's Card: each relation's
// facts, and each argument's values with how many facts hold each.
func Example_dataCard() {
	v, err := vocabulary(source())
	if err != nil {
		panic(err)
	}
	card, err := nl.DataCard(context.Background(), datalog.MustBase(v, source()))
	if err != nil {
		panic(err)
	}
	fmt.Print(card)
	// Output:
	// # The data: each relation's facts, and each argument's values with how many facts hold each (or how many values, and the commonest, when there are many).
	// # imports(from, to): 3 facts
	// #   from: "api" (1), "app" (1), "log" (1)
	// #   to: "api" (1), "log" (1), "util" (1)
	// # pkg(name, license): 4 facts
	// #   name: "api" (1), "app" (1), "log" (1), "util" (1)
	// #   license: absent (1), "" (1), "Apache-2.0" (1), "MIT" (1)
}

// A pack is the host's domain: what the data is about, what it holds,
// worked examples, and a check that refuses an answer that can't be right.
func Example_pack() {
	v, err := vocabulary(source())
	if err != nil {
		panic(err)
	}
	b := datalog.MustBase(v, source())
	pack := nl.Pack{
		About: "The packages of a Go service and what each imports.",
		Card:  "# Package names are lower case, such as app and util.",
		Examples: []nl.Example{
			{Question: "what does api import?", Program: `imports("api", ?d) => ?d`},
		},
		Check: func(_ context.Context, p nl.Program) error {
			if len(p.Rows) > 1 {
				return fmt.Errorf("the question asks for one package, and this answered %d", len(p.Rows))
			}
			return nil
		},
	}
	if err := pack.Validate(v); err != nil {
		panic(err)
	}
	fmt.Print(pack.Prefix())

	m := draftsByQuestion{"which package imports log?": {
		`imports(?p, ?q) => ?p`,
		`imports(?p, "log") => ?p`,
	}}
	p, tr, err := nl.Translate(context.Background(), "which package imports log?", v, nil, m, nl.Options{Base: b, Pack: &pack})
	if err != nil {
		fmt.Println(err)
		return
	}
	for _, t := range tr.Tries {
		fmt.Printf("try %q: %v\n", t.Kind, t.Err)
	}
	fmt.Println(p.Text, "answers", p.Rows[0].Bind["p"].S)
	// Output:
	// # What the data is about:
	// # The packages of a Go service and what each imports.
	//
	// # Package names are lower case, such as app and util.
	//
	// # Examples:
	// # what does api import?
	// imports("api", ?d) => ?d
	// try "answer": the question asks for one package, and this answered 3
	// try "": <nil>
	// imports(?p, "log") => ?p answers api
}

Extracting facts from text

The same loop runs the other way: nl.Extract turns text (a document, a chat log, a ticket thread) into the facts it states, as an ns.Changes the host can apply (#414). The prefix lists the vocabulary’s base relations, each with its labels, a closed domain’s values and its doc, and the model replies with one fact per line:

+ owner("app", "bob") from "Bob owns app now"
- imports("app", "api") from "app no longer imports api"

+ is a fact the text states and - one it says no longer holds. Each fact is read as a goal of one atom and checked by datalog.Validate, so its values are held to the argument’s type, closed domain and units exactly as a query’s constants are, and a refusal goes back with the line, a caret under the value and the engine’s fix (did you mean "Apache-2.0"?). A text value and the words after from read a backslash as keeping the next character alone, so "say \"hi\"" is say "hi", the spelling a host stores and a key finds (#444). The words must appear in the text as written, and an added fact cites them as text:<start>-<end>, byte offsets into the text, so an answer over the facts explains itself back to the words. A derived relation’s facts follow from its rules, so a line adding one is refused. nl.ExtractGrammar is the reply’s GBNF, the base relations at their arities with a closed domain’s values spelled out, which Extract sends as Prompt.Grammar.

The engine never decides which facts are one (#183), so a pack says it: Pack.Keys maps a relation to the argument positions that identify its fact. Under {"owner": {0}}, a line adding owner("app", "bob") removes the owner("app", "alice") that ExtractOptions.Base holds, or that an earlier line added, in the same change set. A relation without keys only gains facts. Pack.ExtractExamples are texts with the lines they should give, which follow the pack’s About and Card; Pack.Validate reads each one, and checks every key names a base relation and a position within its arity, before any model is asked. MaxTries, Fallback, a repeated draft and Budget work as they do for Translate, and the Trace records each try with its kind, span for words the text doesn’t hold and derived for a derived relation among them.

Serving agents

The same pack makes an MCP server, so an agent asks over the vocabulary with its own tools: Serving agents covers frontends/mcp.

Scoring a translator

nl.Eval runs a set of nl.Cases, each a question and the rows a right program answers, through a translator and a base, and reports how it went (#326). The translator is a function from a question to Translate’s results, usually Translate closed over a vocabulary, a model and options, so a host can wrap its own fallback or tool around it. A program whose rows Translate already ran over Options.Base is scored on those; any other runs over the base given to Eval. A case failing, in translating or evaluating, fails alone.

The model chooses the program’s columns, so a row matches by its values whatever their names and order: => ?b, ?a answers {"app", "api"} as well as => ?a, ?b does. Rows match as a multiset, in order only when the case sets Ordered, and an absent value is written absent. The last example above scores three questions, two needing a second draft.

The nl.Report holds each case’s Result (the program, its trace, the rows it got and whether they passed) and the totals: passes on the first try and after retries, tries, tokens in, out and cached, latency, the refused tries by kind, and how many of the refused check errors carried a fix. That last pair says how often a model was told how to fix its mistake, and a retry spent on an error without one is where an error message could say more. Report.String writes it as the example does. When Translate ran the programs over a Base, it adds each case’s work and the three costliest drafts with the literal that spent the most (Report.Costliest), ranked by work rather than time, so two runs rank them the same. jaala’s own TestEval (cd frontends/nl && go test -run TestEval -v .) scores a scripted model and fallback the same way, over eight questions on this vocabulary.