Serving
Control and serving planes
Section titled “Control and serving planes”mainpublishes a versioned config snapshot: routes, current versions, key hashes and expiry, engine replicas per model, the request-log switch.- Gateways keep the last good snapshot on disk and serve from it. A
mainoutage or upgrade pauses the UI and new jobs, not inference. - Rolling back a deployment is republishing an earlier snapshot.
- Endpoints (
/v1,/internal/gateway):web/openapi.json.
Request path
Section titled “Request path”An inference request never touches the database.
| Step | Source |
|---|---|
| Key, scope, expiry | In-memory config snapshot (key hashes) |
| Route, version, adapter | Same snapshot |
| Engine replicas for the model | Same snapshot, refreshed from engine announcements |
| Replica choice | Per-gateway memory: in-flight counts, sticky map, health from 5 s checks |
| Request log, key last-used | Buffered, written in batches every ~2 s |
- Config is up to ~1 s stale: a revoked key stays valid for at most a second.
- Scale out: more gateway replicas behind HAProxy, Envoy or a cloud load balancer, then more engines. GPUs are the bottleneck long before gateways or the database.
- Very high log volume moves to ClickHouse or Parquet in object storage, or to sampling.
Streaming
Section titled “Streaming”stream: true sends chat.completion.chunk events as the engine generates them: transformers through a token
streamer, MLX through stream_generate, llama.cpp through its own stream, SGLang and vLLM by relaying their events,
and remote engines as newline-delimited JSON from /generate. The gateway splits each piece into content,
reasoning_content and tool_calls deltas as it goes (a tool call is sent whole once its closing tag arrives).
- An answer held to
response_formator to a custom model’s contract is generated in full, checked, then sent as events, so a 422 still arrives as a status code. stream_options: {"include_usage": true}adds OpenAI’s final chunk with emptychoicesandusage; otherwise the usage rides on the chunk withfinish_reason.- A failure before the first token is an ordinary error response; one after it ends the stream with an
errorevent anddata: [DONE]. The request log is written when the stream ends, with its tokens and status. - A gateway fails over to another replica only before the first byte of a stream.
Sampling
Section titled “Sampling”/v1/chat/completions takes OpenAI’s sampling fields and the engine extensions vLLM and SGLang share. A field left
out is not sent, so the model’s default applies: its generation_config.json on transformers and MLX (greedy when
it has none), the engine’s own default on SGLang, vLLM and llama.cpp. A custom model answers at temperature 0 unless
the request sets one, the way its scores were measured.
| Field | Range | SGLang, vLLM | llama.cpp | MLX | transformers |
|---|---|---|---|---|---|
max_tokens, max_completion_tokens |
1 or more; unset: the context limit | Passed on | Passed on | Passed on | Passed on |
temperature |
0 to 2 | Passed on | Passed on | make_sampler |
GenerationConfig |
top_p |
above 0, up to 1 | Passed on | Passed on | make_sampler |
GenerationConfig |
top_k |
1 or more | Passed on | Passed on | make_sampler |
GenerationConfig |
min_p |
0 to 1 | Passed on | Passed on | make_sampler |
GenerationConfig |
presence_penalty, frequency_penalty |
-2 to 2 | Passed on | Passed on | Logits processor | Logits processor |
repetition_penalty |
above 0, up to 2 | Passed on | repeat_penalty |
make_logits_processors |
GenerationConfig |
seed |
any integer | Passed on | Passed on | mx.random.seed |
torch.manual_seed |
stop |
up to 4 texts | Passed on, after the model’s stop tokens | Passed on | Checked as it decodes | stop_strings |
- A value out of range,
nabove 1,logprobsorlogit_biasis a 400 with an OpenAIerrorobject naming the field inparam. So is any body that fails validation on/v1/chat/completions. - An engine that cannot honour a field (an older
mlx-lmwithouttop_kormin_p, or a server that refuses one) answers 400 withcode: unsupported_parameterand the field inparam, never silently ignoring it. - Without
max_tokensthe answer runs to the end of the model’s context (max_position_embeddings, or 4,096 tokens for llama.cpp’s window).
Text completions
Section titled “Text completions”/v1/completions is OpenAI’s legacy text completion API, for code completion and base models: prompt goes to the
engine as it is, without the chat template, and choices[].text is what follows it. It shares the chat path’s keys,
name@N addressing, sampling fields, safety model, request log (kind completion, shown as the Endpoint on
Monitoring › Requests) and streaming (text_completion chunks ending with data: [DONE]).
- Each engine takes its raw-prompt path:
transformerstokenizes the text with the tokenizer’s BOS token in front (when it has one and the prompt lacks it), MLX encodes it with special tokens, llama.cpp passes it tocreate_completion, and SGLang and vLLM get it on their own/v1/completions. Remote engines carry it asprompton/generate. The answer is read as plain text: no reasoning split, no tool calls. - A custom model applies its live (or pinned) version’s adapter but not its instructions, few-shot turns or contract; it answers at temperature 0 unless the request sets one, as in chat. A prompt-only version is its base model.
max_tokensdefaults to 16, as OpenAI’s does.promptmay be a list holding one text; a longer list or token ids are a 400. So arenorbest_ofabove 1,logprobs,echo,suffixandlogit_bias;useris accepted and ignored.- A blocked call answers with empty
text,finish_reason: content_filterand thesafetyverdict.
Structured output
Section titled “Structured output”/v1/chat/completions takes OpenAI’s response_format: {"type": "text"}, {"type": "json_object"} or
{"type": "json_schema", "json_schema": {"name", "schema", "strict"}}.
| Request | SGLang | llama.cpp | MLX, transformers |
|---|---|---|---|
Classify model, no response_format or text |
Held to the labels (EBNF grammar) | Held to the labels (GBNF grammar) | Held to the labels (token trie) |
json_object |
Constrained (schema {"type": "object"}) |
Constrained (grammar from the schema) | Free, then checked |
json_schema |
Constrained (response_format passed on) |
Constrained (grammar from the schema) | Constrained when the schema has a finite set of answers (enums, booleans, objects of them); otherwise free, then checked |
- A custom model’s contract is the default and stays in force: a request’s
response_formatcan only narrow a model whose contract is free text. A model that answers with labels or its own JSON schema refusesjson_objectandjson_schemawith a 400 (param: response_format,code: unsupported_value);{"type": "text"}is accepted and changes nothing. Base models take any format. - A Classify answer stays the bare label, as before. If an engine refuses the label grammar, the gateway answers without it and logs a warning.
- Every JSON answer is checked against the schema in the gateway, whatever the engine. One that does not parse or
match is a 422 with
error.codeinvalid_jsonorschema_mismatch; an answer cut off bymax_tokens(finish_reason: length) comes back as it is. A JSON answer inside a Markdown code fence is unwrapped. - A malformed
response_formator schema is a 400 with an OpenAIerrorobject (message,type,param,code). A schema the engine cannot compile is a 400 too. - Streaming works: the answer is checked, then streamed.
- The check covers
type,enum,const,properties,required,additionalProperties,items,prefixItems, item and length bounds, number bounds,anyOf,oneOf,allOf,notand local$ref. It does not runpattern(a caller’s regular expression could stall the gateway; llama.cpp applies it while decoding) or checkformat,multipleOfandpatternProperties. strictis accepted and changes nothing: every answer is constrained, checked, or both.
Tool calling
Section titled “Tool calling”/v1/chat/completions takes OpenAI’s tools (functions with JSON-schema parameters), tool_choice (auto,
none, required or a named function) and parallel_tool_calls. Answers carry tool_calls with
finish_reason: tool_calls, and follow-up turns send assistant messages with their tool_calls and role: tool
messages with each tool_call_id. Streams carry the calls as delta.tool_calls.
| Format | Read from | Families | SGLang parser | vLLM parser |
|---|---|---|---|---|
hermes |
<tool_call> JSON </tool_call> |
Qwen 2.5 and 3, Hermes, SmolLM3 | qwen25 |
hermes |
llama3 |
{"name", "parameters"} JSON, after `< |
python_tag | >or alone;;` between calls |
Llama 3.1, 3.2, 3.3 |
mistral |
[TOOL_CALLS] and a JSON list |
Mistral 7B v0.3, Nemo, Small | mistral |
mistral |
harmony |
the commentary to=functions.<name> channel |
gpt-oss | gpt-oss |
openai |
- The format comes from the base model’s own chat template (
chat_template.jinja,tokenizer_config.json, or the GGUF metadata), read once per weights hash, so imported and Hugging Face models get it too. A model whose template has none of these refusestoolswith a 400. - The template renders the tools:
transformers, MLX and llama.cpp passtoolsto the tokenizer’s chat template (llama.cpp through the same Jinja renderer astransformers) and read the calls back from the raw answer; SGLang and vLLM start with the matching--tool-call-parserand gettoolsas they are. requiredand a named function prefill the answer with the format’s call opening (<tool_call>plus the name for Qwen), so the model can only complete a call.nonerenders no tools.parallel_tool_calls: falsekeeps the first call.- A model answering with labels or its own JSON schema takes no tools; a free-text custom model does, with its
instructions first. Arguments the model writes as invalid JSON stay in
content. functionsandfunction_call, OpenAI’s old names, are refused with a pointer totools; content parts other than text are refused.
Reasoning
Section titled “Reasoning”reasoning_effort takes none, low, medium or high; unset keeps the model’s default. The reasoning comes back
as message.reasoning_content (delta.reasoning_content when streaming), never in content, so request logs,
feedback and graders see only the answer.
| Style | Detected from the template | Families | none |
low, medium, high |
Read from |
|---|---|---|---|---|---|
| Toggle | enable_thinking |
Qwen3 (not the 2507 Instruct models), GLM-4.5, SmolLM3 | Thinking off | Thinking on | <think> … </think> |
| Effort | reasoning_effort |
gpt-oss | 400 | The effort | the harmony analysis channel |
| Always | <think> opened by the generation prompt |
DeepSeek-R1 and its distills, Qwen3 Thinking 2507 | 400 | 400 | up to </think> |
| None | none of these | everything else | Accepted, changes nothing | 400 |
- The default stays off for custom models and the catalogue’s Qwen3 bases (
enable_thinking: falsein their template settings); a base imported without that setting follows its own template’s default. - A model answering with labels or its own schema does not reason:
noneis accepted, an effort is a 400. transformers, MLX and llama.cpp pass the setting to the chat template and split the answer; SGLang and vLLM getchat_template_kwargs(andreasoning_effortfor gpt-oss) and start with--reasoning-parser(qwen3,deepseek-r1orgpt-osson SGLang;qwen3,deepseek_r1oropenai_gptosson vLLM).- Evaluation and judge jobs strip the reasoning too, so a reasoning base is scored on its answers.
Safety model
Section titled “Safety model”A guard model installed like any base checks each request before the model answers it, and the answer too when
set. A blocked call answers 200 with OpenAI’s refusal shape: message.content: null, message.refusal naming the
safety model and the categories it found, finish_reason: content_filter, and a safety object (model, stage
prompt or answer, blocked, categories). Streams send the refusal as delta.refusal.
| Guard | Detected from | Prompt | Categories |
|---|---|---|---|
| Llama Guard 3 (1B, 8B) | llama-guard in the repo or name |
its chat template, roles merged to alternate | S1 to S14, named |
| ShieldGemma (2B, 9B) | shieldgemma |
its template with each of four policies as guideline |
the policy that answered Yes |
| Qwen3Guard Gen (0.6B, 4B, 8B) | qwen3guard |
its template (request or answer moderation) | its own; Controversial passes |
| gpt-oss-safeguard (20B, 120B) | gpt-oss-safeguard |
a written policy as the system message, low effort | the category in its JSON answer |
- Which guard applies: a model’s own choice on its Deployments page (a guard, or none), else the install’s under
Settings, General (none by default); base models served through
/v1follow the install’s. Both can also check answers. A request’ssafetyfield ({"model": ..., "answers": true}, an extension) can pick a guard for a model that has none and add the answer check, but cannot change or turn off the one set (400safety.model). - The guard runs greedily through the same engines as any base, and its engine starts with the configuration when it is set. If it cannot answer, the call fails with a 503 rather than going unchecked. Checking answers means the answer is generated in full before it streams.
- Blocked
/v1calls keep their request log, withfinish_reason: content_filterand the safety verdict (Monitor, Requests shows “Blocked by”), and add asafety.blockaudit event with the API key as actor, the stage, categories and response ID, never the content. Playground calls are audited with the person as actor. Changes to the settings are audited assettings.safetyandmodel.safety.
Engines
Section titled “Engines”One engine process serves one base model with many adapters. It picks its engine when it starts, from its own hardware and the model’s weight format:
| Node | Weights | Engine |
|---|---|---|
| Linux, NVIDIA or AMD (ROCm), SGLang installed | safetensors | SGLang (--enable-lora, adapters loaded at runtime) |
| Linux, NVIDIA or AMD (ROCm), vLLM installed, no SGLang | safetensors | vLLM (--enable-lora, runtime LoRA updates on) |
Apple silicon, minibrain[mlx] |
safetensors | MLX (mlx-lm; PEFT LoRA converted on load, swapped per request) |
| Anything else | safetensors | transformers + PEFT |
Any, minibrain[llamacpp] |
GGUF | llama.cpp (Metal, CUDA or CPU); base models only |
--engineoverrides the choice. An automatic choice that fails to load falls back totransformers.- Engines of different kinds serve at the same time: a Mac engine on MLX and a rack engine on SGLang can be replicas of one model.
- Remote engines:
minibrain engine --model KEY --main-url URL --pool NAME --host 0.0.0.0, with the pool’s join token inMINIBRAIN_JOIN_TOKENor on a machine already approved (joining), pulls adapters frommainand base weights from wherever Storage keeps them, never from Hugging Face, and announces its URL, engine and hardware every 10 s. With a bucket,mainhands out a presigned link and the data key per file and the engine reads the bucket directly; with the folder, it reads the files frommain. Either way it keeps a working copy in<home>/engines/<key>/weights, checks every file’s size and SHA-256, and removes copies of modelsmainno longer has when it starts and every 10 minutes. - Local engines on
mainusemain’s working copy,<home>/models/<id>, fetched from the bucket the first time they start when the model lives there. - Fetching the working copy is its own step, before the engine loads the model: the engine answers
/healthwith 503 and{"state": "fetching", "done", "total", "message"}, announces statefetching(Infrastructure and the Models list say “Fetching weights”) and logs its progress every 30 s. Each file gets a freshly signed link, so a fetch longer than the links’ 15 minutes still completes. A request to a model that is still fetching waits up to 60 s, then gets a 503 that says how far the fetch is; the 600 s startup window starts only when loading starts. A fetch that fails says so and loads nothing. mainstarts a local engine for a model deployed on this machine’s pool (local) only when no live replica in that pool serves it, and starts one again when they go away. A model with versions but no deployment, or a safety model, starts one when no live replica serves it anywhere. A model deployed on another pool never starts onmain.- Scale is horizontal: more replicas of the same model, on Kubernetes autoscaled. Heavy engines are optional extras;
SGLang and vLLM pin their own torch and CUDA, so each runs from its own environment (
MINIBRAIN_SGLANG_PYTHON,MINIBRAIN_VLLM_PYTHON). The vLLM engine shares SGLang’s request path (an OpenAI-compatible server per base model) and has not been run on a GPU yet.
Machine heartbeats
Section titled “Machine heartbeats”Every call a worker makes to claim work or ask for engines (about every 10 to 25 s), and every engine announcement
(every 10 s), carries the machine’s static hardware and, since this version, a metrics sample:
| Field | Source | Unknown |
|---|---|---|
gpus[]: index, memory_used_bytes, memory_total_bytes, utilization (percent) |
nvidia-smi, else rocm-smi, when on the PATH |
No GPU tool: an empty list; a field the tool prints as [N/A]: null |
ram_used_bytes, ram_total_bytes, cpu_percent |
psutil | null; cpu_percent is null on the first sample |
disks[]: path, mount, total_bytes, free_bytes, minibrain_bytes |
The disk under MINIBRAIN_HOME; minibrain’s share is recounted every 5 minutes |
No disk found: an empty list |
- Samples are taken at most every 15 s per process. Apple MPS and CPU machines report no GPU entry; their measured memory is host RAM. Nothing is guessed: a value the machine cannot read stays null.
mainkeeps the last hour per machine in memory, at most one sample every 10 s, and samples itself the same way. A restart ofmainstarts the history again.GET /api/compute/poolsreturns each machine’s latest sample while it is under 2 minutes old (metrics, withmemory_used_bytesandmemory_total_bytessummed where models run) and its last reported disks even when it is offline (disks);GET /api/compute/pools/{pool}/machines/{host}/metricsreturns the latest sample and the hour of history.- Each pool has
room: how many more models of one size still fit (the largest size the pool already serves that fits, else the largest catalogue size that fits) and the largest catalogue model the planner can train, from the measured free memory of its online active machines, or estimates from weights, adapters and training plans where a machine does not measure. SGLang and vLLM reserve most of a GPU up front, so measured free memory is small there.
Drain and remove
Section titled “Drain and remove”| Action | Effect |
|---|---|
Drain (POST …/machines/{host}/drain) |
The machine’s claims return no job; jobs it runs finish. It drops out of engine placement; each engine it runs keeps serving until a replica on another machine of the pool is ready, then stops. |
| Undrain | It takes jobs and engines again. |
Remove (DELETE …/machines/{host}) |
Refused (409) while it runs a job or a live engine, and while it is online and not drained. It disappears from the pool; its next claim or engine call gets 410 and the worker exits. minibrain worker started again on it registers and rejoins. |
All three are admin only and audited (machine.drain, machine.undrain, machine.remove). Main cannot be drained
or removed.
Deployments on a pool
Section titled “Deployments on a pool”A deployment is a model’s live version on one compute pool with N replicas (1 by default), kept true continuously.
Deploy and Promote take pool: a pool name, or auto (the default for a model not deployed yet) for the pool where
it fits, this machine first when it fits, then the pool with the most memory left after the deployments already on
it, among the pools used for serving (Pool use). A pool that does not exist, is not used for serving or
where the model does not fit, is a 422 with the reason; a deployment already on a pool that stopped serving may stay
there. Deploying a deployed model
with another pool moves it; left out, it stays where it is. GET /api/models/{name}/placement lists each pool’s fit
for the model and the pool auto would pick; the Deploy dialog’s Pool row reads it (“Automatic · gpu-a10”, then each
pool with “Runs · 2.2 of 24 GB”, “Too small” or, disabled, “Not used for serving”). The model.deploy audit event
records the pool, the replicas and,
on a move, from_pool.
| Pool | Who starts the engine |
|---|---|
local (this machine) |
main, as a supervised child, as before |
| Any other | A worker in that pool (minibrain worker --pool NAME) runs minibrain engine as its child |
-
Each worker in a pool other than
localasksmainevery 10 s which engines to run (POST /internal/worker/engines, with its pool token or approval) and starts or stops children to match. The answer carries an engine token for that pool (engine.<t|m><id>.<pool>.<HMAC-SHA256 of both>,tfor a pool token andmfor an approval, keyed by<home>/secrets/engine_tokenon main), handed to the engines asMINIBRAIN_ENGINE_TOKEN, so a worker needs only its own credential and never sees the gateway token. The engine token lives as long as that credential: once the pool token is rotated or the approved machine removed, its engines get 401 (security). An engine token reaches only what an engine needs: announcing engines in its own pool, base weights and the adapters of versions on a base model deployed or running in its pool. The gateway config (routes and API key digests), request logs, feedback, gateway reports and other adapters answer it with 403, an announcement for another pool too, and a worker inlocalgets no engine token at all. -
mainplaces replicas on the pool’s machines with a worker seen in the last 30 s, one worker per machine, skipping machines too small for the model, by rendezvous hashing on the model and machine name: a model stays on the same machines as long as they stay up, and moves when one goes away. -
Replicas are bounded by the pool: one per online machine where the model fits (
roomon each pool inGET /api/models/{name}/placement;localhas room for one). Asking for more is a 422 (“Pool gpu-a10 has room for 2 replicas of Support triage: one per online machine where it fits”). Deploy and Promote takereplicas; the Deploy dialogs have a Replicas row under Pool, and Settings › Deployment changes them for a live deployment. A move to a pool with less room keeps as many as fit. The header shows ”· 2 replicas” when there are several, and Overview “1 of 2 replicas” while they start. -
The Models list’s Deploy (the ”…” menu on a variant) opens the same Deploy dialog; Deploy on several selected variants opens a compact one with one Pool choice (Automatic, or a pool with how many of them fit) and each variant’s pool and fit, and deploys only those that fit.
-
The engine binds and announces the worker’s
--advertiseaddress (the machine’s name by default;--engine-bindsets another interface) on a free port. It pulls weights and adapters frommainlike any remote engine and dies with the worker.--no-servekeeps a worker to jobs only. -
A name that resolves only to loopback (common on Linux, where
/etc/hostsmaps the hostname to127.0.1.1) would announce an address no gateway can reach. Without--advertise, the worker then takes the address its own traffic tomainleaves from, which is the one interface that is known to reachmain, and logs it. When that is loopback too, ormainis unreachable, it refuses to start and says how to fix it:minibrain worker: This machine's name gpu-01 resolves to 127.0.1.1, which gateways on other machines cannotreach, and no other address of this machine reaches main at http://main.corp:8000. Pass the address or name othermachines reach it at: minibrain worker --pool rack --advertise <address> (add --engine-bind 0.0.0.0 to listen onevery interface), or --no-serve to run jobs only.With
mainon the same machine (--api-url http://127.0.0.1:8000) a loopback name is kept:main’s gateway reaches it, gateways on other machines do not. -
Gateways route a deployed name only to that deployment’s pool (
poolon the route in the config snapshot), ready replicas first; a pinned version (name@N) and the Playground’s versions use any replica. -
Moving a deployment to another pool is a rolling handover. The old pool keeps its engines and its traffic (the route’s
poolstays the old one) while the new pool’s engines start (placementslists both pools). Once the new pool has a ready engine on every machine it was given (one onlocal), main switches the route to it, and the old engines stop within about 10 s. An engine in the new pool that reportsfailed, or no machine of the new pool online for 5 minutes, rolls the move back: the deployment stays on the old pool and says why (“Moving to gpu-a10 failed: its engine in pool gpu-a10 failed to start. It stays on local.”) until it is deployed again. Deploying it back to the old pool before the move ends cancels it; deploying to a third pool retargets it, and the first pool keeps serving. Main checks moves every time it rebuilds the gateway config (every second); each move that ends writes amodel.moveaudit event (outcomemovedorrolled_back, withfrom_pool,pooland the error). -
Status: the header, Overview, Models list and Monitoring say where it runs (“Deployed · v2 on gpu-a10”), and during a move where it goes (“Deployed · v2 on local · moving to gpu-a10”). While the engine fetches and loads it shows “Deploying” or “Fetching weights”; with no worker of the pool online it shows “Waiting for a machine in gpu-a10”.
Routing
Section titled “Routing”Each gateway routes per model across its replicas, with no new setting:
| Rule | Detail |
|---|---|
| Health | Ready replicas first, then loading; a failed request or health check marks a replica down for 10 s |
| Least busy | New work goes to the replica with the fewest requests in flight |
| Sticky | By adapter (so the adapter stays loaded), by prompt prefix for base models (so the KV cache is reused) |
| Overflow | A sticky replica with 4 requests in flight hands new requests to a less busy one; the binding stays |
| Failover | Another replica is tried at once; the key rebinds to the replica that answered |
In front of gateways: HAProxy, Envoy or a cloud load balancer. Under the gateway, per pool, later: sgl-router or llm-d. llm-d needs Kubernetes (Gateway API inference extension, its own scheduler), and minibrain must run without it. sgl-router only routes SGLang and pays off with many SGLang replicas of one model; then it slots under the gateway as one replica per pool, and the gateway keeps keys, routes and adapters.
Compute pools
Section titled “Compute pools”A pool is a named group of machines. Workers and engines join one with --pool NAME (default local, shown as “This
machine”) and report their hardware when they register, each time a worker asks for work and each time an engine
announces itself.
| Reported | Example |
|---|---|
| Platform, architecture | linux, x86_64 |
| Accelerator | cuda, rocm, xpu, mps, cpu, with the GPU name |
| GPUs | Each with its memory |
| Memory for models | Sum of GPU memory; Apple GPU-usable unified memory; RAM on CPU |
| CPU, RAM | Cores, bytes |
| Engines installed | sglang, vllm, mlx, transformers, llamacpp |
- Infrastructure › Machines lists pools, their machines (online if seen in the last 2 minutes) and the engines serving: one row per machine with its GPUs, memory, CPU, RAM, models and jobs, sortable by each, grouped under a row per pool (machines online, GPUs, memory, engines, jobs, room, join token, use). Group by Model lists each deployed model with the machines that serve it.
- Discover judges hardware fit against the selected pool: the best machine in the pool decides. Multi-GPU (XL) models need a machine with two or more GPUs.
- Machines unseen for 7 days and engines unseen for an hour are forgotten. An approved machine stays listed, offline, until it is removed, which revokes its approval.
Pool use
Section titled “Pool use”Each pool is used for Any work (the default), Serving or Training. Admins set it in the pool row’s menu
under Machines (GET/PUT /api/compute/pools/{pool}/settings, {"workloads": ["serving"]}); GET /api/compute/pools returns workloads on each pool. A pool keeps the default until it is changed, so one machine
works out of the box.
| Use | Deployments | Training and scoring jobs (train, eval, judge) |
|---|---|---|
| Any | Placed here | Run here |
| Serving | Placed here | Never start here |
| Training | Never placed here | Run here |
- Stored as a list of workloads per pool (
pool_settings), so a new kind of work is one more value, not a schema change. Any with no description is no row at all, so it also takes kinds added later. - The same row holds an optional one-line description (up to 120 characters, whitespace collapsed; empty clears it),
returned as
descriptionon pools, deploy placements, the catalogue’s pools and the training plan. ThePUT .../settingsbody takesdescription,workloadsor both; a field left out stays as it is. autoplacement skips pools not used for serving; asking for one is a 422 with the reason. Engines already running there keep serving until their model is undeployed or moved.- Workers in a pool not used for training claim no training or scoring job; a job asked for such a pool is refused with the reason (422). Jobs running when the use changes finish. A pool with queued training or scoring jobs held to it cannot stop taking training: the change is a 409 naming those jobs, so cancel them or let them start first.
- Downloads, imports, checks, repairs and exports run on any pool.
- The Train page’s Pool list and the Deploy dialogs show the other pools disabled, with why.
- Each change writes a
pool.workloadsorpool.descriptionaudit event withfromandto.
Inference request
Section titled “Inference request”Deploy a version
Section titled “Deploy a version”A machine joins
Section titled “A machine joins”Add machine (Infrastructure, admins) asks for a pool, then shows the command to run on the machine:
MINIBRAIN_JOIN_TOKEN=mbj_… minibrain worker --main URL --pool NAME. A pool has one join token; the dialog makes it
the first time, and the pool’s row under Machines shows its prefix, copies the command and rotates it. Without a token
the same command asks for approval:
- A request lasts an hour; Deny ends it and the worker stops with the reason. The approval holds across restarts
(
<home>/worker/approval.json) until the machine is removed.--mainis the address of main (--api-urlalso works). - On an approved machine
--poolis ignored: the pool is the one the admin picked.