Skip to content

Serving

Control plane Serving plane versioned config snapshot versioned config snapshot batched request logs batched request logs announce every 10 s announce every 10 s main database HAProxy / Envoy / cloud LB gateway gateway engine Aadapters: x, y engine Badapters: y, z
  • main publishes a versioned config snapshot: routes, current versions, key hashes and expiry, engine replicas per model, the request-log switch.
  • Gateways keep the last good snapshot on disk and serve from it. A main outage or upgrade pauses the UI and new jobs, not inference.
  • Rolling back a deployment is republishing an earlier snapshot.
  • Endpoints (/v1, /internal/gateway): web/openapi.json.

An inference request never touches the database.

Step Source
Key, scope, expiry In-memory config snapshot (key hashes)
Route, version, adapter Same snapshot
Engine replicas for the model Same snapshot, refreshed from engine announcements
Replica choice Per-gateway memory: in-flight counts, sticky map, health from 5 s checks
Request log, key last-used Buffered, written in batches every ~2 s
  • Config is up to ~1 s stale: a revoked key stays valid for at most a second.
  • Scale out: more gateway replicas behind HAProxy, Envoy or a cloud load balancer, then more engines. GPUs are the bottleneck long before gateways or the database.
  • Very high log volume moves to ClickHouse or Parquet in object storage, or to sampling.

stream: true sends chat.completion.chunk events as the engine generates them: transformers through a token streamer, MLX through stream_generate, llama.cpp through its own stream, SGLang and vLLM by relaying their events, and remote engines as newline-delimited JSON from /generate. The gateway splits each piece into content, reasoning_content and tool_calls deltas as it goes (a tool call is sent whole once its closing tag arrives).

  • An answer held to response_format or to a custom model’s contract is generated in full, checked, then sent as events, so a 422 still arrives as a status code.
  • stream_options: {"include_usage": true} adds OpenAI’s final chunk with empty choices and usage; otherwise the usage rides on the chunk with finish_reason.
  • A failure before the first token is an ordinary error response; one after it ends the stream with an error event and data: [DONE]. The request log is written when the stream ends, with its tokens and status.
  • A gateway fails over to another replica only before the first byte of a stream.

/v1/chat/completions takes OpenAI’s sampling fields and the engine extensions vLLM and SGLang share. A field left out is not sent, so the model’s default applies: its generation_config.json on transformers and MLX (greedy when it has none), the engine’s own default on SGLang, vLLM and llama.cpp. A custom model answers at temperature 0 unless the request sets one, the way its scores were measured.

Field Range SGLang, vLLM llama.cpp MLX transformers
max_tokens, max_completion_tokens 1 or more; unset: the context limit Passed on Passed on Passed on Passed on
temperature 0 to 2 Passed on Passed on make_sampler GenerationConfig
top_p above 0, up to 1 Passed on Passed on make_sampler GenerationConfig
top_k 1 or more Passed on Passed on make_sampler GenerationConfig
min_p 0 to 1 Passed on Passed on make_sampler GenerationConfig
presence_penalty, frequency_penalty -2 to 2 Passed on Passed on Logits processor Logits processor
repetition_penalty above 0, up to 2 Passed on repeat_penalty make_logits_processors GenerationConfig
seed any integer Passed on Passed on mx.random.seed torch.manual_seed
stop up to 4 texts Passed on, after the model’s stop tokens Passed on Checked as it decodes stop_strings
  • A value out of range, n above 1, logprobs or logit_bias is a 400 with an OpenAI error object naming the field in param. So is any body that fails validation on /v1/chat/completions.
  • An engine that cannot honour a field (an older mlx-lm without top_k or min_p, or a server that refuses one) answers 400 with code: unsupported_parameter and the field in param, never silently ignoring it.
  • Without max_tokens the answer runs to the end of the model’s context (max_position_embeddings, or 4,096 tokens for llama.cpp’s window).

/v1/completions is OpenAI’s legacy text completion API, for code completion and base models: prompt goes to the engine as it is, without the chat template, and choices[].text is what follows it. It shares the chat path’s keys, name@N addressing, sampling fields, safety model, request log (kind completion, shown as the Endpoint on Monitoring › Requests) and streaming (text_completion chunks ending with data: [DONE]).

  • Each engine takes its raw-prompt path: transformers tokenizes the text with the tokenizer’s BOS token in front (when it has one and the prompt lacks it), MLX encodes it with special tokens, llama.cpp passes it to create_completion, and SGLang and vLLM get it on their own /v1/completions. Remote engines carry it as prompt on /generate. The answer is read as plain text: no reasoning split, no tool calls.
  • A custom model applies its live (or pinned) version’s adapter but not its instructions, few-shot turns or contract; it answers at temperature 0 unless the request sets one, as in chat. A prompt-only version is its base model.
  • max_tokens defaults to 16, as OpenAI’s does. prompt may be a list holding one text; a longer list or token ids are a 400. So are n or best_of above 1, logprobs, echo, suffix and logit_bias; user is accepted and ignored.
  • A blocked call answers with empty text, finish_reason: content_filter and the safety verdict.

/v1/chat/completions takes OpenAI’s response_format: {"type": "text"}, {"type": "json_object"} or {"type": "json_schema", "json_schema": {"name", "schema", "strict"}}.

Request SGLang llama.cpp MLX, transformers
Classify model, no response_format or text Held to the labels (EBNF grammar) Held to the labels (GBNF grammar) Held to the labels (token trie)
json_object Constrained (schema {"type": "object"}) Constrained (grammar from the schema) Free, then checked
json_schema Constrained (response_format passed on) Constrained (grammar from the schema) Constrained when the schema has a finite set of answers (enums, booleans, objects of them); otherwise free, then checked
  • A custom model’s contract is the default and stays in force: a request’s response_format can only narrow a model whose contract is free text. A model that answers with labels or its own JSON schema refuses json_object and json_schema with a 400 (param: response_format, code: unsupported_value); {"type": "text"} is accepted and changes nothing. Base models take any format.
  • A Classify answer stays the bare label, as before. If an engine refuses the label grammar, the gateway answers without it and logs a warning.
  • Every JSON answer is checked against the schema in the gateway, whatever the engine. One that does not parse or match is a 422 with error.code invalid_json or schema_mismatch; an answer cut off by max_tokens (finish_reason: length) comes back as it is. A JSON answer inside a Markdown code fence is unwrapped.
  • A malformed response_format or schema is a 400 with an OpenAI error object (message, type, param, code). A schema the engine cannot compile is a 400 too.
  • Streaming works: the answer is checked, then streamed.
  • The check covers type, enum, const, properties, required, additionalProperties, items, prefixItems, item and length bounds, number bounds, anyOf, oneOf, allOf, not and local $ref. It does not run pattern (a caller’s regular expression could stall the gateway; llama.cpp applies it while decoding) or check format, multipleOf and patternProperties.
  • strict is accepted and changes nothing: every answer is constrained, checked, or both.

/v1/chat/completions takes OpenAI’s tools (functions with JSON-schema parameters), tool_choice (auto, none, required or a named function) and parallel_tool_calls. Answers carry tool_calls with finish_reason: tool_calls, and follow-up turns send assistant messages with their tool_calls and role: tool messages with each tool_call_id. Streams carry the calls as delta.tool_calls.

Format Read from Families SGLang parser vLLM parser
hermes <tool_call> JSON </tool_call> Qwen 2.5 and 3, Hermes, SmolLM3 qwen25 hermes
llama3 {"name", "parameters"} JSON, after `< python_tag >or alone;;` between calls Llama 3.1, 3.2, 3.3
mistral [TOOL_CALLS] and a JSON list Mistral 7B v0.3, Nemo, Small mistral mistral
harmony the commentary to=functions.<name> channel gpt-oss gpt-oss openai
  • The format comes from the base model’s own chat template (chat_template.jinja, tokenizer_config.json, or the GGUF metadata), read once per weights hash, so imported and Hugging Face models get it too. A model whose template has none of these refuses tools with a 400.
  • The template renders the tools: transformers, MLX and llama.cpp pass tools to the tokenizer’s chat template (llama.cpp through the same Jinja renderer as transformers) and read the calls back from the raw answer; SGLang and vLLM start with the matching --tool-call-parser and get tools as they are.
  • required and a named function prefill the answer with the format’s call opening (<tool_call> plus the name for Qwen), so the model can only complete a call. none renders no tools. parallel_tool_calls: false keeps the first call.
  • A model answering with labels or its own JSON schema takes no tools; a free-text custom model does, with its instructions first. Arguments the model writes as invalid JSON stay in content.
  • functions and function_call, OpenAI’s old names, are refused with a pointer to tools; content parts other than text are refused.

reasoning_effort takes none, low, medium or high; unset keeps the model’s default. The reasoning comes back as message.reasoning_content (delta.reasoning_content when streaming), never in content, so request logs, feedback and graders see only the answer.

Style Detected from the template Families none low, medium, high Read from
Toggle enable_thinking Qwen3 (not the 2507 Instruct models), GLM-4.5, SmolLM3 Thinking off Thinking on <think> … </think>
Effort reasoning_effort gpt-oss 400 The effort the harmony analysis channel
Always <think> opened by the generation prompt DeepSeek-R1 and its distills, Qwen3 Thinking 2507 400 400 up to </think>
None none of these everything else Accepted, changes nothing 400
  • The default stays off for custom models and the catalogue’s Qwen3 bases (enable_thinking: false in their template settings); a base imported without that setting follows its own template’s default.
  • A model answering with labels or its own schema does not reason: none is accepted, an effort is a 400.
  • transformers, MLX and llama.cpp pass the setting to the chat template and split the answer; SGLang and vLLM get chat_template_kwargs (and reasoning_effort for gpt-oss) and start with --reasoning-parser (qwen3, deepseek-r1 or gpt-oss on SGLang; qwen3, deepseek_r1 or openai_gptoss on vLLM).
  • Evaluation and judge jobs strip the reasoning too, so a reasoning base is scored on its answers.

A guard model installed like any base checks each request before the model answers it, and the answer too when set. A blocked call answers 200 with OpenAI’s refusal shape: message.content: null, message.refusal naming the safety model and the categories it found, finish_reason: content_filter, and a safety object (model, stage prompt or answer, blocked, categories). Streams send the refusal as delta.refusal.

Guard Detected from Prompt Categories
Llama Guard 3 (1B, 8B) llama-guard in the repo or name its chat template, roles merged to alternate S1 to S14, named
ShieldGemma (2B, 9B) shieldgemma its template with each of four policies as guideline the policy that answered Yes
Qwen3Guard Gen (0.6B, 4B, 8B) qwen3guard its template (request or answer moderation) its own; Controversial passes
gpt-oss-safeguard (20B, 120B) gpt-oss-safeguard a written policy as the system message, low effort the category in its JSON answer
  • Which guard applies: a model’s own choice on its Deployments page (a guard, or none), else the install’s under Settings, General (none by default); base models served through /v1 follow the install’s. Both can also check answers. A request’s safety field ({"model": ..., "answers": true}, an extension) can pick a guard for a model that has none and add the answer check, but cannot change or turn off the one set (400 safety.model).
  • The guard runs greedily through the same engines as any base, and its engine starts with the configuration when it is set. If it cannot answer, the call fails with a 503 rather than going unchecked. Checking answers means the answer is generated in full before it streams.
  • Blocked /v1 calls keep their request log, with finish_reason: content_filter and the safety verdict (Monitor, Requests shows “Blocked by”), and add a safety.block audit event with the API key as actor, the stage, categories and response ID, never the content. Playground calls are audited with the person as actor. Changes to the settings are audited as settings.safety and model.safety.

One engine process serves one base model with many adapters. It picks its engine when it starts, from its own hardware and the model’s weight format:

Node Weights Engine
Linux, NVIDIA or AMD (ROCm), SGLang installed safetensors SGLang (--enable-lora, adapters loaded at runtime)
Linux, NVIDIA or AMD (ROCm), vLLM installed, no SGLang safetensors vLLM (--enable-lora, runtime LoRA updates on)
Apple silicon, minibrain[mlx] safetensors MLX (mlx-lm; PEFT LoRA converted on load, swapped per request)
Anything else safetensors transformers + PEFT
Any, minibrain[llamacpp] GGUF llama.cpp (Metal, CUDA or CPU); base models only
  • --engine overrides the choice. An automatic choice that fails to load falls back to transformers.
  • Engines of different kinds serve at the same time: a Mac engine on MLX and a rack engine on SGLang can be replicas of one model.
  • Remote engines: minibrain engine --model KEY --main-url URL --pool NAME --host 0.0.0.0, with the pool’s join token in MINIBRAIN_JOIN_TOKEN or on a machine already approved (joining), pulls adapters from main and base weights from wherever Storage keeps them, never from Hugging Face, and announces its URL, engine and hardware every 10 s. With a bucket, main hands out a presigned link and the data key per file and the engine reads the bucket directly; with the folder, it reads the files from main. Either way it keeps a working copy in <home>/engines/<key>/weights, checks every file’s size and SHA-256, and removes copies of models main no longer has when it starts and every 10 minutes.
  • Local engines on main use main’s working copy, <home>/models/<id>, fetched from the bucket the first time they start when the model lives there.
  • Fetching the working copy is its own step, before the engine loads the model: the engine answers /health with 503 and {"state": "fetching", "done", "total", "message"}, announces state fetching (Infrastructure and the Models list say “Fetching weights”) and logs its progress every 30 s. Each file gets a freshly signed link, so a fetch longer than the links’ 15 minutes still completes. A request to a model that is still fetching waits up to 60 s, then gets a 503 that says how far the fetch is; the 600 s startup window starts only when loading starts. A fetch that fails says so and loads nothing.
  • main starts a local engine for a model deployed on this machine’s pool (local) only when no live replica in that pool serves it, and starts one again when they go away. A model with versions but no deployment, or a safety model, starts one when no live replica serves it anywhere. A model deployed on another pool never starts on main.
  • Scale is horizontal: more replicas of the same model, on Kubernetes autoscaled. Heavy engines are optional extras; SGLang and vLLM pin their own torch and CUDA, so each runs from its own environment (MINIBRAIN_SGLANG_PYTHON, MINIBRAIN_VLLM_PYTHON). The vLLM engine shares SGLang’s request path (an OpenAI-compatible server per base model) and has not been run on a GPU yet.

Every call a worker makes to claim work or ask for engines (about every 10 to 25 s), and every engine announcement (every 10 s), carries the machine’s static hardware and, since this version, a metrics sample:

Field Source Unknown
gpus[]: index, memory_used_bytes, memory_total_bytes, utilization (percent) nvidia-smi, else rocm-smi, when on the PATH No GPU tool: an empty list; a field the tool prints as [N/A]: null
ram_used_bytes, ram_total_bytes, cpu_percent psutil null; cpu_percent is null on the first sample
disks[]: path, mount, total_bytes, free_bytes, minibrain_bytes The disk under MINIBRAIN_HOME; minibrain’s share is recounted every 5 minutes No disk found: an empty list
  • Samples are taken at most every 15 s per process. Apple MPS and CPU machines report no GPU entry; their measured memory is host RAM. Nothing is guessed: a value the machine cannot read stays null.
  • main keeps the last hour per machine in memory, at most one sample every 10 s, and samples itself the same way. A restart of main starts the history again. GET /api/compute/pools returns each machine’s latest sample while it is under 2 minutes old (metrics, with memory_used_bytes and memory_total_bytes summed where models run) and its last reported disks even when it is offline (disks); GET /api/compute/pools/{pool}/machines/{host}/metrics returns the latest sample and the hour of history.
  • Each pool has room: how many more models of one size still fit (the largest size the pool already serves that fits, else the largest catalogue size that fits) and the largest catalogue model the planner can train, from the measured free memory of its online active machines, or estimates from weights, adapters and training plans where a machine does not measure. SGLang and vLLM reserve most of a GPU up front, so measured free memory is small there.
Action Effect
Drain (POST …/machines/{host}/drain) The machine’s claims return no job; jobs it runs finish. It drops out of engine placement; each engine it runs keeps serving until a replica on another machine of the pool is ready, then stops.
Undrain It takes jobs and engines again.
Remove (DELETE …/machines/{host}) Refused (409) while it runs a job or a live engine, and while it is online and not drained. It disappears from the pool; its next claim or engine call gets 410 and the worker exits. minibrain worker started again on it registers and rejoins.

All three are admin only and audited (machine.drain, machine.undrain, machine.remove). Main cannot be drained or removed.

A deployment is a model’s live version on one compute pool with N replicas (1 by default), kept true continuously. Deploy and Promote take pool: a pool name, or auto (the default for a model not deployed yet) for the pool where it fits, this machine first when it fits, then the pool with the most memory left after the deployments already on it, among the pools used for serving (Pool use). A pool that does not exist, is not used for serving or where the model does not fit, is a 422 with the reason; a deployment already on a pool that stopped serving may stay there. Deploying a deployed model with another pool moves it; left out, it stays where it is. GET /api/models/{name}/placement lists each pool’s fit for the model and the pool auto would pick; the Deploy dialog’s Pool row reads it (“Automatic · gpu-a10”, then each pool with “Runs · 2.2 of 24 GB”, “Too small” or, disabled, “Not used for serving”). The model.deploy audit event records the pool, the replicas and, on a move, from_pool.

Pool Who starts the engine
local (this machine) main, as a supervised child, as before
Any other A worker in that pool (minibrain worker --pool NAME) runs minibrain engine as its child
  • Each worker in a pool other than local asks main every 10 s which engines to run (POST /internal/worker/engines, with its pool token or approval) and starts or stops children to match. The answer carries an engine token for that pool (engine.<t|m><id>.<pool>.<HMAC-SHA256 of both>, t for a pool token and m for an approval, keyed by <home>/secrets/engine_token on main), handed to the engines as MINIBRAIN_ENGINE_TOKEN, so a worker needs only its own credential and never sees the gateway token. The engine token lives as long as that credential: once the pool token is rotated or the approved machine removed, its engines get 401 (security). An engine token reaches only what an engine needs: announcing engines in its own pool, base weights and the adapters of versions on a base model deployed or running in its pool. The gateway config (routes and API key digests), request logs, feedback, gateway reports and other adapters answer it with 403, an announcement for another pool too, and a worker in local gets no engine token at all.

  • main places replicas on the pool’s machines with a worker seen in the last 30 s, one worker per machine, skipping machines too small for the model, by rendezvous hashing on the model and machine name: a model stays on the same machines as long as they stay up, and moves when one goes away.

  • Replicas are bounded by the pool: one per online machine where the model fits (room on each pool in GET /api/models/{name}/placement; local has room for one). Asking for more is a 422 (“Pool gpu-a10 has room for 2 replicas of Support triage: one per online machine where it fits”). Deploy and Promote take replicas; the Deploy dialogs have a Replicas row under Pool, and Settings › Deployment changes them for a live deployment. A move to a pool with less room keeps as many as fit. The header shows ”· 2 replicas” when there are several, and Overview “1 of 2 replicas” while they start.

  • The Models list’s Deploy (the ”…” menu on a variant) opens the same Deploy dialog; Deploy on several selected variants opens a compact one with one Pool choice (Automatic, or a pool with how many of them fit) and each variant’s pool and fit, and deploys only those that fit.

  • The engine binds and announces the worker’s --advertise address (the machine’s name by default; --engine-bind sets another interface) on a free port. It pulls weights and adapters from main like any remote engine and dies with the worker. --no-serve keeps a worker to jobs only.

  • A name that resolves only to loopback (common on Linux, where /etc/hosts maps the hostname to 127.0.1.1) would announce an address no gateway can reach. Without --advertise, the worker then takes the address its own traffic to main leaves from, which is the one interface that is known to reach main, and logs it. When that is loopback too, or main is unreachable, it refuses to start and says how to fix it:

    minibrain worker: This machine's name gpu-01 resolves to 127.0.1.1, which gateways on other machines cannot
    reach, and no other address of this machine reaches main at http://main.corp:8000. Pass the address or name other
    machines reach it at: minibrain worker --pool rack --advertise <address> (add --engine-bind 0.0.0.0 to listen on
    every interface), or --no-serve to run jobs only.

    With main on the same machine (--api-url http://127.0.0.1:8000) a loopback name is kept: main’s gateway reaches it, gateways on other machines do not.

  • Gateways route a deployed name only to that deployment’s pool (pool on the route in the config snapshot), ready replicas first; a pinned version (name@N) and the Playground’s versions use any replica.

  • Moving a deployment to another pool is a rolling handover. The old pool keeps its engines and its traffic (the route’s pool stays the old one) while the new pool’s engines start (placements lists both pools). Once the new pool has a ready engine on every machine it was given (one on local), main switches the route to it, and the old engines stop within about 10 s. An engine in the new pool that reports failed, or no machine of the new pool online for 5 minutes, rolls the move back: the deployment stays on the old pool and says why (“Moving to gpu-a10 failed: its engine in pool gpu-a10 failed to start. It stays on local.”) until it is deployed again. Deploying it back to the old pool before the move ends cancels it; deploying to a third pool retargets it, and the first pool keeps serving. Main checks moves every time it rebuilds the gateway config (every second); each move that ends writes a model.move audit event (outcome moved or rolled_back, with from_pool, pool and the error).

  • Status: the header, Overview, Models list and Monitoring say where it runs (“Deployed · v2 on gpu-a10”), and during a move where it goes (“Deployed · v2 on local · moving to gpu-a10”). While the engine fetches and loads it shows “Deploying” or “Fetching weights”; with no worker of the pool online it shows “Waiting for a machine in gpu-a10”.

Each gateway routes per model across its replicas, with no new setting:

Rule Detail
Health Ready replicas first, then loading; a failed request or health check marks a replica down for 10 s
Least busy New work goes to the replica with the fewest requests in flight
Sticky By adapter (so the adapter stays loaded), by prompt prefix for base models (so the KV cache is reused)
Overflow A sticky replica with 4 requests in flight hands new requests to a less busy one; the binding stays
Failover Another replica is tried at once; the key rebinds to the replica that answered

In front of gateways: HAProxy, Envoy or a cloud load balancer. Under the gateway, per pool, later: sgl-router or llm-d. llm-d needs Kubernetes (Gateway API inference extension, its own scheduler), and minibrain must run without it. sgl-router only routes SGLang and pays off with many SGLang replicas of one model; then it slots under the gateway as one replica per pool, and the gateway keeps keys, routes and adapters.

A pool is a named group of machines. Workers and engines join one with --pool NAME (default local, shown as “This machine”) and report their hardware when they register, each time a worker asks for work and each time an engine announces itself.

Reported Example
Platform, architecture linux, x86_64
Accelerator cuda, rocm, xpu, mps, cpu, with the GPU name
GPUs Each with its memory
Memory for models Sum of GPU memory; Apple GPU-usable unified memory; RAM on CPU
CPU, RAM Cores, bytes
Engines installed sglang, vllm, mlx, transformers, llamacpp
  • Infrastructure › Machines lists pools, their machines (online if seen in the last 2 minutes) and the engines serving: one row per machine with its GPUs, memory, CPU, RAM, models and jobs, sortable by each, grouped under a row per pool (machines online, GPUs, memory, engines, jobs, room, join token, use). Group by Model lists each deployed model with the machines that serve it.
  • Discover judges hardware fit against the selected pool: the best machine in the pool decides. Multi-GPU (XL) models need a machine with two or more GPUs.
  • Machines unseen for 7 days and engines unseen for an hour are forgotten. An approved machine stays listed, offline, until it is removed, which revokes its approval.

Each pool is used for Any work (the default), Serving or Training. Admins set it in the pool row’s menu under Machines (GET/PUT /api/compute/pools/{pool}/settings, {"workloads": ["serving"]}); GET /api/compute/pools returns workloads on each pool. A pool keeps the default until it is changed, so one machine works out of the box.

Use Deployments Training and scoring jobs (train, eval, judge)
Any Placed here Run here
Serving Placed here Never start here
Training Never placed here Run here
  • Stored as a list of workloads per pool (pool_settings), so a new kind of work is one more value, not a schema change. Any with no description is no row at all, so it also takes kinds added later.
  • The same row holds an optional one-line description (up to 120 characters, whitespace collapsed; empty clears it), returned as description on pools, deploy placements, the catalogue’s pools and the training plan. The PUT .../settings body takes description, workloads or both; a field left out stays as it is.
  • auto placement skips pools not used for serving; asking for one is a 422 with the reason. Engines already running there keep serving until their model is undeployed or moved.
  • Workers in a pool not used for training claim no training or scoring job; a job asked for such a pool is refused with the reason (422). Jobs running when the use changes finish. A pool with queued training or scoring jobs held to it cannot stop taking training: the change is a 409 naming those jobs, so cancel them or let them start first.
  • Downloads, imports, checks, repairs and exports run on any pool.
  • The Train page’s Pool list and the Deploy dialogs show the other pools disabled, with why.
  • Each change writes a pool.workloads or pool.description audit event with from and to.
opt [Adapter not loaded yet] POST /v1/chat/completions (API key) HTTPS :443 to :8100 Check key hash, pick engine (fewest in flight, sticky by adapter) POST /generate (model, adapter ID) Pull adapter by artifact ID Completion Response or stream Request logs, batched every 2 s App Load balancer Gateway Engine main
Promote v2 Set current version Rebuild config snapshot (about 1 s) GET /internal/gateway/config (ETag, long-poll) New snapshot First request with the v2 adapter Pull adapter by artifact ID Next request for the model routes to v2 Browser main Gateway Engine

Add machine (Infrastructure, admins) asks for a pool, then shows the command to run on the machine: MINIBRAIN_JOIN_TOKEN=mbj_… minibrain worker --main URL --pool NAME. A pool has one join token; the dialog makes it the first time, and the pool’s row under Machines shows its prefix, copies the command and rotates it. Without a token the same command asks for approval:

POST /internal/join/requests (request key, host, hardware, pool) Code K7QM2P Machines, Waiting for approval: approve into pool gpu GET /internal/join/requests/self Machine credential for pool gpu and this host Save it in its home folder Register, claim jobs, ask for engines Prints the code, polls every 3 s Worker (new machine) main Admin
  • A request lasts an hour; Deny ends it and the worker stops with the reason. The approval holds across restarts (<home>/worker/approval.json) until the machine is removed. --main is the address of main (--api-url also works).
  • On an approved machine --pool is ignored: the pool is the one the admin picked.
loop [Every 10 s] Announce pool, hardware, engine kind, models Record in pool, add to config Config long-poll Snapshot with the new engine Base file list (presigned links and data keys when in a bucket) Fetch each file from the bucket or main, check SHA-256 Announce: alive, loaded adapters, load No announcement for 30 s: dropped from routing Engine (GPU box) main Gateway