> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/OpenShell/llms.txt
> Use this file to discover all available pages before exploring further.
# Configure Inference Routing
> Set up the managed local inference endpoint with provider credentials and model configuration.
This page covers the managed local inference endpoint (`https://inference.local`). External inference endpoints are controlled by sandbox `network_policies` — see [Policies](/sandboxes/policies) for details.
The configuration requires two values:
| Value | Description |
| --------------- | ----------------------------------------------------------------------------------- |
| Provider record | The credential backend OpenShell uses to authenticate with the upstream model host. |
| Model ID | The model to use for generation requests. |
For a list of tested providers and their base URLs, see [Providers](/sandboxes/providers).
## Configure the inference backend
Create a provider record that holds the backend credentials OpenShell will use when forwarding requests from `inference.local`.
```bash theme={null}
openshell provider create --name nvidia-prod --type nvidia --from-existing
```
This reads `NVIDIA_API_KEY` from your environment.
Any cloud provider that exposes an OpenAI-compatible API works with the `openai` provider type. You need the base URL, an API key, and a model name from your provider.
```bash theme={null}
openshell provider create \
--name my-cloud-provider \
--type openai \
--credential OPENAI_API_KEY= \
--config OPENAI_BASE_URL=https://api.example.com/v1
```
Replace the base URL and API key with values from your provider. For other providers, refer to their documentation for the correct base URL, available models, and API key setup.
Use `--config OPENAI_BASE_URL` to point to any OpenAI-compatible server running where the gateway runs.
```bash theme={null}
openshell provider create \
--name my-local-model \
--type openai \
--credential OPENAI_API_KEY=empty-if-not-required \
--config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1
```
For host-backed local inference, use `host.openshell.internal` or the host's LAN IP. Avoid `127.0.0.1` and `localhost` — requests originate from the gateway or sandbox runtime, not your shell.
Set `OPENAI_API_KEY` to a placeholder value if the server does not require authentication.
For a self-contained setup, the Ollama community sandbox bundles Ollama inside the sandbox itself — no host-level provider is needed. See the [local inference tutorial](/tutorials/local-inference) for details.
```bash theme={null}
openshell provider create --name anthropic-prod --type anthropic --from-existing
```
This reads `ANTHROPIC_API_KEY` from your environment.
Point `inference.local` at the provider you created and choose the model to use:
```bash theme={null}
openshell inference set \
--provider nvidia-prod \
--model nvidia/nemotron-3-nano-30b-a3b
```
By default, `openshell inference set` probes the resolved upstream endpoint before saving. If the endpoint is not live yet, add `--no-verify` to persist the route without the probe.
Confirm the provider and model are set correctly:
```bash theme={null}
openshell inference get
```
```
Gateway inference:
Provider: nvidia-prod
Model: nvidia/nemotron-3-nano-30b-a3b
Version: 1
```
## Update part of the config
Use `openshell inference update` when you want to change only one field without repeating the other:
```bash theme={null}
# Change only the model
openshell inference update --model nvidia/nemotron-3-nano-30b-a3b
# Switch providers without changing the model
openshell inference update --provider openai-prod
```
## Use the local endpoint from a sandbox
After inference is configured, code inside any sandbox can call `https://inference.local` directly:
```python theme={null}
from openai import OpenAI
client = OpenAI(base_url="https://inference.local/v1", api_key="unused")
response = client.chat.completions.create(
model="anything",
messages=[{"role": "user", "content": "Hello"}],
)
```
The `model` and `api_key` values supplied by the client are not sent upstream. The privacy router injects the real credentials from the configured provider and rewrites the model before forwarding.
Some SDKs require a non-empty API key value even though `inference.local` does not use it. Pass any placeholder such as `test` or `unused` in those cases.
### Verify end-to-end from a sandbox
To confirm connectivity from inside a sandbox:
```bash theme={null}
curl https://inference.local/v1/responses \
-H "Content-Type: application/json" \
-d '{
"instructions": "You are a helpful assistant.",
"input": "Hello!"
}'
```
A successful response confirms the privacy router can reach the configured backend and the model is serving requests.
## How gateway-level config applies
* **Gateway-scoped:** The active provider and model apply to every sandbox using that gateway. All sandboxes see the same `inference.local` backend.
* **HTTPS only:** `inference.local` is intercepted only for HTTPS traffic.
* **Hot-refresh:** Provider and inference changes are picked up within about 5 seconds by default. Sandboxes do not need to be restarted.
## Self-hosted NIM endpoint
To configure `inference.local` to forward to a self-hosted NVIDIA NIM instance:
```bash theme={null}
# Create an OpenAI-compatible provider pointing at your NIM endpoint
openshell provider create \
--name my-nim \
--type openai \
--credential OPENAI_API_KEY= \
--config OPENAI_BASE_URL=https://your-nim-host.example.com/v1
# Set inference routing to use it
openshell inference set \
--provider my-nim \
--model meta/llama-3.1-8b-instruct
```
Replace the base URL, API key, and model ID with values from your NIM deployment.
## Local inference with Ollama
To point `inference.local` at an Ollama instance running on the same host as the gateway:
```bash theme={null}
openshell provider create \
--name ollama-local \
--type openai \
--credential OPENAI_API_KEY=ollama \
--config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1
openshell inference set \
--provider ollama-local \
--model llama3.2
```
If the gateway runs on a remote host, `host.openshell.internal` points to that remote machine, not your laptop. A locally running Ollama process is not reachable from a remote gateway unless you add a tunnel or shared network path.
Ollama also supports cloud-hosted models using the `:cloud` tag suffix (for example, `qwen3.5:cloud`), which do not require local hardware.
For a fully self-contained Ollama setup — with Ollama running inside the sandbox itself — see the [local inference tutorial](/tutorials/local-inference).
## Troubleshooting
**Endpoint probe fails on `inference set`**
`openshell inference set` verifies the upstream endpoint before saving by default. If the model server is not running yet, use `--no-verify` to save the config first and retry verification later.
**Requests fail with a connection error inside the sandbox**
Check that the upstream server is bound to `0.0.0.0` rather than `127.0.0.1`. Requests to `inference.local` originate from the gateway runtime, so loopback addresses are not reachable. Use `host.openshell.internal` or the host's LAN IP in the provider's `OPENAI_BASE_URL`.
**SDK rejects an empty API key**
Some SDKs validate that the API key field is non-empty before sending the request. Pass any non-empty placeholder — `inference.local` ignores whatever value the sandbox provides.
**Changes not taking effect**
Hot-refresh propagates within about 5 seconds. If a sandbox still uses the old config after that window, check `openshell inference get` to confirm the saved configuration is correct.
## Next steps
Understand the two routing paths and supported API patterns.
View and update provider credential records.
Complete walkthrough for local inference with Ollama and LM Studio.