> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/OpenShell/llms.txt
> Use this file to discover all available pages before exploring further.

# Run local inference in a sandbox

> Route sandbox inference requests to a local model backend using Ollama or LM Studio, with options for GPU-accelerated and cloud-hosted models.

This tutorial covers two ways to run local inference with OpenShell: using Ollama or using LM Studio. Both approaches expose a local model backend through `inference.local` so that agents inside a sandbox can make inference requests without reaching external APIs.

<Tabs>
  <Tab title="Ollama">
    Ollama offers two approaches: a self-contained community sandbox with Ollama pre-installed, or routing sandbox inference to a host-level Ollama instance shared across multiple sandboxes.

    ## Prerequisites

    * A working OpenShell installation. Complete the [Quickstart](/quickstart) before proceeding.

    ## Option A: Ollama community sandbox (recommended)

    The Ollama community sandbox bundles Ollama, Claude Code, OpenCode, and Codex into a single image. Ollama starts automatically when the sandbox launches.

    <Steps>
      <Step title="Create the sandbox">
        ```console theme={null}
        $ openshell sandbox create --from ollama
        ```

        This pulls the community sandbox image, applies the bundled policy, and drops you into a shell with Ollama running.
      </Step>

      <Step title="Run a model">
        Chat with a local model:

        ```console theme={null}
        $ ollama run qwen3.5
        ```

        Or run a cloud-hosted model (no local GPU required):

        ```console theme={null}
        $ ollama run kimi-k2.5:cloud
        ```

        To start a coding agent with Ollama as the model backend, use `ollama launch`:

        ```console theme={null}
        $ ollama launch claude
        $ ollama launch codex
        $ ollama launch opencode
        ```

        For CI/CD and automated workflows, `ollama launch` supports a headless mode:

        ```console theme={null}
        $ ollama launch claude --yes --model qwen3.5
        ```
      </Step>
    </Steps>

    ### Model recommendations

    | Use case             | Model              | Notes                                                            |
    | -------------------- | ------------------ | ---------------------------------------------------------------- |
    | Smoke test           | `qwen3.5:0.8b`     | Fast and lightweight, good for verifying setup                   |
    | Coding and reasoning | `qwen3.5`          | Strong tool calling support for agentic workflows                |
    | Complex tasks        | `nemotron-3-super` | 122B parameter model, requires 48 GB+ VRAM                       |
    | No local GPU         | `qwen3.5:cloud`    | Runs on Ollama's cloud infrastructure, no `ollama pull` required |

    <Note>
      Cloud models use the `:cloud` tag suffix and do not require local hardware.
    </Note>

    ### Tool calling

    Agentic workflows (Claude Code, Codex, OpenCode) rely on tool calling. The following models have reliable tool calling support: Qwen 3.5, Nemotron-3-Super, GLM-5, and Kimi-K2.5. Check the [Ollama model library](https://ollama.com/library) for the latest additions.

    ### Updating Ollama

    To update Ollama inside a running sandbox:

    ```console theme={null}
    $ update-ollama
    ```

    To auto-update on every sandbox start:

    ```console theme={null}
    $ openshell sandbox create --from ollama -e OLLAMA_UPDATE=1
    ```

    ## Option B: Host-level Ollama

    Use this approach when you want a single Ollama instance on the gateway host, shared across multiple sandboxes through `inference.local`.

    <Note>
      This approach uses Ollama because it is easy to install and run locally, but you can substitute other inference engines such as vLLM, SGLang, TRT-LLM, and NVIDIA NIM by changing the startup command, base URL, and model name.
    </Note>

    <Steps>
      <Step title="Install and start Ollama">
        Install [Ollama](https://ollama.com/) on the gateway host:

        ```console theme={null}
        $ curl -fsSL https://ollama.com/install.sh | sh
        ```

        Start Ollama on all interfaces so it is reachable from sandboxes:

        ```console theme={null}
        $ OLLAMA_HOST=0.0.0.0:11434 ollama serve
        ```

        <Tip>
          If you see `Error: listen tcp 0.0.0.0:11434: bind: address already in use`, Ollama is already running as a system service. Stop it first:

          ```console theme={null}
          $ systemctl stop ollama
          $ OLLAMA_HOST=0.0.0.0:11434 ollama serve
          ```
        </Tip>
      </Step>

      <Step title="Pull a model">
        In a second terminal, pull a model:

        ```console theme={null}
        $ ollama run qwen3.5:0.8b
        ```

        Type `/bye` to exit the interactive session. The model stays loaded.
      </Step>

      <Step title="Create a provider">
        Create an OpenAI-compatible provider pointing at the host Ollama instance:

        ```console theme={null}
        $ openshell provider create \
            --name ollama \
            --type openai \
            --credential OPENAI_API_KEY=empty \
            --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1
        ```

        OpenShell injects `host.openshell.internal` so sandboxes and the gateway can reach the host machine. You can also use the host's LAN IP.
      </Step>

      <Step title="Set inference routing">
        ```console theme={null}
        $ openshell inference set --provider ollama --model qwen3.5:0.8b
        ```

        Confirm the saved config:

        ```console theme={null}
        $ openshell inference get
        ```
      </Step>

      <Step title="Verify from a sandbox">
        ```console theme={null}
        $ openshell sandbox create -- \
            curl https://inference.local/v1/chat/completions \
            --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}'
        ```

        The response should be JSON from the model.
      </Step>
    </Steps>

    ## Troubleshooting

    | Problem                           | Fix                                                                                                        |
    | --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
    | Ollama not reachable from sandbox | Ollama must be bound to `0.0.0.0`, not `127.0.0.1`. The community sandbox handles this automatically.      |
    | Wrong `OPENAI_BASE_URL`           | Use `http://host.openshell.internal:11434/v1`, not `localhost` or `127.0.0.1`.                             |
    | Model not found                   | Run `ollama ps` to confirm the model is loaded. Run `ollama pull <model>` if needed.                       |
    | HTTPS vs HTTP                     | Code inside sandboxes must call `https://inference.local`, not `http://`.                                  |
    | AMD GPU driver issues             | Ollama v0.18+ requires ROCm 7 drivers for AMD GPUs. Update your drivers if you see GPU detection failures. |

    ```console theme={null}
    $ openshell status
    $ openshell inference get
    $ openshell provider get ollama
    ```
  </Tab>

  <Tab title="LM Studio">
    LM Studio provides an easy-to-set-up local inference server with both OpenAI-compatible and Anthropic-compatible endpoints.

    ## Prerequisites

    * A working OpenShell installation. Complete the [Quickstart](/quickstart) before proceeding.
    * [LM Studio](https://lmstudio.ai/download) installed and running in the same environment as your gateway.

    If you prefer to work without keeping the LM Studio app open, install `lms` (headless LM Studio):

    <CodeGroup>
      ```console Linux / macOS theme={null}
      $ curl -fsSL https://lmstudio.ai/install.sh | bash
      ```

      ```console Windows theme={null}
      $ irm https://lmstudio.ai/install.ps1 | iex
      ```
    </CodeGroup>

    Then start the daemon:

    ```console theme={null}
    $ lms daemon up
    ```

    <Steps>
      <Step title="Start the LM Studio local server">
        Start the local server from the **Developer** tab and verify the OpenAI-compatible endpoint is enabled.

        LM Studio listens on `127.0.0.1:1234` by default. For use with OpenShell, configure it to listen on all interfaces (`0.0.0.0`).

        * **GUI**: Go to the Developer tab, select **Server Settings**, then enable **Serve on Local Network**.
        * **Headless**: Run `lms server start --bind 0.0.0.0`.
      </Step>

      <Step title="Download and load a model">
        In the LM Studio app, go to the **Model Search** tab to download a small model such as Qwen3.5 2B.

        Using the CLI:

        ```console theme={null}
        $ lms get qwen/qwen3.5-2b
        $ lms load qwen/qwen3.5-2b
        ```
      </Step>

      <Step title="Add LM Studio as a provider">
        Choose the provider type that matches the client protocol you want to route through `inference.local`.

        <Tabs>
          <Tab title="OpenAI-compatible">
            ```console theme={null}
            $ openshell provider create \
                --name lmstudio \
                --type openai \
                --credential OPENAI_API_KEY=lmstudio \
                --config OPENAI_BASE_URL=http://host.openshell.internal:1234/v1
            ```

            Use this provider for clients that send OpenAI-compatible requests such as `POST /v1/chat/completions` or `POST /v1/responses`.
          </Tab>

          <Tab title="Anthropic-compatible">
            ```console theme={null}
            $ openshell provider create \
                --name lmstudio-anthropic \
                --type anthropic \
                --credential ANTHROPIC_API_KEY=lmstudio \
                --config ANTHROPIC_BASE_URL=http://host.openshell.internal:1234
            ```

            Use this provider for Anthropic-compatible `POST /v1/messages` requests.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Configure LM Studio as the inference provider">
        Set the managed inference route for the active gateway.

        <Tabs>
          <Tab title="OpenAI-compatible">
            ```console theme={null}
            $ openshell inference set --provider lmstudio --model qwen/qwen3.5-2b
            ```
          </Tab>

          <Tab title="Anthropic-compatible">
            ```console theme={null}
            $ openshell inference set --provider lmstudio-anthropic --model qwen/qwen3.5-2b
            ```
          </Tab>
        </Tabs>

        The active `inference.local` route is gateway-scoped, so only one provider and model pair is active at a time. Re-run `openshell inference set` whenever you want to switch between OpenAI-compatible and Anthropic-compatible clients.

        Confirm the saved config:

        ```console theme={null}
        $ openshell inference get
        ```

        You should see `Provider: lmstudio` (or `Provider: lmstudio-anthropic`) along with `Model: qwen/qwen3.5-2b`.
      </Step>

      <Step title="Verify from inside a sandbox">
        Run a request through `https://inference.local`:

        <Tabs>
          <Tab title="OpenAI-compatible">
            ```bash theme={null}
            openshell sandbox create -- \
                curl https://inference.local/v1/chat/completions \
                --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}'
            ```

            Or using the Responses API:

            ```bash theme={null}
            openshell sandbox create -- \
                curl https://inference.local/v1/responses \
                --json '{
                  "instructions": "You are a helpful assistant.",
                  "input": "hello",
                  "max_output_tokens": 10
                }'
            ```
          </Tab>

          <Tab title="Anthropic-compatible">
            ```bash theme={null}
            openshell sandbox create -- \
                curl https://inference.local/v1/messages \
                --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}'
            ```
          </Tab>
        </Tabs>
      </Step>
    </Steps>

    ## Troubleshooting

    | Problem                                     | Fix                                                                                                          |
    | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
    | LM Studio server not reachable              | Confirm the local server is running and listening on `0.0.0.0`, not `127.0.0.1`.                             |
    | Wrong `OPENAI_BASE_URL`                     | Use `http://host.openshell.internal:1234/v1` for OpenAI-compatible providers.                                |
    | Wrong `ANTHROPIC_BASE_URL`                  | Use `http://host.openshell.internal:1234` (no `/v1`) for Anthropic-compatible providers.                     |
    | Model name mismatch                         | The model name in `openshell inference set` must match the name exposed by LM Studio exactly.                |
    | Gateway and LM Studio on different machines | `host.openshell.internal` resolves to the gateway host. If LM Studio runs elsewhere, use its LAN IP instead. |

    ```console theme={null}
    $ openshell status
    $ openshell inference get
    $ openshell provider get lmstudio
    $ openshell provider get lmstudio-anthropic
    ```
  </Tab>
</Tabs>

## GPU support for local inference

Both Ollama and LM Studio can use local GPU resources:

* **NVIDIA GPUs**: Both tools support CUDA automatically when the appropriate drivers are installed. No additional configuration is required in OpenShell.
* **AMD GPUs**: Ollama v0.18+ requires ROCm 7 drivers. LM Studio uses ROCm automatically on supported hardware.
* **Apple Silicon**: Both tools use Metal for hardware acceleration on M-series Macs.
* **CPU fallback**: If no GPU is detected, inference runs on CPU. For most coding assistant workloads, a small quantized model (such as `qwen3.5:0.8b`) runs acceptably on CPU.

<Note>
  GPU resources are available to Ollama and LM Studio running on the gateway host. Sandboxes themselves do not have direct GPU access — inference is routed from the sandbox through `inference.local` to the host-side backend.
</Note>

## What's next

<CardGroup cols={2}>
  <Card title="Managed inference" icon="microchip" href="/inference/overview">
    Learn how OpenShell routes inference requests and manages provider configuration.
  </Card>

  <Card title="Configure inference backends" icon="sliders" href="/inference/configure">
    Configure vLLM, SGLang, TRT-LLM, NVIDIA NIM, or any other OpenAI-compatible backend.
  </Card>

  <Card title="Community sandboxes" icon="box" href="/sandboxes/community-sandboxes">
    Explore pre-built sandbox images for common development workflows.
  </Card>

  <Card title="LM Studio CLI docs" icon="terminal" href="https://lmstudio.ai/docs/cli">
    Learn more about the `lms` CLI for headless LM Studio usage.
  </Card>
</CardGroup>