A reproducible, end-to-end tutorial for using TensorSharp as the local inference backend for Pi Coding Agent. Pi remains responsible for the agent loop, conversation state, and tools. TensorSharp serves the model through its OpenAI-compatible API. No Pi source-code modifications are required. This walkthrough is based on the checked-in example at examples/pi-coding-agent. It covers both the Pi CLI and the Pi SDK, including the exact server configuration, provider setup, commands, and an end-to-end test. What you will build Pi Coding Agent running locally against TensorSharp. Qwen3.8-27B-UD-IQ3_XXS served by TensorSharp on http://127.0.0.1:5000/v1. DFlash2 speculative decoding enabled by the TensorSharp preset. A dedicated Pi agent profile, isolated from your normal Pi configuration. Both an interactive CLI workflow and a Node.js SDK example. An E2E test that checks whether the agent can fix a small project without changing its tests. The validated setup described below used an Apple Silicon Mac. The measured E2E run was performed on an M5 Pro MacBook Pro with 48 GB unified memory. Hardware, model output, and timings will vary. 1. Prerequisites Install: macOS on Apple Silicon for the validated Metal configuration. .NET 10 SDK. CMake 3.20 or later. Xcode Command Line Tools. Node.js 22.19 or later. Pi Coding Agent 1.5.1. Install the Xcode Command Line Tools if they are not already installed: xcode-select --install Install the Pi package used by this example: npm install -g @earendil-works/pi-coding-agent@1.5.1 pi --version Use @earendil-works/pi-coding-agent. The older @mariozechner/pi-coding-agent package is deprecated. Clone TensorSharp: git clone https://github.com/zhongkaifu/TensorSharp.git cd TensorSharp 2. Build TensorSharp Build the server host: dotnet build TensorSharp.Server.Host -c Release -p:TensorSharpSkipMlxNative=true The example's server launcher uses the built host and the checked-in Qwen3.8 preset. It also handles downloading the pinned model artifacts and checking their SHA-256 hashes. 3. Start the inference server From the TensorSharp repository root: examples/pi-coding-agent/scripts/start-server.sh The launcher starts TensorSharp with the model preset and the Pi-specific overlay: dotnet TensorSharp.Server.Host/bin/Release/net10.0/TensorSharp.Server.Host.dll \ --config config/qwen3.8-27b-metal-dflash2.json \ --config examples/pi-coding-agent/tensorsharp-pi.json The launcher may use dotnet run as a fallback if the built host is not present at the expected path. By default, model files are stored in the repository's models directory. To use a different location: export TENSORSHARP_MODELS="$HOME/work/models" examples/pi-coding-agent/scripts/start-server.sh The example downloads these pinned artifacts: Artifact Purpose Qwen/Qwen3.8-27B-UD-IQ3_XXS.gguf Main model Qwen/Qwen3.8-27B-DFlash2-Q4_K_M.gguf DFlash2 draft model The model preset configures the main model for Apple Metal and enables DFlash2 speculative decoding. The example's tested preset uses seven draft tokens per step and greedy decoding. Pi-specific TensorSharp overlay The overlay file is examples/pi-coding-agent/tensorsharp-pi.json: { "host": "127.0.0.1", "port": 5000, "max-tokens": 16384, "mmproj": "none", "kv-cache-dtype": "q8_0", "environment": { "MAX_CONTEXT": "65536" } } This overlay is intentionally small. The model-specific preset supplies the model and backend settings; the overlay adjusts the server for this Pi workflow: Bind to localhost. Listen on port 5000. Set the output-token limit to 16384. Disable the vision projector for this text-only coding example. Use Q8_0 KV cache. Set the context limit to 65,536 tokens. The server is bound to localhost because this example does not configure authentication. Do not expose this endpoint to an untrusted network without adding appropriate access controls. Verify the server In another terminal: curl -s http://127.0.0.1:5000/v1/models The returned model list should include this model ID: Qwen3.8-27B-UD-IQ3_XXS The model ID is important: the Pi provider configuration must use the exact ID returned by this endpoint. 4. Configure Pi to use TensorSharp The example creates a dedicated Pi agent directory so that it does not overwrite or interfere with your normal ~/.pi/agent configuration. The example configuration files are: pi-agent/models.json pi-agent/settings.json models.json { "$schema": "https://pi.dev/schemas/models.schema.json", "providers": { "tensorsharp": { "name": "TensorSharp (local)", "baseUrl": "http://127.0.0.1:5000/v1", "api": "openai-completions", "apiKey": "none", "compat": { "supportsDeveloperRole": true, "supportsReasoningEffort": false, "supportsStore": false, "maxTokensField": "max_completion_tokens", "thinkingFormat": "qwen-chat-template" }, "models": [ { "id": "Qwen3.8-27B-UD-IQ3_XXS", "name": "Qwen3.8 27B (TensorSharp, DFlash2)", "reasoning": true, "input": ["text"], "contextWindow": 65536, "maxTokens": 16384, "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 } } ] } } } A few details worth calling out: baseUrl points to TensorSharp's OpenAI-compatible /v1 endpoint. api is openai-completions. The model id must match the ID reported by /v1/models. maxTokensField tells Pi which completion-token field to send. thinkingFormat: "qwen-chat-template" configures the Qwen thinking-format behavior, including the enable_thinking chat-template setting. apiKey is a placeholder for this local server configuration; the example server does not validate it. settings.json { "defaultProvider": "tensorsharp", "defaultModel": "Qwen3.8-27B-UD-IQ3_XXS", "defaultThinkingLevel": "medium", "enableInstallTelemetry": false } This selects TensorSharp and the Qwen3.8 model as the default for this dedicated Pi profile. 5. Run Pi interactively From the TensorSharp repository root, open a project directory and launch the example wrapper: cd ~/src/my-project /path/to/TensorSharp/examples/pi-coding-agent/scripts/pi.sh For example, if you cloned TensorSharp into ~/work/TensorSharp: cd ~/src/my-project ~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh The wrapper points Pi at the example's dedicated agent directory. It also sets offline/telemetry-related environment variables for this reproducible local workflow: PI_OFFLINE=1 PI_SKIP_VERSION_CHECK=1 PI_TELEMETRY=0 It does not overwrite your existing ~/.pi/agent settings. The example profile is kept separately under ~/.pi/tensorsharp-agent. 6. Run Pi non-interactively You can also run a one-shot prompt: ~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \ -p "Explain what src/stats.js does" < /dev/null Redirecting stdin from /dev/null prevents the process from waiting for interactive input. Other useful examples: # Disable model thinking for this invocation ~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \ --thinking off \ -p "Summarize this project" < /dev/null # JSON output, without creating a session ~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh \ --mode json \ --no-session \ -p "List the main modules" < /dev/null # Continue the previous session ~/work/TensorSharp/examples/pi-coding-agent/scripts/pi.sh --continue Check pi --help for the exact CLI options supported by your installed Pi version. 7. Use TensorSharp from the Pi SDK The repository also includes a Node.js SDK example in examples/pi-coding-agent/sdk. The example uses Pi's public SDK APIs to create an agent session, select the TensorSharp model, send a task, and observe text and tool events. Install the SDK example dependencies cd examples/pi-coding-agent/sdk npm install Copy the sample project to a temporary working directory: cp -R ../sample-project /tmp/stats-demo Run the example: PI_CODING_AGENT_DIR="$HOME/.pi/tensorsharp-agent" \ PI_OFFLINE=1 \ node run-task.mjs /tmp/stats-demo Core SDK integration The following is the essential integration pattern used by the example. It shows how to load the provider configuration, resolve the TensorSharp model, create a Pi agent session, send a task, and dispose of the session. import path from "node:path"; import { createAgentSession, ModelRuntime, SessionManager } from "@earendil-works/pi-coding-agent"; const agentDir = process.env.PI_CODING_AGENT_DIR; if (!agentDir) { throw new Error("PI_CODING_AGENT_DIR must point to the Pi agent directory"); } const projectDir = path.resolve(process.argv[2] ?? "."); const modelRuntime = ModelRuntime.create({ modelsPath: path.join(agentDir, "models.json"), authPath: path.join(agentDir, "auth.json") }); const model = modelRuntime.getModel( "tensorsharp", "Qwen3.8-27B-UD-IQ3_XXS" ); const session = await createAgentSession({ cwd: projectDir, agentDir, modelRuntime, model, thinkingLevel: "medium", sessionManager: SessionManager.inMemory() }); try { session.subscribe((event) => { if (event.type === "message_update" && event.assistantMessageEvent?.type === "text_delta") { process.stdout.write(event.assistantMessageEvent.delta); } if (event.type === "tool_execution_start") { console.log(`\n[tool start] ${event.toolName}`); } if (event.type === "tool_execution_end") { console.log(`\n[tool end] ${event.toolName}`); } if (event.type === "message_end") { // The completed assistant message is available from the session. } }); await session.prompt( "Inspect this project, identify the failing behavior, and fix it. " + "Do not modify the tests." ); console.log("\n\nFinal assistant response:"); console.log(session.getLastAssistantText()); } finally { session.dispose(); } The checked-in run-task.mjs is the source of truth for the complete runnable sample, including its event handling and usage reporting. You can override the TensorSharp endpoint for the SDK process with TENSORSHARP_URL when using a different local endpoint. The default endpoint in the provider configuration is http://127.0.0.1:5000/v1. 8. Run the end-to-end test The example includes an E2E script that exercises both the CLI and SDK paths against a small sample project. Start the server in one terminal: examples/pi-coding-agent/scripts/start-server.sh Then run the E2E script from the repository root in another terminal: examples/pi-coding-agent/scripts/e2e.sh Alternatively, let the E2E script start the server: START_SERVER=1 examples/pi-coding-agent/scripts/e2e.sh The test uses fresh copies of a sample Node.js project containing three bugs and five tests. It runs the agent through the CLI and SDK paths, then checks that: The agent completed the task. All five tests pass. The original tests were not modified. This is a functional integration test, not a broad coding benchmark. It verifies the complete path from Pi through TensorSharp inference and tool execution to a working project. 9. What was measured The example's recorded run was performed on: Component Version / configuration Hardware Apple M5 Pro MacBook Pro, 48 GB unified memory OS macOS 27.2 TensorSharp b722bad1 ggml cd6d2b7 Pi Coding Agent 1.5.1 Node.js 26.8.1 Model Qwen3.8-27B-UD-IQ3_XXS Draft model Qwen3.8-27B-DFlash2-Q4_K_M Recorded results: Path Result Wall time Model requests Tool calls Pi CLI PASS, 5/5 tests 88 s 10 10 Pi SDK PASS, 5/5 tests 83.4 s 8 — The SDK run reported: 48,166 prompt tokens. 41,962 tokens served from prefix cache (about 87%). 2,117 generated tokens. Observed latency and throughput in that run: Initial prefill was approximately 3.9K–4.1K tokens, with TTFT around 13–14 seconds. Later requests had TTFT ranging from approximately 0.45 to 3.5 seconds, depending on prompt size and cache reuse. Decode throughput ranged from approximately 20 to 43 tokens/second. These are observations from the recorded run, not guaranteed performance figures. There was one run per path; results vary with hardware, thermal state, model settings, prompt content, and background load. This should not be interpreted as a standardized coding benchmark. 10. Why prefix/KV cache reuse matters with Pi Pi sends the conversation history again when it makes a new model request after a tool call. In a conventional stateless request path, that can mean repeatedly processing a large prompt. TensorSharp can reuse KV state when the new request begins with an exact prefix that matches the previous request. That makes repeated agent turns much cheaper than re-prefilling the entire conversation. The example's recorded TensorSharp revision includes a fix for whitespace differences at the reasoning boundary (). Pi's streamed reasoning text can include a newline that differs from the server-normalized copy. Without handling that boundary consistently, prefix matching may fail and only the system prompt may be reused. Cache reuse still depends on the prefix remaining identical. If the client changes earlier messages, or starts a new session with a different working directory/system prompt, TensorSharp must process the changed portion again. 11. Troubleshooting /v1/models does not return the expected model Check that the TensorSharp server started successfully and that the model ID in models.json exactly matches the ID returned by: curl -s http://127.0.0.1:5000/v1/models Also check the server logs for model download, file-path, or model-loading errors. Pi stops generating around 1,024 tokens Make sure the Pi overlay is loaded after the model preset: dotnet TensorSharp.Server.Host/bin/Release/net10.0/TensorSharp.Server.Host.dll \ --config config/qwen3.8-27b-metal-dflash2.json \ --config examples/pi-coding-agent/tensorsharp-pi.json The overlay sets max-tokens to 16384. Also check the maxTokens value in Pi's models.json. Context length errors The example configures MAX_CONTEXT to 65536 and advertises a 65,536-token context window to Pi. The actual usable context also depends on model configuration, available memory, KV-cache allocation, and the prompt size. If your machine cannot accommodate the configured context, reduce the context limit and keep the Pi model metadata in sync. Port 5000 is already in use Stop the process using port 5000, or change the TensorSharp server port and the Pi provider's baseUrl to the same new port. SDK cannot find the model or agent configuration Ensure PI_CODING_AGENT_DIR points to the directory containing both models.json and the Pi agent configuration. For the example profile: export PI_CODING_AGENT_DIR="$HOME/.pi/tensorsharp-agent" Prefix cache does not appear to be reused Prefix reuse requires the request history to preserve the same leading messages and content. Changes to earlier messages, system prompts, or working-directory context can invalidate the matching prefix. Use the TensorSharp logs/telemetry to inspect cache behavior rather than assuming every request will be a full cache hit. 12. Notes on other backends The repository also contains a Qwen3.8 NInfer preset for NVIDIA configurations (config/qwen3.8-27b-ninfer.json). The exact Pi + CUDA + Q8_0 KV-cache combination was not part of the Apple Silicon E2E validation described above. Treat that as a separate configuration to validate on your own NVIDIA machine rather than assuming the measurements in this post apply to it. 13. Source files All of the integration pieces are checked into the TensorSharp repository: Pi Coding Agent example README TensorSharp Pi overlay Pi provider configuration Pi settings Pi CLI launcher TensorSharp server launcher E2E test script SDK example The key idea: Pi does not need a special inference integration. Configure TensorSharp as an OpenAI-compatible provider, let Pi own the agent loop and tools, and let TensorSharp handle local model inference, speculative decoding, and prefix/KV-cache reuse. If you try this on a different machine or backend, please share the hardware, model, configuration, and E2E result so the setup can be compared reproducibly.   submitted by   /u/fuzhongkai [link]   [comments]