I’m the developer of TensorSharp. Inspired by vLLM PR #57250, I added and optimized a native /v1/systemone endpoint for Jev-style structured decisions using DiffusionGemma GGUF. This uses DiffusionGemma weights; “Jev-style” refers to the API and decision workflow. The main implementation difference: TensorSharp reads the requested label logits directly, rather than asking the model to generate probabilities as JSON. I compared it with the original LocalJev engine on a small benchmark. Across the 27 paired requests where both implementations returned valid responses, the median ratio of LocalJev latency to TensorSharp latency was 3.345×. Benchmark results The test covered 12 cases × 3 measured repetitions, with three decisions per request: 36 requests and 108 expected decisions per implementation. Latency statistics below include successful requests only, so the aggregate columns cover different subsets. The 3.345× figure above is calculated from matched successful pairs—not by dividing the aggregate means or medians. Metric TensorSharp native /v1/systemone Original LocalJev engine Valid requests 36/36 27/36 Failed requests 0 9 Correct decisions / total expected 108/108 81/108* p50 latency 2.877 s 10.479 s p95 latency 3.165 s 26.167 s Mean latency 2.917 s 12.548 s *LocalJev’s 81/108 includes the missing decisions from failed requests. On valid responses, it got 81/81 decisions correct. The failures were schema-validation errors, not incorrect classifications. An important comparison detail: average input length was 192.7 tokens for TensorSharp versus 589.6 tokens for valid LocalJev requests. TensorSharp reads label logits directly; LocalJev builds a larger chat prompt and generates probability JSON. That difference is part of the end-to-end approaches being compared. This is not an identical-prompt, kernel-only performance comparison, and the small test set should not be treated as a general accuracy benchmark. Per-case results Latencies are medians across three measured repetitions. LocalJev latency is shown only for valid responses. Case TensorSharp LocalJev LocalJev failures billing-calm 3.051 s 8.135 s 0/3 outage-angry 3.152 s — 3/3 sales-polite 3.142 s 10.479 s 0/3 payment-immediate 3.117 s 8.060 s 0/3 bug-calm 3.164 s 11.624 s 0/3 sales-angry 3.118 s 10.347 s 0/3 red-small-circle 2.686 s — 3/3 blue-large-square 2.745 s — 3/3 green-medium-circle 2.692 s 7.657 s 0/3 red-large-square 2.697 s 26.182 s 0/3 blue-medium-circle 2.723 s 14.564 s 0/3 green-small-square 2.742 s 15.782 s 0/3 LocalJev failed all three repetitions of outage-angry, red-small-circle, and blue-large-square, returning HTTP 422 because the generated output did not satisfy the requested JSON schema. The benchmark recorded no length-limited attempts. The server applied its configured 256-token output cap, while valid responses used about 60 output tokens. Try it locally Start TensorSharp.Server.Host with a DiffusionGemma GGUF model as usual, then send a Jev-format request to: POST http://127.0.0.1:5000/v1/systemone Python — a minimal billing-classification example: import json from urllib.request import Request, urlopen body = { "model": "jev-latest", "state": "I was charged twice for the same subscription this month.", "questions": { "billing": { "type": "noul", "instructions": "Is this a billing issue?" } }, "samples": 1, "seed": 42 } request = Request( "http://127.0.0.1:5000/v1/systemone", data=json.dumps(body).encode("utf-8"), headers={"Content-Type": "application/json"}, method="POST" ) with urlopen(request, timeout=180) as response: answer = json.load(response) print(answer["answers"]["billing"]["noul"]) curl — using the example request file from the repository root: curl http://127.0.0.1:5000/v1/systemone \ -H 'Content-Type: application/json' \ --data-binary u/docs/examples/jev-ticket.json C# / .NET — calling the model service directly: using System.Text.Json; using TensorSharp.Server; using TensorSharp.Server.Jev; using var service = new ModelService(); service.LoadModel( "C:/Works/models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf", mmProjPath: null, backendStr: "ggml_cuda" ); using var json = JsonDocument.Parse( File.ReadAllText("docs/examples/jev-ticket.json") ); object response = await service.JevAsync( JevRequest.Parse(json.RootElement) ); Console.WriteLine(JsonSerializer.Serialize(response)); The part I find interesting is using a local model for bounded decisions without making it write out a JSON answer first. On this workload, the direct-logit approach was faster and avoided the output-schema failures seen with the generation-based baseline. What workloads would you test next—ticket routing, moderation, agent tool selection, or something else?   submitted by   /u/fuzhongkai [link]   [comments]