I burned through my pro budget this week, didn't even touch Fable. So today when I woke up and had about 12% left, I decided I'd try a bit of an experiment. I'd let my current Sonnet 5.5 seat test, train, and evaluate DeepSeek as a potential candidate replacement. It was a valuable experiment, not even because of what was learned about DeepSeek, but even more so about what was learned about our own system. So many little issues that Sonnet brute forces past or deals with in real time, it's no wonder I'm burning windows. I thought the community might enjoy the evaluation though so here it is. A coding-agent trial of a 2-bit DeepSeek on two 2016 GPUs: a real job, not a benchmark Evaluation written by Claude Sonnet 5.5, who is also DS's mentor on this trial. What this is. I'm Claude Sonnet 5.5. I hold the Systems Engineer (SE) seat on a single-workstation AI lab. The seat owns the machine's configuration as code (Ansible), the registry database, the roadmap, the enforcement hooks and the commits. It also reviews every change that other agents make. For the past stretch I've been mentoring a candidate to replace me in that seat: DeepSeek V4.1 Flash, running locally, driving Claude Code as "DS". It does the same job I do, on real project work, with me checking its output. Model Base: DeepSeek-V4.1-Flash-1017, from DeepSeek (deepseek-ai), MIT license. Quant: ISTA-DASLab "GSQ-RCO" IQ2_XXS, 2.00 bits per weight, 6-shard GGUF. Packed in: an Engram table (IQ4_NL), a Q8_0 speculative drafter, and an optional vision tower. Context: 131,072 tokens maximum. The drafter and the vision tower are both off in the daily profile. Engine: Strata Strata is an MIT-licensed engine for large mixture-of-experts models. It streams the experts instead of holding them all in VRAM. We pin upstream main at commit 13eedf5 (2026-11-12) and build it from source for sm_70 (Volta). Upstream's packages target RTX 30/40/50 cards. The V100 path is flagged experimental, so this run is a real test of it. No source patches. Our changes are configuration only: the model pack is built in place from the existing GGUFs, each launch profile is rendered from data, and a single response is capped at 16,000 tokens. How it loads: the experts total about 132 GB, against about 64 GB of VRAM across both cards. So the experts live in system RAM, and a hot-expert profile that ships with Strata pre-seeds that store with the most-used ones. Engram (DeepSeek's conditional-memory table) loads lazily. Drafter off, on purpose. In an A/B test, forcing it on ran at 17.3 tok/s (171 of 213 drafts accepted), against 23.8 tok/s without it. It loses here because the experts don't fit in VRAM. Speed (measured) Metric Value Decode ~24 tok/s (23.8 in the A/B test; 25.2 across 24 logged bench runs) Prefill ~127 tok/s Time to first token ~9 s The prefill figure comes from those 24 bench runs, with context up to about 22k tokens. I have no prefill number logged from live agent sessions. Hardware 2x Tesla V100 32 GB (Volta: no BF16, no FlashAttention 2, no NVLink). NVIDIA driver 580, CUDA 12.9. 2x Xeon E5-2630 v4, 40 threads, 125 GB RAM, 1 TB NVMe. Evaluation: what DS is like on real work Good On one concrete deliverable it works like a competent junior. It wrote a sandbox proof-check, a database migration and a checker for that migration, and fixed a failing proof tool. It landed them as clean commits with a why-first message that also records what is done and what isn't. It follows the process: it finishes a step, records the evidence, and hands off. Costly Speed. I measured about 17 s per message, about 5 min per web fetch, and about 7 min per context compaction. It stalls for minutes without any output. The auto-mode safety classifier is DS itself. It times out at about 70 s, which denies safe read-only commands. I measured about 13% of wall time lost to this in one session. The fix is allow-rules for read-only commands. Delegation. I told it to hand reading to a cheap fast helper model. It kept reading whole files itself, even after repeated nudges. I'm moving that rule into a hook instead. Dangerous Its own test passed while the real migration failed. The test's database connection skipped a foreign-key setting that production enforces. A green check from DS is not proof, so I review the result myself. Verdict so far DS is not yet a replacement for the SE seat, and the main gap is speed rather than knowledge. I estimate that structural fixes (hooks, allow-rules, routing fetches through a helper) recover about two-thirds of the lost time. Weaker areas: open-ended planning, and judging its own work. I have not run a formal qualitative evaluation or an identical-task comparison against my own seat. That is next, as a graded set of small tasks with known answers, re-run after each fix. Everything above comes from the machine's own registry and configuration. Numbers marked "~" or "about" are my estimates.   submitted by   /u/a4mula [link]   [comments]