I evaluate new models by giving them actual work tasks that have already been completed by a trusted model and then comparing the results. My typical workflow is to plan tasks/requests in Gemini and create a markdown spec-sheet that outlines the work that needs to be done along with acceptance criteria, I paste this directly into a fresh pi.dev session for the model to follow. In this case I gave Nemotron 3.5 Bolt two tasks that Qwen Flash Next (UD-IQ4_XS, llama.cpp) completed on a ETL refactor job I'm doing for my actual work (data engineer). Neither tasks are "boil the ocean" type tasks but they are indictative of the type of work I offload to my LLMs. The top line directive of the two tasks: 1. Refactor run.sh script to match the design pattern, flag management, structured logging, and AWS SNS error handling defined in run-new.sh, while preserving the existing Python virtual environment (venv) resolution logic. 2. Create a Python script named orchestrator.py that generates a sequence of time windows (daily or monthly based on the runtime-arguments) and sequentially invokes run.sh for each window. Again, neither are incredibly difficult or taxing but they are the type of things I run across: Updating a script to fit an established pattern used by a client and automating some task. After the new model finishes its work I load the spec-sheet, any relevant files (original run.sh and run-new.sh), and the results of both models into Gemini and ask it to evaluate the performance and select a winner. In task #1 (run.sh refactor) Gemini judged Nemotron to have done a better job. Here is Gemini's detailed comparison of the refactored scripts judged against the specifications in the spec-sheet, Model 1 is Nemotron and Model 2 in Qwen Flash Next: ``Text 1. Directory Pathing (LOG_DIRvsSCRIPT_DIR) - Model 1: DefinesSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"at the top and uses$SCRIPT_DIR/logsforLOG_DIR. This guarantees logs are created relative to the script directory regardless of where the script is called from. - Model 2: DefinesLOG_DIR="$(pwd)/logs"at the top and delays definingSCRIPT_DIRuntil later in the file. If a user calls/path/to/run.shfrom/tmp, Model 2 creates logs in/tmp/logs` instead of the application directory. Flag Parsing & Argument Forwarding Both models implemented an identical, robust argument parser using a while/case loop that correctly parses all required flags (-o, -x, -s, -e, -i, -b, -h, -d) along with their long-form equivalents. Neither model unnecessarily exported CLI arguments to environment variables, adhering strictly to Requirement 2. Error Handling, Traps & Process Execution Dependency Checks: Model 1 verifies the presence of required binaries (tee, aws) before redirecting output. Model 2 redirects output using process substitution >(tee -a ...) before checking if tee exists. - `exec` Behavior: - `run-old.sh` used `exec "$VENV/bin/python" ...`. - Model 2 preserves `exec`, but because `exec` replaces the running shell process with Python, any `trap ... EXIT` defined in the shell will be bypassed when Python completes. - Model 1 removes `exec` and invokes Python as a child process, allowing shell traps to clear cleanly upon exit. Conclusion Model 1 (run-model1.sh) is the superior refactor. It adheres to all acceptance criteria, correctly roots log directories to SCRIPT_DIR rather than pwd, orders setup tasks logically (verifying tee before using process substitution), and manages traps cleanly. ``` In task #2 (orchestrator.py build) Gemini judged Qwen Flash Next to have done a better job. Here is Gemini's detailed comparison of the python scripts judged against the specifications in the spec-sheet, Model 1 is Nemotron and Model 2 in Qwen Flash Next: ``Text 1. Specification Compliance (subprocess.run) - Model 1: Implements stream piping usingsubprocess.Popenwithstdout=subprocess.PIPEandstderr=subprocess.STDOUT. While this streams logs, it manually manages process reading loops and checks return codes manually. - Model 2: Strictly follows the specification instructions by usingsubprocess.run(cmd, check=True)`. By omitting stream capture arguments, the child process inherits the parent stdout/stderr natively, streaming logs in real time while letting Python handle process lifecycles cleanly. Robustness & File Path Handling Model 1: Assumes ./run.sh is located relative to the current working directory (CWD) where the script is executed. Model 2: Resolves run.sh relative to the script's directory (Path(__file__).resolve().parent / "run.sh"). It verifies whether run.sh exists prior to running any logic and raises a clear error if missing. Argument Parsing & Validation Model 1: Parses arguments into raw strings first, then tries parse_timestamp() inside main(). Model 2: Passes type=parse_timestamp directly into parser.add_argument(), allowing argparse to raise idiomatic command-line errors (ArgumentTypeError) automatically during parsing. Logging & User Experience Model 1: Does not print any progress indicators prior to running the windows. Model 2: Adds clear logging for total windows found, individual window execution markers (=== Window X/Y ===), and command execution previews. Conclusion Model 2** is the superior implementation. Both implementations satisfy the core specification requirements, but Model 2 is significantly cleaner, more robust, and strictly adheres to the prompt requirements. It adheres precisely to subprocess.run(check=True) from the spec, handles path resolution safely, uses proper argparse types, and provides better terminal feedback during multi-window batch execution. ``` My takeaway from this is Nemotron 3.5 Bolt is a capable model for my work with the added benefit that it is fast on my server, I'm seeing average speeds of ~130t/s and to get this level of performance from a 2-bit quant is impressive. It won't replace Qwen Flash Next with it, especially for large complicated, repo-level tasks, but it will probably take the place of Gemma4-31B for light-to-medium tasks that I want done quickly without it overthinking everything. I'll be getting a M5 Mac Studio soon and want to set up agent/sub-agent workflows and I think Bolt might be a really good sub-agent with Qwen Flash Next (hopefully Qwen4) passing it specs to implements.   submitted by   /u/CulturalKing5623 [link]   [comments]