For as long as I’ve been involved with nREPL, the question I’ve heard mostoften from people building servers and clients has been some variation of“Will this work with CIDER?”. And for just as long the honest answer was “Tryit and see”. Today I have a better answer for you -proof, a compatibility suite for nREPLservers (and clients). I’ve been working on it since the end of September andproof 0.2 came out a couple of days ago, so I guess it’s high timeI told you a bit about it.Try It and SeeThe nREPL protocol is tiny. A handful of ops, requests and responses that aresimple maps, bencode on the wire. You can write a basic nREPL server in acouple of hours, and lots of people have done exactly that. Making it play nice withCIDER, Calva, Conjure and the rest of the clients out there is a verydifferent matter, though. What clients actually expect from a server isspread across the reference implementation, a draft spec, many years ofclient code and lots of GitHub issues. Which status values mean what? Isops in describe a map or a list? Where does the end of an interruptedevaluation go? What happens to a session when its client goes away in themiddle of an eval? Server authors usually find out one bug report at a time.Client authors don’t have it any easier, as every server has its own quirks,and for many years the only way to find them was to try your client with everyserver you could get your hands on. I’ve been on both sides of thisconversation for well over a decade (CIDER is a client, nREPL is a server), sobelieve me - I know the pain.Bigger Than ClojureWhy bother? Because I’ve always seen nREPL as something bigger than Clojure.Back in 2019 I wrote that nREPL is similar in nature toLSP, only much simpler, and Istill believe that. For any language that can evaluate code at runtime, nREPLis a much simpler alternative to LSP. Implement eval, interrupt andload-file, throw in completions and lookups if you feel like it, and everynREPL client out there becomes an editor integration for your language.That’s not just a dream anymore. These days there are nREPL servers forClojure dialects running on Python (Basilisp), C++ (jank), .NET (ClojureCLR),Go (let-go) and Node.js (nbb), for Janet and Steel Scheme, and even forErlang and Elixir (more on those later). Still, “language-agnostic” doesn’tmean much if everyone who builds a server has to check it against CIDER byhand.Better Late Than NeverThe idea of a compatibility suite is not new. When nREPL 1.0 came out in 2022I wrote that improving the protocol spec and the ability to test againstit was where our focusshould go next, and a year later the lack of a compatibility test suite mademy list of things people hate aboutnREPL. I even asked if one of you waslooking for a fun small OSS project. Nobody volunteered, and the idea spenta few more years on the backburner. This year, thanks to the support ofClojurists Together, I finally made avery focused push on it.It started with the spec. Late last year Phil Hagelberg wrote a draft of thenREPL protocol spec, which landed in May. A spec isgreat, but you can’t run a spec against your server, so I started proof. Thetwo feed each other - every time proof finds a place where the spec and theclients disagree, or something the spec doesn’t cover at all, it ends up ina running list, and from there in the spec itself. (A few ofthose are already pull requests against it.)When the spec and the clients disagree, proof sides with the clients. Forinstance, the spec recommends returning the ops in describe as a list, butCIDER and Calva both expect a map. A server that follows the spec to theletter, but that CIDER can’t connect to, is not much use to anyone. That’salso why proof is a compatibility suite and not a conformance suite. Youcan only conform to an official spec, and nREPL doesn’t have one yet.Checking ServersChecking servers is proof’s main job. It’s a single binary1 that talks toyour server over a socket, the same way any client would. It knows nothingabout the server’s internals and it doesn’t care what language the server iswritten in. What it does need is a profile - a small TOML file that tellsproof how to start your server and gives it a few snippets of code in theserver’s language (an idea I borrowed fromjupyter_kernel_test):name = "My server"[launch]command = ["my-server", "--port", "0"]port-pattern = 'nREPL server started on port (\d+)'[snippets.value]code = "(+ 1 2)"value = "3"Then you install proof and run it:$ go install github.com/nrepl/proof/cmd/proof@latest$ proof run my-server.tomlproof starts your server, runs about 80 checks against it and shuts it downafterwards. (You can also grab a binary from thereleases, if you don’t have Goaround.) Profiles for nREPL, Babashka, nbb, Basilisp, ClojureCLR, jank,let-go, dialtone and repartee come with proof, so proof run babashka justworks if you have Babashka installed.The most important decision in proof’s design is how it grades the checks. Acheck fails only when some client won’t work properly with your server (orwhen users will lose data), and the report tells you which client it is andlinks to the client code in question, pinned to a specific commit. If yourserver simply does something differently from the reference implementation,you get a warning instead. Here’s a failure from a run against Babashka1.13.226:FAIL interrupt.other-connection An interrupted eval ends on its own connection the interrupted eval's done went to the connection the interrupt came from why: neorepl interrupt stops an eval that another neorepl process started, and that process waits for the eval's done on its own connection forever if it goes anywhere else. see: neorepl interrupt stops an eval from another process https://github.com/nrepl/neorepl/blob/462a182c.../src/neorepl/main.clj#L374-L385 see: nrepl/nrepl#498 https://github.com/nrepl/nrepl/issues/498The idea is that a failure should always be worth fixing and never be amatter of taste. If you think some failure isn’t justified, the links show youexactly what you need to argue with.Some checks go a step further and send exactly what a particular clientsends, the way it sends it. cider.connect sends the requests CIDER sendswhile connecting, calva.eval evaluates code the way Calva does, and so onfor Conjure, vim-fireplace, REPLy, neorepl, clojure-cli.repl andbb repl --connect. That way the report tells you directly whether those clients willwork with your server.Once your server passes everything it can pass, you can list the failures youcan’t fix yet as expected failures in its profile and run proof in your CIwith its GitHub Action. When an expected failure starts passing,proof fails the build and asks you to remove it from the list, so the listcan’t go stale.Last, but not least, the bundled profiles run every day and their reports endup in a compatibility matrix, which showswhere the different servers agree, where they don’t and where the spec needssome work.Checking ClientsServers were always proof’s primary focus, but once the foundation was inplace people started asking me how they could check their clients with it.That made a lot of sense to me - after all, the same knowledge cuts both ways.proof has two tools for client authors.proof proxy sits between a client and a server, passes everything along asis and records it on the way. You use your client for a while (or run itstest suite through the proxy), press Ctrl-C and proof grades every requestthe client sent. Here’s a quick neorepl eval going through it:$ proof proxy -listen 127.0.0.1:7888 babashka$ neorepl eval --port 7888 '(println "hi") (+ 1 2)'client PASS client.bencode Every frame is valid bencode PASS client.dict Every request is a dict PASS client.canonical Frames use canonical bencode PASS client.id Every request has an id PASS client.active-id An id isn't reused while its request is active PASS client.field-types Request fields have the right types PASS client.required-fields Requests carry the fields their op needs PASS client.need-input need-input is answered with stdin in the same session WARN client.close Sessions are closed before disconnecting PASS client.unknown-op Requests use ops the server supports9 passed, 0 failed, 1 warnings, 0 skippedThe grading rule is the same, just turned around. A client check fails onlywhen some server breaks, and it links to the server code in question. (Thatwarning is fine, by the way. neorepl keeps its session around on purpose, sothe next neorepl eval can use it.)A proxy sees only what goes over the wire, though, so it can’t tell you howyour client copes with the replies of different servers. That’s whatproof serve is for. It’s a small nREPL server for client test suites, whichbehaves like nREPL unless you ask it to behave like some other server in someparticular way - send only the value of the last form (like Basilisp, jank anddialtone), drop what the code prints to stderr, have no interrupt op,deliver its replies a byte at a time, and so on. None of those scenarios aremade up. Every one of them is something a real server does (or something TCPcan do to any server’s replies). -like turns on all the quirks of a serverat once, so you can run your client’s tests against “jank” without installingjank:$ proof serve -listen 127.0.0.1:7888 -like jankproof in PracticeSo far, so good. But does it actually find anything? You bet! Here are someof the things proof turned up in the past couple of months: nREPL itself sent the end of an interrupted eval to the connection theinterrupt came from (fixed on master), drops the evals waiting behind aninterrupted one (#503) andcan leave a session that stops sending done after its client goes away inthe middle of an eval (#509).Oops! Babashka, nbb, Basilisp, jank and ClojureCLR all had their share of roughedges, e.g. errors without an eval-error status, sessions sharing *1 andthe current namespace, and a request that throws taking its whole connectiondown. CIDER skipped part of its setup when connecting to a server withoutcider-nrepl, Calva and Conjure didn’t sort the keys of the dicts they sent, vim-fireplaceand REPLy never close their sessions, and Conjure’s interrupts don’t saywhich eval to stop.Michiel Borkent fixed the Babashka and nbb issues the same day I reportedthem (or the next one), Conjure fixed its encoder the next day and Calvawithin a week. That’s the Clojure community for you!I was also super happy to see other people picking up proof before I even gotaround to announcing it. Tom Waddington has been running it against his nREPLimplementations for Janet andSteel Scheme since early October(nrepl-janet even tests its client with proof serve), Jeaye Wilkersonsuggested using it forjank’s nREPL server, and Kyle Cesare is alreadyfixing what it found there.That’s exactly what I was hoping for!The projects that benefited from proof the most were my own, though. Thisyear I built a few toy nREPL implementations, mostly for fun and to see howwell the protocol holds up outside of Clojure: nrepl-beam has nREPL servers forErlang (dialtone) and Elixir (repartee). proof found that they didn’t answerclose with session-closed, didn’t echo unknown ops and returned only thelast value of several forms, and fixing those uncovered a race after closeand temporary sessions that never stopped. mezcaml, my little OCaml client, readreplies as if only one request was ever in flight, so late output from oneeval could end the next one early. neat, my language-agnostic client forEmacs, got late output, errors without err and unknown sessions wrong. nrepl-sci, the nREPL server behindthe “Try it” section of nrepl.org, was the first server I wrote with proofaround from day one, and working onit mostly came downto running proof after every change.And then there’sneorepl,the CIDER-inspired terminal REPL I’ve been working on lately, which is just ashappy to be used by coding agents as by humans. neorepl and proof pretty muchgrew up together. I ran neorepl through proof proxy and every scenario ofproof serve, which caught a few bugs before neorepl 0.1 was out, and proofchecks every server with the exact requests neorepl sends. It’s also how Ifound the interrupt bugs in nREPL above, which hit neorepl harder than mostclients. Its commands are separate processes sharing a session, soneorepl interrupt stops an eval that some neorepl eval started, and thatneorepl eval used to wait for its done forever.A Tool for Tool AuthorsI guess I should spell out who proof is for. If you’re using nREPL through CIDERor Calva you’ll probably never run proof yourself (although hopefully you’llfeel its effects). proof is a tool for tool authors - people writing nREPLservers, clients and middleware.I really hope that it will make it a lot easier to grow the nREPL ecosystem.These days anyone can put together an nREPL server or client in no time withthe help of tools like Claude. Writing the code is no longer the hard part -knowing whether the result actually works with CIDER, Calva and friends is.proof gives you (and your coding agent) a fast feedback loop and failures thatexplain themselves: what went wrong, why it matters and which client codedepends on it. That’s usually all an agent needs to fix the problem.Why “proof”?Naming is hard! Luckily, the nREPL world has a tradition that makes it a biteasier, as many nREPL clients are named after drinks (CIDER, neat, mezcaml,chaser and so on). proof is not a client, so it bends the rule a bit, but it stillbelongs at the bar. The proof of a spirit is a measure of its strength, whichis more or less what this tool measures for nREPL servers. And running itgives you proof that your server works with the existing clients (or a list ofthe reasons why it doesn’t). Two meanings for the price of one!2Epilogueproof is still in its early days, but it already covers the core of theprotocol, the exact requests of the most popular clients and the server quirksclients have to deal with. Now it needs more users.So, if you’re working on an nREPL server or client, please take proof out fora spin and let me know how it went! I’d love to see your server in thecompatibility matrix, and if you think some failure isn’t fair, open an issueand let’s argue about it. And if you’re a casual Clojure hacker who’d just likea nice REPL in the terminal, give neorepl a try:$ bbin install io.github.nrepl/neoreplBig thanks to Clojurists Together for supporting my work on nREPL, to PhilHagelberg for getting the spec going, and to the maintainers of all theservers and clients who were so quick to fix the problems proof found. Yourock! In the (n)REPL we trust!That’s all I have for you today. Keep hacking! Written in Go, of all things. A Clojure tool would have been the obvious choice, but proof shouldn’t favour any of the languages it checks, and a single binary without a runtime is easy to drop into any CI. ↩ “hangover” would have been a good name for what you get when a server forgets to send done, but it’s a hard sell for a project. ↩