Chlorine, Clover and Clematis are all implementations of the same library: REPL-Tooling. In this post I will show you how to create a new implementation of it in a way that’s completely disconnected from any editors, so you can grasp the general concepts.
Suppose I want to do an implementation for an editor that doesn’t run JavaScript – so it’ll connect by some kind of socket. In this example I’m going to use WebSockets because… why not?
We’re going to create a shadow-cljs node project and add repl-tooling as a dependency. We will also had some more dependencies: mostly ws for websockets and the same react libraries that we use for reagent (react, create-react-class and react-dom) – repl-tooling still needs reagent, and probably in the future I will split it into two different libraries (one for the REPL handling and other for the visual rendering part). This supposedly is not to much of a problem because ClojureScript compiler will probably remove these parts in the dead code elimination process anyway. So, our package.json file will just be like this:
{
"name": "ws-repl",
"devDependencies": {
"shadow-cljs": "^2.8.83"
},
"dependencies": {
"create-react-class": "^15.6.3",
"install": "^0.13.0",
"react": "^16.12.0",
"react-dom": "^16.12.0",
"ws": "^7.2.1"
}
}
And our shadow-cljs.edn file:
{:source-paths ["src"]
:dependencies [[repl-tooling "0.4.0"]]
:builds {:node {:output-to "index.js"
:target :node-script
:main ws-repl.core/main}}}
The first step is when someone connects to the WebSocket. Then, we’ll just create a connection to the client, and send a list of supported commands – for now, is just the “connect” command:
(ns ws-repl.core
(:require [clojure.reader :as edn]
[repl-tooling.editor-integration.connection :as conn]
["ws" :refer [Server]]))
(def server (new Server #js {:port 8000}))
(defn- send! [^js ws edn]
(.send ws (pr-str edn)))
(defn ^:dev/after-load main []
(. server on "connection"
(fn [ws]
(send! ws
[:commands [:connect-socket]))))
(defn ^:dev/before-load reload []
(.close server))
When the client sends a “connect”, we’ll issue a connection to the socket repl. And this is where things become interesting:
(defn- register-connect! [ws command]
(let [[cmd host port] (edn/read-string command)]
(when (= cmd :connect-socket)
(.. (conn/connect! host port {:on-stdout #(send! ws [:out %])
:on-stderr #(send! ws [:err %])
:editor-data #(,,,,,,)
:get-config (constantly
{:project-paths "."
:eval-mode :prefer-clj})
:notify #(send! ws [:notify %])})
(then #(send! ws [:commands (-> @% :editor/commands keys)]))))))
(defn ^:dev/after-load main []
(. server on "connection"
(fn [ws]
(.on ws "message" #(register-connect! ws %))
(send! ws [:commands [:connect-socket]]))))
We send a connect! to the REPL, and register lots of callbacks. These all are used to integrate with any editor – the idea is that when some command is issued, if it needs some data from the editor (or provoke some changes on the editor), these callbacks will be sent. In this case, we’re just sending then to the WebSocket (except for get-config – this needs to be synchronous).
Now, we can test it. Let’s open up a Babashka socket repl (because, again, why not?) with the following command (I’m using websocat to connect and issue commands to the websocket). Also, I’ll put > in front of every string that needs to be sent to the websocket, and a < in every string that comes back – when you’re testing this code, do not put these characters otherwise it’ll not work:
$ websocat ws://localhost:8000
< [:commands [:connect-socket]]
> [:connect-socket "localhost" 3333]
< [:notify {:type :info, :title "Babaska REPL Connected"}]
< [:commands (:evaluate-top-block :evaluate-block :evaluate-selection :disconnect :doc-for-var :load-file)]
< [:out "\n"]
< [:out "\n"]
< [:out "user=> "]
< [:out "user=> "]
As you can see, there’s already a bunch of things going on: you’ll get a :notify, telling you that it’s connected. There are already a bunch of commands available for you to run, and there’s also a bunch o :out commands (things that were printed from Babashka). Now, to simulate an evaluation, you can register every command that the websocket sent to you, and route then to what REPL-Tooling commands. For the sake of simplicity, I’ll just create an atom with all commands that we support (by default, only connect-socket and editor-data) and when the connection comes, this atom will be updated with the other supported commands. There also should be some data that represents the “current editor state”, so when we start to evaluate something, REPL-Tooling will know what’s written in our “editor”.
So, the updated code will be something like this: first, change the main code to register a command to set the editor data; then, we change the register-connect! to dispatch!, and then use the local atom to dispatch every command that we support to the specific command on REPL-Tooling. We’ll also keep the atom prepared with our two initial commands, so we don’t need to make any code to treat these two specific cases:
(defn- dispatch! [ws state command-str]
(let [[cmd & args] (edn/read-string command-str)]
(when-let [fun (get-in @state [:commands cmd :command])]
(apply fun args))))
(defn ^:dev/after-load main []
(. server on "connection"
(fn [ws]
(let [state (atom nil)]
(reset! state
{:editor-state {}
:commands
{:editor-data
{:command #(set-editor-data state %1 %2 %3)}
:connect-socket
{:command #(connect! ws state %1 %2)}}})
(.on ws "message" #(dispatch! ws state %))
(send! ws [:commands [:connect-socket :editor-data]])))))
Now, we’ll add the connect! and set-editor-data commands:
(defn- set-editor-data [state filename contents range]
(swap! state update :editor-state assoc
:filename filename
:contents contents
:range range))
(defn- send-res! [ws {:keys [result]}]
(send! ws [:result {:res (:as-text result)
:error? (contains? result :error)}]))
(defn- connect! [ws state host port]
(.. (conn/connect! host port {:on-stdout #(send! ws [:out %])
:on-stderr #(send! ws [:err %])
:on-eval #(send-res! ws %)
:editor-data #(:editor-state @state)
:get-config (constantly
{:project-paths "."
:eval-mode :prefer-clj})
:notify #(send! ws [:notify %])})
(then (fn [res]
(swap! state
update :commands
merge (-> @res :editor/commands))
(send! ws [:commands (-> @res :editor/commands keys)])))))
In the code above, there’s also code to answer for :on-eval (that just sends to the WebSocket a :result with the contents of our evaluation result) and :editor-data (that will just deref our atom). Also, the :range parameter is the 0-based [[<begin-row> <begin-col>] [<end-row> <end-col>]] that identifies a position on our editor.
Using the same format of before, we can connect to Babashka again, and send something for evaluation:
$ websocat ws://localhost:8000
< [:commands [:connect-socket :editor-data]]
> [:connect-socket "localhost" 3333]
< [:notify {:type :info, :title "Babaska REPL Connected"}]
< [:out "\n"]
< [:commands (:evaluate-top-block :evaluate-block :evaluate-selection :disconnect :doc-for-var :load-file)]
< [:out "user=> "]
> [:editor-data "foo.clj" "(+ 1 2)" [[0 0] [0 0]]]
> [:evaluate-top-block]
< [:result {:res "3", :error? false}]
> [:editor-data "foo.clj" "(/ 2 0)" [[0 0] [0 0]]]
> [:evaluate-top-block]
< [:result {:res "#error {\n :cause \"Divide by zero\"\n...", :error? true}]
We can also send any other commands we want:
> [:editor-data "/tmp/foo.clj" "(/ 2 0)" [[0 0] [0 0]]]
> [:load-file]
< [:notify {:type :info, :title "Loaded file", :message "/tmp/foo.clj"}]
And, again, it’ll answer with the info we expect. Please notice that all commands that REPL-Tooling supports will rely on the internal state that’s set from the editor integration, so they’ll accept no parameters and will give their results probably by some registered callback (in this case, it just sends a notification that the file was loaded sucessfully).
Is that all?
Incredibly, yes, it is all. That’s one of the reasons I wrote repl-tooling – to make it easy to integrate editors. Now, some things are not that simple: for example, maybe you need to evaluate an infinite list (let’s change our REPL to Clojure, that’s the only one that supports this kind of evaluation):
< [:commands [:connect-socket :editor-data]]
> [:connect-socket "localhost" 4444]
< [:notify {:type :info, :title "Clojure REPL Connected"}]
< [:commands (:evaluate-top-block :evaluate-block :evaluate-selection :disconnect :doc-for-var :load-file :break-evaluation :connect-embedded)]
> [:editor-data "/tmp/foo.clj" "(range)" [[0 0] [0 0]]]
> [:evaluate-top-block]
< [:result {:res "(0 1 2 3 4 5 6 7 8 9 {:repl-tooling/... (unrepl.repl$XkQisH7kdleWD1h1jn$ei_rxBPk/fetch :G__27941)})", :error? false}]
The result is kinda strange… you’ll probably need to understand UNREPL’s results to understand it (I just changed the unrepl/... to :repl-tooling/...). But the idea is that the result is a list from 0 to 9, and then it have a command (this unrepl.repl$.... one) that can be sent to the REPL and it’ll return more elements from that list.
Now, to integrate this kind of tooling is not that simple: on vscode, for example, the place that renders the result can’t evaluate on the REPL. So, there’s some “serializing, de-serializing”, and a little dance to make these things work. On Atom, things are easier.
And that wraps it up. There’s still room for lots of improvements on these toolings, but for now, I’m really satisfied on how things are coming together!