nREPL on… Chlorine?

When I started the Chlorine project, I just thought it would be great if I could target all Clojure-like REPLs that already exist but didn’t have tooling support. At the time, this would include Lumo and Plank, mostly. Also, Shadow-CLJS and Figwheel have some “clunky run-some-code-and-transform-in-cljs” way of working that simply didn’t click with me.

Now, almost a year later, Chlorine supports Clojure, ClojureScript (Shadow-CLJS, Lumo, Plank, or even over clj), ClojureCLR, Arcadia, Babashka, Clojerl (Clojure on Erlang) and Joker (Clojure on Go, also a linter). But the reality is that working with a pure Socket REPL is really hard – a socket REPL works exactly like a regular one, printing namespaces after each code, and so on. Also, there are some strange decisions on some REPLs, mostly likely ClojureScript (that is the second most used Clojure flavor), so things are not always easy. To put things in perspective, currently Chlorine uses 3 ways to evaluate code: It uses unrepl, that only works on Clojure, or uses internal APIs of shadow-cljs (that obviously only works for shadow-cljs), and for other implementations it uses a kind of a hack – it evaluates the code, inside a trycatch, and it returns a vector where the first element is a symbol in a specific format that Chlorine will understand and then link that with the response. This “hacky way” is currently being used for every other implementation except Clojure and Shadow-CLJS. Things work (autocomplete works too), but it is not pretty and sometimes have strange results.

As a matter of fact, I was already thinking about removing UNREPL (it’s really hard to implement new features on it, and some good ideas only work in theory – for example, the ability to evaluate long strings / collections and render only a part at a time aren’t that good with lots of edge-cases) and, to do it, I though about a better, non-hacky way to evaluate things on some Socket-REPLs (that, again, would only work on some REPLs – ClojureScript REPLs will probably never support “upgradable REPLs” because of the way they work) – the only thing that I had to understand is how to implement this “upgraded REPL”…

Then, recently, Babashka added an initial support for nREPL, with an insane low amount of lines. So, I’ve tried to implement a way to evaluate code over nREPL… and it was really simple to do it, using a npm library that already did it. But implementing like this meant that the user would need to know if the host/port to connect is a Socket REPL, or a nREPL (and the user does not know – lots of tools like lein and shadow-cljs show an nREPL port to be connected).
(more…)

REPL-Tooling Clients

Chlorine, Clover and Clematis are all implementations of the same library: REPL-Tooling. In this post I will show you how to create a new implementation of it in a way that’s completely disconnected from any editors, so you can grasp the general concepts.

Suppose I want to do an implementation for an editor that doesn’t run JavaScript – so it’ll connect by some kind of socket. In this example I’m going to use WebSockets because… why not?

We’re going to create a shadow-cljs node project and add repl-tooling as a dependency. We will also had some more dependencies: mostly ws for websockets and the same react libraries that we use for reagent (react, create-react-class and react-dom) – repl-tooling still needs reagent, and probably in the future I will split it into two different libraries (one for the REPL handling and other for the visual rendering part). This supposedly is not to much of a problem because ClojureScript compiler will probably remove these parts in the dead code elimination process anyway. So, our package.json file will just be like this:

{
  "name": "ws-repl",
  "devDependencies": {
    "shadow-cljs": "^2.8.83"
  },
  "dependencies": {
    "create-react-class": "^15.6.3",
    "install": "^0.13.0",
    "react": "^16.12.0",
    "react-dom": "^16.12.0",
    "ws": "^7.2.1"
  }
}

And our shadow-cljs.edn file:

{:source-paths ["src"]

 :dependencies [[repl-tooling "0.4.0"]]
 :builds {:node {:output-to "index.js"
                 :target :node-script
                 :main ws-repl.core/main}}}

The first step is when someone connects to the WebSocket. Then, we’ll just create a connection to the client, and send a list of supported commands – for now, is just the “connect” command:
(more…)

My Atom editor configuration for working with Clojure/Script, revisited

Sometime ago, I did a post on how I work with Atom to develop Clojure and ClojureScript projects. It is in Portuguese, so I’m gonna re-visit the subject and also update with my current workflow.

There are two packages that I have to install to work with Clojure: lisp-paredit and chlorine. Without lisp-paredit, when I start a newline, the indentation gets all sorts of problematic. I use it on “strict mode” and use the tools to slurp/barf forward only. As for chlorine, it is needed to have autocomplete, evaluation, show documentation, goto var definition and so on. Last, I use also parinfer so I can remove whole lines of text and parinfer will infer the correct closing of parenthesis for me (most of the time at least).

Now, how exactly do I work with Clojure? When you use lein or boot, you’ll get a nREPL port. This is not the port you use with Chlorine, so I need a bit more of work. I can’t just start a REPL with lein repl or clj, I need to inform the tool to open a socket-repl server. The JVM option needed is: '-Dclojure.server.myrepl={:port,5555,:accept,clojure.core.server/repl}'. So, the commands below are what I use with lein or clj:

JAVA_OPTS='-Dclojure.server.myrepl={:port,5555,:accept,clojure.core.server/repl}' lein repl

or

clj -J'-Dclojure.server.myrepl={:port,5555,:accept,clojure.core.server/repl}'

This will open a REPL at port 5555 (or I can change the port if necessary). Then, it’s time to fire up the Atom’s command palette and select “Connect Clojure Socket REPL”, put 5555 on the port, and connect. Then, I’ll use “Refresh Namespaces” or “Load file” command to load my latest version of code into the REPL, and start working.
(more…)

Putting REPL-Tooling to test – VSCode!

Sometimes you make some tooling because you want to use it; sometimes, to experiment; and sometimes to test the waters.

The last tooling in that I did was one of these cases – now there’s a port of Chlorine to Visual Studio Code called Clover!

Now, when I started the project I imagined that VSCode would not have all the features that I have in Atom, nor all the APIs that I want to use – for example I didn’t have any hope of having the inline results in this version.

The thing is, I wasn’t expecting it to be so bad! To begin with, the API: is not really that bad documented, but compared to the Atom, it is incredibly weak. The first thing is that they expect you to use TypeScript so there’s little to no documentation on how to represent objects in pure JavaScript; for example in some cases you can use pure objects, on others you have to instantiate a TypeScript class in the JavaScript code. Also, there are multiple parts of the documentation when they just give you the type signatures and little (or even no) explanation (and let’s make a little detour here: what’s the deal with some people that use static types, that they expect you to understand how any API work just by showing the types of the functions?).

The second part is that VSCode expects you to fit your plug-in infrastructure on what they offer – so, some functionalities will land on the “peek definition” API, others on “Code Lens” and so on. The problem is that they don’t explain what’s a “code lens” for example nor give you any screenshots of the functionality in action – mostly the documentation is some code examples in GitHub repositories so you have to download, install the example extension on your machine and then run it to simply understand how something works.

The second hard part is that you can’t test the API in the devtools – in fact the devtools is almost useless because when you have an error, the stacktrace will point you to a minified JS code in the VSCode internal API. Also, some exception messages are completely obscure and some log errors on the devtools but things work fine on the editor. Well, to summarize: it’s completely useless to depend on the errors.

But the worst part, at least for REPL-Tooling, is that you can’t change the UI of VSCode in any way – and this means no pop-ups, no new elements, no console in the editor, nothing – the only way you have that you can extend the visual components is by implementing a webview – and by webview I am not saying an “electrom webview” when you can access all the Node.JS APIs – I am saying a simple web page when you have to pass your data to and from the editor using JSON. And and that’s all there is – no Date, no JavaScript classes and, of course no Clojure objects.
(more…)

Complex testing – the saga

While I’m developing Chlorine, sometimes I need to test multiple specific implementations of lots of really complicated stuff like REPL state, connection, async stuff (as the project is ClojureScript on Node.JS, all I/O treatment is via callbacks) and the complicated nature of rendering multiple different object types on Atom editor. I had multiple regression errors, then I’ve created some “acceptance” tests (these open up a real Atom editor and evaluate commands on it). The problem with these tests should be obvious: they are slow, and I mean REALLY SLOW, and they need a real Atom editor, lots of setups, and because Atom is not really predictable on its actions (sometimes you connect REPL and it changes the focus on the editor) there are lots of unnecessary interactions on the editor just to have less false-negatives.

Now, as I told before, I’m developing Chlorine together with REPL-Tooling, a library that should contain all tooling for any editor capable of running Javascript to run a port or Chlorine. There’s still too much on Chlorine that relies on internal Atom APIs (for example, detection of the beginning and end of forms is one, detection of namespace’s forms is other) but time is passing and more and more is being moved to REPL-Tooling, and as soon as the detection of forms is on REPL-Tooling (and is stable) there should be possible to port most Clojure parts to REPL-Tooling, and then I can think on how to refactor the ClojureScript part, test it, and then Chlorine will be a very easy project to port. Also, as a proof-of-concept there’s Clematis for NeoVIM (it’s still on the very beginning and nothing much happened after I wrote about it here), and also an “electron fake editor” that I’m using as test.

Wait, What?
(more…)

Chlorine is thriving!

When I started this project, I was experimenting with shadow-cljs to see if I would be able to make an Atom package that would auto reload, run tests on ClojureScript, and se how far could I push ClojureScript in an Atom package.

Now, some months later, I’m seeing the package being used by a bunch of people, and I even discovered some bugs in UNREPL! Now, on this post, I’ll discuss a little bit more in detail the design decisions and my vision on the future of the project.

Chlorine is a clojure and ClojureScript atom package. It connects to a socket repl (opened via lein, boot, clj, shadow-cljs, lumo, or even REBL) and then upgrades the repl to be programmatically oriented with unrepl. Unrepl only works on Clojure, so for ClojureScript we use other techniques. Also, socket repl is a stream protocol, so to emulate a “request-response”, we need to coordinate things so Atom (and other editors) can react to commands and know exactly what’s the correct response for each command sent.

Design decisions

The choice for UNREPL was mostly because there is almost no documentation about prepl so far. Also, Socket REPL is literally everywhere: on Clojure , on ClojureCLR, on Lumo and Plank. Also, I wanted a better way to use ClojureScript, and I still have nightmares trying to use it over nREPL… and with Socket REPL things work fine.

Also, when I started the project Clojure 1.10 was just alpha software, and UNREPL offers us insanely good support for lazy lists, big strings, and other things that I wanted to use out of the box. One of the problems I’m still facing is coordination of evaluate/response, but this will probably be solved after a bunch of other fixes I’ll try.
(more…)

My fading frustration with ClojureScript

I’ve talked about at another post on how ClojureScript frustrates me, mostly because I was doing some Node.JS work and Figwheel simply wasn’t working correctly. Now, it’s time to revisit these points:

A little update: I talked a little with Thomas Heller, Shadow-CLJS creator, and he pointed me some issues with this article, so I’ll update it acordingly

Tooling

Figwheel and Lein are not the best tools to work with ClojureScript. Since I discovered shadow-cljs, things are working way better than before: I can reload ClojureScript code from any target, and I’m even experimenting with Hubot (and it works really fine too). The only thing I’m missing is my profiles.clj file, but I can live with that (and I can always use Shadow with Lein if I need profiles.clj).

Also, I’m working on a new package for Atom (and in the future, for another editors too) called Chlorine. One of the ideas is to offer better ClojureScript support (we have Autocomplete now!), using Socket REPL for solutions (even self-hosted REPLs like Lumo and Plank) and even wrap UNREPL protocol in Clojure. So far, is still in the very beginning but things are looking promising!

The stack

Forget Figwheel at all: Shadow-CLJS is probably the best tooling for ClojureScript ever. It auto-reloads ClojureScript code for the browser, for node.js, for node modules, and it integrates with almost everything that you want. It controls release optimizations, have sensible defaults, and even have post-compile hooks (so you can hook Clojure code to do something after some compilation phases). Also, it integrates with node-modules (no more maven-wrappers for JS libraries!) and have some warnings when you use some kind of ClojureScript code that would break :advanced compilation. And, let’s not forget that you can control the refresh reload phase, it adds a userful :include-macros in ns form (that will include all macros from the namespace being required), and controls exports in a sane manner. But first let’s begin with the feature that I found most useful: :before-load-async.
(more…)