The power of Pathom

Most people don’t know the power of Pathom. Most people that I know of think that Pathom is about graphs.

They are wrong.

I mean, yes, in the end you’ll have a graph, with dependencies, but that’s not the point. The point of Pathom is the ability to transform your code into a soup or attributes. It’s also probably the best usage of qualified keywords for me, if not the only one that justifies the downsides (I’ll not enter in details here – just know that having to convert from qualified keywords to unqualified multiple times is not fun at all).

The name “soup of attributes” may not be a beautiful one, but believe me – it’s incredible. The idea is quite simple – instead of trying to handle all possible conversions from multiple sources to multiple destinations, you just define which attributes can be computed in terms of others, and Pathom does the rest. As always, things are better with examples, so let’s go.

I had to work on a system that somehow had strange rules – it needed to generate a bunch of text files for different companies. Each company expected a different file name and different fields. To generate the file, we had to accumulate data that came from a payload, from an external system that we called via REST API, and also from some data we had on our database. To make things worse, some companies would expect some of the data that was returned from REST on the filename, and there were also some state changes – like, if a file was already processed, the company would send us a return file, and we had to read some content of this file, move this file to another directory, renaming the file in the process.

Implementing shadow.remote API

Since version 0.8.0 of Chlorine, there’s a new way to evaluate ClojureScript code: that’s the Shadow-CLJS Remote API. It is basically a new REPL (not nREPL, no Socket REPL) over WebSockets to try to solve problems when translating other REPLs to ClojureScript. So, to understand why these problems exist, I’ll first introduce the difference between ClojureScript and Clojure.

On Clojure, you’re always inside a JVM. This means that compilation happens on the same JVM that your REPL, and your code is running. If you practice REPL-Driven Development, even your tests are running on the same JVM. In practical terms, it means that when you fire up your REPL, you already have everything ready to run code, compile code, and evaluate forms.

On ClojureScript, the compiler is written in Clojure – that means it’s running on the JVM. So, to produce Javascript code you don’t need a Javascript environment – and that’s when things become confusing, because when exactly will you run the REPL? Let’s try from another angle: if you start the REPL on compilation time, you can’t evaluate code (because there’s no Javascript generated, nor any Javascript engine running). If you start the REPL when you run the compiled code, this REPL can become unusable if you stop the Javascript environment, and also you have to coordinate lots of state and translations between formats.

Reagent Mastermind

One of these days, a friend of mine posted about his experience writing the “Mastermind” game in React (in portuguese only). The game is quite simple:

  1. You have six possible colors
  2. A combination of 4 colors is chosen randomly (they can be repeated – for example, blue,blue,blue,blue is a valid combination) – you have to guess that number
  3. You have up to 10 guesses of 4 colors. For each color on the right position, you win a “black” point. For each color in the wrong position, you win a “white” point
  4. If you can’t guess on the 10th try, you loose.

So, first, we’ll create a shadow-cljs app – create a package.json file, fill it with {"name": "Mastermind"}, then run npm install shadow-cljs. Traditional stuff.

Then, we’ll create the shadow-cljs.edn file. It’ll only contain a single target (:browser), opening up a dev-http server so we can serve our code, and we’ll add reagent library dependency. I also added the material-ui dependency, but you don’t really need it for the code. Now, running npx shadow-cljs watch browser will start a webserver at port 3000, and we can start to develop things.

ClojureScript vs clojure.core.async

I’m going to make a somewhat bold statement: core.async does not work with ClojureScript. And, in this post, I’m going to show some examples why this is true, at least for the current versions of core.async.

So let’s start by understanding a little bit about the runtime: Javascript is a single-threaded runtime that implicitly runs an event-loop. So, for example, when you ask to read a file, you can do it synchronously or asynchronously. If you decide to run in that asynchronously, it means that as soon as you issue the fs.readFile command, you need to register a callback and the control is returned to the “main thread”. It’ll keep running until it runs out of commands to execute, then the runtime will wait the result from the callback; when it returns, the function that you registered will be called with the file contents. When the function ends, the JS runtime will await to see if there’s any other pending call, and it’ll exit if there’s nothing else to do.

The same thing happens in browser environment, but in this case the callbacks are events from the DOM: like clicking on buttons or listening for changes in some elements. The same rules apply here: the runtime is single threaded and when something happens it will first execute everything that needs to be executed, then it will be called back with the event that happened.

So maybe we can change these callbacks with core.async channels right? But the answer is no, because core.asyncs go blocks will not run in different threads (because, again, the runtime is single-threaded). Instead, it creates a state machine and it’ll control of when each of these go blocks will be called, at what time, eventually replacing the event-loop that Javascript environment already have.

The History of Chlorine

Before I even started with Clojure, I was analyzing LightTable – the idea of that editor was to support better integrations between the code you’re writing and the code that’s running. It was a really good experience, but the main problem I had is that it was in the very beginning, with few plug-ins and bad documentation. I tried to make the parinfer plug-in work in the editor, but it had lots of bugs and then I simply changed back to Atom. At the time, proto-repl was the best package to work with Clojure, and I made some small changes to it (so I could add some callbacks to when a new connection to nREPL was made, and other small issues) to improve my workflow.

Fast forwarding a little bit, I started my first Clojure job at Nubank. Most people were using InteliJ, but I felt that by using Atom I had a different approach on problem solving, specially those hard parts where the fast feedback of “run in the REPL and see the results in your editor, then browse over the keys” could give better insights about what’s happening. I tried to implement some features that proto-repl didn’t have at the time (and Chlorine still does not have some) like “automatically add nREPL port”, and “watch expressions” (almost the same as watch variables in a debugger). These ended up in a package called clojure-plus, that still exist today.

I also began to experiment with ClojureScript (at the time, only Figwheel was available – Figwheel-main didn’t even exist!) and found that existing tools didn’t provide the same power that I had with Clojure. It also didn’t have autocomplete, goto var definition, and so on. To ease a little bit these problems, I ended up adding on clojure-plus some CLJS support – when you tried to evaluate a .cljs file, it would try to connect to ClojureScript, reserve a REPL, then evaluate the code over there.

nREPL on… Chlorine?

When I started the Chlorine project, I just thought it would be great if I could target all Clojure-like REPLs that already exist but didn’t have tooling support. At the time, this would include Lumo and Plank, mostly. Also, Shadow-CLJS and Figwheel have some “clunky run-some-code-and-transform-in-cljs” way of working that simply didn’t click with me.

Now, almost a year later, Chlorine supports Clojure, ClojureScript (Shadow-CLJS, Lumo, Plank, or even over clj), ClojureCLR, Arcadia, Babashka, Clojerl (Clojure on Erlang) and Joker (Clojure on Go, also a linter). But the reality is that working with a pure Socket REPL is really hard – a socket REPL works exactly like a regular one, printing namespaces after each code, and so on. Also, there are some strange decisions on some REPLs, mostly likely ClojureScript (that is the second most used Clojure flavor), so things are not always easy. To put things in perspective, currently Chlorine uses 3 ways to evaluate code: It uses unrepl, that only works on Clojure, or uses internal APIs of shadow-cljs (that obviously only works for shadow-cljs), and for other implementations it uses a kind of a hack – it evaluates the code, inside a trycatch, and it returns a vector where the first element is a symbol in a specific format that Chlorine will understand and then link that with the response. This “hacky way” is currently being used for every other implementation except Clojure and Shadow-CLJS. Things work (autocomplete works too), but it is not pretty and sometimes have strange results.

As a matter of fact, I was already thinking about removing UNREPL (it’s really hard to implement new features on it, and some good ideas only work in theory – for example, the ability to evaluate long strings / collections and render only a part at a time aren’t that good with lots of edge-cases) and, to do it, I though about a better, non-hacky way to evaluate things on some Socket-REPLs (that, again, would only work on some REPLs – ClojureScript REPLs will probably never support “upgradable REPLs” because of the way they work) – the only thing that I had to understand is how to implement this “upgraded REPL”…

Then, recently, Babashka added an initial support for nREPL, with an insane low amount of lines. So, I’ve tried to implement a way to evaluate code over nREPL… and it was really simple to do it, using a npm library that already did it. But implementing like this meant that the user would need to know if the host/port to connect is a Socket REPL, or a nREPL (and the user does not know – lots of tools like lein and shadow-cljs show an nREPL port to be connected).

REPL-Tooling Clients

Chlorine, Clover and Clematis are all implementations of the same library: REPL-Tooling. In this post I will show you how to create a new implementation of it in a way that’s completely disconnected from any editors, so you can grasp the general concepts.

Suppose I want to do an implementation for an editor that doesn’t run JavaScript – so it’ll connect by some kind of socket. In this example I’m going to use WebSockets because… why not?

We’re going to create a shadow-cljs node project and add repl-tooling as a dependency. We will also had some more dependencies: mostly ws for websockets and the same react libraries that we use for reagent (react, create-react-class and react-dom) – repl-tooling still needs reagent, and probably in the future I will split it into two different libraries (one for the REPL handling and other for the visual rendering part). This supposedly is not to much of a problem because ClojureScript compiler will probably remove these parts in the dead code elimination process anyway. So, our package.json file will just be like this:

  "name": "ws-repl",
  "devDependencies": {
    "shadow-cljs": "^2.8.83"
  "dependencies": {
    "create-react-class": "^15.6.3",
    "install": "^0.13.0",
    "react": "^16.12.0",
    "react-dom": "^16.12.0",
    "ws": "^7.2.1"

And our shadow-cljs.edn file:

{:source-paths ["src"]

 :dependencies [[repl-tooling "0.4.0"]]
 :builds {:node {:output-to "index.js"
                 :target :node-script
                 :main ws-repl.core/main}}}

The first step is when someone connects to the WebSocket. Then, we’ll just create a connection to the client, and send a list of supported commands – for now, is just the “connect” command:

Ubiquitous interface – how to integrate things in Clojure

Inspired by this thread on Reddit, I decided to write a little bit about my experience integrating things in Clojure.

The first thing to understand is that Clojure have an ubiquitous interface: EDN. And it is important to understand what this means. In the beginning, I made this mistake of “Death By Specificity” on my now abandoned Relational project: to abstract things that don’t need to be abstracted.

But can we do even better? How about we de-abstract (concretize? Is this a real word?) things that are already abstracted?

(check (my-code) => (needs :tests))

So, yesterday I made a talk (in Portuguese only, unfortunately) about the difficulties of testing Clojure and ClojureScript code. Specifically, I think the most problematic issue is the lack of “custom matcher libraries”, and how the default error messages are kinda bad and don’t help you identify the problems.

Then, on Clojurians’ #announcements Slack channel, I found that clojure.test Expectations library have a new version. So, why not integrate it on my Check library, and maybe continue developing it?

What is check?

Midje is too magic. Clojure.test is too little. Thinking about findind a “middle ground” I’ve started the “check” project, and I’m using it to test my personal projects like Chlorine, Clover, REPL-Tooling and Paprika. The problem is that, while the API is stable, but it still doesn’t do all the things I want.

Why I tend to avoid core.async?

It’s no surprise that I don’t like core.async very much. For starters, it make my functional composition looks like imperative programming again. There’s also multiple issues that you need to be aware of (like, don’t use async/put! because you will have problems, deadlocks that are difficult to predict, go blocks don’t compose over functions so you loose lots of helper macros like delay).

But the most important reason is that most of the time, I’m working in ClojureScript. And it’s impossible to migrate callback to core.async.

Well, you may be tempted to write something like:

(js/someFunction "i'm async" "lol" #(async/put! some-channel %))

And one day or another you’ll have the dreadful Assert failed: No more than 1024 pending puts are allowed error. There are multiple ways around this problem, but none of then work if you can’t lose messages.