While I’m developing Chlorine, sometimes I need to test multiple specific implementations of lots of really complicated stuff like REPL state, connection, async stuff (as the project is ClojureScript on Node.JS, all I/O treatment is via callbacks) and the complicated nature of rendering multiple different object types on Atom editor. I had multiple regression errors, then I’ve created some “acceptance” tests (these open up a real Atom editor and evaluate commands on it). The problem with these tests should be obvious: they are slow, and I mean REALLY SLOW, and they need a real Atom editor, lots of setups, and because Atom is not really predictable on its actions (sometimes you connect REPL and it changes the focus on the editor) there are lots of unnecessary interactions on the editor just to have less false-negatives.
Now, as I told before, I’m developing Chlorine together with REPL-Tooling, a library that should contain all tooling for any editor capable of running Javascript to run a port or Chlorine. There’s still too much on Chlorine that relies on internal Atom APIs (for example, detection of the beginning and end of forms is one, detection of namespace’s forms is other) but time is passing and more and more is being moved to REPL-Tooling, and as soon as the detection of forms is on REPL-Tooling (and is stable) there should be possible to port most Clojure parts to REPL-Tooling, and then I can think on how to refactor the ClojureScript part, test it, and then Chlorine will be a very easy project to port. Also, as a proof-of-concept there’s Clematis for NeoVIM (it’s still on the very beginning and nothing much happened after I wrote about it here), and also an “electron fake editor” that I’m using as test.
Wait, What?
As I wrote before, it’s not really viable to keep testing Chlorine with only acceptance tests. Then, to avoid false-positives, I’ve created an electron project on REPL-Tooling, so that I could write tests on a better format (in this case, devcards).
On that project, there are three different kinds of tests: one that is “pure” (like parsers, detections of forms, etc), one that is “impure” (that connects to the REPL and tries to evaluate, break evaluation, and so on) and one that’s a “mini-integration”: it connects to a REPL too, but in this case there’s an UI (and the test interacts with this UI using CSS selectors, clicking buttons, checking results, and so on). Now, the reason for the UI is that I’m a firm believer that your tests should ease you on development of a feature. Using a real UI, with devcards, I can inspect the internal state the way I see fit, and I can interact with the UI without needing to write a test to it (I love writing tests, but the thing is, sometimes is not that easy to write then, specially if you need a test that will connect to a REPL, evaluate a form, render this result somewhere, then click on a link, click on another, and inspect if a change occurred – and yes, there are tests that need to go to that level of detail).
There were some specific problems on this approach: first is that async tests run… asynchronous on devcards… and this posed a problem because when some test connect to the REPL, it needs to disconnect on the end, and when another test would need to connect on the REPL, it was expecting to (1): have a blank state and (2): not being disconnected on the middle of the testcase. So, tests that need to connect to REPL need to be separated on different namespaces, and run serialized. The second problem is that the tests are being run in an UI, so a CI server would need to drive that UI, run all the tests on each namespace, collect failures, and get the right exit code… and I wanted my tests to run on a CI.
So, I wrote a little helper program. Using Specton, another node.js library, I was able to open up Electron, detect the links, click then and inspect the DOM to see if there are failures. It’s not the most beautiful solution, but it works and does give me a feedback on the console. In fact, it is so useful that I’m able to run it from my machine to run all tests. In fact, the whole code is less than 100 lines, so it’s easy to understand and modify. The best part is that I can keep working on an unfamiliar environment (Electron app, connecting to a REPL, running tests inside editors) the same way that I’m used to work on simpler environments (when there’s only a back-end that serves an API, etc). I also am able to publish REPL-Tooling after each merge to master to Clojars, so people can use it on their own projects. Also, the config for CircleCI was really simple because there already exists an image that have both Clojure and Node installed, and the script to run the CI commands is also not a big deal: it just starts a Shadow-CLJS compiler, waits for the compiled file to exist, then run the node.js helper script.
There are lots of rooms to improvement, and I really hope that this opens the door to contributing to Chlorine and future projects they will be born for it!