Atom, community, and Pulsar

Okay, so recently I wrote about the Atom editor and how the community took efforts to revive the editor after the sunset announcement from Microsoft.

By now, things have changed a little bit. There was some disagreement on how the project should be handled from the atom-community maintainers: few people that were on the leadership decided that we should make “graceful changes” – things that do not disrupt too much the editor, small PRs that can be reviewed, and lots of stability. And I actually agree – that’s the way a project should be handled!

Except there’s a big catch: for me, this only works when the project is alive. The Atom editor is dead, for better or for worse. It’s announced already, so there’s no way “graceful changes” will be able to keep the editor alive or even relevant than, for example, VSCode – even for people that want a lightweight alternative or have want a different approach on handling code.

There were other disagreements on the roadmap – for some people, they wanted Atom to have the same functionality that it have today. And, speaking for me, I don’t agree – I think there are functionalities on the the Atom editor that are simply too much: for example, native support for CoffeeScript and TypeScript. I firmly believe that all Atom/Pulsar plug-ins should be transpiled – and that’s not only because the editor should only run JavaScript, it because it’s more reliable that way (the native support can fail between different tooling, language versions, etc. As an example, Atom supports plug-ins written in CoffeeScript 1.12 – and the current version is 2.7.0). Also, if a plug-in is bundled in a single JavaScript file, it’s actually more efficient both in terms of memory and performance for the editor.
(more…)

A sun sets, a pulsar is born

Recently Microsoft announced that the destruction of the Atom editor – I already posted about that, but I want to say that the community response for the sunset was really wonderful – in the beginning, I really thought that the Atom editor would die with the Microsoft announcement. But after posting on a lot of channels, and a very organized “call for arms”, we were able to organize ourselves and create something wonderful – the atom-community is now more active than ever, and indeed there is work being done right now.

People decided join our discord servers; we are reimplementing the API that will be discarded by Microsoft, and modernizing the editor like, for example, bumping tree-sitter and Electron (we now can run Atom on Electron 12); we also will need to rebrand it, and the name chose – Pulsar – could not be more fitting: it’s easy to remember, it basically means a “star that dies, but starts to spin faster and give bursts of radiation”.

I also was able to somehow bootstrap the editor without the original “bootstrap script” from Atom (that is famous to not work correctly, and also need an older version of Node.JS). I was also able to build binaries of Atom with “electron builder” – it’s library from the electron community to build the binaries instead of the way Atom do today (that is a bunch of scripts).

So this makes 5000 lines of code less that we have to keep of Electron scripts – except for the fact that there’s actually a lot of things in Atom that depends on things that these 5k lines do – like some plug-ins that misbehave when you don’t run the scripts, and all the test code that currently will not even run if you don’t bootstrap things.

So what this means for the Saturn editor? Is it the death of the product, like, will Pulsar will be basically the editor that I want to have? And unfortunately, it seems the answer for that question is “no”.
(more…)

Can I split the Atom and, from the parts, generate another thing?

So… my last post was somewhat sad – I tell about how impossible is to keep the Atom editor. Unfortunately, it still holds true – it’s close to impossible to handle the insane amount of code that Atom have.

But maybe is there a different way? So I decided to try: presenting, Project Saturn: a huge editor, but less dense. That’s the idea.

Saturn always fascinated me, and still fascinates: it’s a huge planet, but less dense than water. With a sufficient big bowl of water, Saturn would float. That’s precisely the idea I want in this new editor: to be less dense to keep, while somehow maintaining the power and “wow factor” of the editor. So, how to do it?
(more…)

The impossibility to maintain Atom

Atom Editor was, and still is, an amazing piece of technology. It was the first practical example of web technologies running locally, applied to a really hard problem: text editors. It it was, and still is, darn good at handling code.

But it’s hard, close to impossible, to maintain as the way it is. I want you to take all my opinions with a grain of salt, because these represent my own ideas and feelings on the project – I’m not a full-time Atom developer, nor I intend to be one. I just closely followed its development cycle, helped find bugs and problems in the first editions, and I still am in love with the editor, even when it’s clearly dying.

And while I would love to modernize the editor, making some PRs and fixing some issues, it’s close to impossible to do it. I believe the problem lies on the fact that Atom was an editor, and then Electron was extracted from it. I made the same choice on Chlorine too: I first made the plug-in, then extracted REPL-Tooling from it. Even on a WAY SMALLER codebase (Chlorine), this was HARD: there are still internal, private data that is used inside Chlorine that is not on REPL-Tooling.

Atom is the same. The “setup project” for Atom (prepare Electron, parse cmdline args, etc) is INSANELY huge. There are insane cyclic dependencies (TextEditor depends on components, UI, etc, and these depends back on the editor), there are NPM modules that depend on the editor (so you have code living in Atom that is used on NPM packages, and these also access internal state instead of going thought the public API) and there are outdated web features. There’s also some insanity happening: using newer versions of Node.JS got me 404 trying to run npm install, and even using the node version that Atom says to use in the documentation (that’s also NOT the current LTS version of node, mind you) didn’t work reliably. The only way I was able to install all dependencies was by running yarn install FOUR TIMES, and that did the trick (I got a different error every try, but at least, it worked).

But the problem is not this one: there’s simply too much code that lives “outside” Atom.
(more…)

The History of Chlorine

Before I even started with Clojure, I was analyzing LightTable – the idea of that editor was to support better integrations between the code you’re writing and the code that’s running. It was a really good experience, but the main problem I had is that it was in the very beginning, with few plug-ins and bad documentation. I tried to make the parinfer plug-in work in the editor, but it had lots of bugs and then I simply changed back to Atom. At the time, proto-repl was the best package to work with Clojure, and I made some small changes to it (so I could add some callbacks to when a new connection to nREPL was made, and other small issues) to improve my workflow.

Fast forwarding a little bit, I started my first Clojure job at Nubank. Most people were using InteliJ, but I felt that by using Atom I had a different approach on problem solving, specially those hard parts where the fast feedback of “run in the REPL and see the results in your editor, then browse over the keys” could give better insights about what’s happening. I tried to implement some features that proto-repl didn’t have at the time (and Chlorine still does not have some) like “automatically add nREPL port”, and “watch expressions” (almost the same as watch variables in a debugger). These ended up in a package called clojure-plus, that still exist today.

I also began to experiment with ClojureScript (at the time, only Figwheel was available – Figwheel-main didn’t even exist!) and found that existing tools didn’t provide the same power that I had with Clojure. It also didn’t have autocomplete, goto var definition, and so on. To ease a little bit these problems, I ended up adding on clojure-plus some CLJS support – when you tried to evaluate a .cljs file, it would try to connect to ClojureScript, reserve a REPL, then evaluate the code over there.
(more…)

nREPL on… Chlorine?

When I started the Chlorine project, I just thought it would be great if I could target all Clojure-like REPLs that already exist but didn’t have tooling support. At the time, this would include Lumo and Plank, mostly. Also, Shadow-CLJS and Figwheel have some “clunky run-some-code-and-transform-in-cljs” way of working that simply didn’t click with me.

Now, almost a year later, Chlorine supports Clojure, ClojureScript (Shadow-CLJS, Lumo, Plank, or even over clj), ClojureCLR, Arcadia, Babashka, Clojerl (Clojure on Erlang) and Joker (Clojure on Go, also a linter). But the reality is that working with a pure Socket REPL is really hard – a socket REPL works exactly like a regular one, printing namespaces after each code, and so on. Also, there are some strange decisions on some REPLs, mostly likely ClojureScript (that is the second most used Clojure flavor), so things are not always easy. To put things in perspective, currently Chlorine uses 3 ways to evaluate code: It uses unrepl, that only works on Clojure, or uses internal APIs of shadow-cljs (that obviously only works for shadow-cljs), and for other implementations it uses a kind of a hack – it evaluates the code, inside a trycatch, and it returns a vector where the first element is a symbol in a specific format that Chlorine will understand and then link that with the response. This “hacky way” is currently being used for every other implementation except Clojure and Shadow-CLJS. Things work (autocomplete works too), but it is not pretty and sometimes have strange results.

As a matter of fact, I was already thinking about removing UNREPL (it’s really hard to implement new features on it, and some good ideas only work in theory – for example, the ability to evaluate long strings / collections and render only a part at a time aren’t that good with lots of edge-cases) and, to do it, I though about a better, non-hacky way to evaluate things on some Socket-REPLs (that, again, would only work on some REPLs – ClojureScript REPLs will probably never support “upgradable REPLs” because of the way they work) – the only thing that I had to understand is how to implement this “upgraded REPL”…

Then, recently, Babashka added an initial support for nREPL, with an insane low amount of lines. So, I’ve tried to implement a way to evaluate code over nREPL… and it was really simple to do it, using a npm library that already did it. But implementing like this meant that the user would need to know if the host/port to connect is a Socket REPL, or a nREPL (and the user does not know – lots of tools like lein and shadow-cljs show an nREPL port to be connected).
(more…)

My Atom editor configuration for working with Clojure/Script, revisited

Sometime ago, I did a post on how I work with Atom to develop Clojure and ClojureScript projects. It is in Portuguese, so I’m gonna re-visit the subject and also update with my current workflow.

There are two packages that I have to install to work with Clojure: lisp-paredit and chlorine. Without lisp-paredit, when I start a newline, the indentation gets all sorts of problematic. I use it on “strict mode” and use the tools to slurp/barf forward only. As for chlorine, it is needed to have autocomplete, evaluation, show documentation, goto var definition and so on. Last, I use also parinfer so I can remove whole lines of text and parinfer will infer the correct closing of parenthesis for me (most of the time at least).

Now, how exactly do I work with Clojure? When you use lein or boot, you’ll get a nREPL port. This is not the port you use with Chlorine, so I need a bit more of work. I can’t just start a REPL with lein repl or clj, I need to inform the tool to open a socket-repl server. The JVM option needed is: '-Dclojure.server.myrepl={:port,5555,:accept,clojure.core.server/repl}'. So, the commands below are what I use with lein or clj:

JAVA_OPTS='-Dclojure.server.myrepl={:port,5555,:accept,clojure.core.server/repl}' lein repl

or

clj -J'-Dclojure.server.myrepl={:port,5555,:accept,clojure.core.server/repl}'

This will open a REPL at port 5555 (or I can change the port if necessary). Then, it’s time to fire up the Atom’s command palette and select “Connect Clojure Socket REPL”, put 5555 on the port, and connect. Then, I’ll use “Refresh Namespaces” or “Load file” command to load my latest version of code into the REPL, and start working.
(more…)

Complex testing – the saga

While I’m developing Chlorine, sometimes I need to test multiple specific implementations of lots of really complicated stuff like REPL state, connection, async stuff (as the project is ClojureScript on Node.JS, all I/O treatment is via callbacks) and the complicated nature of rendering multiple different object types on Atom editor. I had multiple regression errors, then I’ve created some “acceptance” tests (these open up a real Atom editor and evaluate commands on it). The problem with these tests should be obvious: they are slow, and I mean REALLY SLOW, and they need a real Atom editor, lots of setups, and because Atom is not really predictable on its actions (sometimes you connect REPL and it changes the focus on the editor) there are lots of unnecessary interactions on the editor just to have less false-negatives.

Now, as I told before, I’m developing Chlorine together with REPL-Tooling, a library that should contain all tooling for any editor capable of running Javascript to run a port or Chlorine. There’s still too much on Chlorine that relies on internal Atom APIs (for example, detection of the beginning and end of forms is one, detection of namespace’s forms is other) but time is passing and more and more is being moved to REPL-Tooling, and as soon as the detection of forms is on REPL-Tooling (and is stable) there should be possible to port most Clojure parts to REPL-Tooling, and then I can think on how to refactor the ClojureScript part, test it, and then Chlorine will be a very easy project to port. Also, as a proof-of-concept there’s Clematis for NeoVIM (it’s still on the very beginning and nothing much happened after I wrote about it here), and also an “electron fake editor” that I’m using as test.

Wait, What?
(more…)

Chlorine, Clematis, REPL Tooling

Recently I began to migrate some of the code on Chlorine to REPL Tooling, so things may become more testable. I already found multiple bugs in this approach and began an integration test methodology using Electron (I say that there are three kinds of developers in the field of automated tests – the ones that don’t test, the ones that do, and test freaks. I’m probably the later).

In the beginning, the idea would be to rewrite the renderer of results and exceptions so that I could fix lots of bugs that they have. So far, it’s progressing slowly, but I already can render datomic results in a meaningful way (the last renderer was interpreting as Java objects, which they indeed are) and fix some bugs with tags and other issues.

Now, one of the things that I wanted in the new renderer is the ability to copy/paste results in a meaningful way – as I’m using UNREPL, there are lots of tags that need to be reinterpreted, like #unrepl/browseable for datafy. These need to be “translated” to something useful for rendering on screen, with links to navigate inner details, but for copy-paste they should have a textual representation that makes sense (probably the hard part).

The results of all these experiments are quite interesting – when you evaluate something and th result is a Java object, there’s a link ... that can be used to navigate inside the object: it’ll show getters, the current class, methods that the object supports, so things like “goto definition for Java object” or “Javadocs” aren’t really necessary.

Then, one of my friends asked for a Vim option. Just for fun, I started Clematis, a port of Chlorine to NeoVim. And it seems they things are progressing faster than I thought!
(more…)

Chlorine is thriving!

When I started this project, I was experimenting with shadow-cljs to see if I would be able to make an Atom package that would auto reload, run tests on ClojureScript, and se how far could I push ClojureScript in an Atom package.

Now, some months later, I’m seeing the package being used by a bunch of people, and I even discovered some bugs in UNREPL! Now, on this post, I’ll discuss a little bit more in detail the design decisions and my vision on the future of the project.

Chlorine is a clojure and ClojureScript atom package. It connects to a socket repl (opened via lein, boot, clj, shadow-cljs, lumo, or even REBL) and then upgrades the repl to be programmatically oriented with unrepl. Unrepl only works on Clojure, so for ClojureScript we use other techniques. Also, socket repl is a stream protocol, so to emulate a “request-response”, we need to coordinate things so Atom (and other editors) can react to commands and know exactly what’s the correct response for each command sent.

Design decisions

The choice for UNREPL was mostly because there is almost no documentation about prepl so far. Also, Socket REPL is literally everywhere: on Clojure , on ClojureCLR, on Lumo and Plank. Also, I wanted a better way to use ClojureScript, and I still have nightmares trying to use it over nREPL… and with Socket REPL things work fine.

Also, when I started the project Clojure 1.10 was just alpha software, and UNREPL offers us insanely good support for lazy lists, big strings, and other things that I wanted to use out of the box. One of the problems I’m still facing is coordination of evaluate/response, but this will probably be solved after a bunch of other fixes I’ll try.
(more…)