clojurescript

Performance optimizations WITHOUT Reagent – part 3 (and probably final)

This is probably the last post about these improvements. At the last post, I showed how I threw every performance optimization I tried away (combine-leafs, other stuff, etc) just to throw it all away because React didn’t like what I did.

So, after a lot of time, I decided to do the horrible, insane thing that I did a lot more than I wanted – rewrite the rendering process of Chlorine to not use Reagent. The reason was actually quite simple: I wanted things to be simpler to implement, and honestly, Reagent was getting in my way. Like, A LOT.

Before I continue explaining the performance improvements that I am doing, I have to say something – in the first version of the rendering code (the one that I was trying to optimize at my last posts) I had a huge beginner’s luck – the actual rendering time is amazing for most cases, and while it is slow for some very specific situations, it’s actually not that bad; but there were also some weird trade-offs that I had to do and I cannot use them anymore.

The reason being, Chlorine started to become very strong in the customization code and “interactive renderers”. If you’re not aware, you can configure Chlorine to add custom commands and other amazing stuff, just by opening up a config file, writing some ClojureScript code, and you’re good to go. Unfortunately, also at the first versions of Chlorine, sometimes you would evaluate a code – let’s say a list of a thousand elements – and you get back only 10 or 20 elements. Worse yet, you would not get back a “List”, but an object called an “Incomplete List”. Which was not ideal, and basically made the whole “customize your editor” difficult to say the least.
(more…)

By Maurício Szabo, ago

A new Chlorine is almost done!

For the last few months, I’ve been working on a new version of Chlorine (now that Pulsar is on a good track). Here I want to share what to expect of this new version.

For a quick TL;DR; it’ll have better ClojureScript (shadow-cljs) support, fix some bugs related to evaluating code and the REPL being “locked” without returning the results or evaluating new commands, and have better configurable parameters. Things will probably break a little bit – it’s not a non-breaking-change, unfortunately – but hopefully it’s good enough that people will want to migrate to the new version.
(more…)

By Maurício Szabo, ago

ClojureScript vs Core.Async debate – the last updates.

So, my old post made to Reddit, and as I expected, lots of people complained about my conclusion. I still find damaging that Clojure community have this huge energy on defending “core” solutions, even when they are low-level, impractical, or (in this case) does not work well, if it works at all. But, well, I decided to clarify some of the posts, and answer some of the comments that people told me would, supposedly, “solve” the problem. I already answered these questions on other forums, so I’ll try to re-use some of my comments:

Interop with <p!

Some people asked about interop with <p!. Honestly, when I made the article, this option did not exist. Even then, a Promise can be rejected and resolved with arbitrary data, so to translate this code to ClojureScript means capturing the original result of <p!, check if it’s an ExceptionInfo, with the right type, and extract the right part. It’s also harder to compose IMHO (like, with Promises, you can chain actions, then if any of them fail, you can catch the error and the promise is back to the “success” phase). But at least, now it’s less complicated – although I would say, if you’re going to add a library to your project, why not add funcool/promesa instead of core.async? Remembering that promesa is both faster than core.async, and its constructions can work with any arbitrary data, not only promises, so you can treat all code as if it’s async (or sync), without needing to remember which ones are promises, and which ones are not…

Just .pause and .resume constantly

This works, indeed. For example, with sockets:

(.on socket &quot;data&quot; (fn [data]
                     (.pause rs)
                     (async/put! chan data #(.resume socket))))
(.on socket &quot;end&quot; #(async/close! chan))

But not all callback-based async code in ClojureScript allow you to pause events. Node.JS Socket do, WebSocket in browser don’t. Also, NPM package pg, when you query with a cursor, also will not allow it. In this case, there’s nothing you can do, really – you will either drop messages, or an exception will happen.

Use offer!

(go (>! and put! both are fragile with callbacks, because you can’t really “park” or “block” threads in Javascript. Keep put!ing messages in a channel, and you’ll hit an exception “No more than 1024 pending operations” (or something like that). So people asked me to use offer! or make a sliding buffer, etc. This doesn’t really solves the problem if you can’t drop messages – and I do have lots of cases when this happens. There were some people that told me that dropping messages is part of the life, and I wholeheartedly disagree – there are LOTs of situations when you can’t loose anything – bank transactions, socket messages (you can’t reconstruct the payload if you lost part of it), downloads, etc. There are ways to mitigate this, for example:
(more…)

By Maurício Szabo, ago

Can I split the Atom and, from the parts, generate another thing?

So… my last post was somewhat sad – I tell about how impossible is to keep the Atom editor. Unfortunately, it still holds true – it’s close to impossible to handle the insane amount of code that Atom have.

But maybe is there a different way? So I decided to try: presenting, Project Saturn: a huge editor, but less dense. That’s the idea.

Saturn always fascinated me, and still fascinates: it’s a huge planet, but less dense than water. With a sufficient big bowl of water, Saturn would float. That’s precisely the idea I want in this new editor: to be less dense to keep, while somehow maintaining the power and “wow factor” of the editor. So, how to do it?
(more…)

By Maurício Szabo, ago

The impossibility to maintain Atom

Atom Editor was, and still is, an amazing piece of technology. It was the first practical example of web technologies running locally, applied to a really hard problem: text editors. It it was, and still is, darn good at handling code.

But it’s hard, close to impossible, to maintain as the way it is. I want you to take all my opinions with a grain of salt, because these represent my own ideas and feelings on the project – I’m not a full-time Atom developer, nor I intend to be one. I just closely followed its development cycle, helped find bugs and problems in the first editions, and I still am in love with the editor, even when it’s clearly dying.

And while I would love to modernize the editor, making some PRs and fixing some issues, it’s close to impossible to do it. I believe the problem lies on the fact that Atom was an editor, and then Electron was extracted from it. I made the same choice on Chlorine too: I first made the plug-in, then extracted REPL-Tooling from it. Even on a WAY SMALLER codebase (Chlorine), this was HARD: there are still internal, private data that is used inside Chlorine that is not on REPL-Tooling.

Atom is the same. The “setup project” for Atom (prepare Electron, parse cmdline args, etc) is INSANELY huge. There are insane cyclic dependencies (TextEditor depends on components, UI, etc, and these depends back on the editor), there are NPM modules that depend on the editor (so you have code living in Atom that is used on NPM packages, and these also access internal state instead of going thought the public API) and there are outdated web features. There’s also some insanity happening: using newer versions of Node.JS got me 404 trying to run npm install, and even using the node version that Atom says to use in the documentation (that’s also NOT the current LTS version of node, mind you) didn’t work reliably. The only way I was able to install all dependencies was by running yarn install FOUR TIMES, and that did the trick (I got a different error every try, but at least, it worked).

But the problem is not this one: there’s simply too much code that lives “outside” Atom.
(more…)

By Maurício Szabo, ago

The magical wonders of programming with pure data

I’ll start with a disclaimer – if you’re developing in ClojureScript because of performance, you’re doing the wrong thing. Although a Clojure code can be really close to Java performance, ClojureScript simply can’t match JS performance – specially because in Javascript, the fundamental data structures like arrays, sets, and objects are implemented in a faster language, and as of today, no “user space” data structure can get even close the “native” ones. And, let’s not forget that JS is slower than Java.

So, if you want to beat JS performance in ClojureScript, you have to make use of what ClojureScript offers you – the REPL, the fast experimentation, and the wonderful experience of programming with data. So, this is a tale of a huge performance improvement that I’m currently working in Chlorine.

Rendering EDN

In Chlorine and Clover, when you evaluate something, the result will be rendered in a tree-like way at the console. This works quite well for Clojure, because UNREPL makes structures that are too big small, and you can query for more data. So far, so good.

In ClojureScript, Clojerl, Babashka, etc, things are not so good. Big data structures will be rendered fully on the view. They can lock or crash your editor, they can occupy all your memory, etc. The reason for that is that tree structures are hard – when you’re rendering the “parent”, you don’t know what the children will be. Currently, in Chlorine, rendering (range 80000) locks my editor for 4 seconds until it calculates everything, layouts all data, etc… and I wanted to change this.

Reagent vs Helix vs React

I was always intrigued on how much performance hit I get when I use Reagent’s Hiccup data structure instead of React objects – after all, there must be some performance problems, right? After all, that’s the promise of Helix – to not pay this performance hit because you’re closer to what React wants.

So, first things first: profiling Javascript inside Electron IS A NIGHTMARE. I decided to install React Devtools extension on Electron (inside the Atom editor) but that got me into so much trouble, false-positives, wrong profiles that I ended up deciding against it and simply used the “performance” tab to do my profiling. There, I could see batching.cljs hogging my performance, so the next move was easy – move away from Reagent and use Helix.

Except… that Helix uses some macros and I ended up doing the code in React. My testcase was simple: render a vector of 80000 elements and see how it would perform, obviously without all the bells and whistles that Chlorine offers today (otherwise this experiment would be waaay longer). And that’s where things get surprising: with Reagent, I was hitting 1800ms of scripting, and about 120ms of rendering. With React… 1650ms of scripting, and about 200ms of rendering. I decided to do more benchmarks and probably because of OS caching, warm-up, or whatever, the results got even closer, with Reagent sometimes performing better than React – but still too slow.
(more…)

By Maurício Szabo, ago

The power of Pathom

Most people don’t know the power of Pathom. Most people that I know of think that Pathom is about graphs.

They are wrong.

I mean, yes, in the end you’ll have a graph, with dependencies, but that’s not the point. The point of Pathom is the ability to transform your code into a soup or attributes. It’s also probably the best usage of qualified keywords for me, if not the only one that justifies the downsides (I’ll not enter in details here – just know that having to convert from qualified keywords to unqualified multiple times is not fun at all).

The name “soup of attributes” may not be a beautiful one, but believe me – it’s incredible. The idea is quite simple – instead of trying to handle all possible conversions from multiple sources to multiple destinations, you just define which attributes can be computed in terms of others, and Pathom does the rest. As always, things are better with examples, so let’s go.

I had to work on a system that somehow had strange rules – it needed to generate a bunch of text files for different companies. Each company expected a different file name and different fields. To generate the file, we had to accumulate data that came from a payload, from an external system that we called via REST API, and also from some data we had on our database. To make things worse, some companies would expect some of the data that was returned from REST on the filename, and there were also some state changes – like, if a file was already processed, the company would send us a return file, and we had to read some content of this file, move this file to another directory, renaming the file in the process.
(more…)

By Maurício Szabo, ago

Implementing shadow.remote API

Since version 0.8.0 of Chlorine, there’s a new way to evaluate ClojureScript code: that’s the Shadow-CLJS Remote API. It is basically a new REPL (not nREPL, no Socket REPL) over WebSockets to try to solve problems when translating other REPLs to ClojureScript. So, to understand why these problems exist, I’ll first introduce the difference between ClojureScript and Clojure.

On Clojure, you’re always inside a JVM. This means that compilation happens on the same JVM that your REPL, and your code is running. If you practice REPL-Driven Development, even your tests are running on the same JVM. In practical terms, it means that when you fire up your REPL, you already have everything ready to run code, compile code, and evaluate forms.

On ClojureScript, the compiler is written in Clojure – that means it’s running on the JVM. So, to produce Javascript code you don’t need a Javascript environment – and that’s when things become confusing, because when exactly will you run the REPL? Let’s try from another angle: if you start the REPL on compilation time, you can’t evaluate code (because there’s no Javascript generated, nor any Javascript engine running). If you start the REPL when you run the compiled code, this REPL can become unusable if you stop the Javascript environment, and also you have to coordinate lots of state and translations between formats.
(more…)

By Maurício Szabo, ago

Reagent Mastermind

One of these days, a friend of mine posted about his experience writing the “Mastermind” game in React (in portuguese only). The game is quite simple:

  1. You have six possible colors
  2. A combination of 4 colors is chosen randomly (they can be repeated – for example, blue,blue,blue,blue is a valid combination) – you have to guess that number
  3. You have up to 10 guesses of 4 colors. For each color on the right position, you win a “black” point. For each color in the wrong position, you win a “white” point
  4. If you can’t guess on the 10th try, you loose.

So, first, we’ll create a shadow-cljs app – create a package.json file, fill it with {"name": "Mastermind"}, then run npm install shadow-cljs. Traditional stuff.

Then, we’ll create the shadow-cljs.edn file. It’ll only contain a single target (:browser), opening up a dev-http server so we can serve our code, and we’ll add reagent library dependency. I also added the material-ui dependency, but you don’t really need it for the code. Now, running npx shadow-cljs watch browser will start a webserver at port 3000, and we can start to develop things.
(more…)

By Maurício Szabo, ago