Exposing Clojure to Ruby and other languages – Java objects in C

In the first post of this series we decided to compile a Clojure library to a native shared library (with the GraalVM native-image command) and we used “isolates” so that a “global variable” in Clojure (and Java) would appear as if they’re “local” in Ruby. Then on the second post of the series we decided to make things local, and instead of “making a resolver then storing it in a global variable” we decided to “make a resolver and return it to Ruby”, and to abstract the nature of “callbacks” we used a CFunctionPointer in the Java side, and we sent the “block” from Ruby as an “opaque object” in Java, using GraalVM’s VoidPointer object.

Now, we need to do the opposite – we need to return a “resolver” to Ruby, and then we need to make a “List of Resolvers” in Ruby-side, and send it to Clojure-side somehow. These Clojure/Java objects (a Resolver, and a List of Resolvers) will also be “opaque objects” in C (meaning – we’ll receive them as void*, because they can’t be represented in C-land, nor in Ruby-land), but only in C – in Java-land (and Clojure, by definition) they’ll need to retain their types. The object GraalVM provides for this task is an ObjectHandle.

We’ll also fix a lot of memory leaks, so let’s move on!
(more…)

Exposing Clojure to Ruby and other languages – Callbacks

At the first part of this series I showed how to expose some Java methods (that delegates to Clojure functions) to a shared library, so that the shared library could be exposed in Ruby via the C bindings. There is a lot of “cutting corners” on the first implementation, for example, we only transmitted strings that would be serialized and de-serialized between languages, and the fact we’re using GraalVM “isolates” to allow multiple instances of “VMs” to exist, so we can fake having multiple objects, etc. In this post, I will fix some of these issues, and then show how to expose complex objects.

So, here’s a quick overview of the Clojure library I want to expose: the library allows you to define “resolvers”, which are functions. The difference between a resolver and a “normal function” is that the resolver always expects and returns a “Map” (HashMap in Ruby). The resolver also have some “metadata” that explains, at least, the dependencies: meaning, what are the keys that the “input” of the resolver must have, and what are they keys that the “output” will return.

The implications of this are huge, although they might not appear so – the idea is that, instead of wiring up things manually (like, for example, querying a database, getting the result, sending to some external service, get the result of the service, do something else) you just define the “dependencies” – for example, “I need the user’s login and email for this external service” as a “resolver” – if hashmap doesn’t have it, you’ll have a different resolver saying “Query the database and return the user’s login and email” and so on.
(more…)

Exposing Clojure to Ruby and other languages

Recently, I’ve been missing Pathom (from Clojure) in the Ruby world. So I decided to check if I could expose an arbitrary Clojure library to Ruby, using the GraalVM’s “Substrate VM” (popularly known as native_image).

The way I tried to do this was to expose a shared library to C, then making a Ruby extension that consumes this shared library and then finally “normalize stuff” into Ruby. This… honestly, worked way better than I expected, and because I needed a lot of help from AI tools, guesswork, and reading some very weirdly-written Java code from Oracle (and auto-generated Javadocs) I decided to make a post explaining the whole process.

The whole experience started as weird, then easy, then VERY hard, and right now… I’m not sure where it will be, because I’m kind of changing the whole project’s core ideas (the reasons will be clear while you read this post). So let’s dive in
(more…)

Porting Clojure libraries to Ruby – Part 2

Last post I gave an introduction on how I’m working with GraalVM to port Clojure things to Ruby. Now, it’s time to handle the hard parts.

The first thing is – some stuff simply doesn’t work. Pathom3 (the library I wanted to port) depends on Guardrails, which requires core.async, which starts a thread pool, which GraalVM doesn’t like. At all.

And the second things is how to serialize some weird stuff like callbacks. So let’s start with the first part – how to make a “Pathom3 shared library” without GraalVM screaming at us
(more…)

Porting Clojure libraries to Ruby

Since I started working with Ruby, in some situations I miss some of the Clojure libraries – especially killer ones like Pathom.

Unfortunately, rewriting the whole library in a different language is completely out of the question, especially because I will reintroduce some of the bugs that are already fixed, and will not have some of the more important features like parallel processing and other stuff.

But it’s not an “end of the world” situation. GraalVM have a very important feature that is to compile Java code to native, using what it’s usually called the “native-image” or, to be more precise, SubtrateVM. I also knew that you could generate shared libraries with this approach, but I never actually did anything and in my last attempts – every time I tried the only result I had was lots of frustrations and weird errors.

So that’s why I’m very proud and very surprised to announce that I have a Pathom working in Ruby right now.

But more on that later – it’s probably not yet ready to be used in production; but what I want to share is how I actually made it work, then challenges I faced (and how to handle them) and how can other people do the same. Finally, I am writing this post to make it easy for other people to do the same – honestly, the whole thing was awfully hard but not because it’s actually hard – but because the documentation is lacking so much that, at some times, I even though things I did were not possible.
(more…)