Ubiquitous interface – how to integrate things in Clojure

Inspired by this thread on Reddit, I decided to write a little bit about my experience integrating things in Clojure.

The first thing to understand is that Clojure have an ubiquitous interface: EDN. And it is important to understand what this means. In the beginning, I made this mistake of “Death By Specificity” on my now abandoned Relational project: to abstract things that don’t need to be abstracted.

But can we do even better? How about we de-abstract (concretize? Is this a real word?) things that are already abstracted?

For example, HoneySQL does this: you write a map, you get back SQL. This means you have the whole core Clojure library to write queries, and you have more than the flexibility you will ever need. You also have meander, specter, and lots of other tools to use that can re-write Clojure collections to write queries. You can also simulate the Ruby’s ActiveRecord behavior!

(Now, a disclaimer: please don’t use the last library I linked. It is still not production ready, and I don’t really know if I’ll keep maintaining it…)

Now, the advantage of de-abstracting to EDN is the integration of everything: I once had to read multiple databases, some inconsistent, some serializing old versions of the same record as JSON, make then consistent and save the results to another DB. What I did was simple: I used the wonderful Clara-Rules library to generate the rules that would correlate old and new data, normalize things, and so on; then I emitted everything as a record (what is a record, if not a super-powerful map?):

(defrule treat-versions-place
  [?place <- Place (= ?place-id id)]
  [?versions <- (acc/all) :from [PlaceVersion (= ?place-id id)]]
  =>
  (insert-all! (normalize-place ?place ?versions)))

Now, the code normalize-place will just emit a Sql record that contained two fields: :honeysql and :order. The :order is just an integer that defines which inserts/upserts I had to run first, and which to run last.

After that, I found that it was necessary to inform external systems which tables were changed after each run of the correlation process. If I just emitted SQL code, or “ORM” models, this would be awfully difficult, maybe even impossible – to the point I would need to change the “rules” to insert! a new fact TableChanged or something. But with this code, it was quite simple:

(def changed-tables [sqls]
  (->> sqls
       (map :honeysql)
       (map :insert-into)
       (map #(cond-> % (coll? %) ffirst))))

And that’s all. This captures {:insert-into :people ,,,,}, for example, or the “insert with select” from PostgreSQL that translates to:

{:insert-into 
 [[:place [:id]] 
  {:select [:place_id]
   :from [:people]
   :where [:= :id 20]}]}

So, to summarize:

1. Query multiple databases with clojure.jdbc, which returns collections of maps
2. Sometimes one of these maps have “versions of the row” in JSON. So cheshire libraru will turn a JSON string into a Clojure Map
3. Convert the maps to records with, for example, map->Place
4. Send these records to Clara-Rules, and return other records
5. These records contain SQL queries, encoded as maps
6. Analyze these maps and return collections of keywords, that are the tables that we changed
7. We can create a “map” with these tables and use cheshire to transform then into JSON, ready to be sent somewhere else
8. Use honeysql to generate the INSERT INTO... SQL, and clojure.jdbc to persist

Now, not a single one of the tools we used was expecting to be integrated with others. Clara-Rules doesn’t know the existence of anywhere else, for example. Clojure.jdbc have no idea that it’ll be sent to a rule-based engine in the future. And yet, everything integrates 100% without issues.

Now that is the power of an ubiquitous interface!