(check (my-code) => (needs :tests))

So, yesterday I made a talk (in Portuguese only, unfortunately) about the difficulties of testing Clojure and ClojureScript code. Specifically, I think the most problematic issue is the lack of “custom matcher libraries”, and how the default error messages are kinda bad and don’t help you identify the problems.

Then, on Clojurians’ #announcements Slack channel, I found that clojure.test Expectations library have a new version. So, why not integrate it on my Check library, and maybe continue developing it?

What is check?

Midje is too magic. Clojure.test is too little. Thinking about findind a “middle ground” I’ve started the “check” project, and I’m using it to test my personal projects like Chlorine, Clover, REPL-Tooling and Paprika. The problem is that, while the API is stable, but it still doesn’t do all the things I want.

Now, I decided to start a new version. I’m still not sure how will be the API for these proposed changes, but the “basic” API will not change: you use check to wrap the first parameter into a expectation, the second parameter is an “arrow” (or any matcher that you want, really) that is comparable to “custom matchers” on other languages, and the third parameter is what you’re expecting.

Now, things work better with examples: Suppose you want to match a string:

(deftest some-strange-test
  (testing "The string is My Friend"
    (check (generate-person) => "My Friend"))

  (testing "The string contains Friend"
    (check (generate-person) => #"Friend")))

There are two matchers for now: => and =includes=>. Both use the expectations library to make the assertion, but I want to migrate to the new “clojure.test/expectations” because it seems “less magic”.

Proposed changes

For now, to extend check you need to implement check.core/assert-arrow and it receives three parametes: cljs? that is true if we’re on ClojureScript environment, the “unevaluated” left side, the matcher as a symbol, and the “unevaluated” right part. You need to return a valid syntax tree that returns a clojure.test map that will be sent to clojure.test/do-report. If it seems too complicated, it’s because it is, and I want to change this. Maybe being inspired by RSpec, get some API like:

(check/define-matcher! is-instance [expected actual]
  {:pass? (= expected actual)
   :failure-message (str "Expected: " expected "\n  Actual: " actual)})

; You should use it like this:
(check "This string" is-instance String)

It would be way better that what we have today.

But why?

Tests are code too. If it’s hard to write tests, people will not do it. Heck, even I that try to test everything I can in my projects suffer from this issue on Chlorine. I do want to make things easier for me and for everyone in the future.

The thing is, there are lots of custom formatters for Clojure, but no library that allows you to define your own. And this is a big issue: sometimes the difference is incredibly huge in productivity. Imagine if you have to read:

Expected: (unnmatch-base64? "extend")

  Actual: (not (unnmatch-base64? "extend" "Rm9yIG5vdywgdG8gZXh0ZW5kIGBjaGVja2AgeW91IG5lZWQgdG8gaW1wbGVtZW50IGBjaGVjay5jb3JlL2Fzc2VydC1hcnJvd2AgYW5kIGl0IHJlY2VpdmVzIHRocmVlIHBhcmFtZXRlczogYGNsanM/YCB0aGF0IGlzIHRydWUgaWYgd2VyZSBvbiBDbG9qdXJlU2NyaXB0IGVudmlyb25tZW50LCB0aGUgInVuZXZhbHVhdGVkIiBsZWZ0IHNpZGUsIHRoZSBtYXRjaGVyIGFzIGEgc3ltYm9sLCBhbmQgdGhlICJ1bmV2YWx1YXRlZCIg"))

Instead of:

Expected: (unnmatch-base64? "extend" "Rm9yIG5vdywgdG8gZXh0...")
   Error: Base64 string decodes to "...now, to extend `check` ..."
   Match:                                      ^^^^^^

Way better isn’t it?

What more?

Async tests. I want to somehow identify when a testing returns a promise and await for it, explicitly or implicitly. Or a core.async channel. Or even a callback, who knows? In the past, I created the API:

(def-async-test "Some test name" {:teardown close-all!}
  (await! (some-async-function)))

;or even:
(def-async-test "Some test name" {:teardown close-all!}
  (some-async-function) =resolves=> 20))

But is limited, and I will try to solve this issue adding support for Clojure’s futures and promises. It already adds a “timeout”, but I want to make it more configurable.

Also, I do want to add some kind of “mock” like Midje:

(deftest some-io-test
  (with-mocked
    (jdbc/query connection? "SELECT * FROM users") => []
    (check ..... => ....)))

But this is probably very low on the list for now. For now, I’m going to concentrate on async tests and custom matchers. Then, I’ll migrate some of my Chlorine tests, and see what happens!