February 14, 2021

Adding an Extra Indirection to Enable Code Reloading in Clojure/Java Interop Code

One of the killer features of Clojure is its ability to reload code and incrementally develop in a running environment. Another strength of Clojure/Clojure script is that it is deliberately a hosted language, leveraging the broader Java/Javascript ecosystem for libraries, frameworks, etc.

However, the dynamic code reloading and interacting with Java libraries can create some friction. Adding another layer of indirection sometimes reduces that friction. Let’s look at an example. Let’s assume we have a large Java library to which you provide a callback functions/object. Like an HTTP-server, network- or some other library. Easy, you implement the interface of the library. Note that the library instance is preserved with defonce.

Java and Clojure Best Friends Forever
Figure 1. Java and Clojure Best Friends Forever
Example Interaction With Java Library:
(ns info.gamlor.sshli.example
  (:import (info.gamlor.example PowerFullJavaServerLibrary LibraryCallbacks)))

(defn authorize-user [user realm]
  (println "Authorize" user realm)
  true)

(defn do-work [user data]
  (println "Do work" user data)
  true)

(defn create-library [do-work-fn]
  (PowerFullJavaServerLibrary/start
    (proxy [LibraryCallbacks] []
      (authorizeUser [role realm]
        (authorize-user role realm))
      (handleRequest [user data]
        (do-work-fn user data)))))

; Keep library instances, because library takes a long time to start or has some state we want to preserve
(defonce instance (create-library do-work))

(println "The Library calls back into our code:")
(.doWork instance)
Output:
The Library calls back into our code:
Authorize user example example
Do work example #object[java.util.ImmutableCollections$Map1 0x51971f1b {test=data}]

Then you change the authorize function, reload the code and you’ll see the change immediately.

Small Change in authorize-user:
(defn authorize-user [user realm]
  (println "Authorize Version 2" user realm)
  true)
Output:
The Library calls back into our code:
Authorize Version 2 example example
Do work example #object[java.util.ImmutableCollections$Map1 0x9c3fda4 {test=data}]

Then you update the do-work function and reload the code. To our surprise, the old version of the implementation is executed. Yikes!

Small Change in do-work:
(defn do-work [user data]
  (println "Do more work" user data)
  true)
Output:
The Library calls back into our code:
Authorize Version 2 example example
Do work example #object[java.util.ImmutableCollections$Map1 0x3533421f {test=data}]
Java and Clojure Not Getting Along
Figure 2. Java and Clojure Not Getting Along

Functions and Vars

Why did the change in the authorize-user work on the reloaded code, while the change in do-work did not? For that we have to take a close look at the create-library again:

create-library:
(defn create-library [do-work-fn]
  (PowerFullJavaServerLibrary/start
    (proxy [LibraryCallbacks] []
      (authorizeUser [role realm]
        (authorize-user role realm))
      (handleRequest [user data]
        (do-work-fn user data)))))

Notice that we call the authorize-user directly by using its name, but the do-work function is passed in as a function. Here lies the difference: The expression (authorize-user role realm) does do two things. It dereferences the var named authorize-user which contains a function and then calls that function. However, when we use (create-library do-work) these two steps happen separately. At (create-library do-work) the var named do-work is dereferenced and its value, the function is passed into create-library. That function reference is then used in the handleRequest.

So, when we change the authorize-user function and reload the code, the existing library instance gets the updated function, because on every call we dereference the authorize-user var, get its latest value and call that function.

However, for the do-work function we dereference it when the library instance was created. It points to the function the do-work referenced when the instance was created. When we change the do-work function, the do-work var points at the new function, but the instance still has the reference to the old version.

We can fix this by not passing do-work directly, but wrap it in a function. Then we have a function registered that does the dereference and call behavior:

Wrapping Function:
(defonce instance (create-library
                    ; Wrapping in anonymous function, which  dereferences of the latest do-work function
                    (fn [user data] (do-work user data))))

After a complete restart or re-defining the instance, changes in the do-work method will show up on reload.

Small Change, Reload Works Now:
(defn do-work [user data]
  (println "Do even more work" user data)
  true)
Output:
The Library calls back into our code:
Authorize Version 2 example example
Do even work example #object[java.util.ImmutableCollections$Map1 0x3533421f {test=data}]

Caveats

Clojure supports more aggressive inlining with direct-linking, avoiding the var lookups and enable more aggressive optimizations. I didn’t try out the behavior in that scenario. However, you probably won’t enable these options for development setup, these options are more for creating production builds.

Tags: Clojure Java Development