Clojure frustrations

One of the nice things about Clojure is that it comes with a huge set of data structures, all of which can be handled using normal Lisp commands. However things are not as simple as they first seem, and it’s a bit of a dark art as there is limited information online. I’ve just spent a day or so on a frustrating voyage of discovery, so I post this here in the hope of saving someone some time in the future.

Lisp generally allows you to read and write data very simply, there is no need to use a separate parser as you store information in lists – the same as the code, for which you already have a parser.

For example if you have a text file called data.txt containing ‘(“one” “two (“three” 4) 5) all you need to do is call (define d (load “data.txt”)) and it gets parsed for you from it’s text form: (first d) returns “one”. This is more efficient than using another kind of format such as XML which you’d need to run a separate chunk of code for.

One of the clojure data types I decided to use was StructMaps, which you define like this (defstruct mystructfieldone :fieldtwo :fieldthree) : and gets created like this:
(struct mystruct 1 2 “three”) => {:fieldone 1, :fieldtwo 2, :fieldthree “three”}

Clojure also comes with some handy io calls, so you can save the StructMap with (spit (struct mystruct 1 2 “three”) “data.txt”) and you get a file created containing {:fieldone 1, :fieldtwo 2, :fieldthree “three”}. This can be read back in simply by using (read-string (slurp “data.txt”)).

The catch is that the result of read-string is a different type to the object we saved:
(type (read-string (slurp “data.txt”))) => clojure.lang.PersistentArrayMap whereas: (type (struct mystruct 1 2 “three”)) => clojure.lang.PersistentStructMap. Presumably the reader interprets the curly braces as a different type. Annoying, but I then got reading about print-dup, which provides more serialisation information for recreating the data:

(print-dup (struct mystruct 1 2 “three”) *out*) => #=(clojure.lang.PersistentStructMap/create {:fieldone 1, :fieldtwo 2, :fieldthree “three”})nil

This seems to output code that should be able to recreate the StructMap. However, to actually write and read this from disk, we have to call it a bit differently:

(use ‘clojure.contrib.io)

(defn serialise [data-structure filename]
    (with-out-writer
        (java.io.File. filename)
        (binding [*print-dup* true] (prn data-structure))))

(defn deserialise [filename]
    (with-open [r (java.io.PushbackReader. (java.io.FileReader. filename))]
        (read r)))

At this point I was getting a bit frustrated, but felt I was close. Unfortunately, deserialising the file created – I got this message:

java.lang.IllegalArgumentException: No matching method found: create (NO_SOURCE_FILE:1)

Great, so the clojure.lang.PersistentStructMap/create doesn’t actually exist? More searching revealed that this is indeed the case, and is not going to be fixed in the short or medium term. Perhaps we can bump down a level and use some serialisation stuff from java – some googling provided these replacements:

(defn serialise [o filename]
    (with-open [outp (-> (java.io.File. filename) java.io.FileOutputStream. java.io.ObjectOutputStream.)]
        (.writeObject outp o)))

(defn deserialise [filename]
    (with-open [inp (-> (java.io.File. filename) java.io.FileInputStream. java.io.ObjectInputStream.)]
        (.readObject inp)))

These seem to do the trick: (type (deserialise “data.txt”)) => clojure.lang.PersistentStructMap. However data.txt is now a (relatively) large binary file, and not human readable. Not a showstopper, but all the same, not very Lisp like. I’ve been aware all this time that the documentation suggests using Records instead of StructMaps, so I assume these will work better – you create and use them in a similar way:

(defrecord myrecord [fieldone fieldtwo fieldthree])
(myrecord. 1 2 “three”) => #:user.myrecord{:fieldone 1, :fieldtwo 2, :fieldthree “three”}

However, they exhibit all the same problems as ArrayMaps when it comes to serialisation – and although they do work with the last method, they exhibit a further problem. If I change the definition of myrecord and then stream in the file, I get this:

java.io.InvalidClassException: user.myrecord; local class incompatible: stream classdesc serialVersionUID = -791607609539721003, local class serialVersionUID = 4065564404893761810 (NO_SOURCE_FILE:1)

The problem here is that I need to be able to change my structures and then convert older saved data using versioning. I can do this with StructMaps as they are not statically typed, so I can write code to check them on load and modify them to the current version (this was used a lot on NoP to keep the game running all the time with the same data while changing the fundamental types). I think a lot of my problems stem from coming from Scheme, so I’m not really aware of the “Java way”.

The other approach is to just abandon looking for an existing solution and go back to writing per-object versioned serialisation/deserialisation parsing like I do in C++.

6 thoughts on “Clojure frustrations

  1. >One of the clojure data types I decided to use was StructMaps

    Out of interest, why did you decide upon StructMaps? As you say, Clojure 1.2 has pretty much deprecated them in favour of Records which are more efficient and offer Java interop. However, it seems that just plain old Maps might work for you here. Sorry if I misunderstood your requirements.

    Unfortunately Clojure doesn’t have a very symmetric reader/printer. There are many such niggles and many of them are due to its hosted nature. If you have any suggestions/queries the mailing list and irc room would be a good place to start a discussion.

    I’m sure that there will be many areas that has a counterpart in Scheme that is more polished. Hopefully people such as yourself can continue to highlight them.

    Sam

  2. Hi Sam,

    I think I decided to use StructMaps because I wanted a map with fixed fields and this sounded like it would be optimised for that (I guess I missed the fact it was deprecated initially). I’m currently using Records, but might just switch to Maps as you suggest so I can modify them on load to do the file versioning.

    I must say in general my impression of Clojure is very good – I had to use the jvm for this project and it’s been much smoother for me as a Scheme programmer than having to go the full Java route. Most documentation I’ve found online is for people coming from Java to parenthetical languages through Clojure, so I’ll keep posting my thoughts from this other direction.

    cheers,

    dave

  3. My particular trajectory was from Ruby (halfway between lisp and Java) to Clojure. Therefore both the parenthetical world and the JVM were (and in many ways still are) new to me.

    I think that Clojure treads a very interesting and innovative path which in many ways becomes a very delicate balancing act. Despite currently working through SICP I’m not really tempted to move to Scheme. The Clojure community is also a very exciting entity to watch.

    The only thing that thoroughly annoys me about the JVM is its lack of a free GC that doesn’t block any threads during execution. This is a real pain when you’re trying to build a musical system with no random pauses whilst the GC stops the world.

  4. I find the abstraction of the JVM not quite as good as I’d like it – probably as I’m not used to using it, but it’s restrictions seem to get in the way sometimes. Your music system sounds interesting, do you have any links to it?

    The way I’ve been doing these things is to run the tightly timed audio thread on a language without a gc, and doing all the sequencing with timestamps in a higher level language.

  5. Faced the same problem, I’ve made a workaround with converting the map into the list on serializing and back.

Leave a Reply

Your email address will not be published. Required fields are marked *