A Harris, 4 shaft table loom:
With my mind on our upcoming AHRC weave/code project (and seeing as Alex has already started writing code) I thought I’d have a go at visualising how computers work in relation to pattern manipulation. These screenshots are from a ZX Spectrum where I’ve modified some library assembler code for higher level arithmetic to display the contents of 7 CPU z80 registers graphically between each instruction – time runs from top to bottom.
Most processors don’t actually have circuits for mathematics, they simply implement ‘add’ along with bitwise instructions for ‘and’, ‘or’, ‘not’, ‘xor’ and a handful of instructions for shifting the bits left and right. This is true even with modern CPU’s where the arithmetic instructions for multiply, divide etc are built with hidden ‘microcode’ routines. For this reason the underlying operation of a computer has more to do with patterns than it does with concepts such as language or even numbers as we normally think of them.
The simplest (and shortest) are multiply in 8 bits. In this function, the ‘a’ register contains one number and the ‘h’ register contains the other – at the end the ‘a’ register contains the result. In the first screenshot the numbers are fairly simple so it’s possible to see what’s going on (ie. in 1*1 the ‘a’ and ‘h’ registers both contain 00000001)
The code for all this is here.
I’ve recently been building the Mongoose 2000 “group composition” tool that the researchers will use for recording information about a whole pack of mongooses (and synchronise data via a Raspberry Pi providing a local wifi node) in their field site in Uganda. As I wrote a bit about before, one of the interesting things about this project is that the interface design has to focus on long term speed and flexibility over immediate ease of use. In this way it seems appropriate that it’s moving in the direction of a musical interface rather than a normal touch screen interface. The different colours in the mongoose selectors show which individuals are present and which have data recorded from them already, the screenshot below is the section where they record relationships between the adult females (at the top) and adult males that may be guarding – or pestering them (below). At the same time, they need to be able to record events that may be occurring with the pack as a whole – in this case an interaction with another pack of mongeese.
The Heliconius Butterfly Wing Pattern Evolver game is finished and ready for it’s debut as part of the Butterfly Evolution Exhibit at the Royal Society Summer Exhibition 2014. Read more about the scientific context on the researcher’s website, and click the image above to play the game.
The source code is here, it’s the first time I’ve used WebGL for a game, and it’s using the browser version of fluxus. It worked out pretty well, even to the extent that the researchers could edit the code themselves to add new explanation screens for the genetics. Like any production code it has niggles, here’s the function to render a butterfly:
(define (render-butterfly s) (with-state ;; set tex based on index (texture (list-ref test-tex (butterfly-texture s))) ;; move to location (translate (butterfly-pos s)) ;; point towards direction (maim (vnormalise (butterfly-dir s)) (vector 0 0 1)) (rotate (vector 0 90 90)) ;; angle correctly (scale (vector 0.5 0.5 0.5)) ;; make smaller (draw-obj 4) ;; draw the body (with-state ;; draw the wings in a new state (rotate (vector 180 0 0)) (translate (vector 0 0 -0.5)) ;; position and angle right ;; calculate the wing angle based on speed (let ((a (- 90 (* (butterfly-flap-amount s) (+ 1 (sin (* (butterfly-speed s) (+ (butterfly-fuzz s) (time))))))))) (with-state (rotate (vector 0 0 a)) (draw-obj 3)) ;; draw left wing (with-state (scale (vector 1 -1 1)) ;; flip (rotate (vector 0 0 a)) (draw-obj 3)))))) ;; draw right wing
There is only immediate mode rendering at the moment, so the transforms are not optimised and little things like draw-obj takes an id of a preloaded chunk of geometry, rather than specifying it by name need to be fixed. However it works well and the thing that was most successful was welding together the Nightjar Game Engine (HTML5 canvas) with fluxus (WebGL) and using them together. This works by having two canvas elements drawn over each other – all the 2D (text, effects and graphs) are drawn using canvas, and the butterflies are drawn in 3D with WebGL. The render loops are run simultaneously with some extra commands to get the canvas pixel coordinates of objects drawn in 3D space.
On Saturday I teamed up with Falmouth University’s Makernow team to do a kids coding event at the Royal Cornwall Show with a new Raspberry Pi cube (based on the one used at the DeerShed Festival last year). We had a constant stream of families and kids wanting to try Scratch coding, and we had a 50/50 gender balance in terms of helpers which I think with these sorts of events is critical.
The part I like best about public events like this are talking with the parents and teachers. The best questions are the fundamental ones: “why should we be teaching them how to program?” which was a great opportunity to get my thoughts straight – the official economic reason is not one I’m so motivated by: “to encourage more talent in the tech sector”. The actual reason I do all this (mostly on a voluntary basis) is the feeling that as we find ourselves living in a computational society, where everything we do is algorithmically processed, the future for people who only know how to consume technology is very different from those who are not afraid to question it, who know it’s possible to take it to bits and rebuild it in new ways.
This is also the reason that I can’t get very excited when teachers tell me they are buying iPads for use in their school – there are some interesting programming environments on them, but the kind of creativity they support cannot, due to Apple’s core business model, encompass this kind of questioning – they can’t escape the sandbox. For example, when I last checked, you can program iPads, but not share the code or work collaboratively as it would bypass the AppStore distribution model to do this.
This relates to the answer I gave to “why should we get a Raspberry Pi” – as it provides a platform that encourages a fearless relationship with technology, it doesn’t have the family email account on it, it’s cheap and nobody cares if you manage to delete the entire operating system, just copy a new sdcard. This stuff has to be possible, and encouraged – if we are to eventually have a society that can have any meaningful debate on increasingly thorny computational/network/society issues such as those related to GCHQ mass spying.
Back to the kids – it was interesting that the majority of the older ones had already used Scratch, either as part of their normal school lessons, or an after school activity (anecdotally, this is sharp improvement over the last year or so). Some of them were keen to show off their skills, which was a great way to demonstrate to the younger ones what was possible.
For the older ones I’m continuing work on the Minecraft API coding project – making simple 3D primitives to demonstrate functional programming in Python. You can explore the results of your programs by walking around and digging into structures generated in a familiar world. More on new versions of that soon.
Bumper crop is an android game I’ve just started working on with Dr Misha Myers as part of the Play to Grow project: “exploring and testing the use of computer games as a method of storytelling and learning to engage urban users in complexities of rural development, agricultural practices and issues facing farmers in India.”
(Warning – contains machine translated Hindi!)
I’m currently working out the details with artist Saswat Mahapatra and Misha, who have been part of the team developing this game based on fieldwork in India working with farmers from different regions. They began by developing a board game, which allowed them to flexibly prototype ideas with lots of people without needing to worry about software related matters. This resulted in a great finished product, super art direction and loads of assets ready to use. I very much like this approach to games design.
From my perspective the project relates very closely to groworld games, germination x, as well as the more recent farm crap app. I’m attempting to capture the essence of the board game and restrict the necessary simplifications to a minimum. The main challenge now that the basics are working is providing an approximation of bartering and resource management between players that board games are so good at, into a simple interface – also with the provision of AI players.
Last week I had the honour of both performing with Alex and presenting at Thinking Digital 2014. Suzy O’Hara invited me to represent the intersection of art, science and education of FoAM kernow and present the work I’ve been doing with the Sensory Ecology Group at Exeter University. I did a quick Egglab game demo and related some thoughts on working with scientists and how it connects with my experience teaching programming in the classroom.
It was an interesting and unusual venue for me, organiser Herb Kim is very much developing on the TED theme – so lots of extremely well considered, motivating and inspiring talks. Much of the context was one of venture capital and startup business so it was interesting to see an explosive talk by Aral Balkan on the implications of Facebook and Google’s business models on the future of our society (he included some of the other presenter’s companies too). This reminded me very much of the themes we explored in Naked on Pluto, but coming from a new angle.
Personally his talk was challenging to me as he roundly attacked the free software movement, for essentially providing a great sandbox for enthusiasts and well funded companies – but incapable of doing much more in terms of data security for real people. As a designer, he sees this as essentially a design problem – one that these companies have solved for themselves but is utterly lacking in devices such as the Firefox phone OS. For Aral, this is fundamentally a design problem that needs it’s own movement, and new business models to be developed. These business models need to take into consideration long term usability (for which user privacy is an essential feature) rather than ultra short term profit ‘pump and dump’, selling of people’s information for vast amounts – i.e. silicon valley ‘business as usual’.
Two things are apparent to me following this talk – one is that I have been labouring under the impression that a particular focus on design is somehow implicitly tied with specific business practices – simplification as wallpapering over data harvesting, and other tactics. This is very much a short sighted developer view, and is wrong – they can of course service different types of businesses.
The other point came during his 3 slide explanation of how to start your own social network (1. fork a github repo, 2. set up a server and 3. install it). Clearly even this satirical simplification is beyond all but existing software developers (many of whom are working for companies reliant on user surveillance in some indirect or direct way). The challenge for me is that I can’t ultimately see a way to make ‘interface as user experience’ ever converge on anything other than exploitation. Can ‘user experience’ ever regard people philosophically as anything but consumers – regardless of the underlying business model?
The problem in solving that is that we now have two problems – the terrible state of software engineering preventing accessibility (i.e programming at large still stuck in the 70′s) and the lack of understanding in society of what a computer is and how it works. The second of these problems is being addressed in some part by the activities of CodeClub (Aral is [correction: was] a director of this organisation) and similar education initiatives. Regarding pushing software engineering forward, in some way I think recent livecoding takeup by musicians over programmers is a fascinating development here, in terms of showing us how programming – when it’s taken and twisted into very strange and new forms, can start to make sense and work for ‘real people’.
I’ve been working lately with the Heliconius research group at the University of Cambridge on a game to explain the evolution of mimicry in butterfly wing patterns. It’s for use at the Summer Science Exhibition at the Royal Society in London, where it’ll be run on a large touch screen for school children and visiting academics to play.
The game models biological processes for education purposes (as opposed to the genetic programming as used on the camouflage egg game), and the process of testing this, and deciding what simplifications are required has become a fascinating part of the design process.
In biosciences, genetics are modelled as frequencies of specific alleles in a given population. An allele is a choice (a bit like a switch) encoded by a gene, so a population can be represented as a list of genes where each gene is a list of frequencies of each allele. In this case the genetics consists of choices of wing patterns. The game is designed to demonstrate the evolution of an edible species mimicking a toxic one – we’ll be publishing the game after the event. A disclaimer, my terminology is probably misaligned in the following code, still working on that.
;; an allele is just a string id and a probability value
(define (allele id probability)
(list id probability))
;; a gene is simply a list of alleles
;; return the id of an allele chosen based on probability
(define (gene-express gene)
(let ((v (rndf)))
(lambda (allele r)
(let ((segment (+ (car r) (allele-probability allele))))
(if (and (not (cadr r))
(< v segment))
(list segment allele)
(list segment (cadr r)))))
(list 0 #f)
;; a chromosome is simple list of genes
;; returns a list of allele ids from the chromosome based on probability
(define (chromosome-express chromo)
(map gene-express chromo))
When an individual is removed from the population, we need to adjust the probabilities by subtracting based on the genetics of the eaten individual, and the adding to the other alleles to keep the probabilities summing to one:
;; prevents the probability from 'fixing' at 0% or 100%
(define (calc-decrease p)
(* (min p (- 1 p)) allele-decrease))
;; remove this genome from the population
(define (gene-remove-expression gene genome)
(let ((dec (calc-decrease (allele-probability (car gene)))))
(let ((inc (allele-increase dec (length gene))))
(if (eq? (allele-id allele) genome)
allele (- (allele-probability allele) dec))
allele (+ (allele-probability allele) inc))))
9,000 players, 20,000 games played and 400,000 tested egg patterns later we have over 30 generations complete on most of our artificial egg populations. The overall average egg difficulty has risen from about 0.4 seconds at the start to 2.5 seconds.
Thank you to everyone who contributed their time to playing the game! We spawned 4 brand new populations last week, and we’ll continue running the game for a while yet.
In the meantime, I’ve started working on ways to visualise the 500Mb of pattern generating code that we’ve evolved so far – here are all the eggs for one of the 20 populations, each row is a generation of 127 eggs starting at the top and ordered in fitness score from left to right:
This tree is perhaps more useful. The ancestor egg at the top is the first generation and you can see how mutations happen and successful variants get selected.