Category Archives: fluxus hacking

Ouya development experiments

The Ouya is a tiny game console which is designed for promoting indy games rather than traditional high budget productions. It’s cheap compared to standard games hardware, and all the games are free to play at least in demo form. It’s very easy to start making games with as it’s based on Android – you can just plug in a USB cable and treat it just the same as any other Android device. You also don’t need to sign anything to start building stuff – it’s just a case of adding one line to your AndroidManifest.xml to tell the Ouya that the program is a game:

 <category android:name="tv.ouya.intent.category.GAME"/>

and adding a 732×412 icon in “res/drawable-xhdpi/ouya_icon.png” so it shows up on the menu.

130327_ouya_0021

There is a lot to like about the Ouya’s philosophy, so I tried out some graphics experiments with it to get better acquainted:

This program was made using Jellyfish, part of my increasingly odd rendering stack which started out as a port of Fluxus to PS2. It’s a type of microcode written inside TinyScheme and running in a separate virtual machine that makes it possible to process a lot of geometry without garbage collection overheads. At some point I might write a compiler for this, but writing the code longhand at the moment means I can tweak the instruction set and get a better understanding of how to use it. Here is the microcode for the ribbons above, they run at 20,000 cycles per frame each (so about 1.2MHz):

;; register section
   8 20000 0 ;; control (pc, cycles, stack)
   mdl-size prim-tristrip 1 ;; graphics (size, primtype, renderhints)
   0 0 0 ;; pos
   0 0 0 ;; sensor addr

   ;; program data section
   mdl-start 0 0     ;; 4 address of current vertex
   mdl-start 0 0     ;; 5 address of accum vertex (loop)
   0 0 0             ;; 6 influence
   0 0 0             ;; temp

   ;; code section 
   ;; add up differences with every other vertex
   ldi  4  0         ;; load current vertex
   ldi  5  0         ;; load accum vertex
   sub  0  0         ;; get the difference
   lda  6  0         ;; load the accumulation
   add  0  0         ;; accumulate
   nrm  0  0         ;; normalise
   sta  6  0         ;; store accumulation
   add.x 5 1         ;; inc accum address

   ;; accumulation iteration 
   lda  5  0         ;; load accum address
   ldl  mdl-end 0    ;; load end address
   jlt  2  0         ;; exit if greater than model end (relative address)
   jmp  8  0         ;; return to accum-loop

   ;; end accum loop
   ;; push away from other verts
   lda  6  0         ;; load accum
   ldl -0.1 0        ;; reverse & make smaller
   mul 0 0

   ;; attract to next vertex
   ldi 4 0           ;; load current
   ldi 4 1           ;; load next
   sub 0 0           ;; get the difference
   ldl 0.4 0
   mul 0 0           ;; make smaller
   add 0 0           ;; add to accum result

   ;; do the animation
   ldi 4 0           ;; load current vertex
   add 0 0           ;; add the accumulation
   sti 4 0           ;; write to model data

   ;; reset the outer loop
   ldl 0 0           ;; load zero
   sta 6 0           ;; reset accum
   ldl mdl-start 0   
   sta 5 0           ;; reset accum address

   add.x 4 1         ;; inc vertex address
   lda 4  0          ;; load vertex address
   ldl mdl-end 0     ;; load end address
   jlt 2  0          ;; if greater than model (relative address)
   jmp 8  0          ;; do next vertex

   ;; end: reset current vertex back to start
   ldl mdl-start 0
   sta 4 0

   ;; waggle around the last point a bit
   lda mdl-end 0     ;; load vertex pos
   rnd 0 0           ;; load random vert
   ldl 0.5 0         
   mul 0 0           ;; make it smaller
   add 0 0           ;; add to vertex pos
   sta mdl-end 0     ;; write to model

   jmp 8 0

Visual livecoding environments: big screenshots

Some decent sized screenshots of al jazari and scheme bricks rendered with fluxus’s tiled frame dump command. This set includes some satisfyingly glitchy al jazari shots – not sure what was causing this, I initially assumed it was the orthographic projection, but the same artefacts occurred on the perspective first-person robot views, so it needs further investigation.

ajbig

ajbig2

ajbig5

ajbig6

schemebricks

schemebricks2

schemebricks3

A fluxus workshop plan

I’ve been getting some emails asking for course notes for fluxus workshops, I don’t really have anything as structured as that but I thought it would be good to document something here. I usually pretty much follow the first part of the fluxus manual pretty closely, trying to flip between visually playful parts and programming concepts. I’ve taught this to teenagers, unemployed people, masters students, professors and artists – it’s very much aimed at first time programmers. I’m also less interested in churning out fluxus users, and more motivated by using it as an introduction to algorithms and programming in general. Generally it’s good to start with an introduction to livecoding, where fluxus comes from, who uses it and what for. I’ve also started discussing the political implications of software and algorithmic literacy too.

So first things first, an introduction to a few key bindings (ctrl-f fullscreen/ctrl-w windowed), then in the console:

  1. Scheme as calculator – parentheses and nesting simple expressions.
  2. Naming values with define.
  3. Naming processes with define to make procedures.

Time to make some graphics, so switch to a workspace with ctrl-1:

  1. A new procedure to draw a cube.
  2. Calling this every frame.
  3. Mouse camera controls, move around the cube.
  4. Different built in shapes, drawing a sphere, cylinder, torus.

Then dive into changing the graphics state, so:

  1. Colours.
  2. Transforms.
  3. Textures.
  4. Multiple objects, graphics state persistent like changing a “pen colour”.
  5. Transform state is applicative (scale multiplies etc).

Then tackle recursion, in order to reduce the size of the code, and make much more complex objects possible.

  1. A row of cubes.
  2. Make it bend with small rotation.
  3. Animation with (time).

At this point they know enough to be able play with what they’ve learnt for a while, making procedural patterns and animated shapes.

After this it’s quite easy to explain how to add another call to create tree recursion, and scope state using (with-state) and it all goes fractal crazy.

This is generally enough for a 2 hour taster workshop. If there is more time, then I go into the scene graph and explain how primitives are built from points, faces and show how texture coords work etc. Also the physics system is great to show as it’s simple to get very different kinds of results.

Planet Fluxus

Fluxus now runs in a browser using WebGL. Not much is working yet – (draw-cube), basic transforms, colours and textures. I’ve also built a small site in django so people can share (or perhaps more likely, corrupt) each other’s scripts. Also much inspired by seeing a load of great live coding at the algoraves by Davide Della Casa and Guy John using livecodelab.

fluxuswebgl2

fluxuswebgl

This is a spin off from the work I did a few weeks ago on a silly Scheme to Javascript compiler. It’s still pretty silly, but in order to explain better, first we take a scheme program like this:

;; a tree
(define (render n)
    (when (not (zero? n))
        (translate (vector 0 1 0))
        (with-state
            (scale (vector 0.1 1 0.1))
            (draw-cube))
        (scale (vector 0.8 0.8 0.8))
        (with-state
            (rotate (vector 0 0 25))
            (render (- n 1)))
        (with-state
            (rotate (vector 0 0 -25))
            (render (- n 1)))))

(every-frame 
    (with-state
        (translate (vector 0 -3 0))
        (render 8)))

Then parse it straight into JSON, so lists become Javascript arrays and everything else is a string, also doing minor things like switching “-” to “_”:

[["define",["render","n"],
    ["when",["not",["zero_q","n"]],
        ["translate",["vector","0","1","0"]],
        ["with_state",
            ["scale",["vector","0.1","1","0.1"]],
            ["draw_cube"]],
        ["scale",["vector","0.8","0.8","0.8"]],
        ["with_state",
            ["rotate",["vector","0","0","25"]],
            ["render",["-","n","1"]]],
        ["with_state",
            ["rotate",["vector","0","0","-25"]],
            ["render",["-","n","1"]]]]],

["every_frame",
    ["with_state",
    ["translate",["vector","0","-3","0"]],
    ["render","8"]]]]

Next we do some syntax expansion, so functions become full lambda definitions, and custom fluxus syntax forms like (with-state) get turned into lets and begins wrapped with state (push) and (pop). These transformations are actually written in Scheme (not quite as define-macros yet), and are compiled at an earlier stage. It now starts to increase in size:

[["define","render",
    ["lambda",["n"],
        ["when",["not",["zero_q","n"]],
            ["translate",["vector","0","1","0"]],
            ["begin",
                ["push"],
                ["let",[["r",["begin",
                        ["scale",["vector","0.1","1","0.1"]],
                        ["draw_cube"]]]],
                    ["pop"],"r"]],
            ["scale",["vector","0.8","0.8","0.8"]],
            ["begin",
                ["push"],
                ["let",[["r",["begin",
                        ["rotate",["vector","0","0","25"]],
                        ["render",["-","n","1"]]]]],
                    ["pop"],"r"]],
            ["begin",
                ["push"],
                ["let",[["r",["begin",
                        ["rotate",["vector","0","0","-25"]],
                        ["render",["-","n","1"]]]]],
                ["pop"],"r"]]]]],

["every_frame_impl",
    ["lambda",[],
        [["begin",
            ["push"],
            ["let",[["r",["begin",
                    ["translate",["vector","0","-3","0"]],
                    ["render","8"]]]],
            ["pop"],"r"]]]]]

Then, finally, we convert this into a bunch of Javascript closures. It’s pretty hard to unpick what’s going on at this point, I’m sure there is quite a bit of optimisation possible, though it does seem to work quite well:

var render = function (n) {
    if (!(zero_q(n))) {
        return (function () {
            translate(vector(0,1,0));
            (function () {
                push()
                return (function (r) {
                    pop()
                    return r
                }((function () {
                    scale(vector(0.1,1,0.1))
                    return draw_cube()
                })()))})();
        scale(vector(0.8,0.8,0.8));
        (function () {
            push()
            return (function (r) {
                pop()
                return r
            }((function () {
                rotate(vector(0,0,25))
                return render((n - 1))
            })()))})()
        return (function () {
            push()
            return (function (r) {
                pop()
                return r
            }((function () {
                rotate(vector(0,0,-25))
                return render((n - 1))
            })()))})()})()}};

every_frame_impl(function () {
    return (function () {
        push()
        return (function (r) {
            pop()
            return r
        }((function () {
            translate(vector(0,-3,0))
            return render(8)
        })()))})()})

Then all that’s needed are definitions for all the fluxus 3D graphics calls – the great thing is that these are also written in Scheme, right down to the low level WebGL stuff, so the only Javascript code needed is half of the compiler (eventually this also can be replaced). I was quite surprised at how easy this is, although it is greatly helped by the similarity of the two languages.

Fluxus at Falmouth University

On Friday Cornwall Locative Arts Network, Cornwall Creative Skills and I took over The Academy for Innovation & Research at Falmouth University with a Fluxus workshop, teaching creative coding via recursive procedural 3D modelling for people new to programming. The thing I like most about Scheme as a programming language is that you can very quickly cover the fundamentals of programming (naming values and processes, recursion, and scope) in a 2 hour course with very little time spent on learning syntax or other fiddly things. We had a great spread of attendants, from local artists and designers to lecturers and PhD students, everyone ending up with their own animated procedural shapes.

IMG_20130531_144314

This was followed by the Cornwall Locative Arts Network meeting at which I presented with Tom Goskar, a digital archaeologist, who uses 3D graphics in fascinating ways to read inscriptions and patterns in ancient monuments. I talked about borrowed scenery, doris and sonic bikes and discussed using ushahidi and beagleboards in artistic and scientific projects.

Later on we were invited by Jowan Sebastian Parker to experience Falmouth University’s MakerNow Lab which is opening soon, providing laser cutters, 3D printers and an electronics workshop. I’m hoping to find time to make extensive use of the lab in the not too distant future! The photo above is of their ‘synths on postcards’ example project.

Fluxus workshop in Falmouth

After rushing around Europe doing a lot of livecoding over the last week (more on that soon) I’m really pleased to announce this workshop closer to home in Falmouth:

CLAN and CREATIVE SKILLS present: CREATIVE CODING taster session for ABSOLUTE BEGINNERS with Dave Griffiths

Friday May 31st 10.30am – 12.30pm
AIR building, Falmouth University, Tremough Campus TR10 9EZ
Places: For 10 people only

Deposit: There are limited places so you will need to pay a £15 deposit to secure your place when you book, which will be refunded before the workshop. If you don’t turn up you lose your deposit. To book: Please email: admin@creativeskills.org.uk

In this workshop for beginners with award winning game designer, creative coder and live coding artist Dave Griffiths you will find out about the emerging art form of live coding and learn how to write simple programmes to create animations in 3D space. You will be introduced to fluxus, an open source game engine for live coding worlds into existence, used by artists, performers and digital practitioners for installations, VJing, games and education.

What will you do? What will you achieve?

You will create 3D animated forms, and have an introduction to fundamental programming concepts, naming of values and processes, recursion and digital representation of colour, 3D shape and texture.

There will be coffees and teas.

At 1pm Dave will talk at a free event-

CLAN – Cornwall Locative/ Interactive Arts Network

1-2pm AIR Sandpit

Speakers: creative coder Dave Griffiths is joined by digital archaeologist Tom Goskar.

Al Jazari – ambient occlusion

A big part of the look of Minecraft comes from it’s Smooth lighting, which includes an illumination model called ambient occlusion. Ambient occlusion darkens areas of an object based on how obscured they are from a wide area light source, for example an entire sky area, as opposed to a point light source. This is often complicated to calculate in realtime, but with simplified geometric voxels it’s fairly fast to do, building on the code from the hollowing out optimisation I mentioned previously. For each point on a cube’s corners you can add up it’s surrounding empty cubes – the more empty space in the 8 voxels, the brighter the corner needs to be. In this way, we are assuming light is coming from all directions (hence ambient) and don’t need to take into account positions of lights etc.

occ

Here is the raw ambient occlusion lighting in Al Jazari 2, rendered as unlit vertex colours on each visible cube:

aj2-2

Then we add standard direct lighting to the ambient occlusion:

aj2-3

And here is the final image with textures added:

aj2-4

There are a few artefacts in the lighting due to the cubes not all being the same size (in order to minimise the complexity of what is drawn), here is the world coloured based on the size of the octree leaf node – the brighter blue areas are bigger cubes:

aj2-1

Once the ambient occlusion is calculated we only need to recalculate it when surrounding voxels are changed. In practice the splitting/joining of the octree levels when blocks are created and destroyed seems to take care of this in most cases.

;; returns a list of samples to test
(define (occ-samples pos) 
    (list 
     (vadd pos (vector  -1 -1 -1))
     (vadd pos (vector  -1 -1  0))
     (vadd pos (vector  -1  0 -1))
     (vadd pos (vector  -1  0  0))
     (vadd pos (vector   0 -1 -1))
     (vadd pos (vector   0 -1  0))
     (vadd pos (vector   0  0 -1))
     (vadd pos (vector   0  0  0))))

;; count up the empty voxels and calculate occlusion value
(define (calc-occlusion o pos)
  (let* ((samples (occ-samples pos))
         (coverage (/ (foldl
                       (lambda (p r)
                         (+ (if (octree-leaf-empty? (octree-ref o p)) 1 0) r))
                       0
                       samples)
                     (length samples))))
    (min 1 (* 2 coverage))))

;; helper to set a bunch of verts in one go
(define (pdata-list-set! k l v)
  (for-each (lambda (i) (pdata-set! k i v)) l))

...

;; uses the cube's position and size to set the vertex colour on each corner
(with-primitive my-cube
  (translate pos)
  (pdata-list-set! "c" '(3 7 19) (calc-occlusion o (vadd pos (vector 0 0 0))))
  (pdata-list-set! "c" '(10 14 21) (calc-occlusion o (vadd pos (vector size size size))))
  (pdata-list-set! "c" '(2 4 23) (calc-occlusion o (vadd pos (vector size 0 0))))
  (pdata-list-set! "c" '(9 15 17) (calc-occlusion o (vadd pos (vector 0 size size))))
  (pdata-list-set! "c" '(0 8 18) (calc-occlusion o (vadd pos (vector 0 size 0))))
  (pdata-list-set! "c" '(5 13 22) (calc-occlusion o (vadd pos (vector size 0 size))))
  (pdata-list-set! "c" '(6 12 16) (calc-occlusion o (vadd pos (vector 0 0 size))))
  (pdata-list-set! "c" '(1 11 20) (calc-occlusion o (vadd pos (vector size size 0)))))

Al Jazari – scooping out the insides of planets with scheme

Optimisation is a game where you write more code in order to do less. In Al Jazari 2 doing less means drawing less blocks. Contiguous blocks of the same type are already automatically collapsed into single larger ones with the Octree – but if we can figure out which blocks are completely surrounded by other blocks, we can save more time by not building or drawing them either.

Here is a large sphere – clipped by the world volume, showing a slice through the internal block structure:

aj2-skin1

The next version has all internal blocks removed, in this case shaving 10% off the total primitives built and drawn:

aj2-skin2

The gaps in the sphere from the clipping allow us to look inside at how the octree has optimised the structure. The gain is higher in a more normal Minecraft set up with a reasonably flat floor covering a large amount of blocks. Here is the code involved, built on top of a functional library I’m building up on to manipulate this kind of data. It maps over each Octree leaf checking all the blocks it touches on each of its six sides, taking into account that the size of the leaf block may be bigger than one.

(define (octree-check-edge f o pos size)
  (define (do-x x y)
    (cond
     ((eq? x -1) #f)
     ((octree-leaf-empty? (octree-ref o (vadd pos (f x y)))) #t)
     (else (do-x (- x 1) y))))
  (define (do-y y)
    (cond
     ((eq? y -1) #f)
     ((do-x size y) #t)
     (else (do-y (- y 1)))))
  (do-y size))

(define (octree-is-visible? o pos size)
  (or
   (octree-check-edge (lambda (x y) (vector size x y)) o pos size)
   (octree-check-edge (lambda (x y) (vector -1 x y)) o pos size)
   (octree-check-edge (lambda (x y) (vector x size y)) o pos size)
   (octree-check-edge (lambda (x y) (vector x -1 y)) o pos size)
   (octree-check-edge (lambda (x y) (vector x y size)) o pos size)
   (octree-check-edge (lambda (x y) (vector x y -1)) o pos size)))

(define (octree-calc-viz o)
  (octree-map
   (lambda (v pos size depth)
     (octree-leaf
      (octree-is-visible?
       o pos size)
      (octree-leaf-value v)))
     o))

Skate/BMX ramp projection

Jaye Louis Douce, Ruth Ross-Macdonald and I took to the ramps of Mount Hawke skate park in deepest darkest Cornwall to test the prototype tracker/projection mapper (now know as ‘The Cyber-Dog system‘) in it’s intended environment for the first time. Mount Hawke consists of 20,000 square feet of ramps of all shapes and sizes, an inspiring place for thinking about projections and tracing the flowing movements of skaters and BMX riders.

IMG_20130328_204629

Finding a good place to mount the projector was the first problem, it was difficult to get it far enough away to cover more than a partial area of our chosen test ramp – even with some creative duct tape application. Meanwhile the Kinect camera was happily tracking the entire ramp, so we’ll be able to fix this by replacing my old battered projector with a better model in a more suitable location.

IMG_20130328_203523

The next challenge is calibrating the projection mapping to align it with what the camera is looking at. As they are in different places this is quite fiddly and time consuming to get right, some improvements to the fluxus script will make it faster. Here is Jaye testing it once we had it lined up:

Next it was time to recruit some BMX test pilots to give it a go:

At higher speed it needs a bit of linear interpolation to ‘connect the dots’, as the visualisation is running at 60fps while the tracking is more like 20fps:

This test proved the fundamental idea, and opens up lots of possibilities, different types of visualisations, recording/replaying paths over time as well as the possibility of identifying individual skaters or BMX riders with computer vision. One great advantage this setup has is once it’s running it will work all the time, with no need for continuous calibration (as with RGB cameras) or the use of any additional tracking devices.