dave's blog of art and programming

missing acceptable use policy

Some images of the Machinima exhibition at FILE 2011 in São Paulo, featuring a movie of a livecoded abstractification of a Missile Command game written/performed/destroyed/recorded in Fluxus.

Some more work on upcycling second hand PS2's into cheap fluxus machines. The next step is to embrace the vector units - strange little processors for doing things with points lines and colours extremely fast.

This is quite a daunting task for various reasons, not only can you run floating point and integer calculations in parallel, the VU's also have instruction pipelining where a calculation can take a number of instructions to finish. The advantage of this is that you can interleave work that you are doing while you wait for things to complete, and make extremely fast programs - but it takes some time to get your head around.

Luckily it's made a whole lot easier by a free software tool called OpenVCL. This lets you write fairly straight forward assembler and it at least makes sure it should run correctly by spacing it out for you - future versions may also start optimising the code automatically.

This is my first very basic renderer, which simply applies the world->screen transform to each vertex in an object. It's job is to load data put in it's memory by the CPU, process and send it to the "graphics synthesizer" to draw the gouraud shaded triangles. It's not only much faster than doing the same job on the CPU, but it leaves it free for other processing (such as running a Scheme interpreter).

        .syntax new
        .name vu1_unlit
        .vu
        .init_vf_all
        .init_vi_all
        --enter
        --endenter
        ; load the matrix row by row
        lq      world_screen_row0, 0(vi00)
        lq      world_screen_row1, 1(vi00)
        lq      world_screen_row2, 2(vi00)
        lq      world_screen_row3, 3(vi00)
        ; load the params and set the addresses for
        ; the giftag and vertex data
        lq      params, 4(vi00)
        iaddiu  giftag_addr, vi00, 5
        iaddiu  vertex_data, vi00, 6
        ; move the vertex count to an integer 
        ; register so we can loop over it
        mtir    vertex_index, params[x]
vertex_loop:
        ; load the colour (just increments vertex_data)
        lqi     colour, (vertex_data++)
        ; load vertex position
        lq      vertex, 0(vertex_data)
        ; apply the transformation
        mul     acc, world_screen_row0, vertex[x]
        madd    acc, world_screen_row1, vertex[y]
        madd    acc, world_screen_row2, vertex[z]
        madd    vertex, world_screen_row3, vertex[w]
        div     q, vf00[w], vertex[w]
        mul.xyz vertex, vertex, q
        ; convert to fixed point
        ftoi4   vertex, vertex
        ; overwrite the old vertex with the transformed one
        sqi     vertex, (vertex_data++)
        ; decrement and loop
        iaddi   vertex_index, vertex_index, -1
        ibne    vertex_index, vi00, vertex_loop
        ; send to gs
        xgkick  giftag_addr
        --exit
        --endexit

Naked on Pluto is a text adventure game, it's world is comprised of descriptions, populated by bits and pieces of text from your facebook profile while the space is largely left to your interpretation and imagination. There are some ways of visualising it though.

We have been archiving versions of the game world every hour for the last year in a git repo. In these files the only facebook profile information we have stored are names, which are considered public information in the agreement with facebook. We are interested in using this data to see how we built the game and potentially how people have been playing it (although one snapshot per hour isn't really enough for that).

I started out experimenting with GraphViz and Gephi but I ended up switching to fluxus and adapting the exquisite code visualiser to read through thousands of dot files to make a dynamically changing graph. This animation shows the process from us building the world to the introduction of the first playtesters over the course of a few months:

A new version of android fluxus, with pdata, multitouch capability, scene inspection and a very pink test script included (see code below) fluxus-0.0.2.apk & source. Press the trackball button to edit the scheme code.

(clear)

(define twirl-shape
  (with-state
   (hint-unlit)
   (build-polygons 40 triangle-strip)))

(define finger-shapes
  (with-state
   (hint-none)
   (hint-unlit)
   (hint-wire)
   (list (build-polygons 30 triangle-strip)
         (build-polygons 30 triangle-strip))))

(define (spiral)
  (line-width 5)
  (pdata-map! (lambda (c) (vector 1 0.29 0.42)) "c")
  (pdata-index-map!
   (lambda (i p)
     (let ((i (* i 0.8)))
       (vmul (vector (sin i) (cos i) 0) (* 0.02 i i))))
   "p")
  (pdata-copy "p" "pref") ; only really needed for animation
  (pdata-map!
   (lambda (p pref)
     (vadd pref (vmul (crndvec) 0.2)))
   "p" "pref"))

(for-each
 (lambda (finger-shape)
   (with-primitive finger-shape (spiral)))
 finger-shapes)

(with-primitive
 twirl-shape
 (pdata-map!
  (lambda (c)
    (vector 1 0.29 0.42))
  "c"))

; using with-primitive is really slow, so directly use grab
; returns the distance between the objects
(define (get-pinch)
  (grab (car finger-shapes))
  (let ((a (vtransform (vector 0 0 0) (get-transform))))
    (ungrab)(grab (cadr finger-shapes))
    (let ((b (vtransform (vector 0 0 0) (get-transform))))
      (ungrab)
      (vdist a b))))

; store pinch as it's slow to calculate
(define pinch 1)

(every-frame
 (begin
   ; do the twirling
   (with-primitive
    twirl-shape
    (pdata-index-map!
     (lambda (i p)
       (let ((i (* i 0.5)))
         (vmul (vector (sin i) (cos i) (* 2 (cos (* i 10.43))))
               (* 5 (sin (* pinch (+ i (* 0.1 (time))) 0.1))))))
     "p"))
   ; check for touch events
   (for-each
    (lambda (touch-id)
      (let ((finger-shape (list-ref finger-shapes touch-id)))
        (with-primitive
         finger-shape
         (identity)
         (translate
          (get-point-from-touch touch-id))
         (rotate (vector 0 0 (* (time) 10)))))
      (set! pinch (get-pinch)))
    (get-touch-ids))))

Getting stuff to work on PS2 wasn't quite as easy as I probably made it sound in the last homebrew post. The problem with loading code from usb stick is that there is no way to debug anything, no remote debugging, no stdout - not even any way to render text unless you write your own program to do that.

The trick is to use the fact that we are rendering a CRT TV signal and that you can control what gets rendered in the overscan area (think 8bit loading screens). There is a register which directly sets the background colour of the scanline - this macro is all you need:

#define gs_p_bgcolor        0x120000e0    // Set CRTC background color

#define GS_SET_BGCOLOR(r,g,b) \
            *(volatile unsigned long *)gs_p_bgcolor =    \
        (unsigned long)((r) & 0x000000FF) <<  0 | \
        (unsigned long)((g) & 0x000000FF) <<  8 | \
        (unsigned long)((b) & 0x000000FF) << 16

Which you can use to set the background to green for example:

GS_SET_BGCOLOR(0,255,0);

Its a good idea to change this at different points in your program. When you get a crash the border colour it's frozen with will tell you what area it was last in, allowing you to track down errors.

There is also a nice side effect that this provides a visual profile of your code at the same time. Rendering is synced to the vertical blank - when the CRT laser shoots back to the top of the screen a new frame is started and you have a 50th of a second (PAL) to get everything done. In the screenshot below you can see how the frame time breaks down rendering 9 animated primitives - and why it might be a good idea to use some of these other processors:

Getting things to render on the PS2 is a little more complicated than using OpenGL and it's also a very different system to a PC. On the right you can see a block diagram of the Emotion Engine - it consists of the EE core, the CPU on the left and the GS - Graphics Synthesiser, on the right. In between are 2 other processors called Vector Units - very fast processors designed to do things to vectors - points, colours etc.

All the Graphics Synthesiser can do is rasterise 2D shapes with points given in screen space and do your texturing & gouraud shading for you. It can draw points, lines, triangles or quads in various configurations (similar to OpenGL). However all the 3D transformations and lighting calculations have to happen elsewhere - in one, or both of the Vector Units or the CPU.

So how do we get the GS to render something? Well you send it chunks of data, called GS Packets, that look a little like this:

GIF Tag 1
Primitive data
GIF Tag 2
Primitive data
...

The GIF Tags contain information on what sort of primitive it should draw and how the primitive data is laid out. The primitive data is the same as the primitive data in fluxus - vertex positions, colours, texture coordinates, texture data etc.

Once I had tinyscheme and the basic scenegraph working that the minimal fluxus build uses, I wrote a very simple renderer running on the EE core to apply the transformation matrices to the primitives (with similar push and pop to OpenGL). It doesn't calculate lighting at the moment, so it's just setting the vertex colours to the normal's values for debugging. This is a literal photographic screen shot of my PS2 running exactly the same test fluxus script as the android was running:

The next thing I wanted to do was see if I could compile the minimal android version of fluxus for the PS2. All the PS2SDK examples are written in C, and when I tried the C++ compiler at link time I got a bunch of these odd errors:

ps2-main.cpp: undefined reference to `__gxx_personality_v0'

It turns out the C++ compiler does not support exceptions so you need to add this line to the makefile:

EE_CXXFLAGS += -fno-exceptions

The other thing to get used to is one of the side effects of a machine with so many processors is that you need send lots of data around between them using DMA transfer, or direct memory access. DMA works on chunks of memory at a time, so your data needs to be aligned on particular byte boundaries. This sounds a lot more complicated than it is in practice (although it does lead to really obscure bugs if you get it wrong).

For instance, when making arrays on the heap you can do this:

struct my_struct
{
    int an_int;
    float my_array[8] __attribute__((__aligned__(16)));
    float a_float;
};

Which tells gcc to sort it out for you by forcing the pointer to my_array to fall on a 16 byte boundary.

When allocating from the heap the EE kernel provides you with a memalign version of malloc:

float *some_floats=(float*)memalign(128, sizeof(float) * 100);

The pointer some_floats will be aligned to a 128 byte boundary. This works as normal with free().

At this point, other than a few changes to tinyscheme for string functions that don't exist on the PS2 libraries, most of the fluxus code was building. The only problem was the OpenGL ES code, as although the PS2 has some attempts at libraries that work a bit like OpenGL, the real point of playing with this machine is to write your own realtime renderer. a bit more on that next...

In order to figure out the source of the crashes I was getting I quickly wrote a linux wrapper around the android code so it could run in a simpler-to-debug environment. This of course was running without any problems, so not much help, but it'll come in useful anyway.

Next step, try it in the emulator - this exhibited the crash, but I couldn't get a good stack trace from gdb. While investigating this I decided I could use a simpler method with the hardware, which is much faster than the emulator. addr2line is a program you can use for making sense out of crash reports, you take the address reported in the crash log (using adb logcat to read them from the android device), feed it to addr2line with the library and it tells you the corresponding line number in the source.

It turned out the crashes were down to something more fundamental as they were happening all over the place in the scheme interpreter - so I had a bit of a wider search and it came down to threads, I hadn't noticed that Android's GLSurfaceView uses a separate thread for rendering, so I was calling the scheme interpreter in the midst of other evaluations, not too great. A simple fix using java locks around the evaluations and rendering calls.

The next thing was implementing more fluxus features. One major improvement is using with-state to wrap push and pop so they are implicitly scoped. This means you can't miss (or add extra) a pops or pushes to your state stack manipulations, so something like:

(push)
(translate (vector 1 2 3))
(build-cube)
(pop)

becomes:

(with-state
    (translate (vector 1 2 3))
    (build-cube))

To do this I had to learn TinyScheme's macro language which is different to Racket's, but this isn't too complex a task:

(define-macro (with-state . args)
  `(begin (push) (let ((r (begin ,@args))) (pop) r)))

Here we call push then basically "paste" the code given to the with-state function into a begin block, and record it's return before calling pop. Then we return the value. There is a very similar macro for with-primitive.

There are now quite a few commands ported - there is a new version packaged here, and this is the test script that I'm using:

(clear)
(colour (vector 0 0.5 1 1))
(define cubes
  (map
   (lambda (p)
     (with-primitive p
       (apply-transform)
       (translate (vector -2.5 2 5))
       (rotate (vector 45 0 0))
       (translate
        (vector
         (quotient (- p 1) 6)
         0
         (modulo (- p 1) 6)))
       p))
   (build-list
    (lambda (n)
      (with-state
       (scale (vector 0.5 0.05 0.5))
       (build-cube)))
    36)))

(every-frame
 (for-each
  (lambda (p)
    (with-primitive
     p
     (rotate (vmul (vector (sin p) 0 (cos p)) 5))))
  cubes))

fluxus installed on the emulator

Mostly my android experience has been good so far, it's been very quick to get things running and be fairly productive. It doesn't come without it's share of yak shaving though. I've spent quite a lot of time trying to get remote debugging of native code working with my phone (a HTC desire) with little success. This seems to be a fairly well documented problem, the symptoms are messages like this when running ndk-gdb:

ERROR: Could not setup network redirection to gdbserver?
Maybe using --port=
to use a different TCP port might help?

When run with --verbose it seems something a little different is wrong:

run-as: Package 'am.fo.nebogeo.fluxus' has corrupt installation

Which looks like some result of having a rooted phone (which I do) as ndk-debug is a possible attack vector for doing nasty things and is therefore very critical of the permissions of the directories on the system. In order to fix this I installed ROM Manager from the market which contains a script to fix permissions. This didn't work at first by pressing the button in the app so after some poking around I found the shell script in: /data/data/com.koushikdutta.rommanager/files/fix_permissions

It turned out that this script depended on an interesting utility called busybox, which provides tiny versions of GNU programs for phones. The easiest way to install this was to install TitaniumBackup and click on it's "Problems" button which would download it, then copy it from /data/data/com.keramidas.TitaniumBackup/files into /system/bin/ and run busybox --install -s /system/bin/

Then the script could run, and additionally I needed to run chmod 774 /data/data/* to get make sure everything was in the correct state, but still no luck with gdb. At this point I decided my time would be better spent on making a linux version of the android app for debugging purposes and get it running in the software emulator. More on this soon.

I'm also documenting bits and pieces of my android development notes on FoAM's wiki here.

I recently took a long train journey and had an idea to use my android phone for taking timelapse photos for Miska Knapek. I had no laptop with me, and realised there was no way I could script anything on the phone itself to do this, or anything more complex. This was the kernel of an idea that ended up as two intense days of hacking fluxus into android.

It's running as native code (underneath the JVM), and as it's got the same type of processor as the NDS it shares some code with Betablocker DS which has similar restrictions. There is only a tiny amount of fluxus code ported at the moment, but the scenegraph and core maths is pretty much the same (with some changes to fixed point). I've used TinyScheme for the interpreter (the same one Impromptu is based on) which gives you an impressively complete language with a very small footprint.

This is just a quick proof of concept (how far I got is testament to the excellent documentation and ease of portability using C++ with the NDK more than my abilities). It would be nice to expose all the sensors to the script (motion, compass, GPS, camera, wifi) and make it quite general purpose. Writing code with the touch screen is do-able but not very usable - but I have some ideas in this area also...

Get the source and, of course the package. Touch the screen when it's started up to view and eval the source.