Category Archives: rendering

Fluxus on the Raspberry Pi

After getting acquainted with the BeagleBoard while working on the Swamp bike opera I decided to have a look at the similar Raspberry Pi, and particularly it’s graphics systems. The Android/PS2 version of fluxus, called nomadic is ported after a bit of fiddling, but no mouse or keyboard input yet (build it with ‘scons TARGET=RPI’). The graphics driver for the Pi’s VideoCore GPU doesn’t work quite like you normally expect with X11, you get access to it via a custom display manager called dispmanx which allows crazy things like alpha compositing on top of the X display like this:

rpi

Everything you need to develop for the Pi’s GPU can be found inside /opt/vc/ (before finding that I installed a bunch of generic OpenGL ES stuff that wouldn’t work). You need to use the headers and link to the driver libraries there. There are some useful examples inside the hello_pi directory – one important thing is to call bcm_host_init(); and link with libbcm_host.so to initialise BroadCom’s driver before you can do any GPU related calls. I started off by trying to port GlutES to the Raspberry Pi but I got further using the example code – I might come back to that as a way of getting more functionality working along with X11.

I’m also experimenting with a new fluxus editor for nomadic, based on Kassen Oud’s work – a text editor written in fluxus for fluxus that you can see in the screenshot. This will eventually be useful for the standard version too, as it will give much more control over the livecoding environment while livecoding :)

Borrowed Scenery – spacesuits for plants and full screen scrolling in HTML5 canvas

Lots more of Theun’s new artwork which is coming on quickly over the last few days has been added to the game, the glass bubbles (more precisely cloches) are space suits for the plants from our world to live in, in their new patabotanical environment. Information about tagged plants is printed in a Voinych Manuscript inspired font.

I’ve been doing a bit more research into HTML5 canvas, turns out the the screen size doesn’t affect rendering framerate too much, and it now adapts to the browser size. Full screen scrolling is working – centring the view on the player. It wasn’t immediately obvious how this could be done without redrawing all the sprites each frame (which is way too slow), in this respect HTML5 development is really reminding me of 16bit era – hacking things like this to work quickly. It turns out it’s possible to use drawImage using the current canvas as it’s own input – so redrawing the whole thing with an offset:

function scroll(diff_x,diff_y) {
    // calculate the source and destination offsets - whether we offset
    // the source or destination rectangle depends on which direction 
    // we are travelling in 
    var sx=0;
    var dx=diff_x;
    var width=this.ctx.canvas.width-diff_x;
    if (diff_x<0) {
        sx=-diff_x;
        dx=0;
        width=this.ctx.canvas.width+diff_x;
    }
        
    var sy=0;
    var dy=diff_y;
    var height=this.ctx.canvas.height-diff_y;
    if (diff_y<0) {
        sy=-diff_y;
        dy=0;
        height=this.ctx.canvas.height+diff_y;
    }

    // remember to convert to ints, subpixel 
    // scrolling results in crazy artifacts
    this.ctx.drawImage(this.ctx.canvas,
                       ~~(sx),~~(sy),~~(width),~~(height),
                       ~~(dx),~~(dy),~~(width),~~(height));
}

Map rendering with OSM and HTML5 canvas

This week I’m trying to get as much code done on the borrowed scenery game as possible – today, fixing the map rendering.

I decided to load map tiles from OpenStreetMaps directly rather than using OpenLayers. This is done quite simply, accessing the millions of pre-rendered png files with URLs in this form:

a.tile.openstreetmaps.org/z/x/y.png

Where x and y are the tile coordinates and z is the zoom level. The question is, given a GPS coordinate – how do you find the right tiles? Luckily the OSM site is very helpful. This is what I’m using, where lat and lon are specified in degrees:

function latlon_to_tile(lat,lon,zoom) {
    with(Math){
        var m=pow(2,zoom);
        var lat_rad=lat*PI/180;
        return [floor((lon+180)/360*m),
                floor((1-Math.log(tan(lat_rad) + 
                       1/cos(lat_rad))/PI)/2*m)];
    }
}

I’m then doing quite a bit of work chopping the images up to create a 4×4 grid of game tiles for each map tile on the fly before the perspective transform. This brought up an issue with processing images loaded from domains other than the current server – try and do anything with the data you’ve loaded other than display it and this happens:

Unable to get image data from canvas because the canvas has been tainted by cross-origin data.

The answer is quite simple, as presumably it’s quite rare for this to actually cause vulnerabilities (perhaps if executing image data in some way?) you can simple add this property to the image:

image.crossOrigin = "anonymous";

And everything works as normal, cross origin data can be treated as if you’ve loaded it locally. I’ve also spent quite a bit of time fiddling around with the scaling to make the street names visible. There will probably be some more image processing to come. I also found another MMO game that used OSM data: Monopoly City Streets.

Fast HTML5 sprite rendering

After quite a lot of experimentation with HTML5 canvas, I’ve figured out a way to use it with the kind of big isometric game worlds used for Germination X which are built from hundreds of overlapping sprites. There are lots of good resources out there on low level optimisations, but I needed to rethink my general approach in order to get some of these working.

It was quite obvious from the start that the simple clear screen & redraw everything way was going to be far too slow. Luckily HTML5 canvas gives us quite a lot of options for building less naive strategies.

A debug view of the game with 10 frames of changes shown with two plant spirits and one butterfly moving around.

The secret is only drawing the changes for each frame (called partial re-rendering in the link above). To do this we can calculate sprites which have changed and the ones they overlap with. The complication is maintaining the draw order and using clipping to keep the depth correct without needing to redraw everything else too.

In the game update we need to tag all the sprites which change position, rotation, scale, bitmap image, transparency etc.

Then in the render loop we build a list of all sprites that need redrawing, along with a list of bounding boxes for each overlapping sprite of the changed sprites that touch them. There may be more than one bounding box as a single sprite may need to be redrawn for multiple other changed sprites.

For each changed sprite:
    Get the bounding box for the changed sprite
    For each sprite which overlaps with this bounding box: 
        If overlapping sprite has already been stored:
            Add the bounding box to overlapping sprite's list 
        Else:
            Store overlapping sprite and current bounding box.
    Add the changed sprite to the list.

Assuming the sprites have been sorted into depth order, we now draw them using the list we have made (we would only need to loop over the redraw list if we built it in depth sorted order).

For each sprite:
    If it's in the redraw list:
        If it's not one of the originally changed sprites:
            Set a clipping rect for each bounding box.
        Draw the sprite.
        Turn off the clipping, if it was used.

With complex scenes and multiple moving objects, this algorithm means we only need to redraw a small percentage of the total sprites visible – and we start to approach Flash in terms of performance for games (I suspect that flash is doing something similar to this under the hood). The code is here, currently written in HaXE, but will probably end up being ported to Javascript.

Germination X in HTML5 canvas

I’ve spent a couple of days on an experimental not-quite-working-yet port of Germination X to HTML5. Port is probably a slightly misleading word, as it’s actually running exactly the same HaXE code, it’s just compiled to Javascript rather than swf with a load of extra Javascript filling in the stuff needed to replace Flash. The game engine that Germination X is built on is now officially cross platform! It’s great to finally have a non-proprietary browser option for the game, and HTML5 is much faster now than when I first tried it, but it’s still got a long way to go. Problems I’ve noticed so far:

It’s still slow. I’m aware of the limitations and background to browser tech, but it’s depressing that it’s somehow problematic drawing a few hundreds of sprites in 2012 while my ancient Gameboy DS can manage quite fine thankyouverymuch :)

Tinting images is quite a basic thing to do in order to efficiently use images and save bandwidth and memory. Why do I have to loop through pixels in javascript to do this? This is the main cause of slowness in the HTML5 version of the game ATM, hopefully a simple bitmap cache will fix this. There are also some neat ways to do it with offscreen bitmaps and blendmode tricks, but they seem even slower.

Having to write your own mouse event detection for sprite down/up/over/out. I guess this is one of the biggest reasons for the explosion of HTML5 game engines. It’s quite non-trivial and un-fun to set up properly. Actually getting the real mouse coordinates is also quite hard to do.

On the plus side, it’s a breath of fresh air to get back the control of immediate mode – it seems historically that this tends to be the optimal approach, give us the render loop!

Now you’ve got this far – here is some accidental art which happened this morning, and which I’m particularly proud of:

Naked on Pluto: Starting work on a “live world” projection

Following on from the VIDA win, we need to work hard on Naked on Pluto’s gallery installation presence. Although we now have the news website style front page, we need to take the game externalisation to another level, and one of the things required is a realtime projection of the game world. This represents the unfiltered behind-the-scenes view of the game as seen by the bots as they attempt to keep track of what is going on. Technically we decided to do this work using HTML5 canvas, in keeping with the web based themes of the game it needs to work on a browser, which has the added bonus of making gallery setup quite simple.

My first approach was to write a scheme bricks representation for Javascript objects, and see how bits of the game looked if rendered in this way.

This is part of the ArrivalLobby, and all the internal information is present with no explanations, which is great, but it results in very large images. The next thing was to try filtering the objects to remove most of this information:

function node_filter()
{
    this.filter=function(obj)
    {
        return {objects:obj.objects.map(function(object) {
            return object.name;
        })};
    }
}

This code provides a single method for filtering locations in the game – it simply returns an object consisting of a list of names of things found at that location. These filters can be easily changed over time, to include different information or process it in different ways. Rendered with the same code as before, this makes the location diagrams much smaller:

Add a few more locations, put them together in a circular formation (the projection will be onto the floor space in the gallery), add some bezier curves to indicate paths between locations and it looks like this:

There is also some relatively complex jiggery-pokery to detect when bots have moved from one location to another and animate them. The moving bots display more detail including what they are wearing and who has ‘liked’ them. In this image you can see the AdverBot004 moving to the HelpDesk, and the HyperClock and GreenwichClock on the upper right as they move from the Palace Garden.

PS2 vu1 rendering

Some more work on upcycling second hand PS2’s into cheap fluxus machines. The next step is to embrace the vector units – strange little processors for doing things with points lines and colours extremely fast.

This is quite a daunting task for various reasons, not only can you run floating point and integer calculations in parallel, the VU’s also have instruction pipelining where a calculation can take a number of instructions to finish. The advantage of this is that you can interleave work that you are doing while you wait for things to complete, and make extremely fast programs – but it takes some time to get your head around.

Luckily it’s made a whole lot easier by a free software tool called OpenVCL. This lets you write fairly straight forward assembler and it at least makes sure it should run correctly by spacing it out for you – future versions may also start optimising the code automatically.

This is my first very basic renderer, which simply applies the world->screen transform to each vertex in an object. It’s job is to load data put in it’s memory by the CPU, process and send it to the “graphics synthesizer” to draw the gouraud shaded triangles. It’s not only much faster than doing the same job on the CPU, but it leaves it free for other processing (such as running a Scheme interpreter).

        .syntax new
        .name vu1_unlit        
        .vu
        .init_vf_all
        .init_vi_all
        --enter
        --endenter
        ; load the matrix row by row
        lq      world_screen_row0, 0(vi00)
        lq      world_screen_row1, 1(vi00)
        lq      world_screen_row2, 2(vi00)
        lq      world_screen_row3, 3(vi00)
        ; load the params and set the addresses for
        ; the giftag and vertex data
        lq      params, 4(vi00)
        iaddiu  giftag_addr, vi00, 5
        iaddiu  vertex_data, vi00, 6
        ; move the vertex count to an integer 
        ; register so we can loop over it
        mtir    vertex_index, params[x]
vertex_loop:
        ; load the colour (just increments vertex_data)
        lqi     colour, (vertex_data++)
        ; load vertex position
        lq      vertex, 0(vertex_data)
        ; apply the transformation
        mul     acc, world_screen_row0, vertex[x]
        madd    acc, world_screen_row1, vertex[y]
        madd    acc, world_screen_row2, vertex[z]
        madd    vertex, world_screen_row3, vertex[w]
        div     q, vf00[w], vertex[w]
        mul.xyz vertex, vertex, q
        ; convert to fixed point
        ftoi4   vertex, vertex
        ; overwrite the old vertex with the transformed one
        sqi     vertex, (vertex_data++)
        ; decrement and loop
        iaddi   vertex_index, vertex_index, -1
        ibne    vertex_index, vi00, vertex_loop
        ; send to gs
        xgkick  giftag_addr
        --exit
        --endexit

Wilderness in html5 canvas

My exploration of web programming continues, this is a rewrite of the wilderness game world from haxe/flash into javascript and html5 canvas. There are some plans to make this into something more resembling an actual game, but at the moment it’s serving as a good test as I get to grips with these new fangled bits and pieces.

Leaving out all the hot air about small companies beginning with “A”, here are my thoughts on canvas for making games compared with haxe/flash:

Pros:

  • Javascript is much nicer for me than Haxe due to dynamic typing and it’s comparative closeness to Scheme.
  • Immediate mode for graphics allows you to draw a scene how you want with your own containers. The retained mode MovieClip object in Flash is an annoyance for me. Hopefully similar things happen in web graphics as it did in 3D with the early DirectX retained mode, which was eventually abandoned.
  • I’ve started using Chrome for development – the javascript debugger is one of the best I’ve come across, and the Firefox one is pretty good too. When using haxe/flash I ended up completely relying on print statements.
  • It’s nicer to develop in a normal webpage rather than a plugin.
  • Cons:

  • Canvas is currently much much slower than Flash. It’s quite possible I’m doing something stupid, but the canvas version uses about 40% of my processor where as the flash version is 5-10%.
  • Less built in support for text and user interface items like entry dialogs. There is probably much more missing along these lines, and much that will need to be implemented from scratch. Personally speaking I am not too fond of library frameworks implemented by architecture astronauts, so I can live with this at the moment.
  • Update: While I hope HTML5 is the future, it seems on some counts my optimism was premature, it seems it’s still to reach a point where it’s available for the majority of people – and where it is available, the stability and performance is variable. With this in mind : On flash, software art and freedom