Cricket Tales released

Cricket Tales is an ambitious citizen science project. 438 days of CCTV footage from the Wild Crickets Research group – the only record of wild behaviour of insects of it’s kind. It turns out that insects have more complex lives and individuality than we thought, and the game is a way of helping uncover this more precisely. For Foam Kernow, this was also a significant project as the biggest production that all three of us have worked on together.

title

My favorite aspect of this project is that the movies are a strangely different way of viewing an ecosystem, tiny close up areas of a perfectly normal field in northern Spain. The footage is 24 hour, with infrared at night, recording a couple of frames a second only when movement is detected. Some of the videos get triggered when there is simply movement of shadows, but there are plenty of moments that we wouldn’t normally notice. Worms and bugs of all kinds going about their lives, sudden appearances of larger animals or swarms of ants, condensation of dew at dawn. The crickets themselves, mostly with tags stuck to them so we can tell which is which, but other than that – this is their normal habitat and way of life. Compared to the study of insects in lab conditions, it’s not surprising they act in a more complex way.

movie2
Screenshots from the Spanish version, as I’m particularly proud of that (my first experience using GNU gettext with Django).

We combined the task of watching the 1 minute long movies with the ability to build houses for the crickets – we needed to provide a way for people to leave something behind, something that marks progress on this gigantic collective task. You get to design a little house for each burrow, and your name gets recorded on the meadow until the next person takes over by watching more videos.

map2

We’ve had plenty of conversations about what kind of people take part in this sort of citizen science activity, what the motivations may be. We ask a couple of questions when people sign up, and this is something we are interested in doing more research on in general for our projects. In this case, we are interested in depth of involvement more than attracting thousands of brief encounters – it only takes a few motivated people to make the researcher’s jobs much easier and provide some data they need.

For me a bigger objective of Cricket Tales is as a way to present more diverse and personal views of the world that surround us, and tends to go unnoticed. Being asked to contemplate a tiny organism’s view of the world for a minute can be quite an eye opener.

A 6502 lisp compiler, sprite animation and the NES/Famicom

For our new project “what remains”, we’re regrouping the Naked on Pluto team to build a game about climate change. In the spirit of the medium being the message, we’re interested in long term thinking as well as recycling e-waste – so in keeping with a lot of our work, we are unraveling the threads of technology. The game will run on the NES/Famicom console, which was originally released by Nintendo in 1986. This hardware is extremely resilient, the solid state game cartridges still work surprisingly well today, compared to fragile CDROM or the world of online updates. Partly because of this, a flourishing scene of new players are now discovering them. I’m also interested that the older the machine you write software for, the more people have access to it via emulators (there are NES emulators for every mobile device, browser and operating system).

nes
Our NES with everdrive flashcart and comparatively tiny sdcard for storing ROMs.

These ideas combine a couple of previous projects for me – Betablocker DS also uses Nintendo hardware and although much more recent, the Gameboy DS has a similar philosophy and architecture to the NES. As much of the machines of this era, most NES games were written in pure assembly – I had a go at this for the Speccy a while back and while being fun in a mildly perverse way, it requires so much forward planning it doesn’t really encourage creative tweaking – or working collaboratively. In the meantime, for the weavingcodes project I’ve been dabbling with making odd lisp compilers, and found it very productive – so it makes sense to try one for a real processor this time, the 6502.

The NES console was one of the first to bring specialised processors from arcade machines into people’s homes. On older/cheaper 8 bit machines like the Speccy, you had to do everything on the single CPU, which meant most of the time was spent drawing pixels or dealing with sound. On the NES there is a “Picture Processing Unit” or PPU (a forerunner to the modern GPU), and an “Audio Processing Unit” or APU. As in modern consoles and PCs, these free the CPU up to orchestrate a game as a whole, only needing to sporadically update these co-processors when required.

You can’t write code that runs on the PPU or APU, but you can access their memory indirectly via registers and DMA. One of the nice things we can do if we’re writing a language for a compiling is building optimised calls that do specific jobs. One area I’ve been thinking about a lot is sprites – the 64 8×8 tiles that the PPU draws over the background tiles to provide you with animated characters.

spriteemu
Our sprite testing playpen using graphics plundered from Ys II: Ancient Ys Vanished.

The sprites are controlled by 256 bytes of memory that you copy (DMA) from the CPU to the PPU each frame. There are 4 bytes per sprite – 2 for x/y position, 1 for the pattern id and another for color and flipping control attributes. Most games made use of multiple sprites stuck together to get you bigger characters, in the example above there are 4 sprites for each 16×16 pixel character – so it’s handy to be able to group them together.

Heres an example of the the compiler code generation to produce the 6502 assembly needed to animate 4 sprites with one command by setting all their pattern IDs in one go – this manipulates memory which is later sent to the PPU.

(define (emit-animate-sprites-2x2! x)
  (append
   (emit-expr (list-ref x 2)) ;; compiles the pattern offset expression (leaves value in register a)
   (emit "pha")               ;; push the resulting pattern offset onto the stack
   (emit-expr (list-ref x 1)) ;; compile the sprite id expression (leaves value in a again)
   (emit "asl")               ;; *=2 (shift left)      
   (emit "asl")               ;; *=4 (shift left) - sprites are 4 bytes long, so = address
   (emit "tay")               ;; store offset calculation in y
   (emit "iny")               ;; +1 to get us to the pattern id byte position of the first sprite
   (emit "pla")               ;; pop the pattern memory offset back from the stack
   (emit "sta" "$200,y")      ;; sprite data is stored in $200, so add y to it for the first sprite
   (emit "adc" "#$01")        ;; add 1 to a to point to the next pattern location
   (emit "sta" "$204,y")      ;; write this to the next sprite (+ 4 bytes)
   (emit "adc" "#$0f")        ;; add 16 to a to point to the next pattern location
   (emit "sta" "$208,y")      ;; write to sprite 2 (+ 8 bytes)
   (emit "adc" "#$01")        ;; add 1 to a to point to the final pattern location
   (emit "sta" "$20c,y")))    ;; write to sprite 4 (+ 12 bytes)

The job of this function is to return a list of assembler instructions which are later converted into machine code for the NES. It compiles sub-expressions recursively where needed and (most importantly) maintains register state, so the interleaved bits of code don't interfere with each other and crash. (I learned about this stuff from Abdulaziz Ghuloum's amazing paper on compilers). The stack is important here, as the pha and pla push and pop information so we can do something completely different and come back to where we left off and continue.

The actual command is of the form:

(animate-sprites-2x2 sprite-id pattern-offset)

Where either arguments can be sub-expressions of their own, eg.:

(animate-sprites-2x2 sprite-id (+ anim-frame base-pattern))

This code uses a couple of assumptions for optimisation, firstly that sprite information is stored starting at address $200 (quite common on the NES as this is the start of user memory, and maps to a specific DMA address for sending to the PPU). Secondly there is an assumption how the pattern information in memory is laid out in a particular way. The 16 byte offset for the 3rd sprite is simply to allow the data to be easy to see in memory when using a paint package, as it means the sprites sit next to each other (along with their frames for animation) when editing the graphics:

spritepatternoffset

You can find the code and documentation for this programming language on gitlab.

A tanglebots workshop report

I’ve tried a lot of different ways of teaching children programming, starting a few years ago with primary school children in a classroom, then doing inset training days for teachers and finally private tutoring in homes. For the finale to the weavingcodes project we are trying a new approach, teaching families about code, robotics and thread by building “tanglebots”.

25956891515_f3d0ea3ec7

The concept is to combine programming with physical objects, concentrating on sensor input and movement as output. It’s important that we incorporate our weavingcodes research process, so deliberately setting goals we don’t yet know the answers to.

The weaving focus allows us to ground the workshop in loom technology and demonstrate the challenges of manipulating thread, with its enormous history of technological development. For the first Cornwall workshop, Ellen started us off with an introduction using FoAM Kernow’s Harris loom and the fundamentals of weaving. We were also joined by Janet and Jon from lovebytes who are helping us to run these events. When first talking about possible workshops with children, we’d discussed the impossibility of making a functional loom in a couple of hours with only broken toys and lego – and so the focus on tangling was suggested by Alex as a way to turn these difficulties to an advantage. Similarly we created a series of prizes for different categories such as “Most technical effort with least impressive result” – inspired by hebocon events.

25956891515_f3d0ea3ec7_2

25324352694_30f0a1a2a4_k2

The workshop format we used is also influenced by Paul Granjon’s wrekshops – wherever possible we’re recycling by pulling apart e-waste, making use of electronics, motors, gears and ideas from the surprising complexity of what’s inside things people are throwing away. This turned out have a powerful implicit message about recycling, parents I talked to had tried taking things apart to learn about them, but the next step – making use of the parts discovered as we were doing here, needs a bit more help to do.

Also as normal for FoAM projects was the importance of the food, in this case tangled by Amber and Francesca to both provide sustenance and inspiration with cardamom knots, spiralised courgetti and tangle fritters.

25931225896_007b088e5d_k2

The groups ended up a bit lopsided, so in future we plan to pre-arrange them as we did on the machine wilderness workshop. In order to do that we need to ask for more information from participants beforehand such as family ages and backgrounds.

We tried using the small Pi touchscreens – these were a bit too fiddly to get away without a mouse, but are much less oppressive somehow than larger PC monitors – as they are so small, they became incorporated into the tanglebots themselves.

Crocodile clips were the best way to connect to random/plundered electronics as well as the lego motors. These removed the need for soldering (which we had set up anyway, but in a separate space).

A selection of other notes we made:

  • Start with a manual tangling exercise (weaving with rope, tablets etc)
  • Lego has a strange all or nothing effect, once you start using it – everything has to work that way, avoiding it may lead to more creative options than including it
  • A first aid kit is needed for these sorts of things
  • The Pimoroni Explorer Hats are good but needed periodic resets in some cases – the motors seemed to get jammed, not sure if this is short circuits interrupting the i2c comms?
  • The Raspberry Pi docs are riddled with minor errors, e.g. the Scratch GPIO section on the explorer hats has a lot of sometimes confusing typos.

All our resources are being uploaded to the kairotic github repository so other people can make use of the materials.

As well as being supported by AHRC Digital Transformations, this project was part of British Science Week, supported by the British Science Association.

25956947035_a44aa6bdd9_k(1)

Artificially evolved camouflage

As the egglab camouflage experiment continues, here are some recent examples after 40 or so generations. If you want to take part in a newer experiment, we are currently seeing if a similar approach can evolving motion dazzle camouflage in Dazzle Bug.

Each population of eggs is being evolved against a lot of background images, so it’s interesting to see the different strategies in use – it seems like colour is one of the first things to match, often with some dazzle to break up the outline. Later as you can see in some of these examples, there is some quite accurate background matching happening.

It’s important to say that all of this is done entirely by the perception from tens of thousands of people playing the game – there is no analysis of the images at any point.

022

020

019

018

016

012

010

009

005

004

Sonic Bikes to Sonic Kayaks – using puredata

When I first started working on the Sonic Bikes project with Kaffe Matthews in 2013 I had just moved to Cornwall, and I used the Penryn river for developing “The swamp that was” installation we made for Ghent. We’ve always talked about bringing this project here, but the various limitations of cycling (fast roads, stupid drivers and ridiculous hills) were always too much of a problem – so we wondered about sonic kayaks, as a distant vague idea. However now, thanks to help from the British Science Association, Feast Cornwall and the Port Eliot Festival they are fast becoming a reality!

We’re also using this opportunity to convert kayaks into instruments for sensing marine microclimates – an area which is currently lacking in scientific knowledge. In order to do this, we need to expand the sonic potential of our current system – moving it from sample playback to a more open ended synthesis approach. We’re running a open hacklab to trial the use of sensors, and actually get out on the water with Kaffe later in the month.

zones

To do all this – and keep it functioning on a Raspberry Pi, we’re using Pure Data. For the moment it seemed most appropriate to stick to the concept of audio zones, previously these defined areas associated with samples that would play back when you were inside of them. The screenshot above is the sonic bike mapping tool – recently rebuilt by Francesca. Using Pure Data we can associated each zone with a specific patch, which leaves the use of samples or not, effects, interpretation of sensor data and any other musical decisions completely open.

pd

The patch above is the first version of the zone patch mixer – it reads OSC messages from the GPS map system (which is written in Lua) and when a patch is triggered, it turns on audio processing for it and gradually fades it up. When the zone is left it fades it down and deactivates it – this way we can have multiple overlayed patches, much like the sample mixing we used before. We can also have loads of different patches as it’s only processing the active ones at any one time, it won’t stress out the Raspberry Pi too much.

I’ve been testing this today by walking around a lot with headphones on – this is a GPS trace, which gives some ideas of the usual problems of GPS (I didn’t actually switch to kayak halfway through, although it thought I did).

trace

Tanglebots workshop preparation

It’s workshop time again at Foam Kernow. We’re running a Sonic Kayak development open hacklab with Kaffe Matthews (more on this soon) and a series of tanglebots workshops which will be the finale to the weavingcodes project.

Instead of using my cobbled together homemade interface board, we’re using the pimoroni explorer hat (pro). This comes with some nice features, especially a built in breadboard but also 8 touch buttons, 4 LEDs and two motor drivers. The only downside is that it uses the same power source as the Pi for the motors, so you need to be a little careful as it can reset the Pi if the power draw is too great.

2hat

We have a good stock of recycled e-waste robotic toys we’re going to be using to build with (along with some secondhand lego mindstorms):

toys

Also lots of recycled building material from the amazing Cornwall Scrap Store.

material

In order to keep the workshop balanced between programming and building, and fun for all age groups, we want to use Scratch – rather than getting bogged down with python or similar. In a big improvement to previous versions of the Pi OS, the recent raspbian version (jessie) supports lots of extension hardware including the explorer hat. Things like firing the built in LEDs work ‘out of the box’ like this:

2scratch-led

While the two motor controllers (with speed control!) work like this:

2scratch-motor

The touch buttons were a bit harder to get working as they are not supported by default, so I had to write a scratch driver to do this which you can find here. Once the driver script is running (which launches automatically from the desktop icon), it communicates to scratch over the network and registers the 8 touch buttons as sensors. This means they can be accessed in scratch like so:

2scratch-touch

How to warp a tablet loom (/neolithic digital computing device)

Tablet looms have some interesting properties. Firstly, they are very
very old – our neolithic ancestors invented them. Secondly they
are quite straightforward to make and weave but form an extremely
complex structure that incorporates both weaving and braiding (and one I
haven’t managed to simulate correctly yet) – they are also the only form
of weaving that has never been mechanised.

I’ve learned to warp tablets very much by trial and error, so I expect
there are many improvements possible (please let me know), but as I had
to warp a load of tablet looms for the weavecoding workshop in Düsseldorf last
week, I thought I’d document the process here.

The first thing you need to do is make the tablets themselves from
fairly stiff card. You need squares of about 5cm, and holes punched out
from the corners. Beer mats or playing cards are good, I’m just using recycled card here. It saves a bit of time later if you can get the holes lined up when
the tablets are stacked together – but I’ve never managed to do this
very well. A good number of cards to start with are 8 or 10, fewer in
number are easier to manipulate and use less yarn – if you don’t have
much to spare.

IMG_20160122_120729

The second step is to prepare the warp yarn. You need four separate
balls or cones of wool – it’s easiest to start with four different
colours, although you can make more patterns with double faced weave
(two colours opposite each other on the cards). Fluffy knitting wool
works fine but can catch and be annoying sometimes, cotton is better. In
order to help prevent the yarn getting tangled together which is
probably the biggest problem with this job – it’s a good idea to set
this up so the yarn passes through the back of a chair with the yarn on
the floor like this – it restricts the distance the balls roll as you
pull the yarn.

IMG_20160122_120905

You need two sticks you can easily loop the warp over, I’m using a cut
broom stick clamped to the chair here. The distance between them determines
how long the final woven band will be, a metre or so is good.

Next you need to thread each of the four warp threads through the
corners of the tablets – each thread needs it’s own corner so be careful
not to mix them up. If the holes line up you can do them all at once,
otherwise it’s one by one.

IMG_20160122_121329

Here are the tablets with all the corners threaded.

IMG_20160122_122040

Now we tie the threads together looped over the warp stick furthest from
the yarn, this knot is temporary.

IMG_20160122_122324

Hook the other end of the warp over the stick on the other side, and
leave one of the tablets half way along. Loop it back again leaving
another tablet – go backwards and forwards repeating.

IMG_20160122_122645

I can never manage to keep the tablets in order, and usually end up with a mess like this – don’t panic if you get a similar result!

IMG_20160122_123113

When you have done all of the tablets, quickly check that every warp
pass has a card associated with it and then cut the first knot and tie
the last warp threads to the first.

IMG_20160122_123216

This gives you a continuous warp, which is good for adjusting the
tension. Group all the cards together and arrange them so the colours
are aligned – rotate them until the same colours are at the top and the bottom.

IMG_20160122_123849

This is also a good point to check the twist of the tablets so they alternate in
terms of the direction the threads are coming from. This is quite
difficult to explain with text but these images may help. It’s basically a bit easier if they are consistent when you start weaving, then you can see how changing this alters the patterns as you go.

IMG_20160122_123910

IMG_20160122_123927

You can actually reorder and flip the tablets at any time, even after
starting weaving – so none of this is critical. It’s handy to equalise
the tension between the warp threads at this point though, so grab all
the cards in alignment, put some tension on the warp and drag them up
and down the warp – if the loops at either end are not too close this
should get the lengths about the same.

Once this is done, tie all the tablets together to preserve their order
and alignment.

IMG_20160122_124223

Then tie loops of strong cord around both ends of the warp.

IMG_20160122_124426

Then you’re done – you can pull the ends off the sticks and start tablet
weaving!

IMG_20160122_124528

As I needed to transport the tablet looms I wrapped the warps (keeping the upper/lower threads separated) around cardboard tubes to keep the setup from getting tangled up. This seemed to work well:

IMG_20160122_130602

If you find one of the warp threads is too long and is causing
a tablet to droop when the warp is under tension, you can pull it tight
and tie it back temporarily, after weaving a few wefts it will hold in
place.

The two most important things I’ve learned about weaving – the older the
technique the more forgiving it is to mistakes, and you can never have
too many sticks.

Red King – listening to coevolution

Scientific models are used by researchers in order to understand interactions that are going on around us all the time. They are like microscopes – but rather than observing objects and structures, they focus on specific processes. Models are built from the ground up from mathematical rules that we infer from studying ecosystems, and they allow us to run and re-run experiments to gain understanding, in a way which is not possible using other methods.

I’ve managed to reproduce many of the patterns of co-evolution between the hosts and parasites in the red king model by tweaking the parameters, but the points at which certain patterns emerge is very difficult to pin down. I thought a good way to start building an understanding of this would be to pick random parameter settings (within viable limits) and ‘sweep’ paths between them – looking for any sudden points of change, for example:

random_path-21-0-big

This is a row of simulations which are each run for 600 timesteps, with time running downwards. The parasite is red and the host is blue, and both organism types are overlayed so you can see them reacting to each other through time. Each run has a slightly different parameter setting, gradually changing between two settings as endpoints. Halfway through there is a sudden state change – from being unstable it suddenly locks into a stable situation on the right hand side.

I’ve actually mainly been exploring this through sound so far – I’ve built a setup where the trait values are fed into additive synthesis (adding sine waves together). It seems appropriate to keep the audio technique as direct as possible at this stage so any underlying signals are not lost. Here is another parameter sweep image (100 simulations) and the sonified version, which comprises 2500 simulations, overlapped to increase sound density.

random_path-23-0-big

You can hear quite a few shifts and discontinuities between different branching patterns that emerge at different points – writing this I realise an animated version might be a good idea to try too.

Stereo is done by slightly changing one parameter (the host tradeoff curve) across the left and right channels – so it gives the changes a sense of direction, and you are actually hearing 5000 simulations being run in total, in both ears. All the code so far (very experimental at present) is here. The next thing to do is to take a step back and think about the best way to invite people in to experience this strange world.

Here are some more tests:

random_path-21-0-big

random_path-21-0-big

random_path-38-0-big