Docker and caching apt-get for guests on the same host

Docker is great for isolation, you probably still want some caching, the Dockerfile from extellisys is a great start:

https://github.com/extellisys/docker-image-apt-cacher-ng

Running the above, runs apt-cache-ng and exposes it’s port as 3142.

This is great for a LAN, if you want to use this for caching on your dev box then read on.

If you just want lines to paste into other Dockerfiles, skip to the end.

To avoid surprises I tend to change the “FROM” part of any Ubuntu or Debian based Dockerfiles to use the same distro as the host (in this case “ubuntu:14.10″), this should save a little space too.

Optionally use the cache with Docker guests on the same host.

Desired result – use the cache if available, otherwise don’t.

Avahi is the usual way to do this, unfortunately it doesn’t work easily in between host and guest. We can make our own way of doing this.

So all we need to do is find the host IP from another guest –

$ route -n | awk '/^0.0.0.0/ {print $2}'

The other part is optionally using a proxy – the undocumented apt.conf.d setting Acquire::http::ProxyAutoDetect is what we want. It runs a discovery script that either outputs an IP of a proxy, or DIRECT if there is none.

Discovery script and setting

/usr/local/bin/apt-ng-host-discover:
#!/bin/bash
HOST_IP=$(route -n | awk '/^0.0.0.0/ {print $2}')
if nc -w1 -z $HOST_IP 3142; then printf http://$HOST_IP:3142; else printf DIRECT; fi

/etc/apt/apt.conf.d/30proxy:
Acquire::http::ProxyAutoDetect "/usr/local/bin/apt-ng-host-discover";

Putting it together / tl;dr;

Everything can be munged into 5 lines you paste into your Dockerfile:
RUN HOST_IP=$(route -n | awk '/^0.0.0.0/ {print $2}'); \
echo "#!/bin/bash" > /usr/local/bin/apt-ng-host-discover; \
echo "if nc -w1 -z $HOST_IP 3142; then printf http://$HOST_IP:3142; else printf DIRECT; fi" >> /usr/local/bin/apt-ng-host-discover && \
chmod +x /usr/local/bin/apt-ng-host-discover
RUN echo 'Acquire::http::ProxyAutoDetect "/usr/local/bin/apt-ng-host-discover";' > /etc/apt/apt.conf.d/30proxy

Further work

Run the extellisys cache on boot, and disable any other caching ( http://ask.xmodulo.com/disable-apt-cache-debian-ubuntu.html ).

3d in the era of J2me

Back in the deep dark past (2004-2007) I worked on a 3D game for phones of that era (before iphone, android). I recently found some of the test files for the engine, here are some videos.

Very low poly racetrack

The ubiquitous cube

Spacekid in the spacestation corridors

Our Maskot wanders the corridors .. you can’t look too far ahead, to keep things really low, which means popup is a little…severe.
Movement is a better in reality – the capture has made the movement a bit jerky.

Dire Straits

Obviously we didn’t make this, but it heavily inspired our 3D engine … textures were just too slow, so we aimed for this look.

Drawing to the same content to many windows, cairo and zeromq

Playing with python, zeromq and Gtk, I made this code to send graphics commands to several windows.

Commands are sent over a zeromq PUB socket, the windows recieve them using zeromq SUB.

Only a few commands are wrapped as it’s a proof of concept.

I’ll clean it up and abstract some bits and it should be quite easy for building something like a live coding environment.

Screenshot from 2013-06-15 19:19:37

See below for code –
[UPDATE]
Posted code to github, update to work with multiprocessing.

https://github.com/stuaxo/zeromq-gtk-playground

[/UPDATE]

Continue reading

Moving things in shoebot – simple particles…

OK, part 3 – now for something fun – extending parts 1 + 2 into a simple particle system.

Particles, generally means – a lot of things moving around (the particles) and a way to generate them, an “emitter”

Here we’re going to take the code from the previous two parts and add a couple of things to make a basic particle system.
Note – shoebot, isn’t the fastest; but we do get nice looking results.

Here’s a video of our arrows as particles (arrowsplosion!):

Continue reading

Moving things in shoebot, adding different behaviours..

In my last post we made an arrow move around the screen, in this post we’ll look to extend things so it’s easy to make many things move around the screen.

This will make the code a little more complex, but as usual it makes things simpler later on.

Note:

This python code runs in shoebot, planar.py is used to handle coordinates

https://github.com/shoebot

planar.py

At the end we’ll have two arrows, a blue one controlled with the keyboard and a pink one that moves on it’s own:

Continue reading

Natural movement using polar coordinates in shoebot

Here’s a little shoebot bot to experiment with natural movement.

This uses polar coordinates to decide the direction and velocity of an arrow on the screen.

Polar coordinates mean we can give an object a sense of ‘forward’, ‘back’, ‘left’ and ‘right’

The code below works on the current version of shoebot

https://github.com/shoebot/shoebot.git

With planar.py to handle the directions and velocity

https://pypi.python.org/pypi/planar

Continue reading

Simple python spectrograph with shoebot

Seeing”Realtime FFT Graph of Audio WAV File or Microphone Input with Python…”  on python.reddit.com reminded me of one I’d built in python with shoebot.

While it works OK, I feel like I’m missing a higher level audio library (especially having seen Minim, for C++ and Java).

To run it in shoebot:

sbot -w audiobot.bot

audiobot.bot

# Major library imports
import atexit
import pyaudio
from numpy import zeros, short, fromstring, array
from numpy.fft import fft

NUM_SAMPLES = 512
SAMPLING_RATE = 11025

def setup():
    size(350, 260)
    speed(SAMPLING_RATE / NUM_SAMPLES)

_stream = None

def read_fft():
    global _stream
    pa = None

    def cleanup_audio():
        if _stream:
            _stream.stop_stream()
            _stream.close()
        pa.terminate()

    if _stream is None:
        pa = pyaudio.PyAudio()
        _stream = pa.open(format=pyaudio.paInt16, channels=1,
                           rate=SAMPLING_RATE,
                           input=True, frames_per_buffer=NUM_SAMPLES)
        atexit.register(cleanup_audio)

    audio_data  = fromstring(_stream.read(NUM_SAMPLES), dtype=short)
    normalized_data = audio_data / 32768.0

    return fft(normalized_data)[1:1+NUM_SAMPLES/2]

def flatten_fft(scale = 1.0):
    """
    Produces a nicer graph, I'm not sure if this is correct
    """
    for i, v in enumerate(read_fft()):
        yield scale * (i * v) / NUM_SAMPLES

def triple(audio):
    '''return bass/mid/treble'''
    c = audio.copy()
    c.resize(3, 255 / 3)
    return c

def draw():
    '''Draw 3 different colour graphs'''
    global NUM_SAMPLES
    audio = array(list(flatten_fft(scale = 80)))
    freqs = len(audio)
    bass, mid, treble = triple(audio)

    colours = (0.5, 1.0, 0.5), (1, 1, 0), (1, 0.2, 0.5)

    fill(0, 0, 1)
    rect(0, 0, WIDTH, 400)
    translate(50, 200)

    for spectrum, col in zip((bass, mid, treble), colours):
        fill(col)
        for i, s in enumerate(spectrum):
            rect(i, 0, 1, -abs(s))
        else:
            translate(i, 0)

    audio = array(list(flatten_fft(scale = 80)))

Cairo: Surface for recording commands and playback

An update on my latest cairo adventures…

When cairo 1.10 comes out we’ll get a RecordingSurface so you can record commands and play them back to another surface, but how to do something similar now ?

Skip to the end if you just want the code, explanation of how I got there ahead:

My first advice was to try using PDFSurface, and set the file object to None.

# Record a red rectangle onto a surface.
#
# Create another surface and draw a background on it
# Draw the first surface onto this surface
#
from cairo import PDFSurface, ImageSurface, FORMAT_ARGB32, Context

recording = PDFSurface(None, 200, 200)
target = ImageSurface(FORMAT_ARGB32, 200, 200)

# Record a red rectangle
cr = Context(recording)
cr.set_source_rgb(1.0, 0.0, 1.0)
cr.rectangle(20, 20, 10, 10)

cr.fill_preserve()
cr.set_source_rgb(0.0, 0.0, 1.0)
cr.stroke()

target_context = Context(target)

# Draw background
target_context.set_source_rgb(1.0, 1.0, 0)
target_context.paint()

# Replay recording to target
target_context.set_source_surface(recording)
target_context.paint()

target.write_to_png('output.png')

That seems to work, except when I tried in Windows, when it complained that the file object was wrong.

OK, we can work round that:

def RecordingSurface(w, h):
    if os.name == 'nt':
        fobj = 'nul'
    else:
        fobj = None
    return PDFSurface(fobj, w, h)

This seems to be working, but my animation seemed slow… time for some benchmarking; I rendered 1000 frames and got a rough wall clock time:
1m48.

Hmm… perhaps SVGSurface will be quicker:
1m28

This is much better, 20 seconds difference just by changing what kind of surface is returned!

Animation not smooth though

The animation still seemed jerky it occured to me that when the surfaces are disposed they will attempt to out their content to the file object !

Luckily, get_similar_surface() comes to the rescue; it returns a surface not associated with a file object. Using this the original surface can be kept around forever, and never output.

Wallclock time:
50 seconds!

And here it is:

_svg_surface = None
def RecordingSurface(*size):
    '''
    We don't have RecordingSurfaces until cairo 1.10, so this kludge is used

    SVGSurfaces are created, but to stop them ever attempting to output, they
    are kept in a dict.

    When a surface is needed, create_similar is called to get a Surface from
    the SVGSurface of the same size
    '''
    global _svg_surface
    if os.name == 'nt':
        fobj = 'nul'
    else:
        fobj = None
    if _svg_surface is None:
        _svg_surface = SVGSurface(fobj, 0, 0)
    return _svg_surface.create_similar(cairo.CONTENT_COLOR_ALPHA, *size)

This is really useful, you can record commands and play them back to other surfaces.

A simple cairo draw queue using closures

Often it’s useful to be able to store up drawing commands so you can use them later somewhere else (or even just pass them to another thread).

This is a simple drawing model, implemented in cairo, hopefully somebody will find it useful.

Queue

class DrawQueue:
    '''
    A list of draw commands, stored as callables that, are
    passed a set of parameters to draw on from the canvas
    implementation.
    '''
    def __init__(self, render_callables = None):
        self.render_callables = render_callables or deque()

    def append(self, render_callable):
        '''
        Add a render callable to the queue
        '''
        self.render_callables.append(render_callable)

    def render(self, cairo_ctx):
        '''
        Call all the render callables with cairo_ctx
        '''
        for render_callable in self.render_callables:
            render_callable(cairo_ctx)

The queue just accepts callables (any old function), and calls them when you call render, passing them a cairo context you pass in.

To get useful functions you can call closure functions like these:

def paint_closure():
    def paint(ctx):
        ctx.paint()
    return paint

def fill_closure():
    def fill(ctx):
        ctx.fill()
    return fill

def set_source_rgb_closure(r, g, b):
    def set_source_rgb(ctx):
        ctx.set_source_rgb(r, g, b)
    return set_source_rgb

def moveto_closure(x, y):
    def moveto(ctx):
        ctx.move_to(x, y)
    return moveto

def rectangle_closure(x, y, w, h):
    def rectangle(ctx):
        ctx.rectangle(x, y, w, h)
    return rectangle

Adding commands to the queue is simple:

dq = DrawQueue()
dq.append(set_source_rgb_closure(1, 1, 1))
dq.append(paint_closure())
dq.append(moveto_closure(10, 0))
dq.append(rectangle_closure(0, 0, 20, 20))
dq.append(set_source_rgb_closure(0, 0, 0))
dq.append(fill_closure())

This is the same drawing model I’m using in my branch of shoebot, I’m hoping to expand it to be multithreaded; while a foreground thread adds commands a background thread is executing them.

Here it is all put together in a simple example to draw a black rectangle

from collections import deque
import cairo

class DrawQueue:
    '''
    A list of draw commands, stored as callables that, are
    passed a set of parameters to draw on from the canvas
    implementation.
    '''
    def __init__(self, render_callables = None):
        self.render_callables = render_callables or deque()

    def append(self, render_callable):
        '''
        Add a render callable to the queue
        '''
        self.render_callables.append(render_callable)

    def render(self, cairo_ctx):
        '''
        Call all the render callables with cairo_ctx
        '''
        for render_callable in self.render_callables:
            render_callable(cairo_ctx)

#### drawing closures
def paint_closure():
    def paint(ctx):
        ctx.paint()
    return paint

def fill_closure():
    def fill(ctx):
        ctx.fill()
    return fill

def set_source_rgb_closure(r, g, b):
    def set_source_rgb(ctx):
        ctx.set_source_rgb(r, g, b)
    return set_source_rgb

def moveto_closure(x, y):
    def moveto(ctx):
        ctx.move_to(x, y)
    return moveto

def rectangle_closure(x, y, w, h):
    def rectangle(ctx):
        ctx.rectangle(x, y, w, h)
    return rectangle

#### /drawing closures

dq = DrawQueue()

# Add some commands to the drawing queue
dq.append(set_source_rgb_closure(1, 1, 1))
dq.append(paint_closure())
dq.append(moveto_closure(10, 0))
dq.append(rectangle_closure(0, 0, 20, 20))
dq.append(set_source_rgb_closure(0, 0, 0))
dq.append(fill_closure())

# Create a surface and context
surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, 200, 200)
ctx = cairo.Context(surface)

# run defered rendering
dq.render(ctx)

surface.write_to_png('output.png')