PSA – Use ‘SNA’ if you are on optimus

"Optimus" is the tech where your laptop has an Intel AND an Nvidia card, the amount of problems this can cause are more than the sum of the parts on Linux. Trying to run apps on the Nvidia card had all sorts of issues until I switched the intel card on 'SNA' ("the successor to UXA"). It seems like every few years intel changes how their graphics cards work, and the new hotness is SNA. Before this, glitches included black windows, and apps just freezing until I resized their windows, or if you ran more than one GL app. As per the answer here at https://ask.fedoraproject.org/en/question/32960/black-windows-in-gnome-3-suddenly add the following to your /etc/X11/xorg.conf
Section "Device"
 Identifier "Intel Graphics"
 Driver "intel"
 Option "AccelMethod" "sna"
 EndSection
I'm not sure it's entirely problem free, I have had one or two logouts, but at least I can run more than one instance of my own app without random freezing and glitching.   [EDIT] It turns out I'm still getting glitchyness and lockups running on intel, at least running things on the Nvidia card with DRI_PRIME=1 works though. [/EDIT]  

Wifi + Moving Country

If you are on linux and having trouble with WIFI in a new country it might be worth letting it know what country you are in - on debian/ubuntu edit the file /etc/default/crda and update the country code.

Setting up Backups

A day of Admin .. Spent a good amount of time getting backups setup on ubuntu .. Insynch to backup google drive to my computer - https://www.insynchq.com/ Setup good old Thunderbird to download email to my computer - the trick is to get an an app specific password. Finally, bought cloud backup from SpiderOak it seems quite good from a security point of view, the app itself is a little 'java', but seems to work OK.

Belated post on libregraphics meeting 2014

Here is my very late post on LGM 2014! Back in April I went to Leipzig for my first in-the-flesh meeting of Shoebot devs .. I met with Ricardo + to collaborate on a workshop on shoebot. To make it more fun we hadn't decided what to do it on ! In the anarchic OSP (open source publishing) house we came with a plan to get people making examples for shoebot. Luckily Ricardo had done a lot of this sort of thing before so did most of the talking, then I showed off some bots - evolution, spirals and also the livecoding work. Overall the workshop seemed to go over well; we got a bunch of examples, and there was even a plugin for sublime text !

Things that came out of the workshop:

People want an integrated editor - this is OK, since the IDE still exists. Differences between the Nodebox/Shoebot API and Cairo are not always intuitive. Livecoding is cool! Shoebot 2 ... or something else ? Going forward, it might be best to take the Nodebox approach and build something new based on these lessons, I'm not sure exactly what yet. What is the most intuitive API, how can we be close to standard APIs.

Non Shoebot Stuff

Outside of the workshops and talks there were plenty of time to drink and chat - apart from talking the head off of one of the mypaint guys I learned quite a lot about OSP off Sarah Magnan and Brendan Howell .. which made me regret missing many of their talks, including Brendans on the screenless office. Leipzig was a really great city to visit, the venue for LGM was particularly impressive being inside an old church that the East Germans repurposed to a university. Importantly for me I learned about the "kebab circle" - the ring of gentrification moving from the inside of the city outwards (beyond which you can still buy kebabs). With any luck LGM will be able to make it next year and meet everyone again.

Adding files to the django auto refresh

Django runserver and the app django-autotest are both good as they'll restart when modifcations are made to files in the app they will restart, however not *all* files will trigger a restart. Wanting to get some other files noticed, I added some in the __init__.py of my app, so it looked like this:
try:
    from django.conf import settings

    if settings.DEBUG:
        import forms, util, tests
except:
    pass
This seemed to be working fine until I swapped to postgres and tried to do syncdb .. and reset, uh oh ! Postgres said (names changed to protect the innocent):
2012-05-10 09:47:57 BST STATEMENT:
                    SELECT c.relname
                    FROM pg_catalog.pg_class c
                    LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
                    WHERE c.relkind IN ('r', 'v', '')
                        AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                        AND pg_catalog.pg_table_is_visible(c.oid)
2012-05-10 09:47:57 BST LOG:  could not receive data from client: Connection reset by peer
2012-05-10 09:47:57 BST LOG:  unexpected EOF on client connection
2012-05-10 09:49:14 BST ERROR:  relation "sandwich_maker_filling" does not exist at character 72
2012-05-10 09:49:14 BST STATEMENT:  SELECT "sandwich_maker_filling"."id", "sandwich_maker_filling"."name" FROM "sandwich_maker_filling"
2012-05-10 09:49:14 BST ERROR:  current transaction is aborted, commands ignored until end of transaction block
And django wasn't too happy either:
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/core/management/commands/syncdb.py", line 60, in handle_noargs
    tables = connection.introspection.table_names()
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/db/backends/__init__.py", line 896, in table_names
    return self.get_table_list(cursor)
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/introspection.py", line 33, in get_table_list
    AND pg_catalog.pg_table_is_visible(c.oid)""")
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/db/backends/util.py", line 40, in execute
    return self.cursor.execute(sql, params)
  File "/home/stu/.virtualenvs/the_app/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 52, in execute
    return self.cursor.execute(query, args)
django.db.utils.DatabaseError: current transaction is aborted, commands ignored until end of transaction block
In the end the solution was simple, just check for the args where we need the refresh to happen automatically:
## Automatically reload when forms or tests changed
try:
    from django.conf import settings

    if settings.DEBUG:
        import sys
        if sys.argv == ['manage.py', 'autotest'] or sys.argv == ['manage.py', 'runserver']:
                # Need to check params as otherwise this can break syncdb, reset and
                # Friends !!
                import forms, util, tests
except:
    pass
Hopefully this will be helpful to somebody with a similar error or that wants autotest or runserver to see more files.   This article was particularly helpful in debugging this:  debugging django syncdb

Cairo with python Ctypes

Uploaded some experiments with python and ctypes to here: https://gitorious.org/pycairo-ctypes/pycairo-ctypes   This is a really rough proof of concept that the pycairo API can be implemented with ctypes + metclasses.     So far only ImageSurface is supported on the SVG backend, along with Contexts.   The nice thing about this is that you can use a pycairo like API on pypy, where things should be faster (in theory).   There's a test that can be run to show the outputs the same for pycairo and cairo ctypes.

Using PyCha with Django

A quick post on using Python Charts to generate nice SVG charts for your django website (I've had the code hanging around for ages - so should just post it). The code is based on the examples there, here I integrate it into Django. To install you'll need to do
pip install pycha
Heres the code for a simple view to output a chart directly to SVG:
# views.py

from StringIO import StringIO
from django.http import HttpResponse

import cairo

def colors(request):
    in_req = 1

    svg_buffer = StringIO()

    width, height = (500, 400)
    surface = cairo.SVGSurface(svg_buffer, width, height)

    dataSet = (
     ('dataSet 1', ((0, 1), (1, 3), (2, 2.5))),
     ('dataSet 2', ((0, 2), (1, 4), (2, 3))),
     ('dataSet 3', ((0, 5), (1, 1), (2, 0.5))),
    )

    options = {
       'legend': {'hide': True},
       'background': {'color': '#f0f0f0'},
    }

    #import pycha.bar
    #chart = pycha.bar.VerticalBarChart(surface, options)

    import pycha.line
    chart = pycha.line.LineChart(surface, options)
    chart.addDataset(dataSet)
    chart.render()

    del chart
    del surface

    response = ''
    response = HttpResponse(mimetype='image/svg+xml')
    svg_buffer.seek(0)
    response.write( svg_buffer.read() )
    return response
The basic idea is that - instead of the chart outputting to an svg file, the output goes to a buffer, this is triggered by calling the destructors of the chart and it's cairo surface.  Once the data is in the buffer, it is rewound and played back to the Http Response.  

Inline SVG

As is, this won't work as inline svg, because outputting the XML preamble in the middle of the page will cause problems.  Below is an example that let's you decide if you need the preamble (full SVG output), or not (SVG fragment for inclusion in a page or another SVG):
# Create your views here.

from StringIO import StringIO
from django.http import HttpResponse
from django.shortcuts import render_to_response

import cairo

XML_PREAMBLE = '<?xml version="1.0" encoding="UTF-8"?>'

def colors_chart(inline = False):
    """
    Generate colours chart

    Set inline to True to disable the XML preamble
    """
    in_req = 1

    svg_buffer = StringIO()

    width, height = (500, 400)
    surface = cairo.SVGSurface(svg_buffer, width, height)

    dataSet = (
     ('dataSet 1', ((0, 1), (1, 3), (2, 2.5))),
     ('dataSet 2', ((0, 2), (1, 4), (2, 3))),
     ('dataSet 3', ((0, 5), (1, 1), (2, 0.5))),
    )

    options = {
       'legend': {'hide': True},
       'background': {'color': '#f0f0f0'},
    }

    import pycha.bar
    chart = pycha.bar.VerticalBarChart(surface, options)

    #import pycha.line
    #chart = pycha.line.LineChart(surface, options)
    chart.addDataset(dataSet)
    chart.render()

    del chart
    del surface

    response = ''

    if inline:
        svg_buffer.seek(len(XML_PREAMBLE))
    else:
        svg_buffer.seek(0)
    return svg_buffer.read()

def colors_svg(request):
    """ render a pure SVG chart """
    response = HttpResponse(mimetype='image/svg+xml')
    response.write(colors_chart(inline = False))
    return response

def index(request):
    """ render a chart into the template """
    chart_svg = colors_chart(inline = True)

    return render_to_response(
        'shapes_index.html',
        { "chart" : chart_svg },
        mimetype='application/xhtml+xml')
An example project with the above code is available here: pycha_django   [EDIT 25/2/2011] Fixed PyCha URL

Set Windows Environment variables from the registry (TCC/LE)

[EDIT: 2010/10/05]

Oops, I've been using TCC/LE by JPSoft as my shell... the following doesn't work in the normal command prompt

[/EDIT]

I just had a look at SET and it lets you set user/system variables:

Display, create, modify, or delete environment variables.

SET [/A /D /E /P /R file... /S /U /V /X] [name[=][value ]]
        file:  One or more files containing variable definitions
        /A(rithmetic)           /S(ystem variables)
        /D(efault variables)    /U(ser variables)
        /E(nv vars)             /V(olatile variables)
        /R(ead from file)       /X override VariableExclude
        /P(ause)

So to set save a variables MYAPP_HOME as the current directory, just do

SET /U MYAPP_HOME=%CD

Wish I'd have looked that up a few years ago, it'll save a few seconds next time I need to setup some dev sdk 🙂

Access files in a Linux virtual machine from Windows

I recently found a neat way of accessing files in a linux VMWare image.   This is really useful, as theres never really a good time to break your VMWare image, this is also handy if you don't want to run the whole machine, but just access the files inside. There is one caveat:
  • It only works if the filesystem is ext2 (ext3 works, and ext4 probably works).

Install VMWare DiskMount Utility

Accept the EULA, download and install the VMWare DiskMount utility. For convenience add the utilities folder to the path:
C:\Program Files\VMware\VMware DiskMount Utility
Do this through the Windows Gui, or even use my addpath utility.
At this point you can mount Windows VMWare images. The usage is:  vmware-mount drive-letter vmdk-image. Heres how I mount my Ubuntu image to the j: drive
[C:\vmware\Ubuntu]vmware-mount j: ubuntu.vmdk
[C:\vmware\Ubuntu]
No output here indicates success. At this point everything seems fine, but a crucial piece of the puzzle is still missing; try and view the files and you still can't:

Failing to see files in an ext2 VMWare image

The next step is to make Windows understand the ext2 filesystem, using a special driver.

Install ext2ifs

Grab ext2ifs from www.fs-driver.org and install. If the following steps don't work then you may need to reboot.

Thats it!

You should be now able to access files inside your VMWare image (assuming it's ext2 and not reiserfs), remount the image and have a go: In my case I did:
[C:\vmware\Ubuntu]vmware-mount j: ubuntu.vmdk
[C:\vmware\Ubuntu] dir j:
Heres the output - hooray, I can copy my work out of the image !

Viewing files inside a VMWare image with ext2fs

This is very useful, especially if you do dist-upgrade in ubuntu and can't access the network.

Bonus tip:

Newer versions of VMWare player let you install VMWare tools from inside the GUI, this is another way to fix the kind of catestrophic problems you can cause yourself by accidentally upgrading the kernel in an image.

Some useful scripts for windows.

Stoyan of PHPIED.com has found the joy using javascript for scripting in the OS... I've put up a few scripts I find useful in windows (download at the end of the article):
  • addpath.js - Add a path to the registry path.
  • dt.cmd - Change to desktop folder.
  • e.cmd - Open explorer in current or specified folder.
  • regpath.js  - Output the path stored in the registry.
  • updateenvironment.js - Updates running apps with any changed settings in the registry.
And a couple that use python and pywin32:
  • ccwd.py - Copy the current working directory to the clipboard
  • cpath.py - Copy the current path to the clipboard
All of have acompanying batch files to run them, I generally have everything in a folder c:\usr\cmd, but they should work from anywhere in the path. Download  cmd.zip