Joe's Lab-book

Sometimes it's useful to use an instutional webcache from home - getting at things on IEEEXplore for example - but many webcaches don't allow access from the internet. SSH to the rescue.

Assuming you have access to a machine inside the institution, host.inst, and that the webcache is called webcache.inst on port 8080 then:

ssh host@inst -L 8000:webcache.inst:8080 -T

Will forward port 8000 on your local machine to port 8080 on the webcache, set up your browser to expect a wep proxy on localhost:8000.

Sometimes to annotate a diagram (made with GraphViz) it would be good to add extra edges. For example, given the final analysis example from KF86 (figure 1) it would be helpful to show the path taken by the algorithm through this tree (see kaf_orig.dot for the dot file).

Figure 1: Final Analysis Data Structure

A naive addition of the algorithm path to the dot file is shown below:

n1 -> n2 -> n3 -> n5 -> n4 -> n7 -> n11 [color=red];

When creating the layout dot treats all edges identically, giving the layout shown in figure 2 (see kaf_unc.dot).

Figure 2: Layout change after additional path

Setting the edge attribute constrained to false prevents the consideration of that edge in setting node ranks, so the code:

n1 -> n2 -> n3 -> n5 -> n4 -> n7 -> n11 [color=red, constrained=false];

Gives the graph shown in figure 3, and in kaf_final.dot.

Figure 3: Original layout, after addition of `constrained` parameter

Following up from Wrapping LibC Functions a couple of months ago, here's how you can build those functions into a dynamically loaded library.

The code to log calloc() and free() looks something like:

#include <stdlib.h>

void *calloc (size_t count, size_t size)
{
     void *ptr = (void *) __libc_calloc(count, size);
     DEBUG("calloc(%d, %d) = %p\n", count, size, ptr);
     return ptr;
}

void free (void * ptr)
{
    __libc_free(ptr);
    DEBUG("free(%p)\n",ptr);
}

If the file's called test_alloc.c build the library with:

$ gcc -shared -ldl -o alloc.so test_alloc.c

Then run a program with these intercept functions:

$ LD_PRELOAD=alloc.so ./program

Say you have a list of country names and you want to highlight those countries on a map, shaded by the number of repitions of that country in the list. You might want to go from a list like this:

locations = ['Slovenia',
         'Slovenia',
         'Italy',
         'Russian Federation',
         'Serbia',
         'United Kingdom',
         'Germany',
         'Norway',
         'Peru',
         'Poland',
         'Moldova',
         ...
         'Germany',
         'France',
         'United Kingdom']

To an image like this, from google charts:

The google charts API requires ISO 3166 country codes, pycountry can translate country names to ISO 3166 codes:

codes = [pycountry.countries.get(name=l).alpha2 for l in locations]

Counting the repitions in codes is easy:

code_count = defaultdict(int)
for code in codes:
    code_count[code] += 1

The last thing to do is save the parameters and make the charts URL:

base = 'http://chart.apis.google.com/chart?'
params = {
    'chco': 'FFFFFF,CCFFCC,88FF88,00FF00',
    'chf': 'bg,s,EAF7FE',
    'chs': '440x220',
    'cht': 't',
    'chtm': 'world'}

params['chd'] = 't:'+
                ",".join(
                str(int((counts[k]/float(max(counts.values()))*100))) 
                for k in sorted(counts.keys()))

params['chld'] = "".join(sorted(counts.keys()))

URL = base + "&".join(
            "=".join([k, params[k]) for k in params.keys())

After all that URL looks like:

http://chart.apis.google.com/chart?chd=t:60,20,20,100,40,100,60,20,20,20,20,20,20,100,20,60,60&chf=bg,s,EAF7FE&chco=FFFFFF,CCFFCC,88FF88,00FF00&chtm=world&chld=BEBYCZDEFRGBITMDNOPEPLPTRSRUSESIUA&chs=440x220&cht=t

Which is the link to the map shown above.

Running two monitors of different sizes as one combined desktop leaves a dead zone - an area of desktop not shown on either monitor. Annoyingly sawfish will someties place a window in this dead zone. To prevent it you can fill the dead zone an x message window, using a command like:

xmessage -geometry 480x480+800+1024 hi &!

With some BIOS versions the eee can drive a 1920x1200 monitor, but it seems it can't do decent power management at the same time! The table below lists those bios versions I've tested.

Gmail will offer to automatically unsubscribe from a mailing list1, so why can't mutt?

Gmail parses the List-Unsubscribe header looking for a mailto URL, so it should be quite possible to knock up something similar for mutt.

A typical List-Unsubscribe looks like:

List-Unsubscribe:
    <http://zgp.org/cgi-bin/mailman/options/linux-elitists>,
    <mailto:linux-elitists-request@zgp.org?subject=unsubscribe>

So a bit of formail and sed action could easily parse out the link.

It's often useful to define element-wise operators for tuples, particularly when they are used to represent coordinate pairs. In python a simple operator factory function can be used to create these tuple operators from their scalar equivalents.

>>> import operator

>>> def tuple_operator_factory (operator):
...     def toperator (*args):
...         return tuple(map(lambda x: operator(*x), zip(*args)))
...     return toperator

>>> tsub = tuple_operator_factory(operator.sub)
>>> tsub((2,3), (1,1))
(1,2)
>>> tmul = tuple_operator_factory(operator.mul)
>>> tmul((2,2), (3,4))
(6,8)

Synopsis

ldiff <filename>

Description

Ldiff highlights changes between version of a latex document. The original file is taken from an automatically discovered git repository and the new file from the path specified by filename.

Requirements

ldiff requires python version 2.5, and the git-python module. The changes ldiff makes to the specified latex file assumes that the following commands are present in the document's preamble:

\usepackage{color}
\usepackage{soul} 
\definecolor{insertcolor}{rgb}{0.81, 1, 0.81}
\definecolor{deletecolor}{rgb}{1, 0.53, 0.53}

Of course the definitions of insertcolor and deletecolor can be amended to taste.

Algorithms

ldiff attempts to locate the relevant git repository by searching the all directories in the absolute path of filename for a directory named .git.

Example

Example Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Ut purus elit, vestibu-lum ut, placerat ac, adipiscing vitae, felis. Curabitur dictum gravida mauris.

Nam arcu libero, nonummy eget, consectetuer id, vulputate a, magna. Donecvehicula augue eu neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Mauris ut leo. Cras viverra metusrhoncus sem. Nulla et lectus vestibulum urna fringilla ultrices. Phasellus eu tellus sit amet tortor gravida placerat. Integer sapien est, iaculis in, pretiumquis, viverra ac, nunc. Praesent eget sem vel leo ultrices bibendum. Aenean faucibus. Morbi dolor nulla, malesuada eu, pulvinar at, mollis ac, nulla. Cur-abitur auctor semper nulla. Donec varius orci eget risus. Duis nibh mi, congue eu, accumsan eleifend, sagittis quis, diam. Duis eget orci sit amet orci dignissimrutrum.→This sentence has been replaced. ------------

Nam dui ligula, fringilla a, euismod sodales, sollicitudin vel, wisi. Morbiauctor lorem non justo. Nam lacus libero, pretium at, lobortis vitae, ultricies et, tellus. Donec aliquet, tortor sed accumsan bibendum, erat ligula aliquet magna,vitae ornare odio metus a mi. Morbi ac orci et nisl hendrerit mollis. Suspendisse ut massa. Cras nec ante. Pellentesque a nulla. Cum sociis natoque penatibus etmagnis dis parturient montes, nascetur ridiculus mus. Aliquam tincidunt urna. Nulla ullamcorper vestibulum turpis. Pellentesque cursus luctus mauris.Nulla malesuada porttitor diam. Donec felis erat, congue non, volutpat at, tincidunt tristique, libero. Vivamus viverra fermentum felis. Donec nonummypellentesque ante. Phasellus adipiscing semper elit. Proin fermentum massa ac quam. Sed diam turpis, molestie vitae, placerat a, molestie nec, leo. Mae-cenas lacinia. Nam ipsum ligula, eleifend at, accumsan nec, suscipit a, ipsum. Morbi blandit ligula feugiat magna. Fusce mauris. Vestibulum luctus nibh at lectus.

This line had been added.

Download

ldiff can be downloaded from my git repository.

If you want to instrument functions from libc then there's no need to mess around with dlsyms, try prepending __libc_ to the function you're interested in:

void *calloc (size_t count, size_t size)
{
     void *ptr = __libc_calloc(count, size);
     DEBUG("calloc(%d, %d) = %p\n", count, size, ptr);
     return ptr;
}

Toggle Pointer Head - Warp the pointer between xinerama heads

Try adding the following code to your sawfish config, and bind the toggle-pointer-head function to a key.

(defun toggle-pointer-head ()
  (defun cons+ (a b)
    (cons
      (+ (car a) (car b))
      (+ (cdr a) (cdr b))))

  (defun head-centre (head)
    (cons+
     (cons-centre (head-dimensions head))
     (head-offset head)))

  (defun cons-centre (a)
    (cons (/ (car a) 2) (/ (cdr a) 2)))

  (defun warp-cursor-cons (c)
    (warp-cursor (car c) (cdr c)))

  (defun warp-pointer-to-head (head)
    (warp-cursor-cons (head-centre head)))

  (cond
    ((>= (1+ (pointer-head)) (head-count))
     (warp-pointer-to-head 0))
    (t (warp-pointer-to-head (1+ (pointer-head))))))

(bind-keys global-keymap  "M-c" '(toggle-pointer-head))

If you have several projects in one git tree, perhaps something like:

repo/
    .git/
    proj1/
    proj2/

And you want several individual git repos:

proj1/
    .git/
proj2/
    .git/

Then to take all sub-folders from ~/git/src and create new git repos in ~/git:

set -e
srcrepo=~/git/src
cd git_tmp
git clone $srcrepo ./orig
mkdir new
cd new
for dir in ../orig/*;
do
    dir=$(basename $dir)
    git clone ../orig $dir
    pushd $dir
    git filter-branch --subdirectory-filter $dir HEAD -- --all
    git reset --hard
    git gc --aggressive
    git prune
    popd
    git clone --bare $dir ~/git/$dir
done
rm -rf ~/git_tmp/*

To convert a set of subversion repos in ~/svn to git repos in ~/git

set -e
git config svn.authorsfile << EOF
joe = Joe Milbourn <joe.milbourn+git@gmail.com>
EOF
mkdir ~/git_tmp
cd ~/git_tmp
for x in ~/svn/*;
do
    repo=$(basename $x)
    mkdir $repo
    pushd $repo
    git svn init file://$HOME/svn/$repo --no-metadata
    git svn fetch
    popd
    git clone --bare $repo ~/git/$repo
done
rm -rf git_tmp

Read comments marked as todo or XXX from source code in many different languages, and display a summary of the output (c.f. eclipse's tasks view).

Example output:

file.java:38 TODO: update loop to consider all options

As an extension allow colouring of the output.

Gnome has a nice onscreen display for volume changes, but the default acpi scripts on the eeepc don't use it. Patch them to use xdotool to send XF86AudioMute, XF86AudioRaiseVolume and XF86AudioLowerVolume. Bind volume up, down and, mute in gnome-keybinding-properties to respond the these events.

Provide a command line interface to the BBC iplayer. Get it here

Usage:

Usage: bbcsearch [options] <search terms>

Options:
  -h, --help            show this help message and exit
  -l, --list-stations   List available stations.
  -s STATION, --station=STATION
                        Limit search to only STATION.
  -f FIELDS, --fields=FIELDS
                        Fields to search.
  -a, --all             Show even un-available programmes.
  -i, --id-only         Print only the ID of matching programmes.
  -d DATE, --date=DATE  Seach on the given date, default is today.

Supported stations bbcsearch -l:

• Radio 7
• Radio 4 (FM)

Date formats can be anything understood by python-dateutils, or "yesterday".

Available search fields are 'programme', 'url', 'available' and 'title'; if unspecified then title and programme are used.

Examples:

Find today's Today programme, searching only radio 4.

   $ bbcsearch -s "Radio 4 (FM)" today 
   <b00kwhzr> 04:45 Farming Today: 19/06/2009
   <b00kwj17> 05:00 Today: 19/06/2009

Find "Old Harry's Game" broadcast on the 17th of June

   $ bbcsearch -d "17th June" Harry                 -- INSERT --
   <b007jvjx> 21:30 Old Harry&#39;s Game: Series 4, Episode 6

Find and play a programme:

   $ bbcplay $(bbcsearch -i -d "17th June" Harry)

Bugs:

• Time reported is always in UTC.

This kind of error message:

$ svk pull
Syncing svn+ssh://des3jjm@vega.dur.ac.uk/home/hudson/pg/des3jjm/subversion/docs
Retrieving log information from 249 to 249
    (in cleanup) Svndiff data ends unexpectedly: Unexpected end of svndiff input
Svndiff data ends unexpectedly: Unexpected end of svndiff input
    (in cleanup) Svndiff data ends unexpectedly: Unexpected end of svniff input

Can be caused by:

$ df                                                          
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/engdebian52-root
                                6890048   6560316         0 100% /

So it's really not worth re-mirroring the whole repository...

Retry a failed command after some time. Retry the command specified if running it the first time fails. The time delay, and the number of retrys, to be configurable.

Example usage:

$ retry -d 10 -t 2 wajig daily-upgrade

That is, run wajig daily-upgrade, if it fails wait ten seconds and try it again. Do that a maximum of two times.

Done: retry.py

Firefox takes the X focus when opening a new tab, set browser.tabs.loadDivertedInBackground to true to stop it.

Downloading previous tracks:

gpsbabel -i garmin -f usb: -o gpx -F file.gpx

Uploading routes

gpsbabel -r -i gpx -f file.gpx -o garmin -F usb:

Creating routes:

1. Draw route on Google Maps.
2. Use GMapToGPX to create a route.
3. Edit created route file to combine all <rte></rte> blocks.

Use GPSVisualizer to plot downloaded routes.

Add handling of multipart messages to mail_to_note.

Anything other than the first part /i.e./ any attachements should be saved as files and linked from a section at the bottom of the message.

The format should be something like:

[1] foo_bar.py
[2] baz.tex

_twyt provides intelligent auto completion for the twitter client Twyt. Features include specific option completion for all commands, and intelligent completion for twitter names and twyt users.

To install pop the file _twyt somewhere in your fpath and restart zsh (assuming you've set up completion).

The latest version of _twyt can autocomplete tweet message IDs too, but it requires a patch to Twyt.

To fetch the link for old harry's game:

$ wget -q -O - http://www.bbc.co.uk/iplayer/console/b007jpd8 | \
link.pl | grep ".ram$" | head -1

To fetch from the link in the clipboard, and place the answer in the clipboard:

$ wget -q -O - $(xclip -o) | link.pl | grep ".ram$" | head -1 | xclip -i

Replacing link.pl and the grep with sed:

$ wget -q -O - $URL | sed -e '/http/!d' -e 's/.*http:\/\/\([^ \"]*\).*/\1/g' -e '/.ram$/!d' | head -1

See also bbcplay.

Attachments:

1. ?link.pl

Assuming you have a privoxy running on remotehost

$ ssh -NL 8118:localhost:8118 remotehost

Will forward localhost:8118 to the privoxy instance listening on remotehost:8118.

The output of /sbin/route contains the default network interface:

route | grep "^default" | awk '{print $8}'

Attachments:

What to do if your long running simulation gets accidentally killed? Have it automatically recover from a checkpoint of course!

Attached is a python script which counts slowly from zero to nine:

$ python2.5 checkpoint.py
0
1
2
3
4
5
6
7
8
9

However, at every step the state of the counter is stored, so if the process dies it can be restored:

$ python2.5 checkpoint.py
0
1
2
^CTraceback (most recent call last):
  File "checkpoint.py", line 41, in <module>
    for i in checkpointed_range(10):
  File "checkpoint.py", line 22, in next
    sleep(1);
Keyboard Interrupt
$ python2.5 checkpoint.py
3
4
5
6
7
8
9

Attachments:

1. checkpoint.py

Look up the position of your current IP address:

$ wget -q -O - "http://api.hostip.info/get_html.php?&position=true"
Country: UNITED KINGDOM (GB)
City: Durham
Latitude: 54.7667
Longitude: -1.566

Look up the position of someone elses IP

$ wget -q -O - "http://api.hostip.info/get_html.php?ip=212.58.226.77"
Country: UNITED KINGDOM (UK)
City: London

Another geo-location service

$  wget -O - -q "http://blogama.org/ip_query.php?ip=129.234.4.1"
<?xml version="1.0" encoding="UTF-8"?>
<Response>
    <Ip>129.234.207.232</Ip>
    <Status>OK</Status>
    <CountryCode>GB</CountryCode>
    <CountryName>United Kingdom</CountryName>
    <RegionCode>D8</RegionCode>
    <RegionName></RegionName>
    <City>Durham</City>
    <ZipPostalCode></ZipPostalCode>
    <Latitude>54.7667</Latitude>
    <Longitude>-1.5667</Longitude>
</Response>

Rather than using cut and grep (to find files present locally, but not checked in):

svk status | grep "^\?" | cut -d " " -f 2

You can use awk:

svk status | awk '/^\?/ {print $2;}'

Or sed:

svk status | sed -e '/^?/!d' -e 's/^?\s*//'

Or, just for fun, perl:

svk status | perl -ne 'print $1."\n" if /^\?\s*(.*)/'

To count the number of rows where time_rec is in the last three days:

mysql> select count(*) from log WHERE DATE_SUB(CURDATE(),INTERVAL 3 DAY) <= time_rec;

The limit expression ~(((~N|~O)!~D)|(~d<5d)) shows only threads with un-read emails, or threads with any message received in the last 5 days.

Better still, ~(((~d<5d!~Q)|(~U))!~D) shows only threads with messages from the last 5 days with un-answered messages, or un-read messages and excludes deleted messages.

My mutt config is in subversion