The Journals of Tarn Barford

debugProxy

tarn@tarnbarford.net (tarn) — Thu, 02 Nov 2023 12:00:00 +0000

Tarn Barford

debugProxy was a project I worked on with David in 2017, I hosted and supported the service at debugproxy.com for several years. I did a "Show HN" thread when we released it which was on the front page for a few hours, this gave us plenty of initial users. Despite it working quite well once setup, configuring devices to use it and trust root certificates was always fiddly and very difficult if a certificate pinning technique was used. I made all the code public in 2019. By 2021 usage was sporadic and I shut it down.

Despite the project not being wildly successful, I enjoyed working on it and am proud of what we built.

Because I've apparently learned nothing, I decided to see if it would still work. It turns out I can't even install the packages to build the frontend code anymore (React 15.6, Webpack 4), but I could find a copy of the compiled JavaScript, which I've added to the repository. To run the backend code, I had to build Python 3.7 and pin a few dependency versions.

I regret to say that I have DebugProxy fully working at debugproxy.tarnbarford.net. I'm definitely not going to provide support for getting specific devices to use the proxy, but if there are issues with the proxy or the website, I'd be interested to hear about it.

Creating Debian virtual machines

tarn@tarnbarford.net (tarn) — Fri, 27 Oct 2023 12:00:00 +0000

Tarn Barford

Creating virtual machines is one of the few things in my personal computer infrastructure I don't configure directly with Ansible. However, I have tried to make the process as fast and simple as possible. For generic Debian based virtual machines, I base new virtual machines off a base disk image.

I shell into the hypervisor to copy the base image.

tarn@hypervisor:~# cp /var/kvm/images/{debian-12-base.qcow2,demo.qcow2}

I then create a new virtual machine with the new disk image.

tarn@thinkpad:~# virt-install \
  --name demo \
  --network bridge=virbr0 \
  --network bridge=br0 \
  --disk /var/kvm/images/demo.qcow2 \
  --vcpu 4 \
  --memory 4096 \
  --os-variant debian11 \
  --console pty,target_type=virtio \
  --serial pty \
  --graphics none \
  --noautoconsole \
  --import

This will create a headless virtual machine with 4 vCPUs and 4GB RAM. On my host a standard virtual machine gets two network interfaces. The virbr0 interface is a virtual IPv4 bridge with DHCP support, this network will be allocated an IP address on an internal network and have a route to the Internet when the machine starts. The br0 inferface is a IPv6 bridge provided by the host system, the network doesn't have DHCP and will need to be configured. The --serial and --console options allow access to the machine via a serial console, the --noautoconsole prevents virt-install automatically connecting to serial console after creating the virtual machine.

The next step is to work out what IP address was assigned to the NIC in the virtual machine. This can be found by looking the DHCP leases, but its made a bit more complicated because every new virtual machine will initially request a lease with the hostname debian-12-base.

tarn@thinkpad:~# virsh net-dhcp-leases default
 Expiry Time           MAC address         Protocol   IP address          Hostname         Client ID or DUID
-----------------------------------------------------------------------------------------------------------------------------------------------------
 2023-10-22 22:26:12   52:54:00:16:b6:7e   ipv4       192.168.122.34/24   debian-12-base   ff:00:16:b6:7e:00:01:00:01:2c:c1:6b:25:52:54:00:16:b6:7e
 2023-10-22 22:33:44   52:54:00:b6:9d:8a   ipv4       192.168.122.66/24   debian-12-base   ff:00:b6:9d:8a:00:01:00:01:2c:c1:6b:25:52:54:00:16:b6:7e

To workout which is the correct IP address, the MAC addresses assigned to the new virtual machine networks can be used.

tarn@thinkpad:~# virsh domiflist demo
 Interface   Type     Source   Model    MAC
-----------------------------------------------------------
 vnet85      bridge   virbr0   virtio   52:54:00:b6:9d:8a
 vnet86      bridge   br0      virtio   52:54:00:07:91:8f

This IP isn't is not directly accessable from over the Internet and I don't have a VPN setup with the hypervisor, however I do have an sshd daemon on it and I can use that as jump-host.

tarn@thinkpad:~# ssh -J hypervisor.tarnbarford.net 192.168.122.66

I wouldn't usually SSH directly in at this point, but rather have Ansible use SSH to configure the system from here, including the networking where I configure static IPv4 and IPv6 addresses. This means on the first run I use the DHCP assigned address as the ansible_host subsequently the staticly configured address.

hosts:
  demo:
    ansible_user: tarn
    ansible_host: 192.168.122.66
    ansible_ssh_common_args: "-o ProxyCommand='ssh -q -W %h:%p tarn@hypervisor.tarnbarford.net'"

The virtual machine can then be accessed from the Internet via its static IPv6 address. To make it accessable via IPv4 I use iptables rules to route a specific port to a specific virtual machine address. All HTTP/HTTPS traffic is routed to load balancer running HAProxy which uses the Host header or Server Name Indication (SNI) to route traffic to a specific VMs.

The process is pretty quick but it is still significantly more fiddly than starting a virtual machine with a cloud provider.

Building a Debian 12 base image

tarn@tarnbarford.net (tarn) — Fri, 27 Oct 2023 11:00:00 +0000

Tarn Barford

I need a new Debian-based image to create new virtual machines since the old one is now two major versions out of date. I previously created the base images using a script that creates a disk image, mounts it as a block device and installs Debian with debootstrap. This process is somewhat tricky because it requires privileges and kernel modules to access system network block devices. It also takes several minutes to install the system, for creating a new virtual machine it's much quicker and simpler to copy a base image. Rather than get the script working, this time I'm just going to create a new virtual machine and use the graphical Debian installer to manually build a new image.

The installation media is downloaded on the host system.

tarn@hypervisor:~/# curl -o /var/kvm/installers/debian-12.1.0-amd64-netinst.iso https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.1.0-amd64-netinst.iso

A qcow2 disk image is created on the server.

tarn@hypervisor:~/# qemu-img create -f qcow2 /var/kvm/images/debian-12-base.qcow2 20G

As I've prevously configured remote libvirt connections with TLS I can connect to a remote host with virt-install, virt-viewer and virsh by exporting an environment variable.

tarn@thinkpad:~/# export LIBVIRT_DEFAULT_URI=qemu+tls://hypervisor.tarnbarford.net:16514/system

I use virt-install to create a new virtual machine.

tarn@thinkpad:~/# virt-install \
    --name debian-12-base \
    --connect qemu+tls://hypervisor.tarnbarford.net:16514/system \
    --network bridge=virbr0 \
    --disk /var/kvm/images/debian-12-base.qcow2 \
    --vcpu 4 \
    --memory 4096 \
    --os-variant debian11 \
    --graphics vnc \
    --console pty,target_type=virtio \
    --serial pty \
    --cdrom /var/kvm/installers/debian-12.1.0-amd64-netinst.iso
Starting install...
Creating domain...
Running graphical console command: virt-viewer --connect qemu+tls://hypervisor.tarnbarford.net:16514/system --wait debian-12-base

The virtual machine will use a network bridge on the host machine called virbr0 the DHCP service will allocate an IP address on an internal network and the bridge provides a route to the Internet. Both the installation media and the disk image are specified. I'll need graphics for the installation, I'll use VNC which I've previously configured to run remote over a secure TLS connection. The --console and --serial options will allow me to make a serial console connection once it's enabled on the virtual machine.

This brings up a virt-viewer window on which I can run through the standard Debian graphic install process. I've taken some screenshot of the process. After completing the installation the system will reboot and the virt-viewer window will close.

For the rest of the setup I prefer to use a serial console rather than VNC, mostly because copy and paste works and I don't have to deal with a window the hijacks my mouse cursor, but this won't work just yet.

First, I need a new VNC connection. I can either use virt-manager or virt-viewer.

tarn@thinkpad:~/# virt-viewer debain-12-base

In the VNC window I just want to configure the system so it supports a serial console. Enabling the serial-getty@ttyS0 serice should be enough, but for some reason this didn't work after I cloned the base image.

root@debian-12-base:~/# systemctl enable serial-getty@ttyS0.service

It's also possible to configure this in grub which worked reliably for me. First edit /etc/default/grub and ensure the following to values are set.

GRUB_CMDLINE_LINUX_DEFAULT="console=tty0 console=ttyS0,115200n8"
GRUB_TERMINAL=console

Once that's done, grub needs to be updated.

root@debian-12-base:~/# update-grub

Now it should be possible to get a serial console with libvirt.

tarn@thinkpad:~/# virsh console debian-12-base
Connected to domain 'debian-12-base'
Escape character is ^] (Ctrl + ])

debian-12-base login: root
Password:
Linux debian-12-base 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64
root@debian-12-base:~#

From this console we do the remaining configuration. Firstly install a couple of packages:

root@debian-12-base:~# apt install sudo vim curl

Add the reguarly user tarn created during the install process to the sudo group.

root@debian-12-base:~# usermod -a -G sudo tarn

Add my SSH key so I can SSH into the server as the tarn user.

root@debian-12-base:~# mkdir -p /home/tarn/.ssh
root@debian-12-base:~# curl -o /home/tarn/.ssh/authorized_keys https://github.com/tarnacious.keys
root@debian-12-base:~# chown -r tarn:tarn /home/tarn/.ssh
root@debian-12-base:~# chmod 600 /home/tarn/.ssh/authorized_keys

Clear the history

root@debian-12-base:~# history -c && history -w

The close the bash shell with ctrl d and then the serial console with ctrl ]. The base image is now configured so we can shut down the VM and delete it's libvirt configuration, the disk is the only thing we care about.

tarn@thinkpad:~/# virsh destroy debian-12-base
tarn@thinkpad:~/# virsh undefined debian-12-base

The file /var/kvm/images/debian-12-base.qcow2 on the host is now a bootable Debian image that I can copy and create new virtual machines from.

Building NixOS qcow2 images with flakes

tarn@tarnbarford.net (tarn) — Mon, 25 Sep 2023 12:00:00 +0000

Tarn Barford

I've been thinking again about using NixOS to build bootable disk images I can run on a remote qemu/KVM host. I worked out how to build bootable qcow2 disk images a few years back, which still seems to work, but I wanted something that worked with flakes which I've since adopted.

I initially found a way using nixos-generators which worked pretty well until I wanted to change the diskSize. This wasn't a big issue, but I realised I didn't really need nixos-generators either.

This is a pretty standard looking configuration.nix module that describes an example system to build.

{ config, lib, pkgs, ... }: {
  boot.kernelPackages = pkgs.linuxPackages_5_15;

  users.users = {
    tarn = {
      isNormalUser = true;
      extraGroups = [ "wheel" ];
      password = "";
    };
  };

  environment.systemPackages = with pkgs; [
    python310
  ];

  system.stateVersion = "23.05";
}

The configuration for building qcow2 disk images image can be separated into it's own module called qcow.nix. The attribute system.build.qcow2 is set to use the NixOS make-disk-image.nix magic.

{ config, lib, pkgs, modulesPath, ... }: {
  imports = [
    "${toString modulesPath}/profiles/qemu-guest.nix"
  ];

  fileSystems."/" = {
    device = "/dev/disk/by-label/nixos";
    autoResize = true;
    fsType = "ext4";
  };

  boot.kernelParams = ["console=ttyS0"];
  boot.loader.grub.device = lib.mkDefault "/dev/vda";

  system.build.qcow2 = import "${modulesPath}/../lib/make-disk-image.nix" {
    inherit lib config pkgs;
    diskSize = 10240;
    format = "qcow2";
    partitionTableType = "hybrid";
  };
}

To tie it all together we can create a flake.nix file which describes an input and uses the modules above to build an output.

{
  description = "Example Virtual Machine Configuration";
  inputs =  {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
  };
  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      # configuration for builidng qcow2 images
      build-qcow2 = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./configuration.nix
          ./qcow.nix
        ];
      };
    };
  };
}

To run a flake all the configuration needs to be added to a git repository.

$ git init
$ git add .

To build the qcow2 images nix build is used. Here we use our build_qcow2 configuration and the system.build.qcow2 attribute from earlier which is now somehow available.

$ nix build .#nixosConfigurations.build-qcow2.config.system.build.qcow2

This builds a qcow2 disk image in the nix store and creates a result symlink to the the containing directory in the store. To try out the disk image, we'll need a writable copy that can be mounted as a writable disk by qemu.

$ cp result/nixos.qcow2 root.qcow2
$ chmod 644 root.qcow2

The image can now be used to boot a qemu virtual machine

$ qemu-kvm -name nixos \
           -m 4G \
           -smp 2  \
           -drive cache=writeback,file=root.qcow2,id=drive1,if=none,index=1,werror=report \
           -device virtio-blk-pci,bootindex=1,drive=drive1 \
           -nographic
...
<<< Welcome to NixOS 23.05.20230924.261abe8 (x86_64) - ttyS0 >>>

Run 'nixos-help' for the NixOS manual.

nixos login: tarn
Password:

[tarn@nixos:~]$ uname -a
Linux nixos 5.15.133 #1-NixOS SMP Sat Sep 23 09:10:03 UTC 2023 x86_64 GNU/Linux

[tarn@nixos:~]$ python --version
Python 3.10.12

To kill the VM shut it down with the poweroff command or use ctrl + a followed by x.

Using libvirtd VNC Remotely

tarn@tarnbarford.net (tarn) — Wed, 20 Sep 2023 12:00:00 +0000

Tarn Barford

Continuing on this foolish path to manage some virtual machines on a remote host with TLS and peer authentication, here I'm going to configure VNC.

The trick here is we're really configuring qemu not libvirtd as qemu itself provides the VNC server. On my Debian system the default configuration file is /etc/libvirt/qemu.conf. By default, TLS for VNC is not enabled.

To listen on all interfaces, use TLS and do peer authentication the following options need to be set.

vnc_listen = "0.0.0.0"
vnc_tls = 1
vnc_tls_x509_verify = 1

The default certificates directories are either the default qemu certificate directory.

/etc/pki/qemu/ca-cert.pem
/etc/pki/qemu/server-cert.pem
/etc/pki/qemu/server-key.pem

Or the qemu VNC certificate directory.

/etc/pki/libvirt-vnc/ca-cert.pem
/etc/pki/libvirt-vnc/server-cert.pem
/etc/pki/libvirt-vnc/server-key.pem

I chose to copy them to the latter.

VNC ports are allocated by qemu starting a port 5900 by default. I'm going to assume these are reused when either a domain is destroyed or deleted and allowed the port range 5900:5910 through the firewall.

I could only find limited documentation about where virt-manager or virt-viewer expects the certificates to be installed for making VNC connections. I ended up using strace to see where it was trying to read them from.

I was a bit surprised to find they actually expect the client certificates in different paths. Here is where virt-manager reads certificates.

~/.pki/CA/cacert.pem
~/.pki/libvirt-vnc/clientcert.pem
~/.pki/libvirt-vnc/private/clientkey.pem

And virt-viewer here, which partially overlaps where libvirt reads its certificates.

~/.pki/CA/cacert.pem
~/.pki/libvirt/clientcert.pem
~/.pki/libvirt/private/clientkey.pem

After copying the missing certificates we should be ready to go.

To create a new virtual machines with a graphical installer, which is what I wanted to do, I still need to download the installer ISO disk image on the server.

$ wget https://channels.nixos.org/nixos-23.05/latest-nixos-gnome-x86_64-linux.iso -P /var/kvm/installers/

It's now possible to create a new virtual machine with virt-manager and then use a graphical installer to install an operating system through a VNC display. 1 2 3 4 5 6

I can also script the virtual machine creation with virt-install which will launch the virt-viewer VNC client. A list of possible --os-variant values can be found run running virt-install --osinfo list.

$ virt-install \
  --connect 'qemu+tls://hypervisor.tarnbarford.net:16514/system' \
  --name demo \
  --memory 2048 \
  --vcpus 2 \
  --disk size=20 \
  --cdrom /var/kvm/installers/latest-nixos-gnome-x86_64-linux.iso \
  --boot cdrom,hd \
  --os-variant nixos-22.05 \
  --graphics vnc

Starting install...
Allocating 'demo.qcow2'                                                                                                    | 3.1 MB  00:00:03 ...
Creating domain...                                                                                                         |    0 B  00:00:00
Running graphical console command: virt-viewer --connect qemu+tls://hypervisor.tarnbarford.net:16514/system --wait demo

Bam! A virt-viewer window appears displying the install media boot screen.

This is pretty great, I can now quickly and easily create new virtual machines on the server and immediately connect to their display and input devices using VPN client.

I'm still not fully satisfied, when I create virtual machines on my laptop I always prefer SPICE over VPN. The main reason is that SPICE has a feature "Redirect USB device" which works really well for many of my workflows that rely on my USB security key. The reason I didn't setup SPICE initially is that it doesn't support TLS peer authentication (!). I have an idea how I could make this work which I'd like to try, but I'll leave that for another time.

Using libvirtd Remotely

tarn@tarnbarford.net (tarn) — Mon, 18 Sep 2023 12:00:00 +0000

Tarn Barford

I have a headless server running libvirtd with a KVM/qemu backend and I wanted to securely connect to it with virsh and virt-manager on my laptop. There are several ways to do this, I decided to try TLS with a private PKI for transport encryption and peer authentication.

I wrote a script to generate a Certificate Authority (CA) certificate, then generate certificates for the client and the server that are both signed by the CA certificate.

The server runnning libvirtd needs to be configured to support TLS connections. On Debian systems the libvirtd configuration is in /etc/libvirt/libvirtd.conf. The relevant default configuration values were good and I didn't make any changes to this file.

listen_tls = 1
tls_port = "16514"
key_file = "/etc/pki/libvirt/private/serverkey.pem"
cert_file = "/etc/pki/libvirt/servercert.pem"
ca_file = "/etc/pki/CA/cacert.pem"
tls_no_verify_certificate = 0

For libvirtd itself, I only needed to copy the server certificates into the locations specified in the configuration files and then restart the service.

On the server, I needed to enable access to the port 16514 through the filewall.

Debian has systemd unit for libvirt socket activation via the TLS port /lib/systemd/system/libvirtd-tls.socket. This means libvirtd will start when a connection to port 16514 is made.

For the client I needed to copy my client certificates to the locations to where libvirt expects them.

I'm now able to connect to my server with virsh on my laptop.

$ virsh -c qemu+tls://hypervisor.tarnbarford.net:16514/system list
 Id   Name           State
------------------------------
 1    bacula         running
 2    loadbalancer   running
 3    mail-server    running
 4    tarnbarford    running
 5    monitoring     running
 7    bab-website    running
 8    owncloud       running
 9    icinga         running
 10   dns            running
 11   ns1            running

I can connect to the serial console of the virtual machines, which is pretty useful when working with headless servers.

$ virsh -c qemu+tls://hypervisor.tarnbarford.net:16514/system console tarnbarford
Connected to domain 'tarnbarford'
Escape character is ^] (Ctrl + ])

tarnbarford login:

And I can connect virt-manager to work with libvirtd on my server.

What isn't yet working is using virt-viewer with VNC or SPICE to conect to the displays of the virtual machines.

The VNC and SPICE connections are directly from virtual machine emulators, not via libvirtd so this needs to be setup separately. I'd really like to get this working smoothly, so I expect to post something about this soon.

Securing TCP Sockets with TLS

tarn@tarnbarford.net (tarn) — Thu, 14 Sep 2023 19:54:00 +0000

Tarn Barford

Using a simple private Public Key Infrastructure (PKI), plain TCP sockets can be secured with TLS. This can be demonstrated with trivial examples that run locally on a loopback device.

A simple echo TCP socket server that listens on port 7000 can be created with socat.

$ socat PIPE TCP-LISTEN:7000,fork

When data is sent to the socket, it will echo it back. The ncat tool from nmap is used because it has TLS features that will be needed later.

$ echo "hello" | ncat localhost 7000
hello

The socket listening on port 7000 won't actually be secured, sorry, instead a new socket listening on port 6000 will provide a secure TLS tunnel/proxy.

The tunnel is created with stunnel. It's configured with local file called stunnel.conf which contains a section for the tunnel.

[service1]
accept = 6000
connect = 7000
CAfile = ca.crt
cert = service1.crt
key = service1.key
verify = 0

The tunnel is called service1, it listens on port 6000 and creates a tunnel to port 7000. The CA and service1 certificates were created by the PKI. The verify option is set to 0 meaning that peer (client) certificates will not be required or verified.

The tunnel is started by running stunnel with the path to the configuration file.

$ stunnel service1.conf

Trying to connect port 6000 like before won't work.

$ echo "hello" | ncat localhost 6000
Ncat: Connection reset by peer.

The server is expecting the client to start a TLS negotiation. Instead, the characters "hello" are being sent, so the server disconnects. The error logged by stunnel is "SSL routines::wrong version number" which isn't particularly helpful.

Fortunately ncat supports TLS/SSL and it be used with the --ssl option.

$ echo "hello" | ncat --ssl localhost 6000
hello

A response is now returned through an encrypted TLS connection.

If this connection was made over an insecure network, there is no certainty that the server being connected to is really service1. With TLS, certificates used by 'service1' can be verified to have been signed by a specific CA and belonging to service1.

To do this with netcat the --ssl-verify option tells ncat to check the server certificate. The certificate should be signed by a specific CA, so ca.crt needs to be specified with the --ssl-trustfile option. The name in the certificate will also be validated, so service1 needs to specified with the --ssl-servername option.

$ echo "hello" | ncat --ssl --ssl-verify --ssl-servername service1 --ssl-trustfile ca_public_key.pem  localhost 6000
hello

Expecting a different server name should result in a certificate error.

echo "hello" | ncat --ssl --ssl-verify --ssl-servername service2 --ssl-trustfile ca_public_key.pem  localhost 6000
Ncat: Certificate verification error. QUITTING.

A really useful tool when debugging TLS connections is s_client the openssl test client.

openssl s_client -servername service1 -connect localhost:6000 -CAfile ca.crt

This will attempt to negotiate a connection and output a heaps of useful information about the certificate, the signatures, the alorithms used, the validation results and more.

TLS also supports a "peer authentication" mode in which the server checks the client certificates. Servers, or tunnels, can be configured to only allow clients with certificates signed by a specific CA to connect.

A secure tunnel configuration that uses strict peer authentication can be added to the stunnel.conf file.

[service1-secure]
accept = 5000
connect = 7000
CAfile = ca.crt
cert = service1.crt
key = service1.key
verify = 2

This tunnel is configuration is similar to the previous one except it'll listen on a different port (5000) and verify is set to 2, meaning it will require client certificates and will validate they are signed by the CA (ca.crt)

After restarting the stunnel process with the updated configuration, connecting as before to the new tunnel on port 5000 does not work.

$ echo "hello" | ncat --ssl --ssl-verify --ssl-servername service1 --ssl-trustfile ca.crt  localhost 5000

This time there is no response, this is because no client certificates are being sent. The error logged by stunnel is "peer did not return a certificate".

To connect to this service, a client certificate pair signed by the CA is required. The process to generate a certificate pair for the client is the same process used to create the service1 certificate pair, but this certificate pair will be called client1.

SUBJECT="/C=DE/ST=Berlin/L=Berlin/O=tarnbarford.net/OU=tarnbarford.net/CN=client1"

openssl ecparam -genkey -name prime256v1 -noout -out client1.key
openssl req -new -key client1.key -subj $SUBJECT -out client1.csr
openssl x509 -req -days 365 -sha256 -in client1.csr -CA ca.crt -CAkey ca.key -set_serial 1 -out client1.crt

The will generate a private key client1.key and a public key client1.crt.

To configure ncat to use the client certificates, client1.key is provided with the --ssl-key option and client1.crt is provided with the --ssl-cert option.

$ echo "hello" | ncat --ssl --ssl-verify --ssl-servername service1 --ssl-trustfile ca.crt --ssl-cert client1.crt --ssl-key client1.key  localhost 5000
hello

A response is now returned, but only if the client and server certificates are signed by the CA.

Simple Private Public Key Infrastructure

tarn@tarnbarford.net (tarn) — Thu, 14 Sep 2023 12:40:00 +0000

Tarn Barford

In this private Public Key Infrastructure (PKI) a secret key and a self signed certificate pair is the Certificate Authority (CA) root. The CA can sign certificates containing identity information with its secret key. Anyone can verify a certificate was signed by the CA using the CA root certificate.

The openssl tool is used for all PKI operations including generating the CA certificate pair. Elliptic Curve (EC) keys are being generated and the prime256v1 curve (a.k.a secp256r1) has been chosen. The CA certificate is valid for 365 days. The Subject field contains the identity attributes in the certificate, for this CA they're not important and some example values have been chosen.

SUBJECT="/C=DE/ST=Berlin/L=Berlin/O=tarnbarford.net/OU=tarnbarford.net/CN=ca"

openssl ecparam -genkey -name prime256v1 -noout -out ca.key
openssl req -new -x509 -sha256 -days 365 -key ca.key -subj $SUBJECT -out ca.crt

A private key ca.key and a public certificate ca.crt are generated.

A certificate pair for an example service service1 can be generated and then signed with the CA created above. This has an intermediate step where a certificate signing request (CSR) is created, typically this would be sent to the CA who would verify it then return a signed public certificate. For these certificates the Common Name (CN) attribute in the subject field is important, as this is the part of the identity that will be verified later.

SUBJECT="/C=DE/ST=Berlin/L=Berlin/O=tarnbarford.net/OU=tarnbarford.net/CN=service1"

openssl ecparam -genkey -name prime256v1 -noout -out service1.key
openssl req -new -key service1.key -subj $SUBJECT -out service1.csr
openssl x509 -req -days 365 -sha256 -in service1.csr -CA ca.crt -CAkey ca.key -set_serial 1 -out service1.crt

A private key service1.key and a public key service1.crt are generated. The CSR certificate service1.csr can be discarded.

The identity information can be extracted from a certificate

$ openssl x509 -noout -subject -in service1.crt
subject=C = DE, ST = Berlin, L = Berlin, O = tarnbarford.net, OU = tarnbarford.net, CN = service1

The public key can be verified to have been signed by the CA.

$ openssl verify -verbose -CAfile ca.crt service1.crt
service1.crt: OK

The keys generated can also be used for signing and verifying signatures. For this the public key needs to be extracted from the certificate which also contains an identity and a signature from a CA.

openssl x509 -pubkey -noout -in service1.crt -out service1.pem

The extracted public key is called service1.pem.

The private key can be used to create a signature data.sig of some data, then the public key that was extracted from the public cerificate can be used to verify the signature of this exact data was made with the private key.

$ echo "hello world" > data.txt
$ openssl pkeyutl -sign -in data.txt -inkey service1.key -out data.sig
$ openssl pkeyutl -verify -pubin -inkey service1.pem -sigfile data.sig -in data.txt
Signature Verified Successfully

Encrypting data with an EC public key is a lot more involved than with an RSA public key and will not be explored here. What's probably more interesting is how to use the PKI to secure connections with TLS. This is covered in a follow-up post, securing sockets with TLS.

Linux One-Liner for Monitoring File Changes

tarn@tarnbarford.net (tarn) — Sun, 10 Sep 2023 12:00:00 +0000

Tarn Barford

A Linux one-liner to run a command when a file in a directory tree is created, modified or deleted. Uses inotifywait from the inotify-tools package.

while inotifywait  -e modify,create,delete -r .; do echo "file changed"; done

This is easy but annoying to look up. When I asked an LLM, it omitted the recursive flag (-r) unless I explicitly asked for it.

Static virtual bridge IP addresses with DHCP

tarn@tarnbarford.net (tarn) — Tue, 27 Nov 2018 19:54:00 +0000

Tarn Barford

It's sometimes desirable to be able to specify a static IP address for a virtual machine that is using a virtual network bridge. For example, this can be useful when you have additional iptables rules for a specific virtual machines on the host.

With libvirt networks, one way to do this is by configuring the DHCP server to allocate IP addresses to virtual machines based on their MAC address. This doesn't require any special configuration in the virtual machine as it obtains its IP via DHCP.

Configuring this requires the MAC address of the virtual machines network interface, when using libvirt this can be found in the virtual machines configuration.

root@demo ~ # virsh dumpxml guest-1
<domain type='kvm' id='1'>
..
  <devices>
    ..
    <interface type='bridge'>
      <mac address='52:54:00:b3:e8:b7'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    ..
  </devices>
..
</domain>

This shows the MAC address of this virtual machines interface is 52:54:00:b3:e8:b7. As this virtual machine had previously obtained an IP address from the DHCP server it will appear in the net-dhcp-leases for the network.

root@demo ~ # virsh net-dhcp-leases  default
Expiry Time          MAC address        Protocol  IP address                Hostname        Client ID or DUID
-------------------------------------------------------------------------------------------------------------------
2018-11-27 12:04:27  52:54:00:b3:e8:b7  ipv4      192.168.122.14/24         guest-1         -

Currently the virtual machine has been allocated 192.168.122.14.

Within this /24 subnetwork there are 254 available IP addresses, the range 192.168.122.2 to 192.168.122.254 is available to allocate to virtual machines.

To allow both dynamically allocated and static IP addresses, the available range for dynamically allocated IP addresses can be limited to a subset so that the rest are can be allocated to specific MAC addresses.

To do this the default network need to be modified and the dynamic IP range updated. To edit the default network the virsh net-edit default command is used, this will open the configuration in the users default editor.

<network>
  <name>default</name>
  <uuid>4690aefc-dfa3-4a0a-bd4c-e4297126a845</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:95:92:6b'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.128'/>
    </dhcp>
  </ip>
</network>

In the above configuration the upper DHCP range has been limited from 192.168.122.254 to 192.168.122.128.

The next task is to assign a specific IP outside this range to a specific MAC address. This done by adding a host element containing the MAC address found earlier, the desired IP address in this example 192.168.122.129 and the name of the virtual machine.

<network>
  <name>default</name>
  <uuid>4690aefc-dfa3-4a0a-bd4c-e4297126a845</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:95:92:6b'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.128'/>
      <host mac='52:54:00:b3:e8:b7' name='guest-1' ip='192.168.122.129'/>
    </dhcp>
  </ip>
</network>

Modifying this network configuration has no effect until the network is restarted. To do this the network is destroyed and started again, in virtualisation terminology destroying means stopping. This will cause virtual machines using the network to lose network connectivity.

root@demo ~ # virsh net-destroy default
Network default destroyed

root@demo ~ # virsh net-start default
Network default started

Even after the network is restarted, running virtual machines using the network will not be able to use the new network, they need to be destroyed and restarted for the interface to work again.

root@demo ~ # virsh destroy guest-1
Domain guest-1 destroyed

root@demo ~ # virsh start guest-1
Domain guest-1 started

Looking at the leases again, the virtual machine guest-1 has now been allocated 192.168.122.129.

root@demo ~ # virsh net-dhcp-leases default
Expiry Time          MAC address        Protocol  IP address                Hostname        Client ID or DUID
-------------------------------------------------------------------------------------------------------------------
 2018-11-27 13:49:09  52:54:00:b3:e8:b7  ipv4      192.168.122.129/24        guest-1        -

The VM has the IP address we specified, but taking down the entire network is a real problem, especially if there are other virtual machines using the network.

It is also possible to make some live configurations to the network such as the IP address assigned to a virtual machine, this change is immediate, the interface in the virtual machine doesn't even need to be restarted.

root@demo ~ # virsh net-update default modify ip-dhcp-host \
      '<host mac="52:54:00:b3:e8:b7" name="guest-1" ip="192.168.122.130"/>' \
      --live --config
Updated network default persistent config and live state

Host elements can also be deleted. This won't automatically allocate a new IP in the virtual machine, in fact it will keep using the previously allocated one. When the interface is restarted it will be allocated a new address.

root@demo ~ # virsh net-update default delete ip-dhcp-host \
      '<host mac="52:54:00:b3:e8:b7" name="guest-1" ip="192.168.122.130"/>' \
      --live --config
Updated network default persistent config and live state

Host elements can also be added, the interface in the virtual machine will need to be restarted to obtain the new IP address.

root@demo ~ # virsh net-update default add ip-dhcp-host \
      '<host mac="52:54:00:b3:e8:b7" name="guest-1" ip="192.168.122.131"/>' \
      --live --config
Updated network default persistent config and live state

Modifying the networks with commands is a bit unwieldy but may be preferable than taking down the entire virtual network. It's also possible to specify IPv6 addresses that will be allocated to virtual machines, but I intend to use a physical bridge network for IPv6 networking, so I'm not going to write about it. For more information on that, check out the libvirt networking documentation or this great libvirt tutial by Jamie Nguyen.

Virtual bridge networking with libvirt

tarn@tarnbarford.net (tarn) — Mon, 26 Nov 2018 19:54:00 +0000

Tarn Barford

The virtual bridged networking provided by libvirt is the easiest way to enable networking on KVM virtual machines. In most cases it should be as simple as starting the default network and specifying the network when defining new virtual machines.

The default libvirt network creates a virtual bridge called virbr0 and it has its own IP address 192.168.122.1/24. When a virtual machine is configured to use this bridge, a new interface is created called vbox1, vbox2 etc. The interface is attached to the virtual bridge and shared with the virtual machine.

The virtual bridge network supports DHCP so that virtual machines can enable DHCP to obtain an IP address on the bridged network. Some iptables rules are added by libvirt so that packets to addresses outside the host are correctly routed.

The default libvirt network can be started with the virsh command.

root@demo ~ # virsh net-start default
Network default started

The configuration of the network looks like this:

root@demo ~ # virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>4690aefc-dfa3-4a0a-bd4c-e4297126a845</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:95:92:6b'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

On the host it has created two new interfaces, the virbr0 interface and a virbr0-nic interface. The later isn't so important, it's added to the bridge by default so that the virbr0 has a MAC address.

root@demo ~ # ip addr
..
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:95:92:6b brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:95:92:6b brd ff:ff:ff:ff:ff:ff

When --network bridge=virbr0 is passed to virt-install, an interface will be added as a PCI device.

root@demo ~ # virsh dumpxml guest-1
<domain type='kvm' id='1'>
..
  <devices>
    ..
    <interface type='bridge'>
      <mac address='52:54:00:8c:7f:94'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    ..
  </devices>
..
</domain>

If the virtual machine disk image doesn't configure a network interface, for example if /etc/network/interfaces only contains a loopback configuration like this:

auto lo
iface lo inet loopback

Then when we access the machine via its console, we'll find an interface called ens2 that is down.

root@guest-1:~# ip addr
..
2: ens2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:b3:e8:b7 brd ff:ff:ff:ff:ff:ff
root@guest-nonw:~# ping tarnbarford.net

And we can't access the internet.

root@guest-1:~# ping tarnbarford.net
ping: tarnbarford.net: Temporary failure in name resolution

As the network supports DHCP, we just need to run dhclient on the interface to obtain an IP address and bring the interface up.

root@guest-1:~# dhclient

Now the interface is up and we have been allocated an IP address, 192.168.122.14.

root@guest-1:~# ip addr
..
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:b3:e8:b7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.14/24 brd 192.168.122.255 scope global ens2
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:feb3:e8b7/64 scope link
       valid_lft forever preferred_lft forever

Access to the internet is possible.

root@guest-1:~# ping -c 1 tarnbarford.net
PING tarnbarford.net (178.63.44.71) 56(84) bytes of data.
64 bytes from mail01.tarnbarford.net (178.63.44.71): icmp_seq=1 ttl=59 time=0.401 ms

To make this more permanent, it can be configured in /etc/network/interfaces by appending the following lines.

auto ens2
iface ens2 inet dhcp

After a reboot, the network should be up.

The virtual machine can now access the internet as well as other virtual machines running on the host that are using the same virtual bridge. However, it's still inaccessible from the internet.

Cheap root servers from Hetzner auctions

tarn@tarnbarford.net (tarn) — Sun, 25 Nov 2018 19:54:00 +0000

Tarn Barford

I rent a couple of root servers from Hetzner that run various services I host. You can rent standard and custom configurations but these have an initial setup cost and setup time. For impatient hobbyists like myself, Hetzner also allows you to rent unused servers from their server auctions. These servers can be rented with a one month notice period, have no up front costs, and are available to use almost immediately.

I'm experimenting with IPv6 bridge setups, for this I've found a cheap server from the auctions and am trying to document the process.

When you order a root server from Hetzner you will be given an IPv4 address and an IPv6 subnet for the server. When you SSH to the IP address you will reach the Hetzner rescue system, a shell environment where you can format, mount and install an operating system on the disks connected to your server.

Note: when SSHing into the rescue system you may want to disable "strict host checking" as the rescue system host key will be different than that of your host.

tarn@laptop $ ssh -o StrictHostKeyChecking=no [server IP address]

It's possible to mount the disks connected to the server yourself and manually install whatever you want. For convenience Hetzner also provides an installimage script that starts an ncurses interface that leads you through the setup of various distributions and will handle partitioning, grub setup, software RAID, prompt for root password, network configuration, install sshd and add your SSH key as an authorized key.

root@rescue ~ # installimage

I chose Debian 9 and took all the default options except the hostname, which I changed to demo. This isn't very interesting, but I took some screen shots of the process anyway. 1 2 3 4 5 6 7

After installing the image you can reboot from the rescue system and your server will boot up.

root@rescue ~ # reboot

The next time you SSH to your IP address you'll be connecting to your own server. This time you want strict host checking so when we connect via SSH going forward you can be confident you're connecting to your server.

tarn@laptop $ ssh [server IP address]
root@demo ~ #

If all goes well you should have an ssh shell on your new physical server!

If you ever find yourself in a situation where you can't SSH into your own server because the network is misconfigured or the server doesn't boot, you can re-enable the rescue system from the Hetzner web interface and either mount the disks and fix the configuration or install a new operating system.

If you're having problems with the boot process you can also request VNC access to the servers video and input devices. This is a manual process that requires raising a ticket, the few times I've made a request it's taken less than half an hour to get setup.

Techbikers 2016 and World Bicyle Relief

tarn@tarnbarford.net (tarn) — Mon, 13 Jun 2016 00:27:00 +0000

Tarn Barford

On July 1st, I will join Techbikers Germany for a charity bike ride from Poznan to Berlin to raise money for World Bicycle Relief. The ride relies on donations, so I hope to explain why the ride is important to me, that the charity is not only a good cause but also effective, and hopefully convince you to donate.

As most of you will know, I am pretty passionate about tech and love cycling, despite an accident or two. Some of you will have even donated in previous years, for which I am eternally grateful, we made it and I am really excited to be doing it all again.

The Techbikers initiative started in London in 2012, the idea was to organize a ride from London to Paris every year and have participants raise money for charity. The ride successfully raised over £170,000 for Room to Read over the first 3 years.

In 2013, Lea and Jan who participated in the inaugural ride decided to bring the idea to Berlin. Techbikers Germany's first ride was from Prague to Berlin. I joined the group a week before that ride and later wrote about it. It's been a pleasure to ride together in 2014 from Copenhagen to Berlin and in 2015 from Hamburg to Berlin.

Since the 2014 ride, we have been raising money for World Bicycle Relief, a charity whose work nicely aligns with our common interest, cycling. World Bicycle Relief designs and manufactures rugged bicycle parts engineered specifically for rural terrain and load requirements and provides training and tools to assemble and maintain the bikes locally. The bikes are mostly distributed to students and healthcare workers, but also entrepreneurs and family businesses in a number of African countries, but also in Thailand and Colombia. I am really impressed by how efficient they are: they distribute a high quality bike for €134 raised. 64% of their income comes from individual contributions, so rides such as this one are important to them. And they are passionate about what they do!

Cycling is my primary form of transport, even though I live in a city with a world class public transport system. It's also pretty flat which helps, but nevertheless I personally really believe in the power of bicycles. I am happy to be supporting a program that gets quality bikes to people for whom this really makes life a lot easier and who could otherwise not afford them.

This year, 34 of us will cycle 400km from Poznan to Berlin. It will be a little different for me as I had to take a break from cycling this year and I'm leaving my much trusted Koga Miyata at home and riding my fixed gear bike (hey 8bar-bikes, it would be great if you could donate too!). It should be fun.

This year we are targeting €25,000. My personal goal is €750 and every donation no matter how large or small will help reach this goal. To the companies I work with, a percentage of this is small relative to the value of our relationships and I will thank you in a follow up post. To friends and family who donate, I will thank you in person or if that's not possible by phone. For my Australian friends an extra hug when I'm back later this year!

Click here to visit my fundraising page directly

Update 15/6: The link should be in english now.

(The above link is in German, you can visit the Techbikers group page, here, and change the language in the top right corner then scroll down to find my page, don't donate to the other people, they are already mostly ahead of me)

Thanks

Tarn

Cross-compiling with Rust

tarn@tarnbarford.net (tarn) — Tue, 29 Dec 2015 19:55:00 +0000

Tarn Barford

I found a guide for cross-compiling Rust which basically involves configuring it with a --target option. The target needs to include the cross-compile target and the host target.

$ wget https://static.rust-lang.org/dist/rustc-1.5.0-src.tar.gz
$ tar xf rustc-1.5.0-src.tar.gz
$ ./configure --target=arm-unknown-linux-gnueabi,x86_64-unknown-linux-gnu --prefix "$(PWD)"/../bin
$ make -j$(nproc) install

This didn't work with my target arm-unknown-linux-gneabi as at some point in the build it started trying to use arm-linux-gneabi-gcc and arm-linux-gneabi-ar, which didn't work as they are called arm-unknown-linux-gneabi-gcc and arm-unknown-linux-gneabi-gcc. I guess this is a problem with the build script, but I couldn't easily work out a fix, so I just created some symlinks with the names it was expecting. This worked well.

$ export LD_LIBRARY_PATH=~/projects/arm-rust-cross-compiler/bin/lib/
$ export PATH=$PATH:~/projects/arm-rust-cross-compiler/bin/bin/
$ bin/bin/rustc --version
rustc 1.5.0-dev

Cargo can also be built from source. I don't think Cargo needs to be cross-compiled as it supports compiling for different targets and I don't actually want to run Cargo on the target machine.

$ git clone https://github.com/rust-lang/cargo
$ cd cargo
$ git submodule update --init
$ python -B src/etc/install-deps.py
$ ./configure --local-rust-root="$PWD"/../bin --prefix "$(PWD)"/../bin
$ make
$ make install

To cross-compile with Cargo, a linker needs to be specified for the target. In my case:

$ cat > ~/.cargo/config << EOF
[target.arm-unknown-linux-gnueabi]
linker = "arm-unknown-linux-gnueabi-gcc-5.2.0"
EOF

Now Cargo can be used to create a new project and build a binary for an ARM target.

$ cargo new --bin hello-rust
$ cd hello
$ cargo build --target=arm-unknown-linux-gnueabi
$ file target/arm-unknown-linux-gnueabi/debug/hello-rust
target/arm-unknown-linux-gnueabi/debug/hello: ELF 32-bit LSB shared object,
ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter
/lib/ld-linux.so.3, for GNU/Linux 4.3.0, not stripped

I was pretty confident at this point, until I tried to run it in my emulator and the program immediately terminated with the message "Illegal instruction". So far, my cross-compiler toolchain and emulator had worked well. Why is now generating illegal instructions?

I eventually found that I wasn't specifying a CPU type and qemu was using the default ARM926 processor on the Versatile board. This can be fixed by adding --cpu arm1176 (which is the CPU on the first generation Raspberry Pi which I am trying to emulate). However, now the Kernel doesn't boot as it was built for an ARM926 processor as specified by versatile_defconfig.

The Linux ARM Versatile config restricts the possible CPU targets to CPUs that can actually fit in the board. Fortunately I found this patch which removes this restriction. Now make ARCH=arm versatile_defconfig && make ARCH=arm menuconfig allows the System Type -> Support ARM V6 processor option to be selected.

Finally, after rebuilding the Kernel and adding the Rust built hello binary to the filesystem, the program can run in the emulator.

$ qemu-system-arm -M versatilepb -cpu arm1176 -m 128M -kernel zImage -initrd ./rootfs.img.gz -append "root=/dev/ram rdinit=/sbin/init"

Building Linux and BusyBox for ARM and emulating it on QEMU

tarn@tarnbarford.net (tarn) — Sun, 27 Dec 2015 19:55:00 +0000

Tarn Barford

Cross compiling a minimal Linux sytem for the ARM architecture and emulating it in QEMU.

Almost all the information from this and my previous post can be found in this nice tutorial by Kamel Messaoudi, which also includes debugging user programs. I'm posting my notes as I go through them again and which will use as reference points for future posts.

A Linux kernel that can be emulated by QEMU can be built using the versatile_defconfig and a working cross-compiler.

wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.3.3.tar.xz
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.3.3.tar.sign

gpg --recv 6092693E
unxz linux-4.3.3.tar.xz
gpg --verify linux-4.3.3.tar.sign

tar xf linux-4.3.3.tar

cd linux-4.3.3
make ARCH=arm versatile_defconfig
# to further configure the build
#make ARCH=arm menuconfig
make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- all

# this builds a kernal image to `./arch/arm/boot/zImage`
cp linux-4.3.3/arch/arm/boot/zImage .

Busybox is a single binary combining many Unix utilities including a shell.

wget https://busybox.net/downloads/busybox-1.24.1.tar.bz2
wget https://busybox.net/downloads/busybox-1.24.1.tar.bz2.sign

gpg --recv ACC9965B
gpg --verify busybox-1.24.1.tar.bz2.sign

tar xf busybox-1.24.1.tar.bz2
cd xf busybox-1.24.1
make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- defconfig
make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- install

This builds a minimal filesystem in ./_install. Some additional directories are required.

cd _install
mkdir -p lib proc sys dev etc etc/init.d

Some shared libraries are required by the BusyBox binary. The exact libraries can be found using the arm-unknown-linux-gnueabi-readelf program. I'm just going to copy all the libraries from my toolchain to the target.

cp ~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/lib/* lib

Next we add an init script at /etc/init.d/rcS which will be executed by the BusyBox init script.

cat > ./etc/init.d/rcS << EOF
#!/bin/sh

# Mount the /proc and /sys filesystems
mount -t proc none /proc
mount -t sysfs none /sys

# Populate /dev
/sbin/mdev -s

# Enable the localhost interface
ifconfig lo up
EOF

It needs to be executable.

chmod +x etc/init.d/rcS

A compressed disk image of this filesystem which can be read by qemu can be made using the cpio program.

find . | cpio -o --format=newc | gzip > ../../rootfs.img.gz

I use my system qemu package, but it is not hard to build.

qemu-system-arm -M versatilepb -m 128M -kernel ./zImage -initrd ./rootfs.img.gz -append "root=/dev/ram rdinit=/sbin/init"

Compiling an arm-unkown-linux-gnueabi toolchain with crosstool-ng

tarn@tarnbarford.net (tarn) — Sun, 27 Dec 2015 19:54:00 +0000

Tarn Barford

Building an arm-unkown-linux-gnueabi GNU cross-compiler toolchain using crosstool-ng in order to compile code from an x86_64 GNU/Linux system to an ARM GNU/Linux system.

Building cross-compiler toolchains is complicated and in some cases using a prebuilt one might be preferable. For example, Ubuntu has a package for an arm-unkown-linux-gnueabi toolchain. However, crosstool-ng makes compiling toolchains quite straight forward and makes customization possible if required.

First specify the crosstools-ng version and some build directories. The final toolchain is built to ~/x-tools by default.

CROSSTOOL_VERSION=1.22.0
BASEDIR=~/arm-tool-chain
CROSS_TOOL_HOME=$BASDIR/crosstool/source/
CROSS_TOOL_PREFIX=$BASDIR/crosstool/bin/
TOOLCHAIN_HOME=$BASDIR/toolchain/

Make the directories used to build the cross-compiler.

mkdir -p $CROSS_TOOL_HOME
mkdir -p $CROSS_TOOL_PREFIX
mkdir -p $TOOLCHAIN_HOME

Downloading, verifying and building crosstools-ng should be no problem in sane build environment. I have found that I sometimes need to apply a minor patch to get the entire toolchain to build. So far these have not been too difficult to work out or find a patch for.

cd $CROSS_TOOL_HOME
wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-$CROSSTOOL_VERSION.tar.bz2
tar xjf crosstool-ng-$CROSSTOOL_VERSION.tar.bz2

# The package signed with Bryan Hundven's gpg key, this can be verified by
# downloading the key are verifying the signature.
wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-$CROSSTOOL_VERSION.tar.bz2.sig
gpg --recv-keys 35B871D1
gpg --verify crosstool-ng-$CROSSTOOL_VERSION.tar.bz3.sig


cd crosstool-ng

# I had to apply this patch to get the native gdb to compile,
# https://github.com/crosstool-ng/crosstool-ng/pull/273/files

./configure --prefix=$CROSS_TOOL_PREFIX
make
make install
PATH="${PATH}:$CROSS_TOOL_PREFIX/bin"

In this case configuring the toolchain is as simple as selecting a predefined sample. In other cases crosstool-ng has a Linux kernel like menuconfig command to browse and update settings.

cd $TOOLCHAIN_HOME
ct-ng arm-unknown-linux-gnueabi

# If you want to further configure the build, I didn't make any changes here,
# but if your target has hardware floating point support for example you might
# want to enable it here.
#   ct-ng menuconfig

ct-ng build

If all works out well, the tools and libraries should have been built to ~/x-tools/arm-unknown-linux-gnueabi/.

$ ls -1a ~/x-tools/arm-unknown-linux-gnueabi/bin/
.
..
arm-unknown-linux-gnueabi-addr2line
arm-unknown-linux-gnueabi-ar
arm-unknown-linux-gnueabi-as
arm-unknown-linux-gnueabi-c++
arm-unknown-linux-gnueabi-cc
arm-unknown-linux-gnueabi-c++filt
arm-unknown-linux-gnueabi-cpp
arm-unknown-linux-gnueabi-ct-ng.config
arm-unknown-linux-gnueabi-dwp
arm-unknown-linux-gnueabi-elfedit
arm-unknown-linux-gnueabi-g++
arm-unknown-linux-gnueabi-gcc
arm-unknown-linux-gnueabi-gcc-5.2.0
arm-unknown-linux-gnueabi-gcc-ar
arm-unknown-linux-gnueabi-gcc-nm
arm-unknown-linux-gnueabi-gcc-ranlib
arm-unknown-linux-gnueabi-gcov
arm-unknown-linux-gnueabi-gcov-tool
arm-unknown-linux-gnueabi-gdb
arm-unknown-linux-gnueabi-gprof
arm-unknown-linux-gnueabi-ld
arm-unknown-linux-gnueabi-ld.bfd
arm-unknown-linux-gnueabi-ldd
arm-unknown-linux-gnueabi-ld.gold
arm-unknown-linux-gnueabi-nm
arm-unknown-linux-gnueabi-objcopy
arm-unknown-linux-gnueabi-objdump
arm-unknown-linux-gnueabi-populate
arm-unknown-linux-gnueabi-ranlib
arm-unknown-linux-gnueabi-readelf
arm-unknown-linux-gnueabi-size
arm-unknown-linux-gnueabi-strings
arm-unknown-linux-gnueabi-strip

We should add this to our path if we want to use them:

PATH=$PATH:~/x-tools/arm-unknown-linux-gnueabi/

The -v option will tell us a little about the toolchain:

$ arm-unknown-linux-gnueabi-gcc -v
Using built-in specs.
COLLECT_GCC=arm-unknown-linux-gnueabi-gcc
COLLECT_LTO_WRAPPER=/home/dev/x-tools/arm-unknown-linux-gnueabi/libexec/gcc/arm-unknown-linux-gnueabi/5.2.0/lto-wrapper
Target: arm-unknown-linux-gnueabi
Configured with:
/home/dev/projects/arm-tool-chain/build/bin/.build/src/gcc-5.2.0/configure
--build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu
--target=arm-unknown-linux-gnueabi
--prefix=/home/dev/x-tools/arm-unknown-linux-gnueabi
--with-sysroot=/home/dev/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot
--enable-languages=c,c++ --with-float=soft --with-pkgversion='crosstool-NG crosstool-ng-1.22.0' --disable-sjlj-exceptions --enable-__cxa_atexit
--disable-libmudflap --disable-libgomp --disable-libssp
--disable-libquadmath --disable-libquadmath-support --disable-libsanitizer
--with-gmp=/home/dev/projects/arm-tool-chain/build/bin/.build/arm-unknown-linux-gnueabi/buildtools
--with-mpfr=/home/dev/projects/arm-tool-chain/build/bin/.build/arm-unknown-linux-gnueabi/buildtools
--with-mpc=/home/dev/projects/arm-tool-chain/build/bin/.build/arm-unknown-linux-gnueabi/buildtools
--with-isl=/home/dev/projects/arm-tool-chain/build/bin/.build/arm-unknown-linux-gnueabi/buildtools
--with-cloog=/home/dev/projects/arm-tool-chain/build/bin/.build/arm-unknown-linux-gnueabi/buildtools
--with-libelf=/home/dev/projects/arm-tool-chain/build/bin/.build/arm-unknown-linux-gnueabi/buildtools
--enable-lto --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --enable-threads=posix
--enable-target-optspace --enable-plugin --enable-gold --disable-nls
--disable-multilib
--with-local-prefix=/home/dev/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot
--enable-long-long
Thread model: posix
gcc version 5.2.0 (crosstool-NG crosstool-ng-1.22.0)

We can also try using it to build something simple:

$ cd /tmp
$ cat > ./etc/init.d/rcS << EOF
#include <stdio.h>

int main (void) {
    printf ("Hello, ARM!\n");
    return 0;
}
EOF
$ arm-unknown-linux-gnueabi-gcc hello.c -o hello

Obviously we can't run it on the host as it is compiled for an ARM system:

$ ./hello
bash: ./hello: cannot execute binary file: Exec format error

We can get a bit of information about the binary using the file command:

$ file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 4.3.0, not stripped

We can get a lot more information using the readelf (some of the output removed for verbosity):

$ arm-unknown-linux-gnueabi-readelf -a hello
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x102c8
  Start of program headers:          52 (bytes into file)
  Start of section headers:          4676 (bytes into file)
  Flags:                             0x5000202, has entry point, Version5 EABI, soft-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         8
  Size of section headers:           40 (bytes)
  Number of section headers:         29
  Section header string table index: 26

..

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x0004a4 0x000104a4 0x000104a4 0x00008 0x00008 R   0x4
  PHDR           0x000034 0x00010034 0x00010034 0x00100 0x00100 R E 0x4
  INTERP         0x000134 0x00010134 0x00010134 0x00013 0x00013 R   0x1
      [Requesting program interpreter: /lib/ld-linux.so.3]
  LOAD           0x000000 0x00010000 0x00010000 0x004b0 0x004b0 R E 0x10000
  LOAD           0x0004b0 0x000204b0 0x000204b0 0x0011c 0x00120 RW  0x10000
  DYNAMIC        0x0004bc 0x000204bc 0x000204bc 0x000e8 0x000e8 RW  0x4
  NOTE           0x000148 0x00010148 0x00010148 0x00020 0x00020 R   0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x10

..

Dynamic section at offset 0x4bc contains 24 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]

..

Amongst the information here we can see that it links to the /lib/ld-linux.so.3 and libc.so.6 and shared libraries. These libraries were built with the cross-compiler and will need to be present in the /lib directory on the target system.

$ ls -1a ~/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/lib
.
..
ld-2.22.so
ld-linux.so.3
libanl-2.22.so
libanl.so.1
libatomic.a
libatomic.so
libatomic.so.1
libatomic.so.1.1.0
libBrokenLocale-2.22.so
libBrokenLocale.so.1
libc-2.22.so
libcrypt-2.22.so
libcrypt.so.1
libc.so.6
libdl-2.22.so
libdl.so.2
libgcc_s.so
libgcc_s.so.1
libitm.a
libitm.so
libitm.so.1
libitm.so.1.0.0
libitm.spec
libm-2.22.so
libmemusage.so
libm.so.6
libnsl-2.22.so
libnsl.so.1
libnss_compat-2.22.so
libnss_compat.so.2
libnss_db-2.22.so
libnss_db.so.2
libnss_dns-2.22.so
libnss_dns.so.2
libnss_files-2.22.so
libnss_files.so.2
libnss_hesiod-2.22.so
libnss_hesiod.so.2
libnss_nis-2.22.so
libnss_nisplus-2.22.so
libnss_nisplus.so.2
libnss_nis.so.2
libpcprofile.so
libpthread-2.22.so
libpthread.so.0
libresolv-2.22.so
libresolv.so.2
librt-2.22.so
librt.so.1
libSegFault.so
libstdc++.a
libstdc++.so
libstdc++.so.6
libstdc++.so.6.0.21
libstdc++.so.6.0.21-gdb.py
libsupc++.a
libthread_db-1.0.so
libthread_db.so.1
libutil-2.22.so
libutil.so.1

In a follow up post I write about using this cross compiler toolchain to compile a Linux kernel and Busybox and then run them in the qemu emulator.

Securing my Raspberry Pi backup server

tarn@tarnbarford.net (tarn) — Thu, 30 Jul 2015 19:00:00 +0000

Tarn Barford

I run my own email server, ownCloud server, this blog and git server on a VM running on a physical machine we rent from Hetzner. I depend on not losing data from these servers so I setup my Raspberry Pi as a backup server. Here are some thoughts on how I have tried to secure everything, perhaps it is interesting and I appreciate feedback.

Ideally the backup server shouldn't accept connections from the Internet at all, thereby reducing its surface area for attack. For this my Raspberry Pi securely protected from in-bound connections by my home router! Actually this isn't very secure at all as I don't trust the router or the other devices connected to it, although it is likely to see less random attacks then on the open Internet. As I run the server headless I need sshd running so I added some basic security; disallowed password and root SSH login and installed fail2ban.

The SSH keys that are used to access the server need to be stored on the Raspberry PI, this means someone could just take the SD card and copy the keys off it. To prevent this I encrypted the root partition and set it up so it will try and read the secret key from a USB stick during the boot. This way if I remove the USB stick after it has booted and the Raspberry Pi and SD card were stolen, the thief would still need to break the disk encryption to get the SSH keys. Of course this also means it won't restart after a power outage unless the USB stick is inserted.

It is tempting to allow the backup server root access on the target machine as it makes it easier to backup everything, but this is not ideal. If the backup server is somehow compromised then root access to the target machine is also given. To limit this risk I created an unprivileged user the backup server could connect to the as. I granted this user permissions to dump databases I wanted to backup and added secondary groups to the user for www-data and the private groups of the users home directories I wanted it to backup. This way the backup server has read access to all the data it needs much not much more.

I sync everything I want to backup from the server to a local encrypted disk connected to the Raspberry Pi. As the root partition of the Raspberry PI is itself encrypted it is reasonably safe to keep the key for the local disk on the root partition.

I don't really trust my local disk (disk failure, apartment disaster!), so I also transfer my backups to Amazon S3 who I trust not to lose data but not really for privacy. To protect my data privacy on Amazon I encrypt it with my GPG key. This is secure enough that it would cost more than I would bet anyone would pay to crack my backups.

Below is the script I run as a daily cron job. It backs up several databases and directories. It snapshots the backup directory every day to the local disk and every Sunday it pushes a PGP encrypted snapshot to Amazon S3. When it finishes, successfully or not, it emails me with the result and the logs. It also prevents multiple backups running simultaneously and cleans up old snapshots on the local disk. I don't bother removing old snapshot from S3 yet, but might consider it when I get an Amazon storage bill of more than $2.

#!/bin/bash
set -e

ssh_host=someone@somehost.com
ssh_port=22
email=someone@somehost
s3_bucket=some.bucket
log_path=/mnt/backup/logs
backup_path=/mnt/backup/current
snapshot_path=/mnt/backup/snapshots
snapshot_name=`date +"%Y%m%d_%H%M%S"`.tar.gz
snapshot_file=$snapshot_path/$snapshot_name
scriptname=$(basename $0)
pidfile="/tmp/${scriptname}"

function log {
    echo "[$(date)]: $*"
}

initialize_directories() {
    log "Initializing directories"
    mkdir -p $backup_path
    mkdir -p $snapshot_path
    mkdir -p $log_path
}

backup_databases() {
    cat databases | while read database; do
        log "Backing up database: $database"
        ssh -p $ssh_port $ssh_host "pg_dump --data-only $database" > $backup_path/$database.sql
    done
}

backup_files() {
    cat locations | while read remote_path local_path; do 
        log "Backing up $remote_path to $local_path"

        log "Ignoring files:"
        ssh -n -p $ssh_port $ssh_host "find $remote_path -type d -a ! -executable -prune -o ! -readable -a ! -type l" | 
            cut -c ${#remote_path}- > /tmp/unreadable
        cat /tmp/unreadable

        log "Start rsync"
        rsync --delete -rltzuv --exclude-from="/tmp/unreadable" -e "ssh -p $ssh_port" $ssh_host:$remote_path $backup_path/$local_path
    done
}

snapshot() {
    log "Taking snapshot $snapshot_name"
    tar -czf $snapshot_file $backup_path
}

remove_old_snapshots() {
        $snapshot_path -name "*.tar.gz" | sort -r | tail -n +10 | xargs rm
}

should_push_offsite() {
    # Returns successfully if the backups should be pushed offsite.
    # Currently it returns successfully on Sundays, but it would be better
    # if it checked how long since the last successful offsite push.
    day_of_week=$(date +%u)
    sunday=7
    if [ $day_of_week = $sunday ]; then
        return 0;
    else
        return 1;
    fi
}


push_offsite() {
    log "Encrypting snapshot"
    gpg --encrypt -o $snapshot_file.gpg --recipient $email $snapshot_file 
    log "Send encrypting snapshot"
    s3cmd put $snapshot_file.gpg s3://$s3_bucket/$snapshot_name.gpg
}

get_lock() {
    log "Attempting to aquire lock"
    exec 200>$pidfile
    if flock -n 200; then 
        pid=$$
        log "Lock aquired for process $pid"
        echo $pid 1>&200
        return 0;
    else 
        log "Failed to aquire lock"
        return 1;
    fi
}

do_backup() {
    get_lock
    initialize_directories
    backup_databases
    backup_files
    snapshot
    if should_push_offsite; then
        push_offsite
        remove_old_snapshots
    else
        log "Skipping offsite storage"
    fi
    log "Completed backup."
}

main() {
    do_backup | tee $log_path/$snapshot_name.log

    if [ ${PIPESTATUS[0]} -eq 0 ]; then
        log "Sending report to $email."
        gpg -ea -r $email -o $log_path/$snapshot_name.log.asc $log_path/$snapshot_name.log
        echo "Results attached" | mail -s "Buckup Succeeded" -a $snapshot_file.log.asc $email
    else
        log "Backup failed, sending report to $email."
        gpg -ea -r $email -o $log_path/$snapshot_name.log.asc $log_path/$snapshot_name.log
        echo "Results attached" | mail -s "Buckup Failed" -a $snapshot_file.log.asc $email
    fi
    log "Completed."
}

main

Now I think my backup system is relatively secure from attackers and unlikely to lose data, my main risk of data loss is probably me losing a key or un-plugging the Raspberry Pi.

How I delivered a million spam emails

tarn@tarnbarford.net (tarn) — Tue, 02 Jun 2015 19:54:00 +0000

Tarn Barford

I've been running my own personal mail server for almost two years. On Friday May 15, I tried to send an email and immediately received a undeliverable mail response from my mail server.

This is the mail system at host tarnbarford.net.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                   The mail system

<xxxxxxxx@xxxxxxxx>: host gmail-smtp-in.l.google.com[74.125.136.26] said:
    550-5.7.1 [144.76.187.44      12] Our system has detected that this message
    is 550-5.7.1 likely unsolicited mail. To reduce the amount of spam sent to
    Gmail, 550-5.7.1 this message has been blocked. Please visit 550-5.7.1
    http://support.google.com/mail/bin/answer.py?hl=en&answer=188131 for 550
    5.7.1 more information. y7si5709303wjw.181 - gsmtp (in reply to end of DATA
    command)

A week earlier a friend told me that one of my mails had been marked as spam by GMail, which was concerning, but I figured maybe I could take some steps to ensure my mail server looked more legitimate, like setting up reverse DNS lookup, SPF and DKIM correctly. This seemed much more concerning. I took a break outside to think about and what I should do next and why Google was picking on me all of a sudden? Slowly it dawned on me that maybe my server had been compromised.

I rushed back to my computer, shelled into the server and ran tail -f on the mail log. I was immediately presented a continuous stream of log messages. Fuck. I stopped the tail to see what was happening, it didn't look good "delivery temporarily suspended", "ERROR: Mail Refused", "Retrying will NOT succeed.". I stopped postfix and re-ran the tail command, the tail of log appeared but thankfully new log messages were not appearing.

"Table tennis?" a colleague had approached me from behind and shocked me. "Not now, I'm busy" I snapped back, shocking him and me. I broke my trance like stare from the shell on my screen, turned around and explained that my mail server had been compromised and appeared to by trying to send spam.

I turned back to my computer to see a message from Michal, we share a physical server which we run virtual machines on using KVM. "Hey", this cannot have been unrelated so I beat him to the punch, "my server was spamming :'(". He'd received an abuse email from our hosting provider.

15:41:48 piwoni: hey
15:41:54 tarnacious: yes?
15:42:05 tarnacious: my server was spamming :'(
15:42:09 piwoni: yes
15:42:14 tarnacious: I don't know how
15:42:16 piwoni: wanted to tell you
15:42:19 tarnacious: thanks
15:42:22 tarnacious: too late
15:42:44 tarnacious: you get email from hetzner?
15:42:45 tarnacious: :D
15:42:49 piwoni: yes
15:42:53 tarnacious: today?
15:42:55 piwoni: yes
15:42:58 tarnacious: ok.
15:43:05 tarnacious: has been 4 days.
15:43:11 tarnacious: ip address is ruined :(
15:43:33 piwoni: how did it happen?
15:43:42 tarnacious: I don't know yet.
15:43:45 piwoni: ok
15:44:05 piwoni: do you have anyting on your
15:44:08 piwoni: site?
15:44:13 piwoni: form?
15:44:36 tarnacious: nope.
15:44:37 piwoni: is the smtp with auth
15:44:45 tarnacious: yep.
15:44:56 tarnacious: maybe brute force the auth
15:45:10 tarnacious: I do not know yet.
15:45:49 tarnacious: queue was pretty massive, but most mail servers have stopped talking to me.

What happened

I still hadn't worked out how my server had been compromised. I've been running it for almost two years and I was pretty confident it wasn't an open relay. I looked through the logs and found a message from android@tarnbarford.net. I ran grep from= /var/log/mail.log, this account was trying to send a lot of mail.

15:51:43 tarnacious: ok, i see what has happened :(
15:51:49 piwoni: ?
15:52:39 tarnacious: in my mail logs...
15:52:47 tarnacious: sasl_username=android connects.
15:52:51 tarnacious: and me.
15:53:06 piwoni: android?
15:53:11 piwoni: username
15:53:11 piwoni: :P
15:53:17 tarnacious: android is an account I created to build android.
15:53:21 piwoni: heh
15:53:25 piwoni: and pass ?
15:53:28 tarnacious: I guess I must have given him a week password.
15:53:29 piwoni: empty ?
15:53:53 piwoni: how do you store them ?
15:54:57 tarnacious: unix password
15:55:13 piwoni: change it
15:56:44 tarnacious: no shit

I use this machine for many other things including; a web server, a VPN server, a git server, for connecting to IRC servers, for pair programming, and for running arbitrary long running processes. I try to manage the system like I think a Unix system administer would; I create user accounts for everything, I care about file permissions and try to run a much as possible as non-privileged users.

The android account was created for building a custom CyanogenMod. Aside from taking ages to build, it also requires significantly more disk space than I can free on my Macbook Air. As the user was not in the sudoers file and password SSH connections are not allowed, I hadn't thought the password was very important and presumably created an extremely weak one.

What I had not thought of was that the password could be used to authenticate with the SMTP server, this is because I have postfix set-up to use Linux accounts as users. It appeared that someone had managed to guess a username/password combination on the SMTP server. A friend later joked that the attacker may have been surprised to get access so early in their dictionary attack.

I changed the android users password, cleared the entire postfix mail queue, restarted postfix and watched the logs. There were a few intermittent failed login attempts, but the server didn't seem to be trying to send any more mail.

What was sent

Feeling like I had done enough to temporarily safely start running the server again while still keeping an eye on the logs I moved onto working out what had happened. For this I had copied all the logs to a working directory and unzipped them. I had logs from April 19th to May 17th, 12GB in total, which itself was pretty ominous.

I initially attempted to analyse the logs myself with some awk scripts but found the postfix logs quite difficult to extract useful summaries from. The problem is there can be many log entries for a single message and useful information is spread across them. There is some sort of ID in each log message and I initially thought that the ID would be unique per message and I could group on that and work out what happened for each ID, but it turns out it is a little more complicated than that as messages can have multiple recipients. Probably worth looking around, surely someone has solved the problem.

After a quick search I found a script, pflogsumm.pl, that seemed to summarise the type of information I was looking for. I'm not sure how accurate it is or what these numbers mean, but it is very clear that my little mail server had been quite busy.

Grand Totals
------------
messages

 493404   received
   1543k  delivered
      0   forwarded
 311685   deferred  (33608k deferrals)
 448708   bounced
    486   rejected (0%)
      0   reject warnings
      0   held
      0   discarded (0%)

   2406m  bytes received
   5826m  bytes delivered
    672   senders
   1166   sending hosts/domains
   1053k  recipients
 105817   recipient hosts/domains

The per day traffic summary was quite interesting to me as I shows that the account had been compromised a few weeks earlier than I thought. I also find the usage patterns quite interesting, it looks like it was compromised on April 26 and a few thousand emails were sent, possibly to test if the server could be used to send spam. Then it is used in bursts, with no email at been sent between May 7 and May 9.

Per-Day Traffic Summary
-----------------------
    date          received  delivered   deferred    bounced     rejected
    --------------------------------------------------------------------
    Apr 19 2015        21         21          0          0         12
    Apr 20 2015        35         35          0          0         30
    Apr 21 2015        32         32          0          0         14
    Apr 22 2015        30         30          0          0         44
    Apr 23 2015        34         34          0          0          8
    Apr 24 2015        28         28          0          0         13
    Apr 25 2015        22         22          0          0         10
    Apr 26 2015       110       2030          0       2039          9
    Apr 27 2015        61         60          0          0         18
    Apr 28 2015     67547     193033        565k     52301         15
    Apr 29 2015       149      11711     453347      10425         21
    Apr 30 2015    136912     460833       1244k     90682         17
    May  1 2015     11679     194702       4523k     26787         15
    May  2 2015       113      10189       3798k      2113         10
    May  3 2015        69       7538       3601k       850          9
    May  4 2015        44       5902       3344k      6094         10
    May  5 2015       109      19998       3064k        64         14
    May  6 2015        63       7587     187980          0          9
    May  7 2015        52         54         43          0         18
    May  8 2015        65         64         21          0         17
    May  9 2015        50         50         21          0         27
    May 10 2015     35576      74756     335577      22229         14
    May 11 2015     48658     125790        668k     68149         22
    May 12 2015     88920     203938       3024k    115446         26
    May 13 2015        73       5227       3056k     11819         13
    May 14 2015        76       3157       2885k      6634         13
    May 15 2015    102763     253975       2876k     33070         12
    May 16 2015        48         48          0          3         30
    May 17 2015        65         65          0          3         16

The top domains is pretty uninteresting, except that I can clearly see why GMail decided to start blocking emails from me.

Host/Domain Summary: Message Delivery
--------------------------------------
 sent cnt  bytes   defers   avg dly max dly host/domain
 -------- -------  -------  ------- ------- -----------
 281109     1116m    1540k    4.8 h   87.7 h  hotmail.com
 247760      705m   15277k   13.1 h  121.2 h  yahoo.com
 152088     1129m       0     1.2 m    4.9 h  tarnbarford.net
 118590   371368k  132586     5.3 h  120.9 h  gmail.com
  40608   130322k  359555     2.5 h   37.5 h  aol.com
  35961   132388k       1     2.8 h   15.5 h  telenet.be
  23607    73362k      12     7.3 h   17.3 h  163.com
  21060    77012k  206635    11.9 h  120.8 h  skynet.be
  18418    73872k  481938    25.0 h   76.2 h  hanmail.net
  17039    61274k   88959     4.4 h   82.1 h  msn.com
  15737    57300k   62322     5.4 h   15.5 h  live.nl
  13636    42315k       9     6.2 h   17.3 h  126.com
  13020    34618k   58867     6.9 h   21.1 h  ameritech.net
  11876    36937k   95879    10.5 h   77.1 h  vip.sina.com
  11828    30012k    1176k    8.0 h  121.1 h  adelphia.net
  11168    40958k   38749     5.1 h   31.7 h  hotmail.fr
  10203    33512k  267776    15.7 h  120.9 h  yahoo.fr
   9005    29039k       0     6.8 h   17.3 h  sina.com

These are the top two recipients, the next ten are genuine and then a huge list of addresses used three or less times.

Recipients by message count
---------------------------
 150742   android@tarnbarford.net
  34696   ppway2323@hotmail.com

I have no idea why ppway2323@hotmail.com was sent so many emails and I also don't know where the 150742 emails to android@tarnbarford.net are. There are only 87 emails in the Maildir of the user, they are all delivery status notifications. Usually the delivery status emails contain the original email, so we get some information on what was sent. They are not all the same, but they are all spam, this one is quite common.

Subject: Re: Part time job opportunity
To: Recipients <android@tarnbarford.net>
From: android@tarnbarford.net
Date: Thu, 14 May 2015 15:49:24 -0700
Reply-To: tinagerardi9470@gmail.com

Our Ref: Bny/23/9swd/34
Your Ref: ABWN/NYT/87E3

Position: Retail Product Research Coordinator.

We are a reputable Survey Company handling products and services survey
evaluation for most Fortune 500 companies in the United States of America. We
are seeking Dedicated Part time staff for the position of Retail Product
Research Coordinator. You will work as a team to test products pricing and
product display in your Geographical Zone.It is probationary assignments that
will last 18 months.Your basic pay per assignment will be $900.00. You will be
entitled to a pay review after 90 days. You only need 1hr in a day to carry out
your assignment. If interested in this position please send your reply to
tinagerardi9470@gmail.com. Include your full name, Address and a Day time
Telephone number. We will send you a package with your first Assignment as soon
as we we hear from you.

Sincerely;  Tina Gerardi.
Survey  Coordinator.

Adding security and anti-spam measures

My initial thoughts were to build a new mail server on its own machine, I had wanted to do this anyway as the mail server was was both relatively important to me and relatively difficult to set up. If not automate the set up, I wanted to at least document it. I decided against building it now as it appeared the server itself wasn't compromised, I didn't have the time and I would probably have to deal with my host name being on email blacklists even if I changed IP address. So I decided to see if I could harden this server and get myself off the blacklists.

I immediately added smtpd_sender_restrictions to only allow mail to be sent by specified users. Although I will probably create users with --disabled-password in the future, there are potentially other accounts on the system with weak passwords. It was quite easy to verify this was working, I just removed myself from the list, restarted postfix and tried to send a mail.

$ cat mail.eml | msmtp -a tarn tarn.barford@retresco.de
msmtp: recipient address tarn.barford@retresco.de not accepted by the server
msmtp: server message: 554 5.7.1 <tarn@tarnbarford.net>: Sender address rejected: Access denied
msmtp: could not send mail (account tarn from /Users/tarn/.msmtprc)

I enabled fail2ban jails for postfix and sasl which means any IP address that fails to authenticate 3 times will be banned for an hour. This was just a matter of enabling some sections in the fail2ban configuration as I already had it running for ssh connections. I found out this works the hard way by accidentally banning myself while testing other SMTP logins failed.

I used policyd to setup per-address rate limiting for sending and receiving mail. This was a bit more tricky, but I found a guide that helped me get it set-up. I tested this works by setting the quota really low and sending a few messages.

$ cat mail.eml | msmtp -a tarn tarn.barford@retresco.de
$ cat mail.eml | msmtp -a tarn tarn.barford@retresco.de
$ cat mail.eml | msmtp -a tarn tarn.barford@retresco.de
$ cat mail.eml | msmtp -a tarn tarn.barford@retresco.de
msmtp: recipient address tarn.barford@retresco.de not accepted by the server
msmtp: server message: 554 5.7.1 <tarn@tarnbarford.net>: Sender address rejected: Sender quota exceed.
msmtp: could not send mail (account tarn from /Users/tarn/.msmtprc)

Setting up a Sender Policy Framework (SPF) is just a matter setting a TXT record on the domain name.

$ dig +short -t txt tarnbarford.net
"v=spf1 mx a -all"

In my case all mail from my domain should come from the IP that is in the MX record, and if not then it should be rejected.

To send DomainKeys Identified Mail (DKIM) you need to create a key-pair, publish the public key as a domain record and add a header each email with a signature of the contents. There is a Debian package for OpenDKIM that includes a daemon that can sign mails and is easy to integrate with Postfix. Below is an example of a DKIM signed message I sent to myself.

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tarnbarford.net;
    s=mail; t=1432809266;
    bh=YocFhtTgjr2exNi9eb3SUWb8spcRxFH1Tqh/0OcRcfA=;
    h=Date:From:To:Subject:From;
    b=PAy0LzlmVsAQaQ9NzI4d+RSLMpz8Fg/eqNrVYU9rFlc7WHnIkBwBpFol2NQy8T+yg
    Dxuw3OAjDh0kjV7W9LF0Y2rnjVfVJ5RLQRVe0HJiVOkLyS9cESoZZ63Ki4uUb/oB8N
    hhvWPjrKw6hukPZYGSCOhGg8Zz8da1EviMHB6oP4=
Date: Thu, 28 May 2015 12:34:24 +0200
From: Tarn Barford <tarn@tarnbarford.net>
To: tarn@tarnbarford.net
Subject: DKIM
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.23 (2014-03-12)

hello, DKIM.

My DKIM public key for this message can be found based on the from header and the s field in the DKIM signature.

$ dig +short -t txt mail._domainkey.tarnbarford.net
"v=DKIM1\; k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC0LNf8wdRCk2eXt1+EAIAdDSfq4aLMR/a6hCdPzJdAZ5OB1Z2LvOJlUGtx81MAttOG2lztjMmdrEq4mmGe0LUXOmOTqMcY8/woNspqvj9N4zPUEZXFP6yrYlgVGuVcLYWV6huCviwlt49KciaB9al+E2PogZJxDda5/cYffrI4PQIDAQAB"

There are plenty of tools to create and validate DKIM mail signatures including a Python library. This validated that my DKIM signature was valid for the above mail.

$ virtualenv .
$ ./bin/pip install dkimpy
$ ./bin/pip install dnspython
$ ./bin/dkimverify.py < test.eml
signature ok

While this verifies the signature is ok, I wasn't entirely satisfied and wanted to see what was going on. The DKIM-Signature header has a body hash field bh which is specified in the header as being an rsa-sha256 hash. Once this is verified the signature itself can be verified on the headers alone as the headers contain a hash of the body contents.

$ printf "hello, DKIM.\r\n" | openssl dgst -binary -sha256 | openssl base64
YocFhtTgjr2exNi9eb3SUWb8spcRxFH1Tqh/0OcRcfA=

This matches the body hash in the signature, so we can continue. To verify the authenticity of this mails headers a signature, a public key and the exact headers that were signed with the private key are required.

The b field in the DKIM-Signature header is the signature. It is base64 encoded and openssl requires the binary form, so it needs to be decoded and saved.

$ printf "PAy0LzlmVsAQaQ9NzI4d+RSLMpz8Fg/eqNrVYU9rFlc7WHnIkBwBpFol2NQy8T+ygDxuw3OAjDh0kjV7W9LF0Y2rnjVfVJ5RLQRVe0HJiVOkLyS9cESoZZ63Ki4uUb/oB8NhhvWPjrKw6hukPZYGSCOhGg8Zz8da1EviMHB6oP4=" | openssl enc -base64 -d -A > signature

The p field in the mail._domainkey is a base64 encoded, DER encoded public key.

$ printf "MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC0LNf8wdRCk2eXt1+EAIAdDSfq4aLMR/a6hCdPzJdAZ5OB1Z2LvOJlUGtx81MAttOG2lztjMmdrEq4mmGe0LUXOmOTqMcY8/woNspqvj9N4zPUEZXFP6yrYlgVGuVcLYWV6huCviwlt49KciaB9al+E2PogZJxDda5/cYffrI4PQIDAQAB" | openssl enc -base64 -d -A > key.der

This can be verified this by converting it back into a familiar looking public key and saving it as key.pub.

$ openssl rsa  -pubin -inform DER < key.derwriting RSA key | tee key.pub
writing RSA key
-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC0LNf8wdRCk2eXt1+EAIAdDSfq
4aLMR/a6hCdPzJdAZ5OB1Z2LvOJlUGtx81MAttOG2lztjMmdrEq4mmGe0LUXOmOT
qMcY8/woNspqvj9N4zPUEZXFP6yrYlgVGuVcLYWV6huCviwlt49KciaB9al+E2Po
gZJxDda5/cYffrI4PQIDAQAB
-----END PUBLIC KEY-----

The modulus and exponent of this public key can also be checked and should be rejected if it isn't long enough.

$ openssl rsa -noout -text -pubin < key.pub
Modulus (1024 bit):
    00:b4:2c:d7:fc:c1:d4:42:93:67:97:b7:5f:84:00:
    80:1d:0d:27:ea:e1:a2:cc:47:f6:ba:84:27:4f:cc:
    97:40:67:93:81:d5:9d:8b:bc:e2:65:50:6b:71:f3:
    53:00:b6:d3:86:da:5c:ed:8c:c9:9d:ac:4a:b8:9a:
    61:9e:d0:b5:17:3a:63:93:a8:c7:18:f3:fc:28:36:
    ca:6a:be:3f:4d:e3:33:d4:11:95:c5:3f:ac:ab:62:
    58:15:1a:e5:5c:2d:85:95:ea:1b:82:be:2c:25:b7:
    8f:4a:72:26:81:f5:a9:7e:13:63:e8:81:92:71:0d:
    d6:b9:fd:c6:1f:7e:b2:38:3d
Exponent: 65537 (0x10001)

The DKIM-Signature also specifies a list of headers in the h field. These are the headers, followed by the DKIM-Signature header itself minus the signature field b that are hashed and then signed. There are some rules about formatting these headers before they are signed; line wrapping must be removed, the header key and value are separated by a single colon without spaces and headers are separated with a Carriage Return and Line Feed.

$ printf "date:Thu, 28 May 2015 12:34:24 +0200\r\n" > headers
$ printf "from:Tarn Barford <tarn@tarnbarford.net>\r\n" >> headers
$ printf "to:tarn@tarnbarford.net\r\n" >> headers
$ printf "subject:DKIM\r\ndkim-signature:v=1; a=rsa-sha256; c=relaxed/simple; d=tarnbarford.net; s=mail; t=1432809266; bh=YocFhtTgjr2exNi9eb3SUWb8spcRxFH1Tqh/0OcRcfA=; h=Date:From:To:Subject:From; b=" >> headers

Now openssl can again be used, this time to verify the key, the signature and the headers.

$ openssl dgst -sha256 -verify key.der -keyform DER -signature signature headers
Verified OK

Great, it verifies as expected. There is obviously more detail in the RFCs or by looking at an open source implementation, but I found it interesting to see how it worked for one email.

A Domain-based Message Authentication, Reporting and Conformance (DMARC) policy is simply another DNS record that allows a domain to specify what action other mail servers should take with mail purportedly from this domain based on the combined results of SPF and DKIM checks (allow, reject, hold, etc). It also provides an email address for participating mail servers to send reports.

$ dig +short -t txt _dmarc.tarnbarford.net
"v=DMARC1\;p=reject\;pct=100\;rua=mailto:postmaster@tarnbarford.net"

My DMARC policy says to reject 100% of mails that fail either SPF or DKIM checks and mail conformance information to postmaster@tarnbarford.net. I now get digest DMARC reports from participating mail servers, here is what Google sent me.

<?xml version="1.0" encoding="UTF-8" ?>
<feedback>
  <report_metadata>
    <org_name>google.com</org_name>
    <email>noreply-dmarc-support@google.com</email>
    <extra_contact_info>https://support.google.com/a/answer/2466580</extra_contact_info>
    <report_id>12005498305772839030</report_id>
    <date_range>
      <begin>1432684800</begin>
      <end>1432771199</end>
    </date_range>
  </report_metadata>
  <policy_published>
    <domain>tarnbarford.net</domain>
    <adkim>r</adkim>
    <aspf>r</aspf>
    <p>reject</p>
    <sp>reject</sp>
    <pct>100</pct>
  </policy_published>
  <record>
    <row>
      <source_ip>144.76.187.44</source_ip>
      <count>1</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>pass</dkim>
        <spf>pass</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>tarnbarford.net</header_from>
    </identifiers>
    <auth_results>
      <spf>
        <domain>tarnbarford.net</domain>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>
</feedback>

Blacklists

I found MXToolbox which checks a domain or IP address against quite a few email blacklists. It is quite useful as it also provides some information and links to get your IP off each list. My IP was listed in eight blacklists, I was able to get it off six almost immediately by following the delisting procedures. I believe I was on the other two remaining lists already as our other IP addresses in the network are also on the list, and are not even mail servers.

Google has been more difficult, "How to Remove Your IP from the Gmail Blacklist" goes into some detail and links to a Google Support page where you can submit delivery issues. The post says it will take 3-7 days to process provided you fixed the reason it was blocked in the first place. I filled the form out a few weeks ago after stopping the spamming and adding the sending restrictions. I thought I had done enough at that stage to be unblocked, but apparently Google didn't think so as I am still blocked. I have since added SPF, DKIM, DMARC and rate limiting and I will fill the form out again after I post this story.

Conclusion

This story shouldn't be seen as a reason not to run your own mail server, if you want to you should, it's fun! This was a pretty embarrassing security failure, but the exploit was crude and relatively harmless. In working out what happened and putting systems in place to prevent it occurring again I have learned a lot and added a range of anti-spam and security features to my little mail server.

Hopefully soon I will again be able to mail Google addresses, I will update this post when this happens (or doesn't).

Update June 19, 2014: I never actually re-sent that request to Google to unblock me, but today it seems I am able to send mail to GMail accounts again. Thanks GMail!

Packaging a Flask Application

tarn@tarnbarford.net (tarn) — Tue, 09 Sep 2014 19:54:00 +0000

Tarn Barford

Packages are not very exciting but the only sane way to deploy. I've tried to git clone and make install on the servers with Puppet, that was silly. It's about time I learnt how to build packages.

I run a few Flask applications (including this site) that I would like to package for Debian linux, so packaging a minimal Flask application seemed a good place to start.

I created a very simple Flask application called demosite. The basic requirement of the site is that python setup.py install works and that it provides some mechanism changing the configuration file and logging paths.

The package should install the application, all its dependencies, add the configuration files to /etc/demosite, set up the logging to log to /var/log/demosite, create a un-privileged user and install an init script in /etc/init.d/demosite.

I found this post: Packaging a Flask app in a Debian package very helpful getting me started.

The package is built and packaged on a Debian machine provisioned by Vagrant and run in Virtualbox. The application itself is installed inside a virtualenv environment which is then packaged by fpm. Below are all the steps needed to create a package and install it.

Create a vagrant box and ssh into it:

$ vagrant init chef/debian-7.4
$ vagrant up && vagrant ssh

Switch to the root user, update the system and install build dependencies:

vagrant:~$ sudo su
vagrant:~$ apt-get update
vagrant:~$ apt-get install ruby1.8 rubygems git python-dev python-setuptools python-virtualenv ruby-dev curl
vagrant:~$ gem install fpm pleaserun

Installing the application into a virtualenv is quite straight forward. Download the source, create a virtualenv and install the application in it, a lot problems with generated paths can be avoided by creating the virtualenv in the desired install path.

vagrant:~$ git clone https://github.com/tarnacious/demosite.git demosite-src
vagrant:~$ virtualenv /usr/share/demosite
vagrant:~$ cd /home/vagrant/demosite-src
vagrant:~/demosite-src$ /usr/share/demosite/bin/python setup.py install

I used pleaserun to create the init file. The gem requires the ruby-dev package to be installed.

vagrant:~$ pleaserun --user demosite --install --name demosite /usr/share/demosite/bin/gunicorn demosite:app
No platform selected. Autodetecting... {:platform=>"sysv", :version=>"lsb-3.1", :level=>:warn}
Writing file {:destination=>"/etc/init.d/demosite"}
Writing file {:destination=>"/etc/default/demosite"}

The file /etc/default is sourced using the dot operator in the init script. Environment variables can be added here, for the demosite application these are the locations of the config file and the logging config file.

vagrant:~$ cat > /etc/default/demosite << EOF
export DEMOSITE_CONFIG_PATH=/etc/demosite/config.cfg
export DEMOSITE_LOGGING_CONFIG_PATH=/etc/demosite/logging.conf
EOF

Create the config.cfg expected by the application.

vagrant:~$ mkdir /etc/demosite
vagrant:~$ cat > /etc/demosite/config.cfg << EOF
TAGLINE="this is all so far"
EOF

Create the logging.conf expected by the application.

vagrant:~$ cat > /etc/demosite/logging.conf << EOF
[loggers]
keys = root, demosite

[handlers]
keys = root, demosite

[formatters]
keys = default

[formatter_default]
format = [%(asctime)s] - %(name)s - %(levelname)s - %(message)s
class = logging.Formatter

[logger_root]
level = DEBUG
qualname = root
handlers = root

[handler_root]
class = logging.handlers.RotatingFileHandler
formatter = default
args = ("/var/log/demosite/root.log",)

[logger_demosite]
level = DEBUG
qualname = demosite
handlers = demosite

[handler_demosite]
class = logging.handlers.RotatingFileHandler
formatter = default
args = ("/var/log/demosite/demosite.log",)
EOF

Debian packages have hooks for a postinst, preinst and prerm. This postinst script creates a user demosite, a directory /var/log/demosite and changes ownership over to the demosite user.

vagrant:~$ cat > postconfig << EOF
#!/bin/sh -e

action="\$1"
oldversion="\$2"

. /usr/share/debconf/confmodule
db_version 2.0

if [ "\$action" != configure ]; then
    exit 0
fi

if ! getent passwd demosite >/dev/null; then
    adduser --quiet --system --no-create-home --home /home/vagrant/demosite --shell /usr/sbin/nologin demosite
fi

mkdir -p /var/log/demosite
chown demosite /var/log/demosite
EOF

In this package the preinst and prerm hooks are the same for now: stop the service if the init script is there.

vagrant:~$ cat >stopservice << EOF
#!/bin/sh
set -e

if [ -f /etc/init.d/demosite ]; then
    /etc/init.d/demosite stop
fi
EOF

Package the application into a deb package with fpm.

vagrant:~$ fpm -s dir \
    -t deb \
    -n demosite \
    -v 0.1 \
    --deb-init /etc/init.d/demosite \
    --before-install stopservice \
    --after-install postconfig \
    --before-remove stopservice \
    -d "python" \
    /etc/demosite/=/etc/demosite \
    /etc/default/demosite=/etc/default/ \
    /usr/share/demosite/=/usr/share/demosite

Now copy the package to the host machine, to a build directory:

vagrant:~$ mkdir -p /vagrant/build
vagrant:~$ cp demosite_0.1_amd64.deb /vagrant/build/

Unless there was any problems with the build we can exit this box and even destroy it.

vagrant:~$ exit

Ok, let's try it out in a new vagrant box!

$ cd build
$ vagrant init chef/debian-7.4
$ vagrant up && vagrant ssh
vagrant:~$ sudo dpkg -i /vagrant/demosite_0.1_amd64.deb

Does it work? Let's try start it and find out.

vagrant:~$ sudo /etc/init.d/demosite start
demosite started.

Looks good, let's see if it works.

vagrant:~$ curl localhost:8000 
...

Woohoo!

To automate all this one could use the Vagrant Shell Provisioner or something like Fabric.

Currently the server is not binding to an external network interface and would not be able to bind a privileged port as it is not running as a privileged user. This is OK as I intend to proxy it though nginx, this is quite simple to configure but it would be nice to manage this configuration via Puppet, which I hope to write about in my next post.

Running of the Bulls

tarn@tarnbarford.net (tarn) — Sun, 17 Aug 2014 19:54:00 +0000

Tarn Barford

I'm going to Scoopcamp for the hackathon again this year as I had a lot of fun last year. I'm joining the Henri-Nannen-Schule team who to want to make some news games. To help ensure I can help with their ideas and not get bogged down in boring technical details, I decided to have a look at what tools, resources, libraries and frameworks exist these days that might be useful.

This morning I decided I would build a game with the Phaser HTML5 game framework, Tiled map editor and Texture Packer (trail version) and post it on this blog. I decided the game would be about bull fighting as I was trying to animate a bull earlier and because I knew Inés wouldn't approve.

I um, used, some peoples art. I hope this is ok, here is where I found it:

The awesome Batman style collisions from Morkii
I found the Vega character spites in a forum
The map tiles are by Buch
The background is from the Phaser Examples project

It's a bit of a mess, but was fun. I think I'm better prepared and now looking forward to the hackathon.

Chess

tarn@tarnbarford.net (tarn) — Sun, 20 Apr 2014 19:54:00 +0000

Tarn Barford

This week's post is a playable chess engine that runs in modern browsers that have support for websockets. The engine is my fork of the chess-at-nite engine which is written in C++. The engine is compiled with clang to LLVM byte code and then to asmjs with Emscripten. The engine runs in a web worker and the UI is built with a minimal amount of Javascript that makes use of the excellent chess.js and chessboard.js libraries.

Game is Drawn

New Game

Black Pieces

White Pieces

Position

To get started I compiled Emscripten, and the Emscripten forks of LLVM and clang as Enscripten currently prefers to use the "fastcomp" LLVM backend which is not the version I had previously installed with brew. After configuring my Emscripten to use the new clang I could run emcc which is pretty much a drop-in replacement for gcc.

I found the chess-at-nite randomly by googling for chess engine source code. I would have liked to have used the GNU Chess engine, but I have looked at the source previously and did not think it was a great codebase to adapt. Fortunately the chess-at-nite source compiles with gcc and looked like a pretty nice codebase to hack on.

When I initially tried to compile the code with Enscripten I got a heap of 'error: reference to 'move' is ambiguous' errors. Unfortunately many of the source files use 'using namespace std;' and apparently the Emscripten std namespace defines move. After updating all the usages to only include the referenced parts of the std namespace, it compiled and even printed the menu in the Emscripten generated web page! And then crashed my browser.

At this point I thought it would be awesome if I could compile it as a worker and have it automagically use postMessage and onmessage for stdin/stdout and console.error for stderr. I was disappointed, but this is a compiler option I would consider working on!

I also couldn't get all of the Worker API working for some reason I have not investigated yet. However, I could get enough of it working to hack together this post. I created a new entry point in the chess-at-nite code that exports a C style function suitable to be called using the Emscripten Worker API, this partly worked. I was unable to get the Worker API calling the worker, but noticed I could create and call the worker with normal Javascript, provided I posted and handled the internal message formats used by the Worker API.

For this post the worker takes a FEN game state and returns a new FEN string with an updated game state. Luckily chess.js and chessboard.js support FEN strings which made wiring up the UI quite straight forward. As FEN strings do not provide enough information about previous moves to detect threefold repetitions, the engine is quite susceptible to threefold repetition draws, as can be seen when it plays itself.

Ideally I would have liked to have the Chess engine running in the worker, comunicating with the xboard protocol using "standard" stdio over the web worker onmessage/postMessage interface. However I am pretty happy with the results and may have found a project I want to contribute too.

Haste Web Workers

tarn@tarnbarford.net (tarn) — Sun, 13 Apr 2014 19:54:00 +0000

Tarn Barford

This week I learnt of Haste, a Haskell to Javascript compiler intended to be used to build web applications. I have been trying to learn Haskell as a side project for a while and have previously looked into options for compiling it to Javascript.

I was very happy that after a little playing around I was able to get an incomplete Haskell chess implementation compiling to Javascript and playing itself in a browser.

Getting it running was just a matter of including the modules, generating some moves and writing the state to an element. However as it does some computation calculating moves it also blocked the page. This can be fixed by running the code in a web worker, I'd used web workers with ClojureScript last week to do the calculations for my Swipe Keyboard and figured I could probably use them for this too.

There does not appear to be any support for web workers in Haste, but it does support a foreign function interface which allows support to be added.

To support creating and using Worker objects I created three functions in Javascript. One to create a Worker object, one to bind an onmessage function and one to call the workers postMessage function. The A function is provided by Haste to call callbacks and provide arguments.

function make_worker(url) {
    return new Worker(url);
}

function on_message(worker, callback) {
    worker.onmessage = function (e) {
        A(callback, [[0, e.data], 0]);
    }
}

function send_message(worker, message) {
    worker.postMessage(message);
}

In Haskell I create a worker, send it a message, wait for messages from the worker and write them to an element on the page.

{-# LANGUAGE ForeignFunctionInterface #-}

module Main where
import Haste

newtype Worker = Worker JSAny

foreign import ccall make_worker :: JSString -> IO Worker
foreign import ccall on_message :: Worker -> JSFun (JSString -> IO ()) -> IO ()
foreign import ccall send_message :: Worker -> JSString -> IO ()

message :: Worker -> (JSString -> IO ()) -> IO ()
message worker f = on_message worker $ mkCallback f

handle_message :: Elem -> JSString -> IO ()
handle_message el s = setProp el "innerHTML" s'
    where Just s' = fromJSString s

main = do
    Just el <- elemById "output"
    worker <- make_worker (toJSString "worker.js")
    message worker (\s -> handle_message el s)
    send_message worker $ toJSString "Hello worker!"

Worker scripts run in an environment where an onmessage event can be bound and a postMessage function is provided. I created some more wrapper functions to use these in Haskell.

function on_message(callback) {
    self.onmessage = function (e) {
        A(callback, [[0, e.data], 0]);
    }
}

function send_message(message) {
    postMessage(message);
}

The Haskell web worker script sends a message once it has loaded and adds a prefix to every message it receives and sends the message back.

{-# LANGUAGE ForeignFunctionInterface #-}

module Main where
import Haste

foreign import ccall on_message :: JSFun (JSString -> IO ()) -> IO ()
foreign import ccall send_message :: JSString -> IO ()

message :: (JSString -> IO ()) -> IO ()
message f = on_message $ mkCallback f

response :: JSString ->  JSString
response s = toJSString $ "Worker recieved, " ++ s'
    where Just s' = fromJSString s

main = do
    send_message $ toJSString "Worker: Hello"
    message (\x -> send_message (response x))

I didn't need all this for the Chess worker as it just posts updated game stages and doesn't receive any messages.

I still don't know Haskell well enough to really know what I'm doing, but hopefully Haste will encourage me to engage with more Haskell code.

I've pushed all the code for this post to Github.

Swipe Keyboard

tarn@tarnbarford.net (tarn) — Sun, 06 Apr 2014 19:54:00 +0000

Tarn Barford

When I first tried a Swype keyboard I was impressed how effective it was. Even though I don't use the feature on my phone I was interested in how it could be built, so I implemented this otherwise useless swipe-able keyboard below. It probably doesn't work on mobile devices, but works on modern browsers with mouse pointers (although I've only really tried Chrome and Firefox).

Loading

I initially tried to solve this using the technique Peter Norvig famously uses in his spell checker. He takes a sequence of characters and generates a set of word candidates by adding, removing and swapping characters in the original sequence, the generated candidates are removed if they are not found a dictionary. This can work but to be effective too many combinations need to be generated.

If the dictionary is indexed into a trie the number of combinations generated can be reduced significantly by traversing the trie and only generating valid letter combinations. This is a pretty bare implementation of that, it requires:

The first and last characters of the initial sequence are used
Intermediate characters in the initial sequence can be repeated or discarded
No characters are added or swapped

Basically, if you swipe through all the characters in a word in order, then the word will be found if it is in the index regardless how many characters are swiped in between. It is surprisingly quick and effective.

This implementation uses these 10000 words, I intended to use digital books but never got around to it as these words demonstrate the concept well enough.

This is the first thing I've written in ClojureScript or Clojure, so my code my vary from non-idiomatic to shamblolic. I initially used a zipper to build the trie with immutable data structures, but found the indexing took to long with my zipper implementation so I switched to using native Javascript maps.

I found that core.async library is really awesome, the Google closure library and compiler integration with Leiningen the cljsbuild plug-in to be impressive. My main pains were the slow JVM start-up time, the advanced closure compiler build of the web worker script fails silently when run (but the main script works fine when compiled with the advanced compiler), and at times I felt some compile time type checking would be nice.

I would like to extend this experiment to index the word occurrence counts and proceeding word counts in original text and rank the found words as most likely. Support casing, umlauts, special characters, spelling correction and compound words in the indexing and lookup. I think a live lookup while swiping would also be possible.

Overall this was fun, turned out OK I think, and was a great learning experience.

Mozilla Persona

tarn@tarnbarford.net (tarn) — Sat, 10 Aug 2013 19:54:00 +0000

Tarn Barford

I added the ability to sign into this site to comment with Mozilla Persona. Not that I'm expecting people to use it, I've only had a handful of genuine comments since I started the site some years ago. I was just interested in trying it out, which is why I have this site.

It was pretty straight forward as I already only support the concept of signing in with 3rd party authentication providers such at OpenID, Google and Twitter. Adding Persona was just adding another provider, I didn't need to change how the authentication system works. Here is the commit.

A difference from the other providers is that I had to use Javascript. It's not that I don't like Javascript, many posts here have running Javascript in the content. It's just over the several re-writes of this site in different languages I've included less Javascript in the site itself. There was none on this site, I liked it that way.

Another thing I liked is OpenID. I understand it had usability and adoption issues. With OpenID though, I could choose who manages my identity or I could manage it myself. Apparently an email provider (in my case, me) can implement the Persona (BrowswerID) protocol and not depend on login.persona.org. Perhaps that is something I will look into after posting this.

Finally a note on adoption; I was having dinner with Özlem last night and tried to explain Persona, but was unable to convince her to even try it to sign into this site. To be fair it's only a sample size of one and I think Özlem was hoping dinner conversation wouldn't be about online identity management.

Prague to Berlin

tarn@tarnbarford.net (tarn) — Thu, 08 Aug 2013 19:54:00 +0000

Tarn Barford

On July 26-28 I joined 32 other entrepreneurs, VCs and engineers to cycle from Prague to Berlin and raise money for dotHIV.

It all started on Thursday a week before; our communications manager, Agnes, emailed me a link to the techbikers.de site and made some joke about how I should do it on my hipster bike. Very funny, yet another hipster bike joke. However I thought it sounded pretty exciting, and it would be fun to call Agnes's joke. "Very funny, I'm going to do it." I emailed back.

I got in touch with Lea who informed me a space had just become available as another participant had withdrawn. The organizers encouraged riders to commit to raising €750, for their chosen charity dotHIV. This sounded like quite a bit in a week, so I asked our CEO if Retresco could get me started with a few hundred Euro. He agreed to contribute €300 and my fund-raising drive was on its way.

The weekend before the ride was lovely in Berlin so my entire training program consisted of cycling about 100km in total to some lovely lakes around Berlin. Certainly a lot more pleasant than training for a charity run back in Melbourne.

In the days leading up to the ride I started worrying the maybe taking my beloved bike might not be such a great idea if there were any serious mountains. It's a 1987 Koga Miyata frame fitted with a single 52/17 free wheel gear, perfect for cycling around Berlin. When I emailed the group the first response was "I really can NOT recommend to use a single speed bike" but then there was another saying that there would be a couple of spare bikes if I really struggled. Of course I decided to take my bike, determined to ride her all the way back to Berlin.

Following the advise given to the riders I visited a cycling shop and purchased padded cycling shorts, a shirt, gloves, a water bottle holder, sunglasses and a helmet, which was mandatory. I'd previously promised I'd never wear Lycra cycling shorts, but this seemed an appropriate exception. I left €250 lighter, but feeling like a bit of a pro.

I met up with most of the organizers and cyclists who took the organized bus from Berlin to the start of our ride in Prague. I was a little surprised that only a handful of people including myself took their own bikes, it was comforting to find that most people, like me, had no experience cycling such distances.

We arrived in Prague to check-in and have diner at our place for the night, a Boatel called The Admiral. On my dinner table amongst others was Casey, a fellow Australian who runs the Google office in France and Eugine, an early-stage investor at Mangrove Capital Partners. Dinner and conversations were good.

As we were in Prague, most of us decided to checkout the city. Our group diminished to four as sensible people headed back to the Boatel to get some sleep. We decided we should a least checkout the famous and trashy Karlovy Lázně night club.

Over breakfast we were given our days route, timings, break stops etc. Most of us seemed a bit freaked out, but mostly excited by the 100km+ days schedule.

The ride out of Prague involved lots of very slow cycling, cheering, cobble stones, tram tracks and quite a steep hill. As we reached the top of the hill one of the guides cycled up to me and told me that if I could do this hill on my single gear, I would probably be alright on all the accents we would encounter. This was very good to hear and he was right as never ended up using one of the spare bikes.

I wasn't the only one who was surprised that a lunch of dried fruits and nuts seemed a bit lean for a day we planned to cycle more than 100km. I eventually worked out that it was the morning break and we'd only cycled 30km not 60km.

That break was good though as Bjorn produced boxes of Wonderpots yoghurt and fruit tubs he'd kept frozen for a day and a half on dried ice in our support car. I think he found a few fans of the Yoghurt stores he founded in Berlin. He was also one of several riders I got along with really well and hope to meet up with again.

The basic pattern for a days cycling was leave 8-8:30. Morning and afternoon breaks with mixed nuts, fruits, drinks etc. Lunch was a buffet cold meats, salads and bread. There were also periodic breaks every 10-30km for water re-fills and regrouping.

Due to some flooding at the hotel we were originally booked at for the second night, it was rebooked at another hotel. This resulted in the first day being a bit shorter and the second day being longer. As it was very hot, the organizers gave us the option to cycle the initial distance planned for the second day and get driven the remaining distance. Most people including me decided we wanted to ride the additional 40km making it am epic 172km day.

Despite the long distance and the heat, we finished day two with a sprint over the last couple of kilometres to the hotel. I hammered up to top four, but didn't realise how far it was and watched almost the entire field overtake me in the last kilometre or so.

I expected the first day to be OK, then wake up sore and suffer through the last two. However the opposite seems to be true, on the second and third days I felt I was getting stronger. While I was just surviving on the first day, on the second and third I was using extra energy to chase down group ahead and move to the front of groups to help carry them. This felt good, because I spent the first day and a half sticking to the back wheel of the stronger riders.

It was fun on the third morning when we got the route for the day back to Berlin, 109km with no elevation gain, our smallest day. Unlike the first day there was a sense of confidence, this was our Champs-Élysées, we'd done 172km the day before and survived. This would be a cakewalk back to Berlin.

I was incredibly impressed with the group as a whole. Everyone made it back, there were no major crashes or any issues I was aware of. Over the three days of cycling I think I spoke to everyone for at least a bit and everyone seemed to be really enjoying themselves.

I'm very thankful to everyone who helped organize the event and everyone who donated to dotHIV. I still have €100 to reach my target, so please help get me over the line.

PGP/GPG

tarn@tarnbarford.net (tarn) — Tue, 06 Aug 2013 19:54:00 +0000

Tarn Barford

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I just learnt how to use PGP/GPG to sign messages, encrypt
message for people and decrypt messages encrypted for me!

If you haven't learnt about PGP yet, maybe you should. It's fun,
easy and very useful if you want some Pretty Good Privacy.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (Darwin)

iQIcBAEBAgAGBQJSATOgAAoJEAKnAXIHeOzVmz0P/RR1jVIJTtmYnj++19/nIv00
oosR0Ib/4E7klc+8Mnzc5CdQ2NFQLqX87+sSRd+tJZScuP6NVLzK95qKA+Ci5FHY
lWZ1sMeNun2VTuVmUmPQnoeNGl4DBM2O9O2mftNXkrtUJrmZLH4vlRoQb6eNcZF+
82oFay2ZZHBq5+cnw+TXFFohGUeqFcUTUs+ThrqUU6FlKScIMiUzskRwIA3/q7in
2qEhAE+0B68KWqRAgq2xvR9bemcfRTmAeGzY+pgEJTIYw1qtgOh3EN5JXXj3tzeC
NR0STttEuSm7SEIk/hUc5eoM/rAcHF8HGrfYoKh2ZckiivfUV+yERmNZs4E8S/lr
fMAwiDSioy0yXRB/ceZL3jv4XI5Oy//Gm35u9VU5ZPHtW2FcsLW6n1D/IhaieMLm
9LApWNxm3uxz2na+Q+YyM4YDJMqeUdYnqCe4XTtclUNvDOY+ZK7mABnaZkYHtHir
00Hp+2Bl/vpSZ0tzSedKvMnVJNYopNMK54iuKjgBT7GuMMirSiWtXUALer33cRII
llbC5Gf3lVf6/SsyX+/SuaiCrZRIGou6fJuIacl3VM3/kIoPjwy5cpr1fzxPMtg0
CYwq8MjuF0Uzyz8rHxskVQbfW0nWKPcK9JqCw89fLM/nv+LpuNB2o7mUzqvxuizi
WqFWSMBg/bWpb5lB9nx/
=6hUP
-----END PGP SIGNATURE-----

Setting up a pair programming environment on Amazon EC2

tarn@tarnbarford.net (tarn) — Sun, 01 Apr 2012 09:53:00 +0000

Tarn Barford

Arch Linux is a great distribution for a programming environment and there is an Arch AMI for Amazon EC2. I choose to run it on a Large EC2 instance in Tokyo, I downloaded the generated key-pair and copied its public domain name.

Connecting to the instance

The EC2 security group needs to be setup to allow inbound requests to port 22 for SSH.

The key-pair is used to SSH into the instance as root. The permissions on the key file must be set so only the owner can read it. I also move keys to my ~/.ssh directory.

$ mv ~/Downloads/mykey.pem ~/.ssh/
$ chmod 400 ~/.ssh/mykey.pem

SSH into the public name using the key.

$ ssh -i ~/.ssh/mykey.pem root@[ec2-instance-name]
[root@ip-xx-xxx-xx-xx ~]$

Update the system

Firstly the system should be updated with the Arch Linux package management tool.

[root@ec2]$ pacman -Syyu

It asked me something about a dependency conflict, I said yes. Things seem to be fine.

Creating users

adduser is an interactive tool which prompts for new user information and provides defaults where possible. A username and the defaults are fine for now. User passwords can be set with passwd

Great, users for everyone.

Allowing users remote access

The SSH configuration doesn't allow for plain text passwords sent through the tunnel. This can be allowed by updating /etc/ssh/sshd_config and setting PasswordAuthentication to yes. But SSH keys are better, use them.

The SSH daemon checks for the key in ~/.ssh/authorized_keys. Put the users public key there.

[root@ec2]$ mkdir /home/username/.ssh
[root@ec2]$ mv pub-key /home/username/.ssh/authorized_keys

The user can now login, the -i flag can be used if the default key is not used.

$ ssh username@[ec2-instance-name]

All users are not equal

Privileged users should be allowed to perform actions as the root user. This can be done by giving permission to the wheel group and then adding users to the group.

These rules are in /etc/sudoers, because the file is important there is a tool called visudo which checks for syntax errors before saving.

[root@ec2]$ visudo

The line that allows the wheel group to perform actions as the root needs to be uncommented.

%wheel ALL=(ALL) ALL

We can then add users we trust to the wheel group.

[root@ec2]$ gpasswd username wheel

The sudo package needs to be installed at sometime.

[root@ec2]$ pacman -S sudo

Prevent root login

Now a user can connect and perform actions as root, we can prevent logging in remotely as root. To do this we update /etc/ssh/sshd_config and set PermitRootLogin to no. We need to restart the SSH process for this to take effect.

[tarn@ec2]$ rc.d restart sshd

Install some development tools system wide

We will need heaps more, but here are some to get us started.

[tarn@ec2]$ sudo pacman -S vim git screen gcc erlang \
               python mono openjdk6 nodejs v8 patch \
               curl zlib readline libxml2 libxslt \
               autoconf automake diffutils make \
               libtool bison subversion

Installing packages requires root access as the packages install system wide, only users with permission to sudo will be permitted to do it.

Multiuser Screen

To share a screen process, one user starts screen passing a socket name

[person1@ec2]$ screen -S hack-session

They then must enable multiuser by invoking screen with CTRL-a and typing

:multiuser on

Finally they must explictly allow a user, again by invoke screen with CTRL-a.

:acladd person2

The user person2 can now attach to the screen session

[person2@ec2]$ screen -x person1/hack-session

You can then even split your screen CTRL-S and share two shells, perhaps Vim and a REPL? Awesome.

Note: When you allow someone to connect to your screen process you are effectively giving them permission to run as you. Unless you want to remotely pair on system administration, create an un-privleged user and use that to share a screen process.

Build it and they will come

With a shared remote programming environment I'll be able to pair with people I wouldn't otherwise. I'm nervious about pairing with programmers I know are awesome, but I also know it's a great way to learn, so lets bring it!

vim-slime

tarn@tarnbarford.net (tarn) — Mon, 26 Mar 2012 19:54:00 +0000

Tarn Barford

Today I found the awesomeness that is vim-slime, it's been an exciting day for me. Slime is the "The Superior Lisp Interaction Mode for Emacs", I can almost hear the emacs crowd laughing.

For those that use vim and haven't used Slime, vim-slime or something similar, this is why it's awesome:

Text can be sent from any process to the stdin of a gnu screen or tmux session. The process in this case is vim and the screen/tmux session is a terminal.

Screen is a really neat terminal multiplexer (you can run multiple terminals in a terminal window). The multiplexed shell processes are children of the screen process, which itself is not a child of the terminal window process. This means a screen process and its child processes keep running if you close the terminal window. Later you can re-connect to it, this is what makes vim-slime possible.

Here is an screen shot, on the left is me in gVim writing some awful Clojure [1]. On the right is a screen buffer in which I started a Clojure REPL. When I want to try run some code I can send any vim text selection to the REPL in a keystroke (or two).

It doesn't have to be a Clojure REPL either, we can send anything to a screen shell. We could run git commands, find, grep, sed, etc. Like with the Clojure REPL we can even interact with any terminal programs that use STDIN.

This concept can be taken even further, You can even connect to a tmux session over SSH and share a terminal or a terminal program like vim to do remote pairing!

Hopefully remote pairing is the topic of my next post as there are a couple geographically distant people I know who are keen to do some pair hacking. I stand to learn a lot!

[1] I learnt almost everything I know about Lisp from The Little Schemer. Great book.

UHC: Haskell to JavaScript

tarn@tarnbarford.net (tarn) — Sun, 05 Feb 2012 18:06:00 +0000

Tarn Barford

If all goes well some JavaScript included on this page should have generated a list of all the primes numbers between 1 and 100 and printed them below.

To generate these primes I wrote simple Primes module.

module Primes where

factors :: Int -> [Int]
factors n = [x | x <- [1..n], n `mod` x == 0]

prime :: Int -> Bool
prime n = factors n == [1,n]

primes :: Int -> [Int]
primes n = [x | x <- [2..n], prime x]

And a Main module to generate some primes and print them.

module Main where
import Primes

main = print $ primes 100

This is basic Haskell which can be compiled by GHC.

$ ghc Main.hs
[1 of 2] Compiling Primes           ( Primes.hs, Primes.o )
[2 of 2] Compiling Main             ( Main.hs, Main.o )
Linking Main ...
$ ./Main
[2,3,5,7,11,13,17,19,23,29,31,37,41,43,47,53,59,61,67,71,73,79,83,89,97]

It can also be compiled by UHC which I found interesting because UHC can compile to a JavaScript backend.

$ ls
Main.hs  Primes.hs
$ uhc -tjs Main.hs
[1/2] Compiling Haskell      Primes      (Primes.hs)
[2/2] Compiling Haskell      Main        (Main.hs)
$ ls
Main.core  Main.hi  Main.hs  Main.html  Main.js
Main.mjs  Primes.core  Primes.hi  Primes.hs  Primes.mjs

The -tjs sets the target to JavaScript and uhc then generates JavaScript files, some intermediate files and a html page. UHC must be configured then built with the --enable-js flag for the JavaScript backend to work.

The html page references the two generated JavaScript files and quite a few framework/runtime/library JavaScript files. These are all in the source of this page.

The Primes program doesn't interact with the DOM at all, it just dumps a list out where it was called. This makes everything a lot simpler, but is not very useful. The uhc-javascript project implements typed Foreign function interfaces for browser JavaScript functions, the DOM and also some JavaScript libraries like jQuery and Backbone. This makes it possible to write an entire client side application in Haskell, although I'm not sure if it's a good idea.

I also don't know how complete or good UHC is as Haskell compiler. I checked it could compile a simple Chess program which I have been playing with thought it might be fun to try and get running in the browser for a post. Turns out I'm going to have to level up my Haskell a fair bit to get the UI stuff working.

Basic long polling chat with Manos de Mono

tarn@tarnbarford.net (tarn) — Fri, 22 Jul 2011 19:54:00 +0000

Tarn Barford

I fucking love the Manos do Mono manifesto. It has very grand ideals and a philosophy of simplicity which I really dig, it is also excellently illustrated with pictures of cats. I've wanted try it since listening to Jackson Harper talking about it on a Herding Code podcast last year.

Installing Mono

I'm not sure if I needed to upgrade from version 2.6.7 to use Manos, which is in included with the Linux Mint distribution I use, but I knew I would need a newer version to for some other stuff I want to do. So I built and installed the latest Mono (2.11) from source, which was pretty straight forward.

$ git clone git://github.com/mono/mono.git
$ cd mono
$ ./autogen.sh --prefix=/usr/local
**Error**: You must have 'libtool' installed to compile Mono.

Ok, install libtool.

$ sudo apt-get install libtool
$ ./autogen.sh --prefix=/usr/local
..
configure: error: msgfmt not found. You need to install the 'gettext' package, or pass --enable-nls=no to configure.

I think I have gettext, oh well.

$ ./autogen.sh --prefix=/usr/local --enable-nls=no
$ make
$ make install

And make sure the new Mono works, awesome.

$ mono --version
Mono JIT compiler version 2.11 (master/fbff787 Fri Jul 15 17:54:49 EST 2011)
Copyright (C) 2002-2011 Novell, Inc, Xamarin, Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            Included Boehm (with typed GC and Parallel Mark)

Installing Manos

Manos was easy to build and install from source.

$ git clone https://jacksonh@github.com/jacksonh/manos.git
$ ./autogen.sh
$ make
$ sudo make install

Getting started

Manos has a command line tool like rails to create, build and serve websites.

$ manos
manos usage is: manos [command] [options]

  -h, -?, --help
      --init, -i
      --server, -s
      --docs, -d
      --build, -b
      --show-environment, --se

      --run, -r=VALUE

The --init option creates a new application.

$ manos --init ManosChat
initing: ManosChat
$ cd ManosChat/ && ls
ManosChat.cs  StaticContentModule.cs

Which creates a new folder with two files. The initial app is setup to serve static content from a content folder.

We'll replace those files with a ridiculously simple long polling "chat" application which will consist of three files; a C# file, a HTML page and a JavaScript file.

The Manos application has four routes, one for the HTML page, the JavaScript file and routes to post messages and wait for updates. The routing metadata hopefully looks pretty familiar.

The Wait method simply adds the request context to a list of waiting requests and leaves the browser requesting it waiting while the server moves on to handling the next request.
The Send method pulls the message out of the request context and then loops through all the waiting connections and forwards the message on.
The Home and Script methods serve content by simply reading it off disk. You read the files at start up and serve the from memory, but then I'd have to restart the server to change the contents! :)

That is it!

using Manos;
using System;
using System.IO;
using System.Threading;
using System.Collections.Generic;

namespace ManosChat {

    public class ManosChat : ManosApp {

        public List<IManosContext> _waiting;

        public ManosChat ()
        {
            _waiting = new List<IManosContext>();
        }

        [Post ("/send")]
        public void Send(IManosContext context)
        {
            var message = context.Request.PostData["message"];

            foreach(var listener in _waiting) {
                try {
                    listener.Response.End(message);
                } catch(Exception) {
                }
            }
            _waiting.Clear();

            context.Response.End();
        }

        [Post ("/wait")]
        public void Wait(IManosContext context)
        {
            _waiting.Add(context);
        }

        [Get("/")]
        public void Home(IManosContext context)
        {
            var content = File.ReadAllText("index.html");
            context.Response.End(content);
        }

        [Get("/app.js")]
        public void Script(IManosContext context)
        {
            var content = File.ReadAllText("app.js");
            context.Response.End(content);
        }

    }
}

My HTML looks something like this.

<!DOCTYPE HTML>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Manos Chat</title>
    <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>
    <script type="text/javascript" src="/app.js"></script>
</head>
<body>
    <h1>Chat</h1>
    <input type="text" id="message" name="message" />
    <input type="submit" id="send-message" name="Send" />
    <ul id="messages">
    </ul>
</body>
</html>

And my JavaScript something like this.

$(function() {

    var showMessage = function(message) {
        $('#messages').prepend($('<li/>').append(message));
        };

    var getMessages = function() {
        $.ajax({
              type: 'POST',
              url: '/wait',
              data: 'data=none',
              success: function(data) {
                showMessage(data);
                getMessages(); }
              });
        };

    var sendMessage = function(message) {
         $.ajax({
              type: 'POST',
              url: '/send',
              data: 'message=' +  message,
              success: function(data) {
                }
              });
    }

    $('#send-message').click(function(e) {
        e.preventDefault();
            var m = $('#message').val();
            sendMessage(m);
        });

    getMessages();

});

Building and Running

The web application can be built and served using the Manos command-line tool

$ manos --build
$ manos --server
Running ManosChat.ManosChat on port 8080.

Now we point some browser windows at the server and start chatting.

Some Notes

The chat browser windows are "Waiting for localhost" even though the page has fully loaded. This is because we are not responding to the request until we have a message, this is known as long polling. A common way to avoid this is to make the request from inside an iframe. And yes, it is possible for clients to miss messages between requests to wait in this chat application. Also requests timeouts are not handled correctly.

The reason we can access the list of clients without thread synchronization mechanisms is because all our requests a called on the same thread. This is great for this example, but it also means that if you block the handler thread (by making database query or something) then all other requests will have to wait. This just means you need to do any time consuming work or IO on another thread. It may be worth noting that request are queued in another thread, so the server isn't blocking while we are handling a request.

I hope to take this a little further and add Riak storage and maybe fix some of the obvious short failings of this example. Finally, props to Jackson Harper and everyone who has contributed to Manos and Mono, you are all awesome.

Play by Play with Zed Shaw

tarn@tarnbarford.net (tarn) — Mon, 11 Jul 2011 19:54:00 +0000

Tarn Barford

I watched the Zed Shaw episode from the Peepcode "Play by Play" series last night. Each episode in the series follows a different gun programmer doing their stuff solving a problem. Like "Coders at work" it has some interesting insights into how people go about it.

I became interested in Zed after reading his infamous post Rails is a Ghetto, which ironically warns people against listening to people like himself. I also enjoyed this entertaining video of Zed talking about ACL's. More recently Zed has published "Learn Python the Hard Way" which looks like a great introduction to programming. Oh, and he wrote the Mongrel web server.

Whilst didn't learn a lot technically from the screen cast, I did learn I should use R for statistics and plotting, that Fossil is a wrapper around various source control systems which adds issue tracking, wiki and such things, and a few other little tips.

What I found really interesting is the tools he used and how he went about it, mostly because I saw a lot of similarities with how I like to code. Thankfully I was also reminded that I should take breaks more often.

The tools he used are Screen (which I've previously blogged about), Vim, Python, R, gcc, Fossil and Vimperator. I have used all those tools except Fossil and use all but R and gcc regularly.

I also liked the way he just got in and started hacking on the problem, saying that once he understood the problem he would typically throw the code away and start again and warned against trying to refactor your way out from your inital attemp. This advice really resonates with me, of course you write code better and cleaner when you've written it before and understand the problem.

I also really liked his take on testing which was to use a variety of testing methods, including building out test clients. He didn't sound like he would waste any time trying to unit-test something that was difficult to unit test. A refreshing, no dogma, take on testing.

He also talked about how he used heuristics to improve his coding efficiency and effectiveness (such as the code metrics tools he was building in the screencast). I would like to do more of this myself as it makes more sense to me than techniques such as the pomodoro.

There is no doubt I have a way to go before getting anywhere near the productivity, skill and experience of Zed but I did find it encouraging the direction I am heading is a proven one. I just need to keep enjoying building stuff, and maybe take a break more often.

This rant is no more.

Actor Model

tarn@tarnbarford.net (tarn) — Fri, 01 Jul 2011 19:54:00 +0000

Tarn Barford

A couple of months back Andrew Bienert organised a Melbourne Alt.NET meeting on functional programming.

Andrew Browne and myself put our hands up to do mini talks on Scala and Erlang and Andrew Bienert was keen to talk about F#.

A few days before the meeting was originally scheduled Andrew Browne and I caught up on Skype to work out if we could somehow make our talks complimentary, either by each talking about different concepts or showing the same concepts in Scala and Erlang. We decided it would be best to show the same concepts in both Languages.

We had previously also talked over beers about the Actor model in Erlang and Scala and wanted to spend some time on that. I initially wanted to do the exercise proposed by Joe Armstrong from his book Programming Erlang.

Write a ring benchmark. Create N processes in a ring. Send a message round the ring M times. So that a total of N * M messages get sent. Time how long this takes for different values of N and M.

It is a fantastic exercise but we decided it was overly complex for a short presentation. We decided on a much simpler example with three actors; a Hello actor, a World actor and a Print actor.

The Hello actor is sent a message and adds "hello" to the message then sends it to the world actor which adds "world" to the message and sends it to the print actor which prints the messages.

Andrew Browne implemented it in Scala, I implemented it in Erlang and Andrew Bienert joined the party and implemented a version in F#.

I found the similarities and the differences in the implementations very interesting. The meeting was eventually held and the presentation generated some interest amongst the attendees, so I thought it would be worthwhile posting about here.

Cube

tarn@tarnbarford.net (tarn) — Thu, 30 Jun 2011 19:54:00 +0000

Tarn Barford

WebGL has recently seen extra attention caused by banter between the major browser players, which seems to becoming pretty common.

Microsoft called it harmful, a developer from Mozilla raised an issue on Microsoft's connect showing the same risk in Silverlight 5 and Opera and Google bloggers weighed in.

This week it seems to be more about enterprise.

I think WebGL is awesome and I am glad people are pushing the boundaries and although I'm not really into computer gaming, I would like see the graphic technology as part of the web medium.

To get a taste I just wanted to try doing something simple like a rotating cube. Spoiled as we are there is a MDN Getting Started with WebGL guide that walks through doing just that.

I roughly followed along writing in CoffeeScript and found a lot about what happens when you don't get it right. There is quite a lot you need to do, correctly, just to draw a shape on screen programming directly against the API.

After some playing I had the cube you should see above, if the browser supports WebGl anyway. Someone has written IEWebGL plugin, but I haven't tried it. I posted the source here.

Tile Land

tarn@tarnbarford.net (tarn) — Sat, 18 Jun 2011 19:54:00 +0000

Tarn Barford

I was originally trying build an effect that pans and rotates around an infinite world of repeating background images. To do this I was using an image the size of the viewing area, of course to panning around will often require drawing parts of several images. If we are panning along the x and y axis we will need to display, at most, four images.

When the camera reaches one of the edges, it jumps back an entire image size along the same axis. As the images are the same, moving any direction by an entire image size results in exactly the view.

Adding rotating made everything a little more complicated. Now up to six images can be visible at times.

Playing around I worked out that with nine tiles I could make sure all the tiles that could be visible were drawn. This also made it possible handle the infinite panning in a very similar way as before.

Another complication was I wanted the world to rotate around the view, not around the worlds origin. I posted about my struggles with that earlier.

I got this working and had achieved what I was trying to do, but I wanted to take it a little further. Rather than just infinite repeating images, I wanted to make an infinite world. All I had to do was keep track of where the view was in this infinite world and instead of drawing the same image everywhere, I would choose each image to draw by where it was in world.

It was after I got this working that I realised adding the ability to scale the view of this tiled world would be pretty straight forward. If the view was zoomed out 100% then the view would be the size of exactly four images or tiles. Adding another tile of padding around those four tiles would be enough to ensure all the tiles possible in the view would be drawn.

Zooming out 200% would mean the view was the size of nine tiles, adding another tile of padding would again ensure all the tiles in the view would be displayed. Clearly there is a pattern here.

Putting all this together and adding some controls and debugging information I came up with this. Note there is no limit on how far you can zoom or pan, but zooming out far enough will eventually bring any browser to it knees.

Move left [h]/right [l] and up [k]/down [j]. Rotate clockwise [d] and anti-clockwise [f]. Zoom in [z] and out [x]

I have some ideas about what I would like to do with this from here.

A game of some sort, possibly similar to the Desktop Racer game I wrote for DevStar 2008.
Use some free map tiles
An infinite world of images from Flickr
Make it 3D

If you have some ideas let me know or grab the source and start hacking. I also published the not-so-pretty source for the diagrams in this post.

Asteroids

tarn@tarnbarford.net (tarn) — Mon, 13 Jun 2011 19:54:00 +0000

Tarn Barford

As threatened, Asteroids!

Update: You don't need coins, you can press [f] to start. Apparently I should have made that more clear.

Sorry no sounds, shields or extra weapons, there are already other good asteroid games for the HTML5 canvas. I wrote this to try CoffeeScript, which has been quite fun. I posted the source as a gist.

I have some ideas about how I could make it multiplayer which I might consider trying, of course that has been done too.

Canvas Transforms

tarn@tarnbarford.net (tarn) — Sat, 04 Jun 2011 19:54:00 +0000

Tarn Barford

I've been playing with HTML canvas transforms and have had to brush up on how transforms work. It's really just matrices math; eigenvalues, dot products, triple products etc. I should know this stuff pretty well as it was extensively covered in my Engineering undergrad studies and I've previously played with them to do 2D and 3D computer graphics.

However when I have the choice between understanding the math and just typing a few lines and seeing the effect, I often just do the latter. I know, I really need to get past that.

I seemed to be able to get transforms to do what I wanted them to do, but until I got them right I was finding they often didn't do what I expected they would.

So I decided I would explore it a bit using the transforms themselves, but I'm not sure if I've cleared up my understanding. I probably should have just studied the math and written an asteroids game.

Rotate on axis then centre

The example transforms I looked at was rotating a square drawn from (0,0) to (50,50) about its centre, then moving it to (50,50).

This is basically a series of transforms, where the order is important. If this makes perfect sense you can probably just check out the animations and hopefully leave a comment explaining what is going on more clearly than I have.

translate(50,50)
rotate(Math.PI/4)
translate(-25,-25)
draw(..)

My brain thought this looked a bit backward, and it wasn't helped when I tried to animate applying the transforms one by one. Sure it worked out, but not as I expected it too.

Click to start animation

I noticed this would look as I expected if I animated the transforms in reverse order while still applying them in the correct order.

Click to start animation

The previous animations illustrate what was bothering me, but they are not exactly what I wanted to show in the first place. The transforms apply to the canvas and the square is drawn at (0,0) after the transforms are applied.

To show that we need to invert everything (kind of, more on that in another post maybe) and move the axis instead.

Click to start animation

You would think by now I would have started to understand all this but I haven't really. What is more annoying is I wanted to make a demo in which a user could add arbitrary transforms one at time, but the only way I could animate that nicely is if I asked for the transforms in reverse order. Perhaps that is the way to think about it.

Anyway I suck, hopefully a commenter can explain this more clearly than I have. Until then I hope this helps you grok how transforms work, but the real lesson here is know ye' math. Stay at school kids.

Fridge Magnets and the GAE Channel API

tarn@tarnbarford.net (tarn) — Sun, 15 May 2011 22:24:00 +0000

Tarn Barford

Fridge magnets have been my playground web application while learning Erlang. The basic idea is some letters which you can drag around to form words or whatever. It is made a little more interesting as users all use the same set of letters, and so users see letters being moved by other users.

The application itself isn't particularly interesting, there has been a Flash version of this for a long time. Of course there are a couple of things I find interesting about building such an application.

Web clients must be updated when letters are moved by other players
The server must signal the listening clients and maintain letter positions

Web browsers have traditionally been geared to making requests and receiving responses. When there is the need to go beyond that to more persistent connections there are several options (and combinations of these options)

Have the web client make periodic requests
Comet/Long-polling
Browser plug-ins with socket support like Flash
The emerging WebSockets standards

At some point since I last used App Engine, Google have added a Channel API for persistent connections. The App Engine and Python are quite fun to play with, so I decided to hack together a version of Fridge Magnets using the API.

As Google serves the JavaScript and the handlers for the persistent connection they could use any of the techniques. It appears they are only currently doing long-polling.

The app is at gae-letters.appspot.com and the source on bitbucket.

It's all a bit hacky, but it weighs in at less than 150 lines of Python so I'm not too worried. While it was fun and pretty easy, there are some pretty seriously limitations of App Engine when trying to build such applications.

No notification when clients disconnect
No synchronization methods I am aware of, which is probably why Google can scale these apps
I'm also seeing problems with browsers not seeing pushes after some time, but I haven't really looked into it

Although didn't use the Task Queue API which looks very useful, it seems traditional web frameworks suck for building this type of application. Alternatively Erlang and probably Node.js and Twisted etc are made for it.

Hopefully a robust, fast Erlang implementation will be the topic of my next post.

Old school, for fun

tarn@tarnbarford.net (tarn) — Sat, 23 Apr 2011 09:54:00 +0000

Tarn Barford

I used to love 4k demos and I learned to program in Pascal, C and Assembler trying to implement effects like this. Good times.

Oh, Screen. Where have you been?

tarn@tarnbarford.net (tarn) — Sat, 08 Jan 2011 11:02:00 +0000

Tarn Barford

So GNU Screen has been around for a while (initially released in 1987!), but it was only recently at Yow! Brisbane 2010 that @rioter kindly suggested I check it out.

For the last year or so I've used Linux exclusively for my personal computing where I play with a little Python, Ruby, Lisp, JavaScript and Erlang with Vim and a terminal. I do like freeing myself from my mouse. But there were a few area of my work flow that were bothering me:

I didn't like how I navigated up the terminal buffer to see what had happed, I could use Ctrl+PageUp/PageDown to avoid the using the mouse, but jumping by pages isn't always ideal.
I used the mouse to copy text from the terminal buffer. This constantly depressed me.
I never worked out how to run a console shell inside Vim. This resulted in using my windows manger to switch between Vim and console windows. It made copying and pasting between the two more difficult as I had to use the system clipboard.
I sometimes had couple of windows running processes like web servers on a separate desktop.

These were my biggest annoyances with my development environment on Linux. Enter Screen into my world to solve them all.

GNU Screen is a terminal multiplexer with windowing support. Programs created in Screen can persist after the shell that created them, it supports disconnecting, re-connecting and sharing programs.

The terminal buffers in Screen don't have scrollbars but they do support a copy mode (Ctrl+a [). This allows you to move up, down and around the window with Vim like keyboard navigation and copy text which can be pasted into other screen buffers (Ctrl+a ]). Yay, two of my problems solved!

Screen provides many functions for creating and switching between programs. A new shell be be created (Ctrl+a c) and named (Ctrl+a a). The connected programs can be listed and selected into the foreground with (Ctrl+a "). Now I can create shells and have them running in the background, then switch to them when I need them.

It supports splitting the current screen window horizontally and vertically (Ctrl+a S and Ctrl+a |). The focus can be switched between windows (Ctrl+a Tab). This allows me to have Vim on the top and a console on the bottom. Awesome, now I don't even need to run the console inside Vim itself.

There are many more use-cases for Screen but just these functions alone have resolved some of my major frustrations working with Vim in a Linux terminal.

Selenium on a Windows CI Server

tarn@tarnbarford.net (tarn) — Fri, 15 Oct 2010 22:24:00 +0000

Tarn Barford

I've been wanting to post about using Selenium and TeamCity to build and test .NET websites on Windows for a while. Unfortunately it hasn't been forthcoming as I only really use Windows/.NET at work these days.

I decided I would get my shit together today and write, what may be my last post on .NET for a while, in response to @DavidBurela's Developer Blog Banter topic "How do you test your applications?"

I'm going skip unit-testing (which do and try to get better at), BDD and JavaScript testing (which I would like to do and get better at) and focus on automated browser testing.

The Build Process

Watching automated test may be fun the first few runs, but that won't last long. Anyway if you want automated browsers tests, surely you want to automate the automated tests, right?

The build process is going to need to somehow automate the browser and run the site the browser is browsing. For this we use IIS which is scriptable with the PowerShell WebAdministration module (maybe I will write another post on .NET).

The problem with having the UI Test step in this iterative process is it very quickly starts taking ages. This sucks, I want feedback as soon as possible. TeamCity provides feedback after each stage, but we don't get our green light.

This could be resolved by putting the UI tests in a nightly build, but I think a staged build or build pipeline would be better.

The first stage compiles and unit tests and produces a deployable website as an artifact, this process is hopefully quite fast. If the first stage is successful the next stage in the pipeline is triggered which can be setup in the TeamCity administration web interface.

If you want to get awesome you can also start using Selenium Grid to distribute test agents or a cloud based solution like source labs

Tools

Our automated build process consists of a few tools.

TeamCity for our build management and CI server. After years of using CC.Net and a very brief and painful exposure to TFS, I love TeamCity. I also hear great thing about Hudson.

NUnit and included NUnit-Console. It does what I want a unit test framework and runner. I am yet to be convinced to changed.

MSBuild with the MSBuild Community Tasks for our build scripts. I think you have to have some sort of build script to do continuous integration when not using TFS. I would like to be using script or DSL instead, but msbuild is doing the job for now.

Selenium for our browsers automation tests.

Creating Tests

SeleniumIDE is a Firefox plug-in for recording and running tests.

SeleniumIDE is pretty slick and importantly it's a tool non-developer-testers-types can quickly get their head round and start producing some tests.

What we really want though, is to have our tests in code so we can run on them on build machine and version them in a repository.

Test Scripts

That should generated something like this.

This is pretty sweet, it generates test files you can pretty much just add to your project as NUnit tests. For me this mean automation tests can be run by the nunit-console just as our unit tests projects were.

The .NET client drivers are in the Selenium Core download.

I think it is worth considering generating C# test scripts may not necessarily be ideal.

Selenium

Selenium server run of the JVM - Yes Java - Suck it up, TeamCity does too :)

Any version of the standard JRE should be fine. If you've installed Java correctly it should be in your path and you should be able to type this in a console.

c:\>java -version

java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Client VM (build 16.0-b13, mixed mode, sharing)

This should output the Java version and build information. If so you have successfully installed a java runtime. Yay!

You need to download the SeleniumRC JAR file.

Once you have Java and the Selenium JAR you should be able to start the server from the command-line.

java -jar selenium-server.jar

Selenium as a Service

I wanted to run SeleniumRC (the same Jar) as a service. Console windows on servers, not so cool. Another, probably better, option is to have the build script itself instantiate the server and take down the server. I went with the service.

I found a solution but it didn't work on Windows 2008, luckily that lead me to the Non Sucking Service Manager.

The parameters I used were "-Xrs -jar [path/to/selenium-server.jar]"

Discussion

I went down the path of browsers automation because I was finding none of our other testing efforts, while valuable, were actually ensuring the website would work.

We were still have problems where a page or an entire site would return a HTTP 500 Server Error. Sometimes it was code and a unit-test would would ensure the fix, Sometimes it was IOC or ORM/Database configuration. What we needed was an end-to-end integration test. Browser automation provides these tests, and it helps.

I actually don't see the tests themselves as being particularly fragile. No doubt they can be, but I think it also encourages clean markup. If the hierarchy or class/id attributes change in the markup then UI tests are not the only thing the could break. Stylesheets, scripts and some server logic also depend on it.

While I have found the testing value of browser automation is good, I feel that mocking out the entire web framework (like gaeunit for appengine) or testing at HTTP level could achieve the same thing more easily and run quicker.

RxJs - Capture

tarn@tarnbarford.net (tarn) — Sun, 15 Aug 2010 22:24:00 +0000

Tarn Barford

This is the presentation for Composing Events with RxJS an Alt.Net Melbourne Lightning Talk.

An observable that yields for every capture click after the start button and before the stop button has been clicked.

// Setup observables
var buttonStart = $("#buttonStart").toObservable("click");
var buttonCapture = $("#buttonCapture").toObservable("click");
var buttonStop = $("#buttonStop").toObservable("click");

// Compose an event
var compose = buttonStart.SelectMany(function () {
    return buttonCapture.Select(function () {
        return "Captured Click";
        }).TakeUntil(buttonStop);
    });

// Subscribe to, and handle event
compose.Subscribe(function (result) {
    var val = $("<li/>").html(result);
    val.appendTo($("#results"));
});

Back (Zip)

RxJs - Merge

tarn@tarnbarford.net (tarn) — Sun, 15 Aug 2010 22:24:00 +0000

Tarn Barford

This is the presentation for Composing Events with RxJS an Alt.Net Melbourne Lightning Talk.

An observable that yields when either button is clicked

// Setup observables
var button1 = $("#button1").toObservable("click").Select(function() {
    return "Button 1" });

var button2 = $("#button2").toObservable("click").Select(function() {
    return "Button 2"; });

// Compose an event
var compose = button1.Merge(button2)

// Subscribe to, and handle event
compose.Subscribe(function (result) {
    var val = $("<li/>").html(result);
    val.appendTo($("#results"));
});

Previous (Index) Next (Zip)

RxJs - Zip

tarn@tarnbarford.net (tarn) — Sun, 15 Aug 2010 22:24:00 +0000

Tarn Barford

This is the presentation for Composing Events with RxJS an Alt.Net Melbourne Lightning Talk.

An observable that yields and event for consecutive clicks. This can be useful when the change since last event is required.

// Setup observables
var button1 = $("#button1").toObservable("click").Select(function() {
    return "Button 1" });

var button2 = $("#button2").toObservable("click").Select(function() {
    return "Button 2"; });

// Compose an event
var compose = button1.Merge(button2)

// Zip pairs of events
var compose = compose.Skip(1).Zip(compose, function (result1, result2) {
    return result1 + " previously " + result2; });

// Subscribe to, and handle event
compose.Subscribe(function (result) {
    var val = $("<li/>").html(result);
    val.appendTo($("#results"));
});

Previous (Merge) Next (Capture)

Composing events wit RxJs

tarn@tarnbarford.net (tarn) — Sun, 15 Aug 2010 20:24:00 +0000

Tarn Barford

This is the presentation for Composing Events with RxJS an Alt.Net Melbourne Lightning Talk.

*** Reactive Extensions ***

Reactive Extensions for JavaScript.
There is also Reactive Extensions for C#.
Reactive Extensions come from Microsoft Labs.

*** What is IEnumerable? ***

Provides Enumerator, allows lazy evaluation
You can implement an IEnumerable with yield and yield break.

*** What is Observable? ***

When you foreach(..) through IEnumerable you are waiting OnNext.
With Observable you are notified OnNext.

*** What is Rx ***

Like Linq adds a generic common API to query, Rx attempts to add a common API to compose generic events.
There are Select, Where, Join, Zip, Merge and many other constructs.
Basically keeps track of events and callbacks.
RxJs provides integration with several frameworks.
Also provides common error handling, which I won't show.

Next (lets see some code!)

The Mouse is Dead, Long Live the Keyboard

tarn@tarnbarford.net (tarn) — Wed, 16 Jun 2010 23:40:00 +0000

Tarn Barford

First up a disclaimer, this is how I like to role. I enjoy it. I'm not saying you shouldn't use a mouse or I'll never use one again.

I spend a lot of time programming on my netbook development environment because I enjoy it. Without all the visual tooling I feel can focus on working with the machine, learning its many tongues and tricks. I've run ubuntu on it for about a year and have learned heaps about linux. There have been many languages, tools and skills I would not have otherwise been able to learn.

One thing I don't like about it is using the crappy mouse pad. It has long been a pain point, even doing trivial things. Recently I saw a tweet by @TheColonial listing some mostly unfamiliar names and adding themouseisdead hash tag. This was exciting, I had recently adopted vim and was loving it.

Last weekend I was a bit under the weather and not thinking clearly enough to write code. So I used to time to set-up and learn a few new tools. I put together a powerful, mouse-less development environment on my tiny netbook, which I think is pretty fun to use.

xmonad

xmonad is a great window manager (~1000 lines of haskell) which seems to plug into most X systems. There are other ways of managing a windowed environment, I never new existed! It is all keyboard driven, it works out non- overlapping window layouts, you can move between multiple desktops and move windows between them.

Here are some typical tasks in a desktop with a few windows using xmonad:

Switch between window layouts, generated by haskell algorithms (mod- space).
Select between windows on screen (mod-tab).
Throw the selected window to another desktop (shift-mod-2).
Change to the desktop (mod-2).

On Ubuntu it installed with:

sudo apt-get install xmonad

Which installed its haskell dependencies and configured itself. It also added an xmonad session option on the login screen, I wanted to use this and keep the normal session intact.

When you start an xmonad session you are presented with a completely empty screen except for a background picture. You can start terminal session with some keys (mod-shift-enter), but a menu would be nice.

dmenu

dmenu is a generic dynamic menu for X systems, it allows menu items to be selected efficiently with a keyboard.

It installed with:

sudo apt-get install dwm-tools

This adds another session to the login screen, now anything can be opened from the xmonad session by bringing up dmenu (mod-p) and typing the first few letters of the desired application.

Vim

I'm surprised and a little annoyed I didn't start using it sooner really, it is a very powerful text editor. I still fumble through it but feel I'm learning to increase my productivity and reduced my frustration working with code.

sudo apt-get install vim

Now vim is available from the terminal. The vim-tutor is the best place to start, then there is a large ecosystem of plugins.

Vimperator

Vimperator is a Firefox extension that provides command from the keyboard, with vim idioms. It provides deep control of firefox, a basic browsing scenario might work like:

You can open a page in a new tab

:tabopen theage.com.au

Then to open a link to a story in a new tab

shift-f

This adds unique two digit numbers to all links. Now any link can be opened by typing the digits or the link text.

To go to the new tab:

gt

Once read the tab can be deleted

TTYtter

Anything typed that doesn't start with "/" is a tweet (the -verify and -slowpost options can help with potential problems that might cause) The forward slash is used to invoke commands like "whois", "replies", "reply", "dm", "follow", "thread" etc. It's built on curl, so we need that first.

apt-get install curl

Then download the perl script from the website, make it executable and move it to the /usr/bin folder.

I like everything about this app down to the ascii art in the menus. The dude has some pretty cool stuff including a network of gopher servers.

Vimium

Vimium is not as powerful as Vimperator, but provides some vim-like keyboard controls for Chrome. Easily installed as a chrome add-in.

Rock'n'roll time?

I can now manage windows over multiple virtual desktops. Use a fast, effective menu to opening applications. Have extensive control of firefox and most importantly, can tweet about it with TTYtter. I am excited and looking forward to working with some code.

My set-up still isn't right yet though; I need a status bar and decent start script (the plain xmonad session just a blank canvas, an internet connection, battery power monitor and a clock would be useful). I'm going to try dzen for this.

And my keyboard skills on the special keys is way worse than than I'd like it to be. The best way to fix that is to keep my hands on the keyboard, right?

Finally thanks to all the people responsible for the OSS platform, languages and many tools. They are a pleasure to use.

Revisiting dragging and inertia with RxJs

tarn@tarnbarford.net (tarn) — Mon, 07 Jun 2010 19:54:00 +0000

Tarn Barford

This is my first look at RxJs the Reactive Extensions for Javascript. The demo isn't remotely useful or practical but it was complicated enough to get a feel for a range of Rx constructs and how they can be used in the Javascript language.

The excellent articles by Matthew Podwysocki at CodeBetter provided the information I needed to get started and his answers to my questions on Twitter (1 2 3 4) helped fill the gaps for me. The dragging, easing and inertia maths is lifted from the excellent project "Rosetta Stone" tutorials but it isn't very complicated.

What's going on here man?

It's a type of drag and release behaviour that is a little bit more complicated than a normal drag and drop and a lot less useful. Instead of just moving the drag target with the mouse, it will ease towards it and once released it will slow to a stop. It allows you to kind-of throw the tiles.

The behaviour involves mouse button events, movements and regular frames to animate the drag items movement. Rx provides and API to co-ordinate these events.

The behavour could be desribed as:

Select mouseDowns
    SelectMany mouseMoves
        Select frames
            Yield positions
        Until mouse
    Concat
        Select frame
            Yield positions
        Until mouseDown Merge (or) itemStopedMoving

This is almost exactly how the Rx API works. The Select, SelectMany, Until, Concat and Merge are all Rx functions. The code looks very similar too except there is a closure for each drag motion but it could be factored differently.

Enough already, show me the codez

No problems, here is the pure Javascript code which can be run outside the browser in something like Rhino. Separating the non-browser dependant part helped enormously allowing me to trigger the relevant events and test the output in a controlled, not-so insane way.

function drag(target, events) {
   function handleOutOfBounds(position, offset) {
       if (position.x - offset.x > 400) { position.x = 400 + offset.x; position.vx *= -1 };
       if (position.x - offset.x < 0) { position.x = 0 + offset.x; position.vx *= -1 };
       if (position.y - offset.y > 400) {position.y = 400 + offset.y; position.vy *= -1 };
       if (position.y - offset.y < 0) { position.y = 0 + offset.y; position.vy *= -1 };
       return position;
    }

    function dragTo(position,moveTo) {
       var newX = position.x + ((moveTo.x - position.x) * 0.2);
       var newY = position.y + ((moveTo.y - position.y) * 0.2);
       return  { x : newX, y : newY, vx : newX - position.x, vy : newY - position.y };
    }

    function dragOut(position) {
        return newPos = { x : position.x + position.vx, 
                          y : position.y + position.vy, 
                          vx : position.vx * 0.96, 
                          vy : position.vy * 0.96 };
    }

    function isDragOutComplete(position) {
        return ( Math.abs(position.vx) < 0.1 && Math.abs(position.vy) < 0.1 );
    }

    function dragMotion(start) {
        var position = start.mouse;
        var movementComplete = new Rx.Subject();

        function targetPosition() {
            return { target : target,
                x : position.x - start.offset.x,
                y : position.y - start.offset.y }
        };

        function easeTo(moveTo) {
            function easeFrame (e) {
               position = handleOutOfBounds(dragTo(position,moveTo),start.offset);
               return  targetPosition();
            }
            return events.frame.Select(easeFrame).TakeUntil(events.mouseMove);
        }

        function easeToStop(frameEvent) {
            position = handleOutOfBounds(dragOut(position),start.offset);

            if (isDragOutComplete(position)) {
                movementComplete.OnNext(); 
            }

            return targetPosition();
        }


        return events.mouseMove.SelectMany(easeTo)
            .TakeUntil(events.mouseUp)
            .Concat(events.frame.Select(easeToStop)
            .TakeUntil(events.mouseDown.Merge(movementComplete)))
    };

    return events.mouseDown.SelectMany(dragMotion);
};

The RxJs extension for jQuery are used to compose the desired observable events from the browser.

function getDomEvents(target) {
    function getMouse(e) {
        return { x : e.clientX,y : e.clientY }; }

    function getDragItem(target) {
        return { x : parseFloat(target.css("left").replace("px","")),
                 y : parseFloat(target.css("top").replace("px","")) }; }

    function getOffset(mouse,position) {
        return { x: mouse.x - position.x, y : mouse.y - position.y }; }

    function getDragStart(mouseEvent) {
        mouseEvent.preventDefault();
        var mouse = getMouse(mouseEvent);
        var position = getDragItem(target);
        return { mouse : mouse, position : position,
                 offset : getOffset(mouse,position) }; 
    }

    return { mouseUp : $(document).toObservable("mouseup"),
             mouseDown : target.toObservable("mousedown").Select(getDragStart),
             mouseMove : $(document).toObservable("mousemove").Select(getMouse),
             frame : Rx.Observable.Interval(10) };
};

function dragElement(target) {
        var events = getDomEvents(target);
        return drag(target, events);
};

Finally we we can subscribe to the composed observable and update the DOM element accordingly.

$(".tile").each(
    function(target) {
        dragElement($(this)).Subscribe( function ( pos ) {
            pos.target.css( { top: pos.y, left: pos.x } );
        })});

What next

It's interesting there is no reason we couldn't take the observable event stream we created here and compose it with more events. For example we could subscribe to this event stream until the tile position was in the correct position or until an escape key is pressed.

For me Rx is a good solution to a very real problem domain. It's been fun learning about Rx and also a bit more about Javascript. I'm looking forward to using RxJs in real web applications.

Map-Reduce on Mongo

tarn@tarnbarford.net (tarn) — Wed, 12 May 2010 19:00:00 +0000

Tarn Barford

I'm doing a presentation on non-relational databases at DDD Melbourne this weekend where I am going to demonstrate a map-reduce example with MongoDB and server side Javascript. I've been interested in both independently recently and it's been fun getting them to working together with some Javascript TDD to boot.

I needed a good example to demonstrate map-reduce and decided finding word occurrences across a series of documented seemed a simple enough scenario that is suited to being solved by a map-reduce query.

Below is an example of how we might solve this in plain C#

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

class wordCounts {

    static void Main(string[] args) {

        // Setup some data
        List<string> lines = new List<string>()
            {
              "Peter Piper picked a peck of pickled peppers",
              "A peck of pickled peppers Peter Piper picked",
              "If Peter Piper picked a peck of pickled peppers",
              "Where's the peck of pickled peppers Peter Piper picked?"
            };

        // select all words, group, count
        var wordCounts = lines.SelectMany(m => m.Split())
                              .GroupBy(m => m.ToLower())
                              .Select(m => new KeyValuePair<string,int>(m.Key, m.Count()));

        // Print out the results
        foreach (var wordCount in wordCounts)
        {
            Console.WriteLine(string.Format("{0} {1}", wordCount.Value, wordCount.Key));
        }
    }
}

This prints each different word in all the lines and the number of the times it occurs. The collection of strings is isomorphic to a collection of documents in the MongoDB for this example.

The SelectMany flattens lists of words from each line to a single list of words and the Group provides keys for each word, this is very similar to what the map function in the map-reduce query does.

The Select function is similar to the reduce function, but as we will see some additional considerations need to be made to allow it to be distributed.

I saw a good diagram ayande published on his blog but I didn't understand why he had multiple instance of the same document being mapped.

I created my own low key diagram to help demonstrate how a functional map- reduce could be distributed. The diagram shows the initial items can be split in half and reduced completely independently. This is interesting as it means our query can be distributed, but it also means we have to handle reducing a little differently.

It's also worth noting that this example shows a balanced tree, but it could be unbalanced and even introduce some redundancy.

MongoDB allows clients to send JavaScript map and reduce functions that will get eval'd and run on the server. Here is the map function.

function wordMap() {

    // try find words in document text
    var words = this.text.match(/\w+/g);

    if (words === null) {
        return;
    }

    // loop every word in the document
    for (var i = 0; i < words.length; i++) {
        // emit every word, with count of one
        emit(words[i], { count : 1 });
    }
}

The misunderstood Javascript "this" will be the context from which the function is called. Mongo will call function each document in the collection we are querying, and we can call it from a test context. Unlike the SelectMany the map function doesn't return a list, instead it calls an emit function which it expects to be defined.

We can write unit tests for this function by calling the function from a test mock context, calling a mock emit function (using Javascript as our mocking framework, wow).

eval(loadFile("src/js/wordMap.js"));

var emit;
var results;
var context;

testCases(test,
    function setUp() {
        emit = function (key, value) {
            results.push({ key : key, value : value });
        };

        context = { text : "", map : wordMap };
        results = [];
    },


    function empty_string_emits_nothing() {
        context.text = "";
        context.map();
        assert.that(results.length, eq(0));
    },


    function single_word_emits_single_word() {
        context.text = "findme";
        context.map();
        assert.that(results.length, eq(1));
        assert.that(results[0].key, eq("findme"));
        assert.that(results[0].value.count, eq(1));
    },


    function two_different_words_emits_twice() {
        context.text = "for bar";
        context.map();
        assert.that(results.length, eq(2));
    },


    function two_same_words_emits_twice() {
        context.text = "test test";
        context.map();
        assert.that(results.length, eq(2));
    },


    function tearDown() {
    }
);

The reduce function must reduce a list of a chosen type to a single value of that same type; it must be transitive so it doesn't matter how the mapped items are grouped.

function wordReduce(key, values) {
        var total = 0;
        for (var i = 0; i < values.length; i++) {
            total += values[i].count;
        }
        return { count : total };
    }

Similarly we can test this method does exactly what we expect it to.

eval(loadFile("src/js/wordReduce.js"));

testCases(test,
    function reduce_one_items_returns_count_of_one() {
        var result = wordReduce("test", [{ count : 1 }]);
        assert.that(result.count, eq(1));
    },

    function reduce_multiple_items_returns_item_count() {
        var result = wordReduce("test", [{ count : 1 }, { count : 1 }, { count : 1 }]);
        assert.that(result.count, eq(3));
    },

    function reduce_sums_counts() {
        var result = wordReduce("test", [{ count : 2 }, { count : 3 }]);
        assert.that(result.count, eq(5));
    },

    function reduce_is_transitive() {
        var result = wordReduce("test", [{ count : 1 }].concat(
                        wordReduce("test", [{ count : 1 }, { count : 1 }]
                     ));
        assert.that(result.count, eq(3));
    }
);

I'm using Rhino to run the Javascript so I used RhinoUnit as a test runner as it also uses the JVM and runs as an ANT scriptdef task, the setup was pretty painless. Here are the relevant ANT script sections

<scriptdef name="rhinounit"
           src="lib/rhinoUnitAnt.js"
           language="javascript">
    <attribute name="options"/>
    <attribute name="ignoredglobalvars"/>
    <attribute name="haltOnFirstFailure"/>
    <attribute name="rhinoUnitUtilPath"/>
    <element name="fileset" type="fileset"/>
</scriptdef>


<target name="javascript-tests">
    <rhinounit options="{verbose:true, stackTrace:true}"
               haltOnFirstFailure="false"
               rhinoUnitUtilPath="lib/rhinoUnitUtil.js">

        <fileset dir="test">
            <include name="*.js"/>
        </fileset>
    </rhinounit>
</target>

The word count example recreated in Mongo using a Python client and passing the map/reduce functions to the server.

from pymongo import Connection;
from pymongo.code import Code;

# open connection and connect to 'ddd' database
connection = Connection()
db = connection.ddd

# remove any existing data
db.drop_collection("messages")

# insert some data
lines = open("data/peter_piper.txt").readlines();

for line in lines:
    db.messages.insert( { "text" : line } )

# load map and reduce functions
map = Code(open("src/js/wordMap.js","r").read())
reduce = Code(open("src/js/wordReduce.js","r").read())

# run the map-reduce query

result = db.messages.map_reduce(map, reduce)

# print the results

for doc in result.find():
    print doc["value"]["count"],doc["_id"]

And it worked! I'd like to run the query on a larger result-set, but there isn't much point on this tiny low-spec'd netbook.

DevEvening NoSql/MongoDB Presentation

tarn@tarnbarford.net (tarn) — Wed, 07 Apr 2010 21:12:00 +0000

Tarn Barford

The slides and demo I'll show in my NoSql presentation for tomorrows DevEvenings Melbourne ORM Smackdown. I hope to take the time and write a more considered post of my findings and opinions as I've found it very interesting.

Here is a link to the slides.

First I'll use some Python and the PyMongo module to connect to MongoDB, list databases, insert documents and get them out again.

Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymongo import Connection
>>> connection = Connection()
>>> connection.database_names()
[u'files', u'working', u'demo', u'downloads', u'posts', u'local',u'admin']
>>> db = connection.demo
>>> import datetime
>>> db.messages.insert( { 'author' : 'tarn', 'date': datetime.now(),'message' : 'Hello Mongo' } )
ObjectId('4bbc4beec73d721445000003')
>>> db.messages.find_one()
{u'date': datetime.datetime(2010, 4, 7, 19, 10, 6, 355000),
 u'message': u'Hello Mongo',
 u'_id': ObjectId('4bbc4beec73d721445000003'),
 u'author': u'tarn'}

Then have a look at some content I have in the database

>>> db = connection.working
>>> db.posts.count()
106
>>> for post in db.posts.find()[:5]:
...     print post["date"],post["title"],"by",post["author"]
...
2010-01-25 18:06:00 Python Silverlight/Moonlight 2 Xapping by tarn
2010-03-12 13:53:00 Devevenings Presentation - IOC/Unit Testing/Mocking in ASP.NET MVC by tarn
2010-02-17 19:25:00 Revisiting Modal Binding an Interface, now with DictionaryAdapterFactory by tarn
2009-12-02 18:35:00 Creating Silverlight apps in the browser by tarn
2009-10-02 23:08:00 #.think.in infoDose #43 (11th September - 22nd September) by brodie

File storage using the GridFS class from the gridfs module. Show some files and then write a file out to the file system.

>>> from gridfs import GridFS
>>> fs = GridFS(connection.files)
>>> len(fs.list())
116
>>> for file in fs.list()[:5]:
...     print file
...
post/debugging-ironpython-with-my-excalibur/image.png
post/debugging-ironpython-with-my-excalibur/image_thumb.png
post/devevenings-presentation---iocunit-testingmocking-in-asp.net-mvc/20102f32fdevevening_presentation.pptx
post/devevenings-presentation---iocunit-testingmocking-in-asp.net-mvc/20102f32fguestbook.zip
post/think.in-infodose-40-5th-august---16th-august/image.png
>>>
>>> with open('image.png','w') as out_file:
...     with fs.open('post/debugging-ironpython-with-my-excalibur/image.png') as in_file:
...             out_file.write(in_file.read())
...

Now to some C# (mono) and a controller class for a basic web application to view the data. It serves files found in the database, but only sends bach the correct MIME type for "image/png". Lazy.

public class HomeController : Controller
{
    BlogRepository _blogRepository;

    public HomeController()
    {
        _blogRepository = new BlogRepository();
    }

    public ActionResult Index ()
    {
        ViewData["posts"] = _blogRepository.GetPosts();
        return View ();
    }

    public ActionResult Entry(string id)
    {
        ViewData["post"] = _blogRepository.GetById(id);
        return View ();
    }

    public ActionResult Resource(string slug, string fileName)
    {
        return new FileStreamResult(_blogRepository.GetFile( "post/" + slug + "/" + fileName ), "image/png");
    }
}

This is a very basic repository that provides the data for the demo application. I did only enough with the C# provider to get it working and try to disconnect my connections.

public class BlogRepository
{
    Mongo _mongo;

    public BlogRepository()
    {
        string connstr = ConfigurationManager.AppSettings["connectionString"];
        _mongo = new Mongo(connstr);
    }

    public Stream GetFile(string name)
    {
        try
        {
            _mongo.Connect();
            var db = _mongo["files"];
            var fs = new GridFile(db);
            Stream data = fs.Open(name, FileMode.Open, FileAccess.Read);
            Stream output = new MemoryStream();
            CopyStream(data,output);
            output.Seek(0,SeekOrigin.Begin);
            return output;
        }
        finally
        {
            _mongo.Disconnect();
        }
    }

    public List<Document> GetPosts()
    {
        try
        {
            _mongo.Connect();
            var db = _mongo["working"];
            var posts = db["posts"];
            using(ICursor all = posts.Find(new Document())){
                return all.Documents.ToList();
            }
        }
        finally
        {
            _mongo.Disconnect();
        }
    }

    public Document GetById(string id)
    {
        try
        {
            _mongo.Connect();
            var db = _mongo["working"];
            var posts = db["posts"];
            Document doc = posts.FindOne( new Document() {{ "_id" , new Oid(id) }} );
            return doc;
        }
        finally
        {
            _mongo.Disconnect();
        }
    }

    public static void CopyStream(Stream input, Stream output)
    {
        byte[] buffer = new byte[32768];
        while (true)
        {
            int read = input.Read (buffer, 0, buffer.Length);
            if (read <= 0)
                return;
            output.Write (buffer, 0, read);
        }
    }
}

So that's where I got for my demo for the DevEvenings ORM Smackdown. No doubt I will continue looking into MongoDB and other object/document databases.

Scraping this blog

tarn@tarnbarford.net (tarn) — Thu, 18 Mar 2010 23:31:00 +0000

Tarn Barford

I have created a monster and this post is about killing it off by scraping the contents of this blog into structured Python objects. Sometime later I will convert the HTML content to markdown and download the images and other resources locally.

I want to put the contents into a ZODB object database to get a feel for working with object database. A greated goal is to migrate the content a new blog engine. I don't want to go into why I felt I need to scrape it or why I want to migrate to another blog engine as it's depressing.

Moving on, I wanted to put the content into these classes

class Post():
    title = ''
    content = ''
    date = ''
    tags = []
    comments = []

class Comment():
    content = ''
    author = ''
    date = ''
    website = ''

The scraping code is not elegant but was quite fun to write as I could write it all from an interactive console session. I found BeautifulSoup was fantastic in making HTML into something that was easy to work with, although I would have liked to have used jQuery/CSS style selectors.

from BeautifulSoup import BeautifulSoup
from datetime import datetime
import urllib2
import re

def ParseComment(soup):
    comment = Comment()
    comment.author = soup.find('p',{"class":"author"}).first().string.strip
    content = soup.find('p',{"class":"content"})
    if content:
        comment.content = content.prettify()

    website = soup.find('p',{"class":"author"}).first()
    if website.has_key('href'):
        comment.website = soup.find('p',{"class":"author"}).first()['href']

    r = re.compile('\d*/\d*/\d* \d*.\d*')
    date = r.findall(soup.find('p',{"class":"date"}).renderContents())[0]
    comment.date = datetime.strptime(date,'%d/%m/%Y %H:%M')
    return comment

def ParsePost(postSoup):
    post = Post()
    post.title = postSoup.find('a',{"class":re.compile('posthead.*')}).string
    post.content = postSoup.find('div', {"class":"entry"})
    date = postSoup.find('div',{"class":"descr"}).contents[0][:-4]
    post.date = datetime.strptime(date,'%B %d, %Y %H:%M')
    post.author = postSoup.find('div',{"class":"descr"}).first().string
    post.tags = map(lambda x: x.string, postSoup('a',{"rel":"tag"}))
    comments = postSoup.find('div',{"id":"commentlist"})('div')
    post.comments = [ParseComment(commentSoup) for commentSoup in comments]
    return post

def DownloadPost(url):
    postHtml = urllib2.urlopen('http://blog.sharpthinking.com.au/' + url).read()
    postSoup = BeautifulSoup(postHtml)
    return ParsePost(postSoup);

def GetPosts():
    page = urllib2.urlopen("http://blog.sharpthinking.com.au/archive.aspx")
    soup = BeautifulSoup(page)
    postUrls = map(lambda x: x['href'], soup('a', href=re.compile('/post/.*')))
    return [DownloadPost(url) for url in postUrls[:-10]]

I'm sure there is better way, but this was better than any way I've used previously. Anyway I've done a lot of work untangling the mess I created.

>>> posts = GetPost()
>>> for post in posts[:5]:
...     print post.date, post.title
...

2010-03-17 22:16:00 OMG. It's a JavaScript Rhino
2010-03-12 13:53:00 Devevenings Presentation - IOC/Unit Testing/Mocking in ASP.NET MVC
2010-02-20 17:18:00 Revisiting Pygments in the browser with Silverlight, now with BackgroundWorker
2010-02-17 19:25:00 Revisiting Modal Binding an Interface, now with DictionaryAdapterFactory
2010-02-16 20:34:00 Modal Binding an Interface with DynamicProxy

I wanted to put the contents into the object database tonight, but I have pickled it to be revisited later.

OMG. It's a JavaScript Rhino

tarn@tarnbarford.net (tarn) — Wed, 17 Mar 2010 22:16:00 +0000

Tarn Barford

JavaScript is a slightly flawed language but it's got elegant parts too. All languages do to some degree, it's just JavaScript seems to have both in extremes. Whatever you think of it, history has made it the language for scripting the client-side web. It has become a mainstream language that shows no sign of falling off.

The excelent book JavaScript: The Good Parts by Douglas Crockford, working with the jQuery library and learning a little Lisp has lead me to really embrace JavaScript.

It's no secret that I like programming with interactive consoles and decided I wanted find out if there was an interactive console for JavaScript. A language that only lives inside web browser environment didn't seem right to me.

Rhino is a JavaScript implementation on the JVM. It has a compiler, a debugger and interactive console.

To get started you obviously need a version of the JVM. That's not to difficult. On Windows I just downloaded a Sun Java 6 installer. On my Ubuntu install I installed openjdk but found Rhino didn't work. So I installed sun-java6, which worked.

sudo apt-get install sun-java6-bin sun-java6-jre sun-java6-jdk

You can find what version you've installed by running

$ java -version
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03)
Java HotSpot(TM) Client VM (build 14.1-b02, mixed mode, sharing)

Excellent. The Rhino binaries includes js.jar which is needed for the console. Now should be able to run the jar

$ java -jar js.jar

When all goes well this will take you into the Rhino shell

Rhino 1.7 release 2 2009 03 22
js>

We can start playing with the language.

js> get_counter = function() { var counter = 0; return function() { 
     print(counter); counter++; } };
..
js> counter1 = get_counter();
js> counter1();
0
js> counter1();
1
js> counter2 = get_counter();
js> counter2()
0
js> counter1()
2

Which is cool and there is some of the weirdness

js> '5' + 3
53
js> '5' - 2
3

And some interesting features

js> parseInt('06')
6
js> parseInt('08')
NaN
js> parseInt('10')
10
js> parseInt('010')
8

I'm looking forward to learning more about writing code in JavaScript.

Devevenings Presentation - IOC/Unit Testing/Mocking in ASP.NET MVC

tarn@tarnbarford.net (tarn) — Fri, 12 Mar 2010 13:53:00 +0000

Tarn Barford

Here are the slides and the mysterious code that was never shown from my DevEvening presentation.

Appologies it's taken a while to get them up, I was hoping to write a bit of a post about what I covered and some of the discussion that came up. That never happened.

Devevening_Presentation.pptx (565.68 kb)

Guestbook.zip (3.42 mb)

I think the presentation went well, the group was really good and we got some good discussions happening before being interrupted by delicious paramas of the world.

I'm looking forward to the next meeting, except it appears I've signed up to represent NoSql (of which I currently know very little about) in an ORM smackdown.

That's what happens when you have meetings at a pub. Anyway it should fun and I'm looking forward to learning enough about NoSql to adequatly represent it in the smackdown.

See ya there.

Revisiting Pygments in the browser with Silverlight, now with BackgroundWorker

tarn@tarnbarford.net (tarn) — Sat, 20 Feb 2010 17:18:00 +0000

Tarn Barford

A couple of week ago I blogged about using Pygments to do live syntax highlighting in the browser using Silverlight.

A major problem with the sample was that it did the pygmentizing on the UI thread which caused most browsers to become unresponsive. Today I wanted to fix that by using the BackgroundWorker to do the pygmentizing in a background thread.

Firstly I refactored the pygmentizing into a method that didn't interact with the UI.

def pygmentize_text(self, text, language):

    # attempt to pygmentize input with current language
    try:

        from pygments import highlight
        from pygments.lexers import get_lexer_by_name
        from pygments.formatters import HtmlFormatter

        lexer = get_lexer_by_name( language, stripall=True)
        formatter = HtmlFormatter(linenos=False, cssclass="source")
        markup = highlight(text, lexer, formatter)
        return markup

    except:

        return "Error Generating Markup"

I then added a method that could be passed into a DoWorkEventHandler. It gets it arguments as a tuple from the event arguments and then sets the event argument result with the marked up HTML. The lack of explicit typing and use of tuples is good example of how some python idioms can be used when working with the .NET framework.

def worker(self, sender, e):

    # do work off UI thread.
    e.Result = self.pygmentize_text(e.Argument[0],e.Argument[1])

The required BackgroundWorker and DoWorkEventHandler can be simply imported from the System.ComponentModel namespace.

from System.ComponentModel import BackgroundWorker, DoWorkEventHandler

The BackgroundWorker can then be setup and started. Again it's syntactically nice how the tuple can be created and passed as a RunWorkerAsync parameter.

def start_pygmentize(self):

    # update application state
    self.input_changed = False
    self.pygmentizing = True
    self.show_message("pygmentizing..")

    # get paremters
    input = self.input.GetProperty("value")
    language = self.language.value

    # setup background worker
    worker = BackgroundWorker()
    worker.DoWork += DoWorkEventHandler(self.worker)
    worker.RunWorkerCompleted += self.complete

    # start the worker
    worker.RunWorkerAsync( (input,language) )

The completed event handler is the responsible for taking the markup generated by the BackgroundWorker and updating the DOM. It also fires off another worker if the source has changed since the last worker started.

def complete(self, sender, e):

    if e.Error:

        # handle errors/exceptions in worker
        self.source.SetProperty("innerHTML",e.Error.Message)

    else:

        # show the result
        self.source.SetProperty("innerHTML",e.Result)

    if self.input_changed:

        # input has changed, starty pygmentize again
        self.start_pygmentize()

    else:

        # no work queued
        self.pygmentizing = False
        self.hide_message()

The update has made the sample much more responsive, however it appears downloading the Silverlight application is still causing some browers to become a little unresponsive which is annoying. I will be interested to find out if this effect can be mitigated.

The actually pygmentizing processing could possibly be made a little faster by reusing the BackgroundWorker and only doing the Pygment imports once but the responsiveness of the browser has improved the sample enormously.

Check out the updated demo here.

Revisiting Modal Binding an Interface, now with DictionaryAdapterFactory

tarn@tarnbarford.net (tarn) — Wed, 17 Feb 2010 19:25:00 +0000

Tarn Barford

After my post yesterday Krzysztof Kozmic (@kkozmic) messaged me saying I should use the Castle DictionaryAdapterFactory instead.

For the record Krzysztof has a great series of posts on DynamicProxy and was also very helpful answering a specific question of mine yesterday.

Anyway this is great news! I can do the same thing only referencing one assembly. The DictionaryAdapterFactory returns an object that passes all my tests from yesterday and slides staight into the model binder.

public class FakeInterfaceModelBinder : DefaultModelBinder
{
    protected override object CreateModel(ControllerContext controllerContext, ModelBindingContext bindingContext, Type modelType)
    {
        Dictionary<string, object> dictionary = new Dictionary<string,object>();
        return new DictionaryAdapterFactory().GetAdapter(modelType,dictionary);
    }
}

It was fun playing with DynamicProxy yesterday, but I did think it might not have been a good idea for a model binder. I'm glad I was pointed in the right direction.

Cheers Krzysztof!

Modal Binding an Interface with DynamicProxy

tarn@tarnbarford.net (tarn) — Tue, 16 Feb 2010 20:34:00 +0000

Tarn Barford

Update: I posted an update to this post which uses the Castle DictionaryAdapterFactory after getting some feedback from this post.

This started while I was playing with some ASP.NET MVC architectural ideas in simple application.

I wanted to less code than other projects I'd worked on but still have it decoupled and testable.

At some point I started ranting on twitter (1, 2, 3) about wanting a Model Binder that could bind to an interface without needing a concrete type.

It was quickly pointed out that what I probably wanted was a Model Binder that resolves an implementation at runtime with IOC.

While this was probably true as I was using already using StuctureMap and it did seemed like a much better idea I still some my reservations. But mainly I wanted to try using the Castle DynamicProxy in an application.

Clearly the reason the default model binder can't bind to an interface is because it can't create an instance of it. Thanks to some nice extensibility hooks in the default model binder, all I had to do was override the CreateModel method and return my DynamicProxy instance. The default model binder then could do the rest of binding work as normal.

public class FakeInterfaceModelBinder : DefaultModelBinder
{
    protected override object CreateModel(ControllerContext controllerContext, ModelBindingContext bindingContext, Type modelType)
    {
        var generator = new ProxyGenerator();
        return generator.CreateInterfaceProxyWithoutTarget(modelType, new FakePropertiesInterceptor());
    }
}

What I needed then was a DynamicProxy object that had properties the behaved like you'd expect properties to on a normal class. I came up with some test scenarios I'd expect the proxy to pass using a test interface.

public interface IDynamicProxyTest
{
    int Integer { get; set; }
    string String { get; set; }
    DateTime Date { get; set; }
}

I figured the Model Binder would set values on the Dynamic Proxy object and the controller would get the properties back to validate and pass them to the data layer or something like that. I came up with these tests.

[Test]
public void If_date_has_been_set_then_return_it()
{
    var date = DateTime.Now;
    var proxy = CreateProxy();
    proxy.Date = date;
    Assert.AreEqual(date, proxy.Date);
}

[Test]
public void If_string_has_been_set_then_return_it()
{
    var proxy = CreateProxy();
    proxy.String = "Hello Proxy";
    Assert.AreEqual("Hello Proxy", proxy.String);
}

[Test]
public void If_int_has_been_set_then_return_it()
{
    var proxy = CreateProxy();
    proxy.Integer = 42;
    Assert.AreEqual(42, proxy.Integer);
}

After implementing these I was surprised to find the default model binder did some gets on the properties before sets. So I added some more tests.

[Test]
public void For_date_instance_if_value_not_set()
{
    var proxy = CreateProxy();
    var b = proxy.Date;
}

[Test]
public void For_int_instance_if_value_not_set()
{
    var post = CreateProxy();
    var b = post.Integer;
}

[Test]
public void For_string_instance_if_value_not_set()
{
    var post = CreateProxy();
    var b = post.String;
}

My property interceptor isn't as clean as I'd like it but did the job passing tests.

public class ProperyInterceptor : IInterceptor
{
    Type _type;
    Dictionary<string, object> properties;

    public ProperyInterceptor(Type type)
    {
        properties = new Dictionary<string, object>();
        _type = type;
        var fields = _type.GetProperties();
    }

    public void Intercept(IInvocation invocation)
    {
        if (IsSetter(invocation))
        {
            properties[Name(invocation)] = invocation.Arguments[0];
        }

        if (IsGetter(invocation))
        {
            if (properties.ContainsKey(Name(invocation)))
            {
                invocation.ReturnValue = properties[Name(invocation)];
            }
            else
            {
                GetDefaultValue(invocation);
            }
        }
    }

    private static void GetDefaultValue(IInvocation invocation)
    {
        if (invocation.Method.ReturnType == typeof(string))
        {
            invocation.ReturnValue = "";
        }
        else
        {
            invocation.ReturnValue = Activator.CreateInstance(invocation.Method.ReturnType);
        }
    }

    private string Name(IInvocation method)
    {
        return method.Method.Name.Substring(4);
    }

    private bool IsSetter(IInvocation method)
    {
        return method.Method.IsSpecialName && 
               method.Method.Name.StartsWith("set_", StringComparison.Ordinal);
    }

    private bool IsGetter(IInvocation method)
    {
        return method.Method.IsSpecialName &&
               method.Method.Name.StartsWith("get_",StringComparison.Ordinal);
    }
}

I used the better model binder Jimmy Bogard discusses as it makes writing binders for derived types and specific "types of type" more straight forward.

public class InterfaceBinder : FakeInterfaceModelBinder, IFilteredModelBinder
{
    public bool IsMatch(Type modelType)
    {
        return modelType.IsInterface;
    }
}

I did try writing a test which ran my model binder, but found it too much trouble. When I tried it out in a sample web project it worked exactly as I expected.

While this was fun I don't actually think it's a very good idea. I was thinking about the interface as DTO's and it breaks down if you want any additional methods on the interface, as they just won't work. Anyway it's been fun as always and maybe I'll find a good place to use these dynamic proxies in the future.

Source code with the sample website is all here.

Scripting your Data Model

tarn@tarnbarford.net (tarn) — Sun, 14 Feb 2010 14:13:00 +0000

Tarn Barford

I really like the way you can script your data model from a python REPL console on the Django and the Google App Engine web frameworks.

For me it is a hands down better way of working with data in your domain model than writing SQL.

I've since wanted to do it in .NET projects I work on, but it wasn't till I was playing with Castle ActiveRecord yesterday that I decided I'd try it out.

The ActiveRecord pattern is an intuitive way of programming with data persistence, so it's also nice to script with.

It turned out to be really easy in the project I was playing with. I didn't need to write a single additional line of C# as I already had the configuration decoupled for integration testing with an in-memory SQLite database.

namespace SimpleBlog.Data
{
    public class Configuration : IBootstrapperTask
    {
        public void Execute()
        {
            Configure(GetDefaultSettings());
        }

        public static void Configure(IDictionary<string, string> properties)
        {
            InPlaceConfigurationSource source = new InPlaceConfigurationSource();
            source.Add(typeof(ActiveRecordBase), properties);
            ActiveRecordStarter.Initialize(source);
            ActiveRecordStarter.RegisterAssemblies(Assembly.GetExecutingAssembly());

        }

        ...

    }
}

I then just wrote this little script to help with the configuration, it uses the static method above and passes in the properties for working with a development database.

import clr

clr.AddReferenceToFile("SimpleBlog.Data.dll")

from System.Collections.Generic import Dictionary
from SimpleBlog.Data import Configuration

# NHibinate Setting
properties = Dictionary[str,str]()

properties.Add("connection.driver_class",
               "NHibernate.Driver.SqlClientDriver");

properties.Add("dialect",
           "NHibernate.Dialect.MsSql2005Dialect");

properties.Add("connection.provider",
           "NHibernate.Connection.DriverConnectionProvider");

properties.Add("connection.connection_string",
           "Data Source=[CONNECTION_STRING]"); # Add

properties.Add("proxyfactory.factory_class",
           "NHibernate.ByteCode.Castle.ProxyFactoryFactory,NHibernate.ByteCode.Castle");

Configuration.Configure(properties)

Using the helper script it's pretty easy to get in and start working with the data in the data model.

IronPython 2.6 (2.6.10920.0) on .NET 2.0.50727.4927
Type "help", "copyright", "credits" or "license" for more information.
>>> import ActiveRecord
>>> from SimpleBlog.Data.Models import *
>>>
>>> post = Post()
>>> post.Title = "Working with SQL sucks!"
>>> post.Content = "Try using a scripting langauge instead. It rocks!"
>>> post.Author = "tarn"
>>> post.Save()
>>>
>>> post.Id

2

>>>
>>> posts = Post().FindAll()
>>>
>>> for p in posts:
...     print p.Title, "by", p.Author
...

Hey, It's alive by tarn
Working with SQL sucks! by tarn

>>>

This is a simple example of a database agnostic data script using your domain model and a powerful scripting language. I think scripting data models like this could add a lot of value in many .NET development scenarios.

Pygments in the browser with Silverlight

tarn@tarnbarford.net (tarn) — Sun, 31 Jan 2010 23:40:00 +0000

Tarn Barford

** 17-02-2010 I've updated the demo to use the BackgroundWorker and posted about the update **

I decided it might be fun to try get Python Markdown and Pygments running in the browser to enhance a markdown preview experience by eliminating the server-side round trips and provide a more responsive preview.

I managed to get it working entirely in Python but I found the application size excessively large (almost 3mb) and, more annoyingly, it seems to block the entire browser when it initially loads the pygments module. I think there is some sort of silverlight background thread I should be using.

I think it would work better with MarkdownSharp as pure C# silverlight applications are a fair bit leaner in size and probably run a little quicker than dynamic language applications. But this was for fun and I prefer coding in Python when not working.

** 17-02-2010 I've since found the MarkdownSharp doesn't do syntax hightlighting **

I ended up having to write so little Python code to get this working that I can include it all here, syntax highlighted with Pygments of course.

from System.Windows import Application
from System.Windows.Controls import UserControl
from System.Windows.Browser import HtmlPage
from System import EventHandler

class App:

    def __init__(self):

        # load relevent HTML DOM elements
        self.input = HtmlPage.Document.GetElementById("input")
        self.source = HtmlPage.Document.GetElementById("output")
        self.language = HtmlPage.Document.GetElementById("lang")

        # fire javascript functions to indicate the application has been load
        HtmlPage.Window.CreateInstance("silverlight_loaded");

        # pygmentize initial
        self.pygmentize()

        # register events
        self.input.AttachEvent('onkeyup', EventHandler(self.update_handler ))
        self.language.AttachEvent('onchange', EventHandler(self.update_handler ))

        # fire javascript function to indicated the pygments has been loaded
        HtmlPage.Window.CreateInstance("pygments_loaded");


    # handle language or input changes by pygmentizing
    def update_handler(self, sender, e):
        self.pygmentize()


    def pygmentize(self):
        input = self.input.GetProperty("value")

        # attempt to pygmentize input with current language
        try:
            from pygments import highlight
            from pygments.lexers import get_lexer_by_name
            from pygments.formatters import HtmlFormatter

            lexer = get_lexer_by_name(self.language.value, stripall=True)
            formatter = HtmlFormatter(linenos=False, cssclass="source")
            markup = highlight(input, lexer, formatter)

            # update the preview
            self.source.SetProperty("innerHTML",markup)

        except:

            # indicate there was an error in pygmentize
            self.source.SetProperty("innerHTML", "Error Generating Markup")


# Do it!
App()

Despite the fact that there isn't much code, the development experience writing silverlight application in python is a bit of a pain in the arse. Granted it's much better with the python console in the browser and better error reporting in most recent SDK, but it still sucks; debugging and logging support is very limited and on some errors the application dies without reporting anything.

Another difficulty is having to manually copy all the python standard library modules required by the module from the library folders into the application (which explains something about the bloated application size). And even though the code in the demo works, some very similar code from the pygments quick start doesn't.

Check out the live demo.

Python Silverlight/Moonlight 2 Xapping

tarn@tarnbarford.net (tarn) — Mon, 25 Jan 2010 18:06:00 +0000

Tarn Barford

Context

I was asked do a presentation about developing Silverlight applications to the Melbourne Python User-group. Hell yeah, I love participating in user groups but I obviously needed some content. Unfortunately I didn't know this local user-group existed, so I've haven't been and have little idea of number of people or the type of stuff they do. This has made preparing relevant content a little challenging, but hopefully I can find the right mix.

Given it's a Python user-group I felt I might look like a bit of dick rolling with Windows and doing a demo with an environment and .NET tools that some part of the group couldn't/don't use. Besides I've always felt the Python language support and being able to build on any platform is a very compelling reason for non-.NET developers get into building Silverlight applications.

I don't yet have a Mac (I'm really starting to want one) so my other option was to do the demo on Linux. This presented some difficulties and some opportunities. Firstly I'm still a bit of Linux noob and, although I'm wrapped to have learned a lot in the last couple of weeks there has been some pain and some compromise.

Also Microsoft don't have a Silverlight browser plug-in for Linux, that support is provided by Moonlight. Moonlight is currently at version 2 and although it supports some of the features of Silverlight 3 it doesn't have the key version 3 features which have really improved the Python development environment (namely IronPython 2.6 support) and application deployment (namely allowing static assemblies that add dynamic languages support to be downloaded independently of the application and therefore cached in the browser).

Plugin and development support on both Mac and Windows is much better, but I'm happy here sacrificing some of the newer features for inclusivity (and apparently inventing words). We're targeting Silverlight 2/Moonlight 2 and effectively using no development tools besides a couple of Python scripts which I'll get onto.

Dynamic Language Silverlight Applications

Silverlight applications are packaged into binary files with an "xap" extension. It turns out these packages are just standard zip files containing all the "stuff" needed by the application. The minimum "stuff" needed in dynamic language application is;

An XML based application manifest file (app.manifest) that describes the target runtime, language type, entry point and some other information needed to run the application
A collection of assemblies that add dynamic language and Python support to the Silverlight runtime.
A plain text file containing the your Python script to make the application dance.

The package can contain more Python scripts, binary assemblies (which can be imported as modules) and XAML files (which are declarative xml type format typically used for describing user interfaces). Resources such as images and video can also be included in the package but it usually make more sense for the application to download them independently when they are required.

Serving the application on a webpage is also quite straight forward. An object tag in the html has attributes to tell the browser the URL to download the application and that the Silverlight/Moonlight plugin should be used to run the application.

Helper Scripts and Templates

With a couple of templates and Python scripts we can get started developing Silverlight applications in any environment with that has Python and a browser with a Silverlight/Moonlight plug-in.

xap.py builds xaps by adding all the files from a source code directory and an assemblies directory into xap file and writing it to disk.

http_server.py is the simplest python server which just serves all the files in the current directory on a specified port.

index.html contains the object tag referencing sample.xap and a tiny bit of Javascript to catch, format and display critical error messages that may be caused by the application.

app.manifest is a template with everything needed to describe a typical Python Silverlight application. The template should be fine as-is unless additional assemblies are required or you want to change the script entry point. The template is surprisingly small and undaunting.

app.py is the script entry point of the application.

assemblies a folder with all the additional assemblies need by the Silverlight runtime to run Python scripts.

Packaging and Running an Application

So you've skipped the previous three sections and jumped straight here. I don't blame you. Here's how to get going with the script and templates described above.

Download this zip containing the assemblies, scripts and templates and unzip it someone, anywhere.

From the terminal/console/command-line go the the place you unzipped it to and run the following command to build an xap.

python xap.py samples.xap assemblies source

A sample.xap should be generated in the same directory.

To fire-up the basic web server just run

python http_server.py

Then just fire up a browsers and go to http://localhost:8000/ and you should see the sample Silverlight application running.

That's it. Now you can edit app.py to add functionality and repeat.

Going further

Clearly this a very bare bones approach to development which has very limited development and debugging tools (and a server that can only serve files). There is more tooling out there, but surprisingly litte for Python developers.

MonoDevelop 2.2, the Moonlight SKD and Python bindings may provide some sort of IDE support but it's only just been released, it is not easy to get up and running and I not sure if it even supports packaging or debugging of dynamic language Silverlight applications.

You might think Visual Studio 2008/2010 would provide significant additional support, but as far as I'm aware only the Windows Debugger which is free in the Windows SDK is much help.

Chiron is a .NET tool provided in the Microsoft Dynamic Language SDK which can package and server XAP's much like I've described above. I found I could compile it on Mono, but couldn't get it serving files. I don't think your missing much here.

XamlPad is my favourite free XAML editor, but it doesn't work on Linux. Blend and Expression Studio are pay to play Microsoft tools for designing and developing XAML. Maybe someone should write a good one in Silverlight.

IronPython can be downloaded and run on Mono.

Some Thoughts

It's easy to get started writing and deploying Silverlight applications in Python on any platform but I think it's fair to say the tooling and support is just not there yet. On any platform.

The Silverlight 2/Moonlight implementations of Python are still missing large parts of the Python standard library, which will come as a bit of a shock to Python developers and makes it difficult to import modules that depend on it. Almost the entire Silverlight 2 Base Class Library is accessible through Python but although the Python/.NET integratation is very impressive, working with the BCL does seem to limit the expressivness of the code.

Silverlight 3 appears to have some additional built-in support for interacting with the application at runtime through a Python REPL console, it supports IronPython 2.6 (which implements a lot more of the Python standard library) and has additional deployment improvements discussed above.

Finally

I hope to do another post before the meeting with some examples I'll be showing, but as always I won't promise anything (in case I get distracted writing a Socket Server in Erlang I've been thinking about). In the mean time I do have some examples on PythonSilverScripting that can be run from the browser.

I'd love to know the tools other people are using to develop Python/Silverlight and thoughts people have of it. If your in Melbourne and your at all interested you should come down to the meeting.

Update 26/01/10: Updated scripts and templates to work in Windows

Here is a zip with the bits and pieces.

Python and Farseer: 2d Physics in Silverlight

tarn@tarnbarford.net (tarn) — Sun, 19 Jul 2009 16:59:00 +0000

Tarn Barford

Ive got a tiny Acer Aspire netbook I use while commuting to work. Its a very low specd machine but it has a very functional keyboard and I love programming on it!

I think I enjoy programming on it because it doesnt have Visual Studios installed. I have the Windows SDK so I can build .NET projects with MsBuild, but I mainly use it for developing Python code. With some free tools and notepad++ plug-ins its a great light weight environment for coding simple Django and Google App Engine web applications and dynamic language .NET applications.

Im not really into computer gaming, but I do like 2d physics games. I remember losing days work when I first found Magic Pen. I decided to try using the cool Farseer Physics Engine, which is described on the Codeplex page as;

The Farseer Physics Engine is an easy to use 2D physics engine designed for Microsofts XNA and Silverlight platforms. The Farseer Physics Engine focuses on simplicity, useful features, and enabling the creation of fun, dynamic games.

Unable to Microsoft Silverlight application. Get Microsoft Silverlight

Getting started

The Farseer engine is written in C# and I found once Id installed the Silverlight SDK I could use MSBuild to build the Farseer engine that targets the Silverlight runtime. The Farseer engine doesnt itself reference anything Silverlight specific, but it does need to be built against the limited .NET framework available in Silverlight.

Ive previously blogged about writing dynamic language Silverlight applications; the Silverlight Dynamic Languages SDK contains the Silverlight IronPython assemblies and a command line application called Chiron. Chiron is a web server that runs from the command line and dynamically packages up XAPs as they are requested.

A little about XAPs

XAPs are packaged Silverlight applications that are downloaded and run on the clients Silverlight runtime. It turns out the XAP is just zip files containing the assemblies, resources and in the case of dynamic language applications, some plain text source code!

Additionally there is an AppManifest.xaml included in the XAP which tells Silverlight about what is included in the XAP and where to find an entry point. Below is an example of an AppManifest.xaml automatically generated by Chiron.

<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment"
            xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
            RuntimeVersion="2.0.31005.0"
            EntryPointAssembly="Microsoft.Scripting.Silverlight"
            EntryPointType="Microsoft.Scripting.Silverlight.DynamicApplication">
  <Deployment.Parts>
    <AssemblyPart Source="Microsoft.Scripting.Silverlight.dll" />
    <AssemblyPart Source="Microsoft.Scripting.ExtensionAttribute.dll" />
    <AssemblyPart Source="Microsoft.Scripting.Core.dll" />
    <AssemblyPart Source="Microsoft.Scripting.dll" />
    <AssemblyPart Source="IronPython.dll" />
    <AssemblyPart Source="IronPython.Modules.dll" />
    <AssemblyPart x:Name="FarseerPhysics" Source="bin/FarseerPhysics.dll"/>
  </Deployment.Parts>
</Deployment>

This would probably be more interesting compared to a non dynamic language manifest file, but here are a few things I found interesting about it;

There is a whole lot of additional referenced assemblies required to compile Pyhton code in the Silverlight client, and yes, they are included in the XAP sent to the client.
The FarseerPhysics assembly is also included the XAP and is referenced in the manifest.
The EntryPointType is really the only difference between XAP manifest files generated for statically typed languages. For dynamic languages there are additional assemblies that compile the dynamic code, there doesnt seem any reason you couldnt feasibly write EntryPoint assemblies that interpreted lol- cat or brainfuck code.
And, Im still targeting Silverlight 2! (Just havent got round to upgrading..)

Writing an application

I started by having a play with the Farseer engine by writing some IronPython scripts and running them from the console. As its just a physics engine, it can be coupled with any user interface, even a console.

When I tried to get the same script working in Silverlight I did encounter some difficulties referencing the Farseer assembly. I had to use the assemblies strong name, which I found using Reflector, but seems unnecessary given it was sent in the same XAP as the code referencing it!

Instead of just hacking this demo together in one giant script I decided to structure my code a little, including separating the view and the simulation model over different namespaces. Generally Im learning heaps about what is pythonic, what I carry over from C/C++/C# and what is good software design.

Im not entirely happy with my view classes, which are just wrappers around Silverlight controls. Even though the Silverlight control can be accessed through a property, I feel this isnt the elegant solution I should be able to build in a dynamic language. It seemed more right for the physics wrappers, but still overly verbose.

Conclusion

I limited the example for this post to a very simple, but interactive 2d physics demo. Clicking any of the objects in the simulation creates a spring between the object and the mouse cursor until the mouse click is released or leaves the simulation view area.

The Farseer physics engine is a lot of fun to use, its easy to get into and you can do some really cool stuff with it.

The source code is sent as plain text and compiled (or interpreted) on client. As the XAP is just a normal zip file, it can unzipped by almost any zip tool. Inside is all the plain text source code, compiled assemblies and manifest file.

Its awesome to be able to develop Silverlight applications with physics libraries using a high level language and light weight development environment. Happy hacking..

NGourd BDD for WPF Applications

tarn@tarnbarford.net (tarn) — Mon, 08 Jun 2009 10:40:00 +0000

Tarn Barford

I checked out NGourd, a project to build a BDD tool based on Cucumber which Michael Minutillo has started. Id never previously used or even heard of Cucumber, but as I read on I liked a lot about it. Its an agile development tool and it has a colourful console!

The concept is illustrated beautifully at http://cukes.info/, but here is my take anyway. Behavior is described in the form of a series of plain text steps. The steps are then implemented as parameterized test methods. Using some reflection and regular expressions the plain text steps are mapped to corresponding implementations and run as red/green light tests.

I started wondering if WPF applications could be tested with NGourd. Theres a great IronPython sample in which a WPF application is started in a new thread and built dynamically from the IronPython interactive console.

Could something like this could be done in NGourd steps?

Scenario: Can Perform basic arithmetic

Given I start the application
When I enter 2 + 2
And I click GO
Then I should see 4

By using a thread to run the application and a dispatcher to invoke methods on it I had a crude framework to write the steps to make a System.Windows.Application dance. The steps class library has references to the application (to run and test it) and to the NGourd.Core (which has the Attributes I need to decorate the step classes and methods). Ive exposed the controls of application publically to make the test easier to write.

[Steps]
public class Behaviour
{
    public Thread _thread;
    public Dispatcher _dispatcher;
    public App _app;


    [Step("start the application")]
    public void StartApplication()
    {
        AutoResetEvent are = new AutoResetEvent(false);
        _thread = new Thread(() =>
        {
            _app = new App();
            _app.Startup += (sender, eventArgs) =>
                    {
                        _dispatcher =
                            Dispatcher.FromThread(_thread);
                        are.Set();
                    };
            _app.Run();
        });

        _thread.SetApartmentState(ApartmentState.STA);
        _thread.Start();
        are.WaitOne();
    }


    [Step(@"enter (.*)")]
    public void EnterScript(string script)
    {
        _dispatcher.Invoke(
            new Action(() =>
            {
                _app.MainWindow.UserScript.Text = script;

            }));
    }


    ...


    public void After()
    {
        _dispatcher.Invoke(
            new Action(() =>
            {
                _app.MainWindow.Close();
            }
            ));

        _thread.Join();
    }
}

Writing the application

The demo application is a trivial calculator that executes Python statements.

The method below is the model for this application. Its worth mentioning its very easy to include C# objects to the scope of the ScriptSource, but only for this post, Ive restrained myself to the more simple behavior I specified earlier (but I did add it to the demo which can be downloaded here)

public string EvaluateExpression(string expression)
{
    try
    {
        ScriptEngine engine = Python.CreateEngine();
        ScriptRuntime runtime = engine.Runtime;
        ScriptSource source = engine.CreateScriptSourceFromString(
                       expression,
                       SourceCodeKind.Expression);

        return (source.Execute() ?? "").ToString();
    }
    catch(Exception ex)
    {
        return ex.Message;
    }
}

Of course this could be unit tested itself.

[TestMethod]
public void ModelCanDoBasicArithmetic()
{
    Model model = new Model();
    string output = model.EvaluateExpression("2+2");
    Assert.AreEqual("4", output);
}

So what are we testing? I guess its this

public void Execute_Click(object sender, EventArgs e)
{
    output.Text = _model.EvaluateExpression(input.Text);
}

The only thing I had to do the application specifically to get it working in test environment was delete an attribute in App.xaml root element

StartupUri="Window1.xaml"

This has consequences. No window loads on start up! By added a constructor in my App class I was able to remedy this. This may have wider reaching consequences Im not aware of, and Im also not sure if using the OnLoad event is better practice. Anyway its working fine for now.

public partial class App : Application
{
    public App()
    {
        MainWindow w = new MainWindow();
        w.Show();
    }
}

It was fun implementing the steps in my scenario then writing the application and watching the behavior test turn green like a cucumber!

Isolating scenarios in separate application domains

Even though everything looked looks like it was working, I wanted to add another scenario (which wont surprise readers of my recent posts)

Scenario: Can Execute Python statements

Given I start the application
When I enter hello.upper()
And I click GO
Then I should see HELLO

It turns out this extra test didnt need any additional step implementations or changes to the WPF application itself, but did show a very real deficiency in what I was doing. I was reminded of the rule enforced by InvalidOperationException:

Cannot create more than one System.Windows.Application instance in the same AppDomain.

I knew of this rule, I still tried to get round it anyway. Can I close one somehow? Delete one? Delete one and de-reference its friends and family?

Creating a new AppDomain seems a clever way round the rules reasoning, but the wary fear this may be some type of pandora's box, at very least additional complexities. Just thinking of MarshalByRefObject proxies makes me feel I should be spending more time enjoying life with my friends and family.

I got some way using DoCallBack until I realized I couldnt make the steps parameterized as DoCallBack doesnt have any parameters! (I also stubbornly know it throws runtime errors if one tries to use the looser lambda scoping to cheekily pass data across application domains, luckily though its the same stubboness that stopped me giving up this post all together).

As I was thinking I should use WCF for communication betweens application domains, another way I could do to it clicked. I just needed to create my original steps class in a new application domain. By deriving the test class from MarshalByRefObject I could call its step methods from the control domain using it dynamic proxy. I then created a wrapper class which creates a new AppDomain, creates the previous steps in the new domain and uses the dynamic proxy to call the steps. This wrapper class now has the Step attributes for NGourd to pick up.

[Steps]
public class BehaviourWrapper
{
    AppDomain domain;
    Behaviour behaviourProxy;

    public void Before()
    {
        domain = AppDomain.CreateDomain("TestDomain");
        behaviourProxy = domain.CreateInstanceFromAndUnwrap
            ("DLRCalculator.BehaviourTests.dll",
             "DLRCalculator.BehaviourTests.Behaviour")
                as Behaviour;
    }

    public void After()
    {
        behaviourProxy.After();
        AppDomain.Unload(domain);
    }

    [Step("start the application")]
    public void StartApplication()
    {
        behaviourProxy.StartApplication();
    }

    [Step(@"enter (.*)")]
    public void EnterScript(string script)
    {
        behaviourProxy.EnterScript(script);
        Thread.Sleep(5000);
    }

    [Step(@"click GO")]
    public void ClickGo()
    {
        behaviourProxy.ValidateOutput();
    }

    [Step(@"should see (.*)")]
    public void ValidateOutput(string expected)
    {
        behaviourProxy.ValidateOutput(expected);
    }
}

Running some tests with NGourd

Im impressed with NGourd and have generally had a lot of fun and success playing with it. Its a little disappointing that no results are displayed until all the tests are complete. Perhaps its because my tests take ages and it would be nice to see what theyre doing (starting an AppDomain and running an Application in it isnt a lightweight task). Mainly though, I wanted to include an action shot of it firing up windows during the test.

Some thoughts and reflection

I didnt write this to actually use it in a real project, this was a personal project deliberately devised to cover some some things I was interesting in learning about and trying out. To do automation on a real project I would look further into White an open source project released by ThoughtWorks, which I hear is really good.

The concept of Cucumber is pretty cool and I was nice to have a look at the NGourd source code. In writing this post I enjoyed putting in practice what I new could be done relatively easily (creating and managing multiple AppDomains, Applications and Threads in a process) and I was happy to find I had no problems at all with NGourd itself getting the UI tests working.

This is all I planned to do with NGourd, the UI steps and the Calculator, but Id be interested to hear what you think of all this rambling. I know I got a lot out of it, did you? What of the other behavior driven development tools MSpec, NSpec or NBehave? What are your experiences?

Ive upload the project from this post here.

IronPython In Action

tarn@tarnbarford.net (tarn) — Mon, 25 May 2009 10:39:00 +0000

Tarn Barford

While twittering (or tweeting) about an IronPython presentation I was doing, @voidspace messaged me asking if I might mention his book. I'd contacted Michael Foord a while back about some code I was playing around with. I also knew of him from his blog, his prolific contribution to IronPythonURLs and as the co-author of the new book IronPython in Action. He's so all over everything IronPython he even reads this blog.

Mainly because I was too cheap to buy myself a copy it, I hadn't read his book. I was wrapped when he offered to get me a copy if I'd review it on this blog. I wanted to read it as I was learning IronPython and I knew it would be a good book, I'd read chapters 1 and 7 which are online. I happily recommended it in my presentation, here's what I thought about the book.

IronPython in Action is a fantastic resource for anyone learning IronPython and for anyone wondering what this dynamic language IronPython is all about and whether they should learn it. It feels nicely balanced for both .NET users looking to learn about IronPython, and for Python users looking to learn about IronPython on .NET.

I prefer software books that have plenty of discussion and can be read from start to finish otherwise I end up skipping through chapters I never go back too. I found this book well written and fun to read with interesting insight all the way through. There is enough about the python language and .NET to get you through the book, and rather than being a reference book it instead provides information on how to learn more.

The book is divided into four main sections. The first section Getting Started with Iron Python contains a good introduction and history of IronPython on the .NET Framework.

The Core Development Techniques section is a walkthrough of building a simple Windows Forms application. I think this section is fantastic as the text places high value on Agile Development philosophy, using a Model-View- Controller pattern and Test Driven Development. I think these are exciting ways to develop software on the .NET framework. There is also a really good section discussing the difficulties, pitfalls and work-arounds using IronPython and dynamic programming languages generally on the .NET framework.

The third section IronPython and Advanced .NET contains chapters about using the major pillars of the .NET framework with IronPython. It has chapters on WPF, Powershell, ASP.NET, Databases, Webservices and Silverlight. It really is amazing how much of the framework can be used in IronPython.

The final section Reaching Out with IronPython has fantastic chapters on extending IronPython with C#/VB and embedding an IronPython Engine in C#/VB applications to provide extensibility.

I highly recommend this book, I enjoyed reading it and learned a lot from it. If you're looking for a little more insight, the forward by Jim Hugunin frames the language, the technology and author well.

IronPython Presentation

tarn@tarnbarford.net (tarn) — Mon, 11 May 2009 10:25:00 +0000

Tarn Barford

Im doing an introduction to IronPython presentation at the Victoria .NET user group tomorrow, I've been attending the user group for about a year and Im pretty excited about presenting. I think python is a fantastic language and its great to be able to build software on the .NET framework with it.

The Slides

Introduction

I wanted to start with the second slide and have a photo of a pub on the contact slide but decided to get through this before trying out PowerPoint jokes.

Why Python

I just plan to discuss Why I started using Python. The cartoon is, of course, XKCD.

Python

Basic information about the language and its background. Would have liked to have added meta programming or at least .. to the paradigms.

I think its a fantastic language so I'll mention its expressive, fun, productive or whatever adjectives I've got in my head at the time.

People using Python

I really want to demonstrate that Python is real language, being used by real people, to write real code.

I probably should also have mentioned Google App Engine here which I've been tinkering with a bit recently and think is pretty cool.

IronPython

The Python language on .NET. Most of the demo is doing cool stuff in .NET with the Python language. I think it goes further than this and allows developers to use their own paradigms and patterns to do .NET development.

C# and IronPython comparison

I saw this example in the awesome Iron Python in Action book. I included it as its a great introduction IronPython code, as it clearly and simply shows differences on similarities with C#.

Dynamic Language Runtime

The Dynamic Language Runtime project is open source. Its quite well described here as:

IronLisp has been superseded by the IronScheme project.

Dynamic Typing

This slide is just a brief overview of what duck typing is.

It is great for writing test objects, but I have a slide for the later.

Interactive Interpreter

I think its awesomely cool, and I use it through the entire demo so I thought it might be worth warning people.

The Demo

The demo is a series of scripts run into the IronPython Interactive Console. I've uploaded the very small collection of scripts for this demo here.

IronPython and the CLR

Create a string, show how its methods can be reflected. Import the CLR and show how the .NET String methods are now available.

# Introduction to IronPython and the CLR

# Standard first Python console command
2 + 2

# Create a string
s = 'Hello IronPython'

# Inspect its methods, note that its only Python methods.
dir(s)

# Python has a method to make the text uppercase
s.upper()

# Import the CLR
import clr

# Now look at the same objects methods,

# We now see the .NET string methods!
dir(s)

# And can use them!
s.ToUpper()

Screen Shot

Using a .NET Classes

Create a WebClient object from the .NET framework to download an RSS feed. Use an xml2py module written by Harry Pierson to deserialise the RSS feed into an object graph. Then send the dates and titles to the Speech Synthesizer. I've seen this done before in an IronPython presentation, and just thought it was to much fun to leave out.

import clr

# Create a WebClient and download a string of XML
from System.Net import WebClient
w = WebClient()
xml = w.DownloadString('http://feeds.theage.com.au/rssheadlines/technology.xml')

# Lets have a look at first 60 characters of the xml..
print xml[0:60]

# If bad stuff happens in the presentation..

#xml = open('theage.rss').read()

# Deserialize to objects using xml2py

from devhawk import xml2py
rss = xml2py.parseString(xml)

# Print the results
for post in rss.channel.item:
  print post.pubDate, post.title

# Import Speech
clr.AddReference('System.Speech')
from System.Speech import *
ss = Synthesis.SpeechSynthesizer()

# Say titles and dates using the SpeechSynthesizer
for post in rss.channel.item:
  s = "%s. %s." % (post.pubDate[:16], post.title)
  ss.SpeakAsync(s)

Screen Shot

SQL

I really dont want to do this in the presentation. I did spend a fair amount of time playing round with asynchronous SQL queries for use in a WPF application, but it seemed too complicated for the demo so I've left it out. I use

# import a simple wrapper around System.Data.SqlClient
# the wrapper is on my blog, I just wanted to show the AdventureWorks
# database, it seems a bit of .NET presentation tradition.

from demo import sql

# setup a query on the AdventureWords database
query = 'SELECT * FROM Production.Product'

sql.connection = 'Data Source=localhost\SQLEXPRESS;InitialCatalog=AdventureWorks;Integrated Security=SSPI;'

# execute the query
products = sql.SqlQuery(query)

# print some results
for p in products[:5]:
  print p.Name

WPF, Data Binding and Exec

I think there is some pretty cool stuff in this demo. Although it takes a lot more to write real applications it does open new ways to develop and test .NET applications.

# More pythonic way of getting the feed.. lets hope I have internet
from devhawk import xml2py
rss = xml2py.parse('http://feeds.theage.com.au/rssheadlines/technology.xml')

# Importing avalon starts a new thread to run the WPF application
# It sets up some dispatures so the console can talk to the GUI

from samples import avalon

# Create a window, show it and load some XAML into it
w = avalon.Window()
w.Show()
xaml = avalon.LoadXaml('demo.xaml')
w.Content = xaml

# Loading names is really cool, it traverses the control tree
# described in the XAML and creates a dictionary of names and
# controls. We can add those dictionary items to the local scope,
# so we get the same effect as a generated designer files in
# Visual Studio.

avalon.LoadNames(xaml,locals())

# We can bind a Python list of Python objects to the ListView
listView.ItemsSource = rss.channel.item

# Create a button click callback that will execute the script
# in the query TextBox.
def OnClick(*args):
  try:
    exec (query.Text)
  except:
    print "Script threw an exception"
  pass

# Wire up the click event to our function
execute.Click += OnClick

# We can now execute this on the client
listView.ItemsSource = [p for p in rss.channel.item if p.title.find('to')>= 0]

Screen Shot

Debugging IronPython with my Excalibur

tarn@tarnbarford.net (tarn) — Sat, 11 Apr 2009 10:50:00 +0000

Tarn Barford

My interest in a light-weight command-line debugger developed a while back after I watched one used in a Ruby on Rails video. I suspect there are many .Net developers with more experience than me who have used command line debuggers and never want to go back. The more I code, the more I long to leave my mouse behind and communicate with my development tools using only my keyboard, Excalibur.

I have been following a series of blogs by the awesome Harry Pierson (aka DevHawk) on writing an IronPython debugger. When I decided I actually wanted to play with the debugger, I found myself going back over these blogs taking notes as I went. This post is an attempt to summarize all this information and provide the links for going deeper, for myself if no one else.

Writing an IronPython Debugger: Introduction

Explains why he built a debugger when there are other alternatives. The Visual Studios debugging was required to much mouse clicking, although he has posted about doing that too in Debugging IronPython Code in Visual Studio. MDbg which is a .Net Command Line Debugger doesn't support Just My Code debugging. He also notes the reason for not using/porting pdb (The Python Debugger) is due to IronPython not yet implementing settrace.

Writing an IronPython Debugger: MDbg 101

Describes the basics of MDbg. I found it interesting there is also an IronPython extension for MDbg which isn't for debugging IronPython code, but to interact with MDbg using IronPython. Sounds cool, I'll have check it out some time.

Writing an IronPython Debugger: Hello, Debugger!

Goes over the worlds simplest debugger, and it really is pretty straight forward. It just passes the path of ipy.exe and the python code to debug as a parameter into CorDebug (provided by MDbg). CorDebug starts the process (it can be used to debug any .Net process) and provides some events to hook into; OnCreateAppDomain and OnProcessExit. The world simplest debugger basically just prints some text when the AppDonain is starts and stops.

Writing an IronPython Debugger: Setting a Breakpoint

Provides some background information:

ipy.exe produces debug information when the -D parameter is used.
IL generation is dynamic and in memory so the debugger API provides the Symbols (equivalent to .pdb files) as a Stream and has a callback when the Symbols change.
The .Net framework has an API to read and write these symbols a files and MDbg provides a wrapper to read them from a Stream.

Then covers the python code to load the symbols, translate document/line into a function/offset, set a breakpoint. Does not implement user defined breakpoints in the post.

Writing and IronPython Debugger: Adding Interactivity

Covers the basics of how the interactivity will happen, which I've just realized isn't that complicated. The process being debugged just runs until a breakpoint event occurs (or the symbols change or the process completes). Once the process has stopped the the breakpoint the interactivity can occur. It would be nice to be able to set breakpoints while the process is running, but it does keep it more simple for now.

Writing an IronPython Debugger: Dynamic Stack Trace

Background info on MDbg:

Two stack chains in a typical managed app; unmanaged and managed. The debugger for managed code so only the managed chain used
Each contains a collection of stack frames which is the familiar call stack. There are three types of stack frame; IL, native and internal.
Explains Dynamic Methods are usually used created by IronPython but can't be debugged and are implemented as non-dynamic methods when the -D option is set on ipy.exe.

MDbg has a wrapper around the unmanaged metadata API to get method information for displaying the call stack.

The command in ipydbg is used to view the trace from the interactive console is "T".

Writing an IronPython Debugger: Refactoring

Mostly discussion about code changes and refactoring. Add code to automatically generate the MTA apartment state if the -X:MTA argument is not used. Explains the effect of this on the debugger design.

Writing an IronPython Debugger: Stepping Thru Code

Introduces console commands for step (S), step in (I), step out (O)

Writing an IronPython Debugger: Debugging Just My Code

Skipped over this one, and feel I may start skipping over a few where I'm not specially interested in the implementation. I can always refer back to them later if I really need to understand something.

Writing an IronPython Debugger: Showing Source Code

Describes an issue when stepping into a Python function the CLR breaks at some infrastructure code, presumably there to manage Pythons decorators and such. Hence there is no line of user code to display. To resolve this an additional automatic step is added when stepping into a function so it can be mapped to a line of user code.

To avoid multiple hits to the file system source files used to retrieve user code are cached.

Writing an IronPython Debugger: Colorful Console

Describes how he choose to implement colors in the console, while bemoaning the stateful nature of the console foreground colours (ie when you change the foreground colour it will stay that way until you change it back). I'm sure I'll come back to this post when I want to add colours to the console but for now I skipped through it, it doesn't have much to do with writing or using an IronPython debugger. I guess that's why he discusses moving it out of the ipydbg and into its own module in his next post.

Writing an IronPython Debugger: A Little HackerrCleanup

Discusses some issues regarding mapping to the debugger COM object instance. This didn't sound like a whole log of fun and I hope I don't have to come back and fully understand this one.

Writing an IronPython Debugger: Getting Local Variables

Uses the GetLocalVariable(int index) and GetLocalVariablesCount() methods from the MDbg CorFame class. This post finally made me look up what lexical scoping meant, its a term I've heard heaps usually in discussions about compilers that I never really understood. Its fair to say that I still not confident in my understanding of it.

Discusses matching up debug symbols with variable names from the user code. Doesn't actually evaluate the variable in this post. Notes get_locals from the IronPython process emits some locals used internally, these are prefixed with a dollar sign.

Writing an IronPython Debugger: Displaying Values

This post covers some pretty tricky interfaces for dealing with all the different types, which requires a pretty good understand of how the CLR handles variables under the cover. I didn't read into to much detail but will consider revisiting it later as the content is quite interesting and it's good to understand core CLR stuff.

Writing an IronPython Debugger: Command Routing

Discusses how console commands are routed to functions. This really has nothing to do with debugging IronPython but it's interesting the way it's implemented. He starts by implementing a switch using a dictionary of input commands and functions (Python has no C# switch keyword equivalent). He then takes it further by making use of Python decorators to bind commands to functions.

Writing an IronPython Debugger: Getting Arguments

Discusses getting function arguments as locals.

Writing an IronPython Debugger: REPL Console

Discusses implementing a REPL console in the debugger with the IPyDebugProcess object available in the console scope. This is awesome for exploring the API using python, just by reflecting the methods of the process object I realised it would be trivial to add a command to list the source files currently being debugged as it's just a property on the object. Explorative coding using reflection is a really powerful concept in Python.

In the current implementation is a new local scope is created and used by the by the REPL console, it does not yet support executing code in the process being debugged.

Writing an IronPython Debugger: Breakpoint Management

Implements console commands for setting Breakpoints, as the original post was about how it worked and only implemented breaking on the first line of the user code. Introduces multi-key commands for breakpoints. Adds a breakpoint function (B) with sub functions add (A), list (L), enable (E), disable (D).

Here's a screenshot of me debugging a really simple app which might help explain what I've been talking about for this whole post.

I'm pretty amazed by the productivity and brilliance of Harry Pierson, I've learned heaps reading his blogs and I'm really impressed by the progress he's making with IpyDbg. The debugger still has a long way to being a really useful tool, but its coming along very quickly. I think I will have to try using MDbg with the Python extensions, but I really hope to make some use of ipydbg and maybe even find something I can contribute to it.

If your interested in trying out the debugger I recommend checking his blog and the latest version of the project on GitHub.

Code Camp Oz 09

tarn@tarnbarford.net (tarn) — Thu, 09 Apr 2009 10:52:00 +0000

Tarn Barford

My first Code Camp Oz was the best Microsoft event I've attended. It's basically a large user group meeting held over an entire weekend at Charles Stuart University in Wagga Wagga. Its a long way to travel but some cool people from the Australian .Net development community attend and it's a great opportunity to get to know them.

I found the meeting people aspect more interesting than the presentations which was the reason I came anyway, I can watch the Mix video anytime. I met an awesome group of guys from Albury outside a Silverlight 3 presentation. Due to lateness we spent the entire presentation outside chatting which was lots of fun. I didn't realize until I signed into Twitter one of the guys was Matt Hamilton who I knew of from his massive Stack Overflow reputation.

The Alt.Net presentation was one I'd penciled in as interesting before the event and was pleased find I shared common development values and principles with the Alt.Net community. I had a good chat with the presenter Liam McLennan and ended up hanging out with him a fair bit during the weekend.

Saturday night at the pub was good fun. I had an interesting chat about what was involved in becoming a Microsoft MVP and the value of vender certification. It turns out you don't need to have any Microsoft Certification to become an MVP which I think is good. For me personally I think there are other things I'd prefer to do to become a better developer.

An interesting aspect of the event was Twitter. I've never really got that into Twitter, but the constant background commentary and discussion really added to the experience. The informal way you can listen and contribute to conversation is great. It meant I didn't take any notes, but I can always search the #ccoz09 tags for quotes, links and photos from the event.

I think it would be really cool break from the power point presentations and maybe looking at some open source code like the ASP.NET MVC framework at this type of event. I could see this working with a series of presenters reviewing the architecture and highlighting certain areas of code for discussion. I'm not sure if this would actually work, but I think it could be really fun and interesting.

I was a little concerned my return train departed Wagga at 3am Monday morning and arrived in Melbourne at 8am, but I met a couple of guys to who gave me a lift back to Melbourne on Sunday and happily abandoned my 3am train.

Overall it was a really well organized and attended event. I extend my gratitude to Mitch Denny and all the people involved in making it happen.

Hopefully I'll see you all at Code Camp Oz 2010.

Geocoding and Maps without JavaScript

tarn@tarnbarford.net (tarn) — Sun, 15 Mar 2009 11:00:00 +0000

Tarn Barford

Since playing with JQuery I've started thinking more about unobtrusive JavaScript and graceful degradation. Despite less than 5% of users browsing without JavaScript enabled, I found it very interesting disabling JavaScript on my Browser and checking out which sites still work and which ones fail. I found the Microsoft Live Search Maps has absolutely no clue your browsing the site without JavaScript, it just renders a loading spinning GIF.

Without JavaScript Enabled

With JavaScript Enabled

Google maps performs much better, providing a rich client side library with JavaScript enabled and a functional mapping service without it.

Without JavaScript Enabled

With JavaScript Enabled

There are also plenty of sites that lose some or all of their functionality, but let you know why. YouTube can't play videos, but you can browse them and it provides an message indicating why the movies don't work.

Hello, you either have JavaScript turned off or an old version of Adobe's Flash Player. Get the latest Flash player.

StackOverflow has graceful degradation implemented well. You can still ask and answer questions but there is a banner at the top of the page

Stack Overflow works best with JavaScript enabled

You then can't expand comments and you don't get the cool Ajax features that indicate when you've earned a badge or someone else has answered the question your currently answering, but its still pretty functional.

Geocoding without JavaScript

It turns out the good guys at Google have provided a REST API to do geocoding without using their fantastic client side JavaScript libraries. Its all documented in the Geocoding API Developer's Guide.

Google do state this in their terms for the REST geocoding service;

This service is designed for geocoding static (known) addresses using a REST interface, for placement of application content on a map. For dynamic geocoding of user-defined addresses (for example, within a user interface element), consult the documentation for the JavaScript Client Geocoder or the Maps API for Flash Client Geocoder.

I don't think this is a huge problem as a well designed website will only use this geocoding service for the 5% of users who don't have Javascript enabled. If JavaScript is enabled, the DOM can be updated using the rich JavaScript functionality.

I wrote a little script to wrap around the service, I wrote the script in Python as I'm hoping to use it on Google App Engine, but more about that later. To use this script you'll need to sign up to get a Google Maps API key and use it to set the GeoCoder.geoCodeKey property.

import urllib2
from xml.dom import minidom

class GeoCoder():

    def __init__(self):
        self.geoCodeUrl = 'http://maps.google.com/maps/geo?q=%(address)s&output=xml&oe=utf8&sensor=false&key=%(key)s'
        self.geoCodeKey = ''

    def GetUrlKey(self,address):
        return self.geoCodeUrl % {'key':self.geoCodeKey,'address':urllib2.quote(address)}

    def GetKml(self, address):
        response = urllib2.urlopen(self.GetUrlKey(address))
        return response.read();

class KmlParser():

    def GetText(self,node,nodeName):
        return node.getElementsByTagName(nodeName)[0].childNodes[0].data

    def GetCoordinates(self,node):
        items = self.GetText(node, "coordinates").split(',')
        return { 'latitude':items[0], 'longitude':items[1] }

    def ParseAddress(self, node):
        address = { 'address':self.GetText(node,'address'),
                    'country':self.GetText(node,'CountryName'),
                    'street':self.GetText(node,'ThoroughfareName'),
                    'postcode':self.GetText(node,'PostalCodeNumber'),
                    'coordinates': self.GetCoordinates(node) }

        return address

    def Parse(self, xml):
        results = []
        dom = minidom.parseString(xml)

        for placeNode in dom.getElementsByTagName('Placemark'):
            results.append(self.ParseAddress(placeNode))
            dom.unlink()

        return results;

    def GetAddresses(address):
        return KmlParser().Parse(GeoCoder().GetKml(address))

if __name__ == '__main__':

    import sys
    if len(sys.argv) > 1:

    for address in GetAddresses(' '.join(sys.argv[1:])):
        print address
else:
    print "Enter Address to Geocode, using the Google Geocoding API"


from xml.dom import minidom

class GeoCoder():
    def __init__(self):
        self.geoCodeUrl = 'http://maps.google.com/maps/geo?q=%(address)s&output=xml&oe=utf8&sensor=false&key=%(key)s'
        self.geoCodeKey = ''

    def GetUrlKey(self,address):

        return self.geoCodeUrl % {'key':self.geoCodeKey,'address':urllib2.quote(address)}

    def GetKml(self, address):

        response = urllib2.urlopen(self.GetUrlKey(address))
        return response.read();

    class KmlParser():

        def GetText(self,node,nodeName):
        return node.getElementsByTagName(nodeName)[0].childNodes[0].data

    def GetCoordinates(self,node):

        items = self.GetText(node, "coordinates").split(',')
        return { 'latitude':items[0], 'longitude':items[1] }

    def ParseAddress(self, node):

        address = { 'address':self.GetText(node,'address'),
                    'country':self.GetText(node,'CountryName'),
                    'street':self.GetText(node,'ThoroughfareName'),
                    'postcode':self.GetText(node,'PostalCodeNumber'),
                    'coordinates': self.GetCoordinates(node) }
        return address

    def Parse(self, xml):

        results = []
        dom = minidom.parseString(xml)

        for placeNode in dom.getElementsByTagName('Placemark'):
            results.append(self.ParseAddress(placeNode))

        dom.unlink()
        return results;

    def GetAddresses(address):
        return KmlParser().Parse(GeoCoder().GetKml(address))

if __name__ == '__main__':

import sys

if len(sys.argv) > 1:

for address in GetAddresses(' '.join(sys.argv[1:])):

print address

else:

print "Enter Address to Geocode, using the Google Geocoding API"

Which makes geocoding pretty easy from Python

Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> import geocoder
>>> addresses = geocoder.GetAddresses('main st, melbourne, victoria')
>>> for a in addresses: print geocoder.AddressToString(a)
...
Main St, Mordialloc VIC 3195, Australia (145.0858270,-38.0058450)
Main St, Lilydale VIC 3140, Australia (145.3507800,-37.7575460)
Main St, Mornington VIC 3931, Australia (145.0407850,-38.2231010)
Main St, Greensborough VIC 3088, Australia (145.1058060,-37.7025570)
Main St, Diamond Creek VIC 3089, Australia (145.1478460,-37.6744410)
Main St, Thomastown VIC 3074, Australia (144.9994220,-37.6787790)
Main St, Blackburn VIC 3130, Australia (145.1491980,-37.8258150)
Main St, Upwey VIC 3158, Australia (145.3366500,-37.9050420)
Main St, Croydon VIC 3136, Australia (145.2811170,-37.7961170)

Google also note this in the terms;

Geocoding is a time and resource intensive task. Whenever possible, pre- geocode known addresses (using the Geocoding Service described here or another geocoding service), and store your results in a temporary cache of your own design.

Even though the geocoding service is actually quite quick, it does make a lot of sense not to make a call to an external service if its not required. I suggest following this advice.

Maps without JavaScript

Normally some very clever JavaScript dynamically adds new map tile images to the DOM when sections of the map need to be rendered out, so the normal Google Maps service won't work without JavaScript. Luckily Google provide a REST Static Maps service which serves up map images. Its all documented in the Static Maps API Developer's Guide.

I wrote this Python script to get a map URL, again a Google Maps API key is required.

key = 'MAPS_API_KEY'
url = 'http://maps.google.com/staticmap?'
colours = ['brown', 'green', 'purple', 'yellow', 'blue', 'gray', 'orange','red', 'white','black']

def GetImageUrl(params, markers):
    if markers:
        marker = [','.join(p) for p in markers]
        params['markers'] = '|'.join(marker)

    if not params.has_key('key') : params['key'] = key
        return url + '&'.join(['='.join(p) for p in params.items()])

if __name__ == '__main__':

    # Generate map URL with 10 locations in Melbourne Australia
    letters = 'ABCDEFGHI'
    address = [['144.9994220','-37.6787790'],
               ['145.2811170','-37.7961170'],
               ['145.0407850','-38.2231010'],
               ['145.3507800','-37.7575460'],
               ['145.3366500','-37.9050420'],
               ['145.1491980','-37.8258150'],
               ['145.1058060','-37.7025570'],
               ['145.1478460','-37.6744410'],
               ['145.0858270','-38.0058450']]

    markers = [ [e[1],e[0],colours[i]] for i, e in zip(range(len(address)),address)]
    params = { 'size':'256x256', 'maptype':'roadmap', 'sensor':'false','markers': markers }
    print GetImageUrl(params, markers)

Using this script and little more Python hacking I could quickly generate some HTML with a map and links to more detailed maps.

A. Main St, Thomastown VIC 3074, Australia

B. Main St, Croydon VIC 3136, Australia

C. Main St, Mornington VIC 3931, Australia

D. Main St, Lilydale VIC 3140, Australia

E. Main St, Upwey VIC 3158, Australia

F. Main St, Blackburn VIC 3130, Australia

G. Main St, Greensborough VIC 3088, Australia

H. Main St, Diamond Creek VIC 3089, Australia

I. Main St, Mordialloc VIC 3195, Australia

Putting it all together

There is still some consideration to put this all together in website that supports server side mapping and geocoding without JavaScript and enhances the mapping with unobtrusive JavaScript.

Page responses need to return HTML that will work without JavaScript. If the browser executes the pages JavaScript, the script can attach an events and disable default behaviors to implement the richer client side functionality.

On a page a user entered address can be geocoded, the requested HTML should contain a form that can post back to the server to do the geocoding. If client side JavaScript is not used the server can handle the post and use the geocoding service to respond appropriately, of course with JavaScript the addresses can be added to the DOM dynamically, probably with some slide or fade effect.

On pages that display maps, the static image URL can be sent in the requested HTML and used if the JavaScript Google Maps can't be loaded. As on Google Maps some additional controls can be added to allow some panning and zooming without JavaScript.

I'm going to see if I can get the all working on the Google App Engine, I'll let you know how I go.

Will you Django with me?

tarn@tarnbarford.net (tarn) — Fri, 13 Mar 2009 11:02:00 +0000

Tarn Barford

I decided I'd have a look at Django to get a feel for a different web framework and to learn more about Python itself. Its no secret that I've been really interested in IronPython recently and I'm excited to give Python a go without the backing of the .NET framework (but with a complete standard library implementation, unlike IronPython).

For this post I basically just followed the installation guide and the first 4 tutorials in the excellent Django Documentation. Its not my intention to reproduce the tutorials in this post, I really just want to discuss things I found interesting while learning the framework.

Installing

After some fooling around I found the following versions of Python, Django and MySql were the easiest way to get started using Django on my Vista box. All of the versions below have windows installers so its pretty straight forward.

Python 2.5.4 (MySQL Python didn't have an installer for 2.6)

Django 1.0.2

MySql 5.0 (I had problems installing 5.1 on my Vista Box, so I reverted to 5.0 which just worked)

MySql Python Module

I deliberately didn't install the GUI tools for MySql as I wanted to get hardcore from the console. Djangos ORM can be used directly from the Python console and MySql has a complete command line interface.

Getting Started

Django comes with a helper script to assist setting up and managing a site. To do things like create a new web project all you need to do is run following command:

python django-admin.py startproject [sitename]

This creates the skeleton project structure and files.

.
..
manage.py
settings.py
urls.py
__init__.py

You can immediately validate everything is on the right track by running the development server.

python manage.py runserver

I was pretty excited I could just create a new database from the same console window

c:\Working>mysql -u root -p
Enter password: *******
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.0.77-community-nt MySQL Community Edition (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> CREATE DATABASE demodb;
Query OK, 1 row affected (0.03 sec)

All I then needed to do was add my database credentials to the settings.py. Once this was setup I could use the syncdb command to build all the tables required by the framework (which includes an authorization module).

python manage.py syncdb

Which, without me doing anything else, creates the tables.

Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site

Models

As with most MVC web application frameworks, there is a standard project structure convention. Django offers another command to quickly create an app, which is basically a small web application.

python manage.py startapp weblog

This creates a sub folder with the following files empty files

__init__.py
models.py
views.py

Then custom models can be added to the models python file. (Yes, I see that we probably would want more characters for a post, but I just want to get something going for now)

class Post(models.Model):
    content = models.CharField(max_length=200)
    pub_date = models.DateTimeField('date published')
    class Comment(models.Model):
    post = models.ForeignKey(Post)
    comment = models.CharField(max_length=200)

To make the next bit of magic happen, we have to register this class in the main settings python file of this project.

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'DjangoSite.weblog',
)

Now the syncdb command can be used again to create the new tables for our model.

c:\Working\DjangoSite\DjangoSite>python manage.py syncdb
Creating table weblog_post
Creating table weblog_comment
Installing index for weblog.Comment model

We can now start checking out the ORM provided by Django in the interactive Python console.

>>> from DjangoSite.weblog.models import *
>>> import datetime
>>>
>>> post = Post(content = "Django Rocks!", pub_date=datetime.datetime.now())
>>> post.save()
>>>
>>> comment = Comment(post = post, comment="+1")
>>> comment.save()
>>>
>>> findPost = Post.objects.get(id=post.id)
>>> findPost.content
u'Django Rocks!'
>>>
>>> comments = findPost.comment_set.all()
>>> for c in comments:
...     print c.comment
...
>>> findPost.delete()

That's pretty cool, and just 6 lines of code to create the models!

Administration

A few lines need to be uncommented in the urls python file, another line added to the installed apps list and the new table needs to added to the database to enable the default administration features. For the mean time this will serve pages to manage user accounts, later we'll be able to use the administration features to create and a web administration interface for any of our models.

The urlpatterns object is pretty cool. Its basically the URL routing and it uses regular expressions to map requests to handler scripts. More on this before this post is finished.

from django.conf.urls.defaults import *

# Uncomment the next two lines to enable the admin:
from django.contrib import admin

admin.autodiscover()

urlpatterns = patterns('',
    (r'^admin/(.*)', admin.site.root) ,
)

Now if we run the server admin pages can be found from /admin directory of the site.

I won't go into to much detail about how its done but by simply registering a model, web pages are provided to manage that model (finding, adding, deleting and updating). Its all very clever, the framework does lots of introspection to display the correct html controls for each field type.

To separate the UI from the model, a class can be created which is used to guide how the admin framework renders the pages for the model, it is then used to map fields to properties on the model object.

Views

Getting some weblog pages setup is amazingly straight forward. Its just a matter of adding a regular expression into the urlpatterns that matches the type of URL you want to handle and a function to handle it. The most basic handler method is what you'd expect, it takes a http request and returns a http response.

from django.http import HttpResponse

def index(request):
    return HttpResponse("This is the posts page!")

This can be expanded to actually use the weblog models. The code below gets the 5 most recent posts and creates a string with each post content separated with a
tag.

from DjangoSite.weblog.models import Post, Comment
from django.http import HttpResponse

def index(request):
    lastestPosts = Post.objects.all().order_by('-pub_date')[:5]
    output = '<br />'.join([p.content for p in lastestPosts])
    return HttpResponse(output)

Well that's pretty cool, but I don't think we'll be writing entire pages using standard Python. Besides, these look more like controllers than views, there must be more to it.

Templates

With templates, we can throw objects at a template to be rendered. The index method can now be separated from the rendering of HTML

from django.shortcuts import render_to_response
from DjangoSite.weblog.models import Post, Comment
from django.http import HttpResponse

def index(request):
    latestPosts = Post.objects.all().order_by('-pub_date')[:5]
    return render_to_response('weblog/index.html', {'posts': latestPosts})

And the template (weblog/index.html) looks something like this

{% if posts %}
<ul>
{% for post in posts %}
<li>{{ post.content }}</li>
{% endfor %}
</ul>
{% else %}
<p>There are no posts</p>
{% endif %}

It worth noting this isn't just Python embedded in HTML. I think this text from the Django documentation describes this design decision:

..you'll want to bear in mind that the Django template system is not simply Python embedded into HTML. This is by design: the template system is meant to express presentation, not program logic

It makes sense to me.

Forms

A forms post data can be retrieved from a dictionary on the request object.

def index(request):
    postTitle = resquest.POST["postTitle"]
    ...

I'm sure Django has support for mapping posts to model objects, but I haven't seen it yet.

Some more magic

It turns out there is heaps more support in Django for handling generic web problems. For example creating list and details pages is so common that Django has some generic functionality to help. You just need to provide a model and write some templates to describe how to render the models.

It appears there is heaps more of the magic throughout the framework some things I haven't yet looked into are:

Advanced form processing
Using the RSS framework
Using the cache framework
Using the comments framework

Summary

I've just stepped through all the basic steps to building a functional database driven web application. Overall I'm pretty impressed; it didn't take much time, code or leaps in faith to get it going. It appears to do generic things, like what I was doing, is even easier as the framework has generic support for it.

I am glad I got into it from the ground up, I think I would have been a lot more skeptical had I written a simple weblog without writing a line of Python. I've come to learn that its not about the 90% things that are easy to do in a web application framework, its about the 10% of things you want to do but don't work out of the box. Of course I haven't had enough exposure with Django to say how deep the application framework actually goes. I personally feel that Django and Python are a fantastic language and framework to deal with the 10% of requirements that don't come out of the box.

On anther note, I personally love coding from the console rather than an IDE, even with limited exposure I was getting pretty quick navigating my project, running the Django web development helpers and managing MySql. I personally think it's faster developing from a console with all your tools accessible without taking your hands of the keyboard.

Its been lots of fun playing with the framework. I'd definitely like to build a site in Django some time, but I'm not sure when I'll find the time.

Executing IronPython in, err.. IronPython

tarn@tarnbarford.net (tarn) — Fri, 06 Mar 2009 10:57:00 +0000

Tarn Barford

I just read the latest post from secretGeek, a brilliantly funny blog which often makes my day with posts like IT Industry Revolutionised By Labour Saving Device and Defensive Programming.

The latest post, A 3 minute guide to embedding IronPython in a C# application, is a demo of a Windows application that can execute an IronPython script from a TextBox at runtime. Another TextBox control on the form is added to the IronPython script scope so the script can interact with it. This is both cool and powerful, the concept has been used in an IronPython powered spreadsheet from Revolver Systems.

I thought it would be kind of fun to create an IronPython script that created a Windows Form which could execute a script. This means I could run the script and get window which I could then use to run the script again and get anther window, then..

I posted the script in the comments and then decided it was pretty fun and it would be worth writing a post about myself. Below is the Form class script:

import clr
clr.AddReferenceByPartialName("System.Windows.Forms")
clr.AddReference('IronPython')
clr.AddReference('Microsoft.Scripting')

from System.Windows.Forms import *
from IronPython.Hosting import Python
from Microsoft.Scripting import SourceCodeKind

class MetaNote(Form):

    def __init__(self):
    self.button = Button()
    self.code = TextBox()
    self.text = TextBox()
    self.Width = 420
    self.Height = 400
    self.code.Multiline = True
    self.code.Height = 300
    self.code.Width = 300;
    self.button.Left = 310
    self.button.Text = "Execute"

    self.Controls.Add(self.button)
    self.Controls.Add(self.code)
    self.button.Click += self.run

def run(self,sender,e):

    engine = Python.CreateEngine()
    source = engine.CreateScriptSourceFromString(self.code.Text,SourceCodeKind.Statements)
    scope = engine.CreateScope()
    scope.SetVariable("txt", self.text);
    source.Execute(scope)

The only trick is, to start the application from the IronPython console (ipy.exe) you need to include this line at the end which creates the class and creates a standard application message loop.

Application.Run(MetaNote())

When the script is executed from within the application itself, you only need to create in instance of the class and call its Show method.

MetaNote().Show()

Then the fun..

Get in the Queue, your locks don't work here

tarn@tarnbarford.net (tarn) — Thu, 26 Feb 2009 01:21:00 +0000

Tarn Barford

I am, of course, talking about a pattern in high scale web application design, Queues. More specifically the Queue service on the Windows Azure cloud computing platform.

So when do you use Queues?

Its common to use thread synchronization objects in a web application to serialize access to a resource, but this thread synchronization approach doesn't work so well when you have multiple applications, over multiple servers competing for exclusive access to a global resource.

The first scenario I've found which suits the Queue service is in a user driven content ranking system. I suspect I'll find many more places to use the Queue services as I continue to learn more about developing and designing applications on the cloud platform.

In the web application I'm building I want users to be able to rate dynamic content and I also want to record the number of views. This is a very common concept which allows the users to provide feedback which can then be shared with other users and used to dynamically rank the content.

I can start by implementing some tables, one to store views and one to store votes. All I need to do is insert a row into the corresponding table for every view or vote. I can then query the tables to get all the views and votes for each content item. While this will work, it won't scale. Imagine querying a YouTube video view table! It's certainly not the sort of task you could do while the user is waiting for a page to render.

You may be able to resolve this problem with a background process, using some thread locks, to periodically calculate and cache this information on each web server. While I may end up using some caching on each web server, storing all the pages and their counts on each web server may not be feasible. Another problem with this is all the web servers are doing the same data analysis, which doesn't seem a efficient use of resources.

A nice solution for this type of problem is to use the Queue service and a table for the summarized data. Instead of adding page views and votes to the original tables, they can be added to a queue. A worker role can then be implemented to periodically check this queue, potentially doing some validation (or other work) on each item, and then update the reference table. This could be scaled even further, if required, by having multiple worker roles processing items from queue.

A not entirely desirable effect of using this pattern is that votes and pages views are not immediately registered when a user votes or visits a page. I guess this is more desirable than your users not being able to view pages when a single server web server reaches capacity or users getting page time-outs as you process billions of table rows while they wait.

Continuing a theme in my recent posts, I'm going to have a play with the service using IronPython.

I've been using this little helper module which just imports the StorageClient assembly and makes it easy to create development StorageAccountInfo objects as the have the development credentials as default values.

import clr
clr.AddReference("StorageClient.dll")
from Microsoft.Samples.ServiceHosting.StorageClient import *
from System import Uri

class Account():

   def __init__(self):
      self.endPointUri = Uri("http://127.0.0.1:10002/")
      self.accountName = 'devstoreaccount1'
      self.accountSharedKey = '...'

   def GetStorageAccountInfo(self):
      return StorageAccountInfo(self.endPointUri, None, self.accountName,self.accountSharedKey)

class BlobAccount(Account):

   def __init__(self):
      Account.__init__(self)
      self.endPointUri = Uri("http://127.0.0.1:10000/")

class QueueAccount(Account):

   def __init__(self):
      Account.__init__(self)
      self.endPointUri = Uri("http://127.0.0.1:10001/")

It's then pretty easy to create a queue and add an item to it

from AzureQueueHelper import *

# create a connection to the development storage service
queueStorage = QueueStorage.Create(QueueAccount().GetStorageAccountInfo())

# create a queue by name
messageQueue = queueStorage.GetQueue("test1")

# this will create the queue if it does't exist, but will do nothing if it already exists
messageQueue.CreateQueue()

# create and send a message
msg = Message("testString")
messageQueue.PutMessage(msg)

Then get it back in another process

from AzureQueueHelper import *

# create a connection to the development storage service
queueStorage = QueueStorage.Create(QueueAccount().GetStorageAccountInfo())

# create a queue by name
messageQueue = queueStorage.GetQueue("test1")

# this will create the queue if it does't exist, but will do nothing if it already exists
messageQueue.CreateQueue()

# the is the time you have exclusive access to the queue item
msg = messageQueue.GetMessage(5)

if msg:

   print msg.ContentAsString()

   # message must be explicitly removed from the queue
   messageQueue.DeleteMessage(msg)

The asynchronous methods can be used to create a simple little event based queue processing server.

from AzureQueueHelper import *
from System.Threading import *

# auto reset event to keep it alive
are = AutoResetEvent(False)

# create a connection to the development storage service
queueStorage = QueueStorage.Create(QueueAccount().GetStorageAccountInfo())

# create a queue by name
messageQueue = queueStorage.GetQueue("test1")

# this will create the queue if it does't exist, but will do nothing if it already exists
messageQueue.CreateQueue()

# handle message from the queue
def MessageReceieved(sender, e):
   print e.Message.ContentAsString()
   messageQueue.DeleteMessage(e.Message)

# setup polling, should keep reading untill the queue is empty, then poll for 5 seconds
messageQueue.MessageReceived += MessageReceieved
messageQueue.PollInterval = 10000;
messageQueue.StartReceiving();

# repeat until are is set (forever in this example)
are.WaitOne()

Its really amazing how little is required to extended this little server to the worker process discussed in this post using just the table services I've previously posted about.

Import AntiGravity and I'll see you on Cloud Azure

tarn@tarnbarford.net (tarn) — Sat, 21 Feb 2009 00:03:00 +0000

Tarn Barford

On the topic of setting up tables on Windows Azure Steve Marx writes

Probably the best solution is to have separate initialization code that creates your tables. This is analogous to the pattern of having CREATE TABLE commands scripted in T-SQL which you run once to set up the database.

In another post Mark Seemann extends on this and demonstrates creating tables with a PowerShell script and writes

You could obviously write a little utility that references StorageClient and your custom TableStorageDataServiceContext.

Another, in my opinion, better option for such a one-off script is a PowerShell script

I agree but I think you can actually take this a lot further. In my experience working on enterprise applications it is also common to use one-off SQL scripts to add, update, maintain and manage data. The problem with this is you often have a very good ORM available which has logic to protect the state of the data and the business requirements. I feel that if you can also expose these business objects to a scripting environment, you potentially have a very powerful way of managing your enterprise information. This is probably even more relevant in the cloud data services where there is no equivalent of low level T-SQL you can optimize, you can have full control from scripting environment with the tools and armour of your enterprise objects.

Anyway, enough rambling. In my previous post I demonstrated working with Azure table storage data in IronPython, and briefly glossed over creating the tables. In this post I'm going to walk through dynamically creating table in the cloud with IronPython and what currently needs to be done to get them working on the local development storage server.

I'm going to create some of my own customs tables and some tables for the standard providers (Membership, Roles and Session) as I am building a simple MVC application for the cloud. I've got DataModels.dll with my custom models and the AspProviders.dll from the Azure SDK Samples with the standard provider models. All the models extend TableStorageEntity.

Both assemblies also extend TableStorageDataServiceContext classes which are similar to the DataContext class in Linq2Sql. They have IQueryable fields templated to model classes, which represent tables. The base class has additional functionality for tracking, adding, deleting and saving model objects. We will be able to create tables by reflecting these TableStorageDataServiceContext classes.

Creating a database the local development storage server

Unfortunately a current restriction of the local development storage server is that you can't dynamically create tables. To create tables on the local development server the DevTableGen.exe tool is used.

DevTableGen /database:TarnsDevDB /forceCreateAspProviders.dll;DataModels.dll

Which should produce an output like below, indicating that the tables had been created. You can see the provider model tables and my custom tables have been created.

Windows(R) Azure(TM) Development Table database generation tool version 1.0.0.0 
for Microsoft(R) .NET Framework 3.5 Copyright (c) Microsoft Corporation. All rights reserved.

DevTableGen : Generating database 'TarnsDevDB'
DevTableGen : Generating table 'Membership' for type 'Microsoft.Samples.ServiceHosting.AspProviders.MembershipRow'
DevTableGen : Generating table 'Roles' for type 'Microsoft.Samples.ServiceHosting.AspProviders.RoleRow'
DevTableGen : Generating table 'Sessions' for type 'Microsoft.Samples.ServiceHosting.AspProviders.SessionRow'
DevTableGen : Generating table 'PostTable' for type 'DataModels.PostDataModel'
DevTableGen : Generating table 'CommentTable' for type 'DataModels.PostDataModel

UI for development storage server should be found in the task bar. As it only supports one concurrent database, you may have to select the database just created.

Dynamically managing a database in the cloud

I created a simple IronPython helper module to assist creating and managing the table schema in the cloud. Like in the previous post, it's built on the StorageClient library that ships with the Windows Azure SDK. You'll notice that its expecting the StorageClient assembly to be in one of its script paths.

There was some frustration writing it; All the TableStorageDataServiceContext classes are internal, this meant I had to write the LoadFromAssembly method and do some reflection to get the types.

import clr
clr.AddReference("StorageClient.dll")

from Microsoft.Samples.ServiceHosting.StorageClient import *
from System.Reflection import Assembly
from System import Uri

class Account():

   def __init__(self):
      self.endPointUri = Uri("http://127.0.0.1:10002/")
      self.accountName = 'devstoreaccount1'
      self.accountSharedKey = 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=='

class TableHelper():

   def __init__(self, account):
      self.account = account
      self.storageInfo = StorageAccountInfo(account.endPointUri, None, account.accountName, account.accountSharedKey)
      self.tableStorage = TableStorage.Create(self.storageInfo)

   def AddTables(self, dataServiceContext):
      TableStorage.CreateTablesFromModel(dataServiceContext,self.storageInfo)

   def DeleteAllTables(self):
      for t in self.tableStorage.ListTables():
         self.tableStorage.DeleteTable(t)

   def PrintTables(self):
      for t in self.tableStorage.ListTables():
         print t

   def LoadFromAssembly(self,assemblyName):
      # A bit of screwing round with reflection, because the types we want are internal
      assembly = Assembly.LoadFrom(assemblyName)
      baseType = TableStorageDataServiceContext
      contexts = filter(lambda t : t.BaseType == baseType, assembly.GetTypes())

      for context in contexts:
         print context
         self.AddTables(context)

Using this module we can easily create and manage tables from the IronPython interactive console, or with a script. Below is an example (with the credentials removed)

# load the helper module
from AzureTableHelper import *

# create an account and set creadentials
account = Account()
account.endPointUri = Uri('[Table Storage Url]')
account.accountSharedKey = '[Shared Key]'
account.accountName = '[Account Name]'

# create the helper with the account details
tableHelper = TableHelper(account)

# delete tables
tableHelper.DeleteAllTables()

# load tables from the AspProviders assembly
tableHelper.LoadFromAssembly("AspProviders.dll")

# load tables from the DataModels assembly
tableHelper.LoadFromAssembly("DataModels.dll")

# print all the tables to the console
tableHelper.PrintTables()

There is a limitation of scripting tables this way as Mark Seemann notes,

Currently, the script has one limitation: Deleting a table using the StorageClient API only marks the table for deletion, so the operation returns much to soon. This means that if you are trying to recreate a table by the same name, a conflict will occur, and the table will not be created. You can work around this limitation by waiting a little while and then run the script again.

I think this part of the Azure framework is pretty exciting and I'm looking forward to checking out other parts.

Azure Table Storage in IronPython

tarn@tarnbarford.net (tarn) — Fri, 20 Feb 2009 14:14:00 +0000

Tarn Barford

My goal was to write a bit of scaffolding to make using the storage service fun and easy from in IronPython. The Windows Azure SDK comes with a library in the samples which does the low level work interfacing with the API and provides some nice classes to work with. There are methods to generate table schemas by reflecting on model classes and another sample implements all the standard .NET providers.

I was hoping to write all the scaffolding and the model classes in IronPython but, in the first of a series of set backs, I found the development storage server behaves differently than the cloud. For some reason you need to create tables on the development storage server using a command line tool, passing your model assemblies as arguments. Apparently this will be fixed soon, but trying to stay focused I decided I'd have to write my models in C# for now.

Once I had an assembly with some models I could use the DevTableGen.exe command line tool that comes with the Azure SDK to create tables on my development storage server.

I don't think there are currently any good tools for visualizing and editing data, but I'm sure by the time its released it will integrate into Server Explorer and Query Analyzer (or perhaps it'll just be a Firefox plug-in). I've seen a presentation where HTTP requests are hand coded to retrieve data, but I wasn't up for that and tried importing the sample library into an IronPython console. I got enough working to convince me everything was going to work out, I just needed find out how to use extension methods in IronPython...

Much to my surprise and disappointment, consuming extension methods in IronPython is still difficult. There does appear to be a way to bind the extension methods, but I haven't seen an example binding all the Linq extension methods. This is a problem as it means Linq expression trees can't be easily built in IronPython. Passing expression trees as queries is ideal as the cloud can do the filtering, sorting and only return the data you want.

I decided to put this problem on ice and write helpers in C# that returned a List of all the rows. This won't work well with lots of data, but it will work find on small tables.

I wrapped the required credentials to connect to the service in an Account class, mainly so I could hard code the development credentials which are always the same.

public class Account
{
    public Uri EndPoint { get; set; }
    public String AccountName { get; set; }
    public String SharedKey { get; set; }

    public Account()
    {
        // development server defaults
        EndPoint = new Uri("http://127.0.0.1:10002/");
        AccountName = "devstoreaccount1";
        SharedKey = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UV
    }

}

I later I will try creating Models on the DLR, but for now I created them in a separate assembly using C#.

namespace DataModels
{
    public class PostDataModel : TableStorageEntity
    {
        public PostDataModel(string partitionKey, string rowKey)
            : base(partitionKey, rowKey)
        {
        }

        public PostDataModel()
            : base()
        {
            PartitionKey = Guid.NewGuid().ToString();
            RowKey = String.Empty;
        }

        public string Name
        {
            get;
            set;
        }

        public string Content
        {
            get;
            set;
        }

    }
}

A context is required that derives from TableStorageDataServiceContext, the field names in the context are used for table names and the type of the field describes the table row. The DataServiceContext works a bit like the LinqToSql data context keeping track of all the objects it returns.

namespace DataModels
{
    public class DataServiceContext : TableStorageDataServiceContext
    {
        public DataServiceContext(StorageAccountInfo accountInfo)
            : base(accountInfo)
        {
        }

        public IQueryable<PostDataModel> PostTable
        {
            get
            {
                return this.CreateQuery<PostDataModel>("PostTable");
            }
        }

        public IQueryable<PostDataModel> CommentTable
        {
            get
            {
                return this.CreateQuery<PostDataModel>("CommentTable");
            }
        }
    }
}

I wrote a generic model class to hide all the details and provide a simple wrapper to access create, read and select operations on a table. There's a couple of tests in the solution that demonstrate how this work, but basically you just need a custom context (V), a model (T) and table name. The generic model can then be used to insert, select and delete from the table.

public class Model<T, V>
    where V : TableStorageDataServiceContext
{
    private V _context;
    public StorageAccountInfo AccountInfo { get; set; }
    public TableStorage TableStorage { get; set; }
    public string TableName { get; set; }

    public Model(Account account, string tableName)
    {
        TableName = tableName;
        AccountInfo = new StorageAccountInfo(account.EndPoint, null, account.AccountName, account.SharedKey);
        TableStorage = TableStorage.Create(AccountInfo);
         _context = Activator.CreateInstance(typeof(V), new object[] { AccountInfo } ) as V;
    }


    public List<T> Select()
    {
        MethodInfo field = _context.GetType().GetMethods().Where(f => f.Name == "get_" + TableName).FirstOrDefault();

        if (field == null) return null; // field doesn't exist
        IQueryable<T> fieldValue = (field.Invoke(_context, null) as DataServiceQuery<T>);
        var results = from c in fieldValue select c;
        TableStorageDataServiceQuery<T> query = new TableStorageDataServiceQuery<T>(results as DataServiceQuery<T>);
        IEnumerable<T> queryResults = query.ExecuteAllWithRetries();
        return queryResults.ToList();
    }


    public void Insert(object item)
    {
        _context.AddObject(TableName, item);
        _context.SaveChanges();
    }


    public void Delete(object item)
    {
        // AttachTo is not required if the item was created by the context
        _context.DeleteObject(item);
        _context.SaveChanges();
    }
}

We've done all the C# code, now lets see how it can all be tied together in an IronPython console.

IronPython 2.0 (2.0.0.0) on .NET 2.0.50727.3053
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> # import everything we need
>>>
>>> import clr
>>> clr.AddReference("DataModels.dll")
>>> clr.AddReference("DataHelper.dll")
>>> clr.AddReference("StorageClient.dll")
>>> from DataModels import *
>>> from DataHelper import *
>>> from Microsoft.Samples.ServiceHosting.StorageClient import *
>>>
>>> # create a generic model, using the default account (development server)
>>>
>>> model = Model[PostDataModel, DataServiceContext](Account(),"PostTable")
>>>
>>> # now we can add some data rows
>>>
>>> for i in range(5):
...    post = PostDataModel()
...    post.Name = "Post Name " + str(i)
...    post.Content = "Some content for post" + str(i)
...    model.Insert(post)
...
>>>
>>> # we can now read them back
>>>
>>> for post in model.Select():
...    print "Name", post.Name
...
Name Post Name 3
Name Post Name 1
Name Post Name 2
Name Post Name 0
Name Post Name 4
>>>
>>> # delete them all
>>>
>>> for post in model.Select():
...    model.Delete(post)
...
>>>
>>> # and finally ensure they have all been removed
>>>
>>> for post in model.Select():
...    print "Name", post.Name
...
>>>
>>>

I'm pretty excited with how it all worked out, despite the setbacks. In future posts hopefully I'll have a go creating tables in the cloud with models and context created on the DLR. I will also try and resolve using Linq Extension Methods in IronPython which are essential to building complex queries to be executed in the cloud. I'm also writing a very simple Azure MVC weblog app which I'll hopefully finish soon.

The project can be downloaded here and I've also uploaded just the compiled DataHelper assembly.

I found these links useful writing this post:

Windows Azure Essential Links

Walkthrough: Simple Table Storage

Creating Azure Tables From Script

Silverlight Dynamic Languages SDK

tarn@tarnbarford.net (tarn) — Mon, 26 Jan 2009 15:12:00 +0000

Tarn Barford

I decided to have a look at creating Silverlight applications with IronPython, this led me to the Silverlight Dynamic Languages SDK. The main piece of the SDK is Chiron which is a development utility for creating Silverlight application packages (XAPs) from dynamic languages.

Chiron can be used to:

Create dynamic language Silverlight projects
Start a local web server which dynamically generates XAPs from the project files
Write generated XAPs to disk.

I needed something to build on Silverlight with IronPython, so I decided to write a small part of the UI for an RSS Feed Visualizer I've been wanting to write. Its only really a demo, the UI could use some more work and I've just used some sample names as data. I hope to build on this example in the future, but I'll need to do some refactoring, the code is pretty scrappy as I wrote most of it after I lost a house poker game. Apparently a stronger understanding of counter intuitive probability problems hasn't helped my game and I don't write my best code after an afternoon of drinking.

(If there is problems viewing the demo on this page, it can also be viewed here)

Here is the IronPython code that drives the demo, I've also uploaded the entire project. I used examples of dragging and inertia from the excellent Project "Rosetta Stone" Tutorials.

from System.Windows import Application
from System.Windows.Controls import UserControl
from System.Windows.Controls import TextBlock
from System.Windows.Controls import Canvas
from System.Windows.Media import Colors
from System.Windows.Media import SolidColorBrush
from System.Windows import Visibility
from System.Windows import FontWeights
from System.Windows import HorizontalAlignment
from System.Windows.Media.Animation import Storyboard
from System import Random;
import math

class App:

  def __init__(self):
    self.root = Application.Current.LoadRootVisual(UserControl(),"app.xaml");
    spinner = Spinner(self.root,['Tarn','Leah','Alex','Mick','John','Rex','Jack','Jill','Elle']);


class Spinner():

  def __init__(self, root, names):
    self.root = root;
    self.names = names
    self.radius = 0;
    self.textBlocks = [];
    self.mainText = Block(self.root.layout_root,self.centreClick);
    self.mainText.SetPosition(100.0,100.0);
    self.mainText.HorizontalAlignment = HorizontalAlignment.Center;
    self.selectedItem = None;
    self.angle = 0;
    self.diff = 0;
    self.target = 0;
    self.drag = False;
    self.createTextBlocks(self.names[0], self.names);
    self.root.MouseLeftButtonDown += self.leftButtonDown;
    self.root.MouseLeftButtonUp += self.leftButtonUp;
    self.root.MouseMove += self.mouseMove;
    self.sb = Storyboard();
    self.root.Resources.Add("sb",self.sb);
    self.sb.Completed += self.enterFrame;
    self.sb.Begin();

  def centreClick(self,item):
    self.selectedItem = item;
    self.state = "Selected";
    pass;

  def createTextBlocks(self, item, list):
    self.state = "Expanding";
    self.mainText.text.Opacity = 1;
    self.mainText.text.Text = item;
    self.velocity = 0.01;

    for textBlock in self.textBlocks:
      textBlock.dispose();

    textBlocks = [];
    i = 0;
    list = filter(lambda m: (m != item), list)

    for title in list:
      textBox = Block(self.root.layout_root,self.centreClick);
      textBox.text.Text = title;
      textBox.angle = i * 2 * math.pi / list.__len__();
      textBox.HorizontalAlignment = HorizontalAlignment.Center;
      textBox.radius = -(100/list.__len__()) * i;
      textBox.SetPosition2(0);
      self.textBlocks.append(textBox);
      i += 1;

  def enterFrame(self, object, e):
    diff = ((self.target - self.diff)-self.angle) ;
    if (self.drag == True):
      oldAngle = self.angle;
      if (diff > math.pi): diff = diff - math.pi * 2;
      if (diff <= -math.pi): diff = diff + math.pi * 2;
      self.angle  = self.angle + ((diff) * 0.2);
      self.velocity = self.angle - oldAngle;
    else:
      self.angle = self.angle + self.velocity;
      if (math.fabs(self.velocity) > 0.01):
        self.velocity = self.velocity * 0.90;

   self.angle = self.angle % (2 * math.pi);

    if (self.state == "Expanding"):
      expandComplete = True;
      for textBlock in self.textBlocks:
        if (textBlock.Expand(100) == False): expandComplete = False;

    if (self.state == "Selected"):
      expandComplete = True;
      if (self.selectedItem != self.mainText):
        self.mainText.text.Opacity -= 0.10;
        self.selectedItem.moveTo(100,100);
      for textBlock in self.textBlocks:
        if (self.selectedItem != textBlock):
          textBlock.text.Opacity -= 0.010;
          if (textBlock.Expand(200) == False): expandComplete = False;
      if (expandComplete == True):
        self.createTextBlocks(self.selectedItem.text.Text, self.names);
    for textBlock in self.textBlocks:
      if (self.selectedItem != textBlock): textBlock.SetPosition2(self.angle);
    self.sb.Begin();

  def leftButtonUp(self,object,e):
    self.root.ReleaseMouseCapture();
    self.drag = False;

  def mouseMove(self, object, e):
    if (self.drag == True):
        self.target = self.getAngle(e);

  def leftButtonDown(self, object, e):
      self.root.CaptureMouse();
      angle = self.getAngle(e);
      self.diff = angle - self.angle;
      self.target = self.getAngle(e);
      self.drag = True;

  def getAngle(self, e):
    x = e.GetPosition(self.root).X-100;
    y = e.GetPosition(self.root).Y-100;
    angle = (math.atan2(x,y));
    if (angle < 0): angle = angle + (math.pi*2);
    return angle;

class Block():
  def __init__(self, canvas, clicked):
    self.clicked = clicked;
    self.canvas = canvas;
    self.text = TextBlock();
    self.text.Foreground = SolidColorBrush(Colors.White);
    self.offset = 100;
    self.radius = 10;
    self.angle = 0;
    canvas.Children.Add(self.text);
    self.text.MouseEnter += self.mouseOver;
    self.text.MouseLeave += self.mouseOut;
    self.text.MouseLeftButtonDown += self.mouseClick;

  def dispose(self):
    self.canvas.Children.Remove(self.text);

  def moveTo(self, x, y):
    self.text.FontWeight = FontWeights.Normal;
    currentX = self.text.GetValue(Canvas.LeftProperty);
    currentY = self.text.GetValue(Canvas.TopProperty);
    newX = currentX + ( (x - currentX) * 0.1);
    newY = currentY + ( (y - currentY) * 0.1);
    self.SetPosition(newX,newY);

  def Expand(self, max):
    if (self.radius < max):
      self.radius += 1;
      return False;

  def mouseClick(self,sender,e):
    self.clicked(self);

  def mouseOver(self,sender,e):
    self.text.FontWeight = FontWeights.Bold;

  def mouseOut(self,sender,e):
    self.text.FontWeight = FontWeights.Normal;

  def SetPosition(self, x, y):
    self.text.SetValue(Canvas.LeftProperty, x);
    self.text.SetValue(Canvas.TopProperty, y);

  def SetPosition2(self, angle):
    if (self.radius < 20):self.text.Visibility = Visibility.Collapsed;
    else:self.text.Visibility=Visibility.Visible;
    self.SetPosition( (math.sin(self.angle + angle) * self.radius) + self.offset, (math.cos(self.angle + angle) * self.radius) + self.offset);

App()

I noticed when I uploaded the XAP that it was quite large (> 1Mb) for a very small demo. I think this is because additional assemblies for dynamic languages and IronPython are not included in the client Silverlight plug-in and have to be included in the XAP.

Goats, Cars and Babies

tarn@tarnbarford.net (tarn) — Sun, 18 Jan 2009 14:36:00 +0000

Tarn Barford

I watched the movie "21" the other night and although I thought it was rubbish, I was slightly intrigued in the explanation to a question posed in the movie, which turns out to be the Monty Hall Problem.

I was surprised then, while catching up on feeds, to see Jeff Atwoods post The Problem of the Unfinished Game on a similar topic. Jeff asks

Let's say, hypothetically speaking, you met someone who told you they had two children, and one of them is a girl. What are the odds that person has a boy and a girl?

It turns out that I've always hated this type of math despite generally enjoying using math to solve engineering problems, having a pretty good understanding of No-Limit Texas Hold'em and perpetually trying to crack all those math tricks that make you do a series of seeded numerical operations to magically reveal some significant number. For me the probability math in quantum physics was bewildering and in statistical process control it was incredibly boring as well.

Asked this question last week and I would probably have meekly replied, "Umm, maybe 50%?" and mumbled something about "stupid probability questions" then maybe attempted to draw out some permutations before giving up with absolutely no confidence in my answer.

Having no confidence in my answer may have been wise in that case, but in his following post about the answer to a similar problem, Jeff made a point that resonated with me:

You don't need to be a mathematician to prove this. I'm just a crappy programmer, and even my crappy code can brute force the answer by simulating results from thousands of games.

As soon as I read that I decided I was going to give the Monty Hall Problem the same treatment, at that stage I'd only heard the explanation in the movie which didn't make much sense to me.

import random;

class GameShow:

  def __init__(self):
    self.prizeDoor = random.randint(1,3);
    print "\nCreating Game, Prize Door is ", self.prizeDoor;

  def SelectDoor(self, door):
    self.firstSelection = door;
    print "Contestant Selects Door", self.firstSelection;

  def HostOpenDoor(self):
    options = filter(lambda m: (m != self.prizeDoor) and (m !=self.firstSelection), range(1,4))
    self.hostOffer = options[random.randint(0, options.__len__()-1)];
    print "Host opens door", self.hostOffer;

  def ChangeDoor(self, swap):
    self.swap = swap;
    print "Contestant", self.swap and "Takes Swap" or "Keeps Original Choice";

  def OpenDoor(self):
    self.alternativeDoor = filter(lambda m: (m != self.hostOffer) and (m!= self.firstSelection), range(1,4))[0];
    self.selectedDoor = (self.swap and self.alternativeDoor or self.firstSelection);
    self.wins = (self.selectedDoor == self.prizeDoor);

    print "Contestant Selects", self.selectedDoor, "and", self.wins and "Wins Prize" or "Looses";

def RunGame(swap):
  game = GameShow();
  game.SelectDoor(random.randint(1,3));
  game.HostOpenDoor();
  game.ChangeDoor(swap);
  game.OpenDoor();
  return game.wins;

def RepeatRun(swap, repeats):
  wins = 0;
  for i in range(repeats):
    if (RunGame(swap)): wins += 1;

  return wins;

if __name__ == "__main__":
  iterations = 1000;

  swapWins = RepeatRun(True, iterations);
  stayWins = RepeatRun(False, iterations);

  print "\nOverall Results\n";
  print "Swap Wins",swapWins/float(iterations),"%";
  print "Stay Wins",stayWins/float(iterations),"%";

After some fooling around with Python I got my brute force results that proved to me making the swap was the right choice. Actually I'd proved it to myself by the time I'd written the code that works out which doors the host can open. It turns out that if you select a door with a goat behind it first, the host can only open one door to reveal a goat, the other is the car. And that is the key; If your first selection is a goat which is a 2/3 chance, then the switch will always yield a car (this is better than the 1/3 chance first selection).

I've satisfied my interest in counter intuitive probability problems for now, which is good because my friends are sick of me talking about them.

IronPython Asynchronous RSS Reader

tarn@tarnbarford.net (tarn) — Tue, 13 Jan 2009 11:25:00 +0000

Tarn Barford

Reading RSS is a common and simple task in Python using the standard library, unfortunately it uses parts of the standard library that are not implemented in IronPython. Luckily it isn't very difficult using the .NET framework either. There is an example Parsing RSS on the IronPython Cookbook site that demonstrates it.

While this example was useful, I wanted to make the web request asynchronously so I could use it in a WPF application without blocking the UI. I was happy to find implementing it was quite familiar and easy, I've used the WebClient object asynchronously many times in C# and I've passed event handlers as parameters in Javascript, both of which are supported by IronPython and are used in this module.

In the implementation below I've used all XML processing code from the Parsing RSS example. I removed the class RSSFeedSubscriptions as I didn't need the functionality and I felt it made the sample more clear. I split the RSSFeedFetcher into an RSSFeed class for the feed data and an RSSFeedReader class for handling the downloading and XML processing. I think it is much a better separation of functionality.

I found the the exception handling code in the original sample that outputs an exception message was itself throwing an exception, so I removed it. I don't think its ideal effectively suppressing exceptions and printing a message but its good enough for what I'm using it for. If anyone has a suggestion or I improve the module myself, I'll update this post.

The example usage is shown in the Main section. I used an AutoResetEvent in the test to keep the tread alive until the asynchronous event is fired and the results are output.

import clr
clr.AddReference('System.Xml')

from System.Net import WebClient
from System.Xml import XmlDocument, XmlTextReader
from System.IO import StreamReader, MemoryStream
import System

class RSSFeedItem(object):

    def __init__(self):
        self.Title = ""
        self.Description = ""
        self.Link = ""
        self.GUID = ""


class RSSFeed():

    def __init__(self, URL):
        self.FeedURL = URL
        self.ChannelName = ""
        self.Items = []
        self.BadCount = 0


class RSSFeedReader(object):

    def __init__(self,URL, onComplete):
        self.onComplete = onComplete
        self.feed = RSSFeed(URL)

    def Parse(self, sender, args):
        rssBytes = args.Result
        try:
            xmlDoc = XmlDocument()
            stream = MemoryStream(rssBytes)
            xmlDoc.Load(stream)
            rssNode = xmlDoc.SelectSingleNode("rss")
            channelNodes = rssNode.ChildNodes

            for channelNode in channelNodes:
                itemNodes = channelNode.SelectNodes("item")
                self.feed.ChannelName = channelNode.SelectSingleNode("title").InnerText

                for itemNode in itemNodes:
                    try:
                        newitem = RSSFeedItem()
                        newitem.Title = str(itemNode.SelectSingleNode("title").InnerText)
                        newitem.Description = str(itemNode.SelectSingleNode("description").InnerText)
                        newitem.Link = str(itemNode.SelectSingleNode("link").InnerText)

                        if itemNode.SelectSingleNode("guid"):
                            newitem.GUID = str(itemNode.SelectSingleNode("guid").InnerText)

                        self.feed.Items.append(newitem)

                    except:
                        self.feed.BadCount = self.feed.BadCount + 1

        except:
            print "Error Parsing Results"

        self.onComplete(self.feed)

    def ReadAsync(self):
        try:
            webClient = WebClient()
            webClient.DownloadDataCompleted += self.Parse
            uri = System.Uri(self.feed.FeedURL)
            webClient.DownloadDataAsync(uri)
        except:
            print "Error Starting Read"

if __name__ == "__main__":
    from System.Threading import *
    are = AutoResetEvent(False)

    def callback(feed):
        print "Results!"
        for item in feed.Items:
            print item.Title
            print item.Description
            print item.Link

        are.Set()

    feedReader = RSSFeedReader("http://feeds.feedburner.com/sharpthinking", callback)
    feedReader.ReadAsync()
    are.WaitOne()

One of the things I like about IronPython 2 is that this code alone can be run from the command line. Simply saving this code as rssreader.py and running it from the command line with ipy rssreader.py will run the test code at the bottom which will output recent posts from this weblog.

A taste of Python

tarn@tarnbarford.net (tarn) — Sun, 28 Dec 2008 13:51:00 +0000

Tarn Barford

I'm on a bit of a break and generally enjoying a holiday from programming, but I am spending a couple of hours here and there learning some Python. I think learning a new programming language is a good way to improve my general development skills and knowledge. As I'm just learning and not under pressure to actually build something its quite a lot of fun.

I decided to learn either Python or Ruby. I understand both are fairly modern high level languages, both are considered fun languages to use by their advocates and both been implemented on top of the .NET dynamic runtime.

I would like to try Ruby on Rails but I felt while I was on break from programming, learning a new language and a new web framework was a bit much. I decided on Python, mostly because I like the name, but also as I understand Python has found use in all sorts of environments and for all sorts of purposes including scientific computation, robotics, game development, and business and web applications.

I found Dive into Python to be awesome book for Java, C# and C++ developers wanting to learn the Python language. And its free online for browsing and downloading! I think it is very succinct in only explaining what you need to know about the language, which makes it very readable.

I was pretty excited about getting into IronPython after watching these two videos from TechEd 2007 in Orlando. There is lots of interesting information about IronPython and the Dynamic Language Runtime but there are also some really cool demos.

From the interactive console, Merlin (who apparently lives in all recent versions of windows as a COM control) is summoned and instructed to do all sorts of stuff, a Windows Form is created and run and then controls and events are added while its running! There are also demos of Iron Python with Visual Studio, ASP.NET and Silverlight. They kind of sold me on IronPython for now.

IronPython 2 has recently been released on CodePlex and its a pretty straight forward download and install. Slightly surprisingly it only contains a command line compiler, an interactive console, some libraries and sample applications.

There is also a project called IronPython Studio which is a Visual Studios extension to add support for IronPython development and debugging. I'm going to start with just the base install and Notepad 2 because I know you can do some pretty cool with it and I'd like to see how it all works.

Although IronPython is apparently compliant with the Python language, the standard library, which is considered very powerful and extensive, is not completely implemented. This basically means most Python code that makes use of the standard library won't work in IronPython. Fortunately you can use most of the .NET framework from IronPython meaning there is a pretty rich framework available.

I hope it is possible to implement the entire standard library on top of the .NET framework, this would mean more Python code would work in IronPython and IronPython code that didn't directly call into the .NET framework would run on other platforms.

As I'm happy to use IronPython and the .NET framework there is a great resource called the IronPython Cookbook which has heaps of examples of common programming tasks in IronPython and information about using IronPython in different environments from the Microsoft Robotics kit to the XNA framework.

I was hoping to write about some examples in this post, but its become much longer than I expected and I might wrap it up with some links I have found useful or interesting along the way.

Introducing IronPython - A good article from DevX

Revolver Systems - An interesting spread sheet / IronPython mash-up.

IronPython and the Dynamic Language Runtime - An MSDN article by Bill Chiles

IronPython and ASP.Net: Two tastes that taste great together - A brief article from ZDNet

IronPython and WPF - Just one of many great article from DevHawk which I've since added to my RSS feed reader.

CLR Inside Out - IronPython - Another good MSDN by James Schementi

The Voidspace IronPython Pages - A collection of IronPython resources

IronPython in Action - A book I haven't read and probably won't buy, but may be of interest to others

Hope you've found this interesting, I'm having a lot of fun and looking forward to getting back to hacking some more IronPython.

Please feel free to add a comment or point out any incorrect information.

A break I had to have

tarn@tarnbarford.net (tarn) — Tue, 23 Dec 2008 17:27:00 +0000

Tarn Barford

I've just finished up working for thinkgroup where I've been writing web based enterprise software in .Net for the last three years. Despite the fact that I enjoyed working there, after three years I really felt I needed a change.

I'm going to take a break before I start looking for new jobs, this may not be a good idea in the current global economic climate, but its a break I think I had to have. I have no clear idea yet about where I want to work or even what I want to do next. And besides Summer in Melbourne is the best time not to have a job.

It seems quite logical to get anther job writing web based software in ASP.NET. I'm very passionate about it, I enjoy coding in C# and I think I am good at designing and writing software systems to solve real world business problems. In this area I would be looking for senior developer roles and good salaries, but there are also other directions I would like to consider.

I choose to study electronic over software engineering at university even though I already really liked programming. I was looking for jobs in electronics when I graduated, but after not finding one due to both a lack of opportunities and lack of effort on my part, I ended up taking a full time role developing software.

I'd still like get a job designing electronics or low level programming but I think I would probably need to get a more recent qualification and consider re-locating to increase opportunities. Another slightly more realistic option I'm considering is getting a job writing in anther language and framework. I'm interested in other languages and its a challenge I'd really like to take and I think I would benefit from in the long term.

While I'm on holiday I'll probably still do a couple of hours here and there building a simple weblog with ASP.NET MVC I started and I'll be playing around with Python and IronPython. I'd like to blog and do a presentation to the VIC.NET User Group on some of the cool stuff you can do with IronPython but I'll wait to see how motivated I am during my break before I commit to anything.

Desktop Racer

tarn@tarnbarford.net (tarn) — Tue, 16 Dec 2008 12:10:00 +0000

Tarn Barford

I've finally got around to publicly hosting my DevSta entry - Desktop Racer. Turns out it was pretty easy, I was just too burnt out by it to want to do anything with it till now.

Play Desktop Racer Now

Looking at it now, I'm a little disappointed I didn't just go a little bit further and add a two player mode as I think it would have actually made it fun to play. But honestly after the sixth day of the competition I was so over it I didn't care.

I would like to add the two player mode and some performance improvements to make it a bit more fun to play, but I seriously doubt I'll ever get round to doing it. I'm already half way into an ASP.NET MVC weblog project and I've also suddenly become really interested in learning Python.

I'm going to upload the source as well for anyone who is interested, but I warn you its probably not a great example Silverlight application as I was just learning what was a beta technology at the time. As the competition was only 200 hours I didn't spend much time pondering what would be the best way to implement it, I pretty much just got in and started coding. Some logic is nicely packaged but some classes just ended up getting way to large and unwieldy. I've since seen some good demos and blogs on patterns to help separate out the presentation and logic.

There are some cool-ish parts in the code; I like how the level and all the level elements are defined in XAML so you can visualize the level at design time in Visual Studios. I also kind of like how the all the level elements the car can collide with implement an interface that has its geometric information. This allowed my to simplify the collision logic to two type of geometric primitive and basically calculate and collision with all the level elements in a single loop.

SharpWeblog - Iteration 1

tarn@tarnbarford.net (tarn) — Sat, 29 Nov 2008 16:28:00 +0000

Tarn Barford

I've finished all the initial requirements so I'll briefly describe how its going.

The Data Layer

The data layer interface has methods to find, create, update and delete stateless model objects. The data layer is implemented with Linq to Sql. As the interface uses model data objects, the data layer implements the repository pattern by mediating these objects with the Linq to Sql classes.

This is nice as it means the rest of the application doesn't have a dependency on the data layer implementation. This is good as I don't want to have to deal with Linq to Sql classes through-out the entire application. There are tests verifying what I would consider a more sane object model is correctly implemented by the data layer.

I personally don't think Linq to Sql should be considered as an ORM for most web applications; It has no built in way of modeling many to many relationships and it's much more difficult to use than most other ORM I've worked with. I'm looking forward to trying the new Entities Framework, and hope they have finally made a legitimate ORM.

Routing

I'm a big fan of both being able to write routing logic and routing logic tests so easily. I highly recommend writing tests verifying your routes. I briefly discussed how this is done in my previous post about testing this project. The requested route is tested against all the routes in sequence until a match it found. This means its very easy to add a route that inadvertently catches a route that was supposed to be mapped by another route further down the sequence.

I think TDD is really important here as the routing can quickly get quite complicated. Adding unit tests verifying specified routes are mapped to the correct controller and actions will protect your routes. It's also generally a quicker development cycle writing some route tests and then writing the route logic than writing the route logic and testing it in the browser.

If you insist on testing your routes in the browser or just want to see how specific are mapped in action, Phil Haack has a neat little route tester tool that renders the route test information in the browser for any URL on your site you enter.

Views

So far there has been no work done on the UI and the views are all pretty basic. I'm using C# with strongly type ViewPages, but I would like to consider using a dynamic language later in the project.

I'm a huge fan of the templated ActionLink helper method, I included the ASP.NET MVC Futures assembly without hesitation when I realised it wasn't included in the Beta.

<%=Html.ActionLink<HomeController>(m => m.Category(category),category) %>

I'm a little more nervously on the fence with regard to the RenderAction method which allows you to call an action on the controller from within a view to render a control. I can see the why its there; It provides a lot of flexibility for making controls that can be responsible for getting there own data and can be used by any view. This also means these user control controller actions can be tested separately. On the other hand it appears to be way to get around the MVC pattern where in many cases the controller itself could call off to get the additional information and pass it down to view.

<% Html.RenderAction<HomeController>(m => m.PostDetail(ViewData.Model.SelectedPost)); %>

If I could call a view from within another view and pass it a model from the data already in the calling view I would be a lot more comfortable with it (You probably can do that, I haven't worked it out yet). That way there would be a nice way to call controls with model data without calling methods on the controller from the view.

EDIT: As soon as I tabbed to Visual Studio after posting this I noticed the login control from the template renders a user control using RenderPartial from the view. The user control inherits from ViewUserControl and can be templated. An overloaded RenderPartial method allows you to pass model data to the view. I've fallen to the lets get on with coding side of the fence.

<% Html.RenderPartial("LoginUserControl"); %>

Membership and Roles

It wasn't in my initial requirements, but the MVC VS2008 Beta template comes with some login in and registration pages using the default SQL Server Membership provider. I am looking to leverage all the technologies I can to build this site, so I decided I would use the default provider, if I could use it and keep the code very testable.

In the default template a database is generated with the membership and roles tables and stored procedures used by the default membership provider already created. As I already had a database, I used a tool called aspnet_regsql to add the tables and procedures to my database. The tool is part of the .NET framework and can also be used to generate scripts and works with SQL Server Express.

I had mixed emotions when I found this in the controller from the template. I liked that they had thought about testing the controller (no tests are actually included).

// This constructor is not used by the MVC framework but is instead provided for ease
// of unit testing this type. See the comments at the end of this file for more
// information.

public AccountController(IFormsAuthentication formsAuth, MembershipProvider provider)
{
    FormsAuth = formsAuth ?? new FormsAuthenticationWrapper();
    Provider = provider ?? Membership.Provider;
}

..

// The FormsAuthentication type is sealed and contains static members, so it is difficult to
// unit test code that calls its members. The interface and helper class below demonstrate
// how to create an abstract wrapper around such a type in order to make the AccountController
// code unit testable.

..

That great! Its a pattern I use in the project to avoid dependencies and improve testability, the only problem is I don't think they've gone far enough. I'm going to take it further and create a wrapper around Provider so I can mock it and not have a dependency on System.Web in my controller unit tests.

EDIT: After writing this post I found I could reference System.Web in my tests and not have a development web server fire up when the tests run. I was a little surprised by this, and now I'm not sure what does fire up the development web server. I'll post an update on this when I understand it more clearly.

SharpWeblog - Testing

tarn@tarnbarford.net (tarn) — Mon, 24 Nov 2008 15:54:00 +0000

Tarn Barford

I'm right into the coding for this project at the moment, but the iteration one requirements are all complete so its probably a good time to write some posts and reflect on what has been implemented so far.

As specified this project is use TDD for development. There has been a lot of work put into ASP.NET MVC to make it a most testable framework. I would like to discuss how various parts of this project have been tested.

I am using Microsoft Unit Test Framework and MOQ for testing this project. I had never used MOQ before, but I love it. For those that don't know about MOQ its another mock testing framework, the thing I love about it is it uses C# 3.5 so you can do a lot with lambda expressions.

Controllers

In the controller tests a real controller is created with a mocked IWeblogRepository interface. This means the controller can be tested independent of an IWeblogRepository implementation.

[TestInitialize]
public void Setup()
{
    _mockRepository = new Mock<IWeblogRepository>();
    _controller = new HomeController(_mockRepository.Object);
}

Testing controllers is really testing two things, firstly that the controller renders the correct view and secondly it sends the right data to that view. The tests shown here validate the "Post" action on the HomeController.

In this test the mock repository is setup to return a post when GetPost("Test- Post-1") is called on it. When we call the controller with these parameters we expect it to make the specified request to the mock repository. This test ensure that when a post is found that "Post" view is rendered.

The sharp eyes will notice that I'm not using the date parameters to find the post from the repository. That functionality has not been implemented in this iteration.

[TestMethod]
public void Post_FindsPost_RendersPostView()
{
    _mockRepository.Expect(m => m.GetPost(It.Is<string>(i => i == "Test-Post-1")))
                   .Returns(new PostModel());

    ViewResult result = _controller.Post(8,8,8,"Test-Post-1") as ViewResult;
    Assert.AreEqual("Post", result.ViewName);
}

We can also test the correct data is being sent to the view. I personally like to separate these two tests, this may not be a universal opinion, but I think its best to keep unit tests short and I think these tests are clearly testing different expectations.

[TestMethod]
public void Post_FindsPost_ReturnPostViewData()
{
    _mockRepository.Expect(m => m.GetPost(It.Is<string>(i => i == "Test-Post-1")))
                   .Returns(new PostModel() {
                                                Title = "Test Post 1",
                                                Content = "Test Post Content 1"
                                            });

    ViewResult result = _controller.Post(8, 8, 8, "Test-Post-1") as ViewResult;
    PostViewData data = result.ViewData.Model as PostViewData;

    Assert.IsNotNull(data);
    Assert.AreEqual("Test Post 1", data.SelectedPost.Title);
    Assert.AreEqual("Test Post Content 1", data.SelectedPost.Content);
}

Two additional test were added to test other expectation of the "Post" controller action. These test that the "Index" view is rendered when the action is called with invalid parameters or can't find the post. This functionality may change to display an error message or alternate posts when the request is invalid in later iterations.

[TestMethod]
public void Post_PostNotFound_RendersIndexView()
{
    _mockRepository.Expect(m => m.GetPost(It.Is<string>(i => i == "Test-Post-1")))
                   .Returns<PostModel>(null);

    ViewResult result = _controller.Post(8, 8, 8, "Non-Existing-Post") as ViewResult;

    Assert.AreEqual("Index", result.ViewName);
}


[TestMethod]
public void Post_InvalidRequest_RendersIndexView()
{
    ViewResult result = _controller.Post(null,null,null,null) as ViewResult;
    Assert.AreEqual("Index", result.ViewName);
}

Routes

Testing routes is testing Http Requests are correctly mapped to the appropriate controller actions with the appropriate parameters. The below is testing "~/Post/2008/10/20/Test-Post" request will call the HomeController with the Post action and will also correctly populate the parameters.

[TestMethod]
public void PostRoute_TestSuccessfullRoute()
{
    var routes = new RouteCollection();
    var context = new Mock<HttpContextBase>();

    context.ExpectGet(m => m.Request.AppRelativeCurrentExecutionFilePath)
           .Returns("~/Post/2008/10/20/Test-Post");

    SharpWeblogRouting.RegisterRoutes(routes);
    var routeData = routes.GetRouteData(context.Object);

    Assert.AreEqual("Home", routeData.Values["controller"]);
    Assert.AreEqual("Post", routeData.Values["action"]);
    Assert.AreEqual("20", routeData.Values["day"]);
    Assert.AreEqual("10", routeData.Values["month"]);
    Assert.AreEqual("2008", routeData.Values["year"]);
    Assert.AreEqual("Test-Post", routeData.Values["name"]);
}

I can write a more route tests to test other requests when I have features saying what should happen.

Data Layer

While I think its great we can "fake out" the IWeblogRepository to avoid using an actual implementation while testing other areas of the system, I strongly believe implementations of the IWeblogRepository should also be tested, separately.

The IWeblogRepository interface basically has methods to save, load, update and delete model objects in a data storage. In this case it is storing the data in a SQL Express 2005 database using LINQ to SQL.

NOTE: The model classes are just data objects that have no information about the data context or data state. This does result in some inefficiencies updating the database, but the design was chosen as it clearly separates the implementation of the data layer from the rest of the system.

Anyway here are some typical tests, no mocking required here.

NOTE: I think these tests should be written against an interface with our implementation injected in. This would mean the same tests could be applied to other IWeblogRepository implementations.

[TestMethod]
public void PostCRUD()
{
    // Create
    PostModel post = CreateTestPost();
    _repository.SavePost(post);

    // Read
    PostModel findPost = _repository.GetPost(post.Id);
    Assert.IsNotNull(findPost);
    ValidatePosts(post, findPost);

    // Update
    findPost.Content = "Updated Content";
    findPost.Title = "Update Content";
    _repository.SavePost(findPost);

    // Read Update
    PostModel findUpdatedPost = _repository.GetPost(post.Id);
    Assert.IsNotNull(findUpdatedPost);

    // Delete
    _repository.DeletePost(findUpdatedPost);
    PostModel findDeletedPost = _repository.GetPost(post.Id);
    Assert.IsNull(findDeletedPost);
}

While I generally like small tests, I think an objects CRUD tests are best tied up in one test because otherwise you end up testing "create" four times to setup for each of the other tests. I would also generally implement the entire CRUD of an object even if it wasn't all required just so as CRUD tests can be performed and each operation can be tested.

I have many more tests for the data layer but they are all pretty standard, so no point discussing them further here.

MetaWeblog

The MetaWeblog API has also been (partly, so far) implemented for this project. I will discuss the implementation further in another post but for now we'll talk about testing it.

The MetaWeblog is implemented in two clearly separate components. One part processes the XML-RPC request and invokes methods on a IMetaWeblog interface and returns XML-RPC responses to Http Response. The other is an implementation of the IMetaWeblog, both can be tested separately.

Testing the IMetaWeblog implementation is pretty straight forward. As the IMetaWeblog implementation uses the repository through the IWeblogRepository interface we can mock the repository and test the expected calls are made against the interface.

NOTE: Sharp eyes will notice I'm not validating the user name and password. This isn't yet implemented.

[TestMethod]
public void NewPost_InvokesSaveMethod()
{
    MWAPost post = TestHelper.CreateTestMWAPost();
    Mock<IWeblogRepository> repository = new Mock<IWeblogRepository>();
    SharpMetaWeblogAPI model = new SharpMetaWeblogAPI(repository.Object);
    model.NewPost("100", "TestUser", "TestPassword", post, false);
    repository.Verify(m => m.SavePost(It.Is<PostModel>(p => ComparePosts(p, post))));
}

We can also test the return values of the IMetaWeblog methods with a known responses from the IWeblogRepository mock.

[TestMethod]
public void GetPost_ReturnsRequestedPost()
{
    Guid id = Guid.NewGuid();

    PostModel post = new PostModel()
    {
         Title = "Test Title",
    };

    Mock<IWeblogRepository> repository = new Mock<IWeblogRepository>();
    repository.Expect<PostModel>(m => m.GetPost(id)).Returns(post);
    SharpMetaWeblogAPI metaWeblog = new SharpMetaWeblogAPI(repository.Object);
    MWAPost findPost = metaWeblog.GetPost(id.ToString(), "none", "none");

    Assert.AreEqual(post.Id.ToString(), findPost.postID);
    Assert.AreEqual(post.Title, findPost.title);
}

Being able to unit test the MetaWeblogFramework is cool, but I may have taken unit testing too far here, but I'll let you decide.

The MetaWeblogFramework had to be implemented against HttpContextBase, with an actual context object passed into the ProcessRequest method. An instance of IMetaWeblogAPI is injected into the constructor to allow the framework to call the interface methods on a real object after processing the request, kind of like a call back or an event.

The structure of most of tests I have written so far is to mock a HttpContextBase request and send it to a real instance of MetaWeblogFramework which has been constructed with a mock IMetaWeblog API.

This way I can verify that with known HTTP requests and payloads the expected methods on the IMetaWeblog interface are called. The test below creates an XmlRpcRequest object which the HttpContextBase mock will return when the MetaWeblogFramework ProcessRequest method requests the HttpContext.Request.InputStream. The test expects the IMetaWeblogAPI.NewPost(..) method to be called with known parameters.

[TestClass]
public class MetaWeblogFrameworkTest
{
    Mock<HttpContextBase> context;
    Mock<IMetaWeblogAPI> api;
    Byte[] _output;

    [TestInitialize]
    public void Setup()
    {
        context = new Mock<HttpContextBase>();
        api = new Mock<IMetaWeblogAPI>();
        _output = new Byte[1000];
        context.ExpectGet(m => m.Response.ContentType).Returns("text/xml");
        context.ExpectGet(m => m.Response.OutputStream).Returns(new MemoryStream(_output));
    }

    [TestMethod]
    public void ProcessRequest_MockRequestNewPost_InvokesNewPostMethod()
    {
        MWAPost sentPost = TestHelper.CreateTestMWAPost();
        XmlRpcRequest mockRequest = new XmlRpcRequest()
        {
            MethodName = "metaWeblog.newPost",
            Params = new List<Param>()
            {
                new StringParam("TestBlogId"),
                new StringParam("TestUserName"),
                new StringParam("TestPassword"),
                new PostParam(sentPost),
                new StringParam("False"),
            }
        };

        context.ExpectGet(m => m.Request.InputStream).Returns(mockRequest.CreateStream());
        MetaWeblogFramework framework = new MetaWeblogFramework(api.Object);
        framework.ProcessRequest(context.Object);

        api.Verify(m => m.NewPost("TestBlogId", "TestUserName","TestPassword", It.Is<MWAPost>(a=>sentPost.Match(a)) , true));
    }

    ..
}

Testing the MetaWeblogFramework was a little difficult, and I had to create a whole range of additional objects and methods to perform the tests. I might re-visit the MetaWeblogFramework implementation later in the project and re- write it to be more testable.

Conclusion

Well I managed to test a whole range of components in this ASP.NET MVC project, but my overall coverage is still pretty low (32% over the entire project). I'll keep you updated with how testing is going though-out the project and I'll try to get those coverage stats up by the drop of the first iteration this week.

image

SharpWeblog Project

tarn@tarnbarford.net (tarn) — Fri, 21 Nov 2008 14:47:00 +0000

Tarn Barford

Introduction

I've just started a small project building a simple weblog site using ASP.NET MVC. I know there are heaps of blog sites out there but I thought it would be a good project to get familiar with ASP.NET MVC, JQuery and Silverlight 2.

I don't like writing about projects I'm going to build, but I'm hoping I can get a couple of the guys from work involved to share the learning and the work. I'm also going to hassle a few graphic designers I know and try to get little free consulting and maybe even some design work.

It will be used it to drive this blog instead of BlogEngine.NET which we are currently using.

To keep things simple the project will use the database schema and parts of the Metaweblog API implementation from the BlogEngine.NET project. LINQ to SQL will be used to quickly get some ORM mappings to the database schema. It will also use the standard ASP.NET Membership and Roles providers for SQL Server. Some Silverlight 2 will be added for some cool menus and navigational controls.

Tools, Technologies and Methodologies

The project will use TDD, Agile methodologies and identify design patterns where appropriate.

For now the code will be hosted on our internal Subversion server, but I would like to move it CodePlex eventually. I will move it sooner if anyone else is interested in checking out the code or contributing to the project. I would like to use a build machine, but I'm not sure if there is support for this at CodePlex, but we'll find out.

The project will be developed with .NET 3.5 SP1 and ASP.NET MVC Beta. The Microsoft Unit Test framework will initially be used for unit testing but a Mock Testing framework may be introduced later.

The project should be able to be run and debugged using the Visual Studios internal development web server and SQL Server Express, basically by pressing F5 after checking out the code.

Initial Requirements

In agile fashion the project will start with some initial requirements, additional requirements can be added as the project evolves.

A default or index page showing recent posts in reverse chronological order
A post page, using the same URI routing as BlogEngine.NET
A Metaweblog API implementation to support Windows Livewriter for publishing and managing content on the weblog

So that's it for now, hopefully we can knock this first iteration together pretty quickly.

Tasks

I don't want to get to far into fine grained tasks as I don't have to estimate the project time or cost, but I do want to cover the basic tasks as it does give an indication of what I expect will need to be done for this phase.

Create the database
Create the LINQ to SQL classes
Create a Data Layer and a Data Layer Interface
Write some Data Layer Tests
Create a Model (tests?)
Create a Controller
Write some Controller Tests
Setup some routes
Create the index view
Create the post view
Implement the Metaweblog API

Lets Rock'n'Roll

Hopefully I'll be posting the learning from phase one and the goals of phase two next week.

SQL Sever 2005 Full Text Indexing

tarn@tarnbarford.net (tarn) — Fri, 14 Nov 2008 09:31:00 +0000

Tarn Barford

Till now I've always had problems getting SQL Server 2005 Full Text Indexing (FTI) to perform well in real world scenarios. Recently I found it was because I wasn't implementing it correctly, so I'll post this tip which will hopefully help myself and others get it right next time.

The query is a user entered keyword search on text columns using the FTI predicates and T-SQL predicates on other relational columns. The query is over a moderately large number of and returns paged results.

FTI is basically the ability to search from text columns where the text has been indexed with natural language, relevance and ranking considerations built in. XML and other data type can also be index natively, but I haven't used that. FTI in SQL Server 2005 uses the Microsoft Search Service. This service is not actually part of SQL Server itself, it is also used by Exchange and Sharepoint. I think you can use the service directly with a Sharepoint SDK.

To use Full Text Indexing in SQL server you must enable full text indexing to the entire database and the table and columns containing the searchable text. It can all be setup from SQL Management Studio, but it does create a new physical folder in addition to the normal database MDF and LDF files. Once this has been done additional FTI predicates can be used on the selected columns.

As SQL Server Express doesn't support FTI, I think it is worth considering optionally supporting normal T-SQL similar to applications support querying using the FTI and functions. Its nice to be able develop or even deploy with SQL Server Express even though the text searching is not indexed using the Search Services. But this really depends on circumstances, I just wanted to note supporting both is an option.

I found a query similar to this example was taking ages over a large amount of data. It was unacceptable, we had to render all pages in less than 4 seconds under moderate load and we expected to get less than two seconds. We we getting search times alone in double figures for some keywords.

SELECT *
FROM
(
      SELECT *, ROW_NUMBER() OVER ([DocumentInformation].PublishDate ASC) as RowNumber,
      FROM Document
      JOIN DocumentInformation ON DocumentInformation.DocumentId = Document.DocumentId
      JOIN UserAccount Owner ON Document.OwnerId = UserAccount.UserId
      JOIN UserAccount Publisher ON Document.PublisherId = Publisher.UserId
      WHERE [Document].Published = 1 AND
            [Owner].IsActive = 1 AND
            [Publisher].IsActive = 1 AND
            CONTAINS([Document].DocumentText, "FTI*" )
) x
WHERE RowNumber BETWEEN 1 AND 10
ORDER BY RowNumber

Below has an additional inner query on the document table alone, using just the CONTAINS predicates on the document text. The additional tables are joined to the results and the normal T-SQL predicates are added.

SELECT *
FROM
(
    SELECT *, ROW_NUMBER() OVER ([DocumentInformation].PublishDate ASC) as RowNumber,
    FROM
        (SELECT *, DocumentId
        FROM dbo.Document
        WHERE CONTAINS(([Documemnt].ProductKeywords, "FTI*" )) p
        JOIN DocumentInformation ON [DocumentInformation].DocumentId = p.DocumentId
        JOIN UserAccount Owner ON Document.OwnerId = UserAccount.UserId
        JOIN UserAccount Publisher ON Document.PublisherId = Publisher.UserId
        WHERE [Document].Published = 1 AND
              [Owner].IsActive = 1 AND
              [Publisher].IsActive = 1
) x
WHERE RowNumber BETWEEN 1 AND 10
ORDER BY RowNumber

I gave this a go after reading an FTI Best Practice article that indicated the searchable text rows couldn't be excluded from a search due to the search service running in a separate process. The result of this subtle change was amazing, the searches were obviously much quicker and returned the same result set. Under load test we found this change alone improved the average execution time of all pages on the website by a massive 30%.

In addition to this the FTI best practices document also recommends the following optimizations to improve full text performance.

We ended up implementing all the recommendations above to try and get the most out of the FTI service. After all this we got search times down to about a second under light load, this was cool but we had grander plans for an in memory search using of expression trees for search filters and LINQ for querying the in memory objects. I hope to chronicle this in future posts as initial performance testing shows much quicker search times and a significant capacity load increase by removing searching from SQL Server which was previously our performance bottleneck.

In SQL Server 2008 they have apparently integrated FTI into the SQL Server process which appears to be the cause of my original performance problem. But so far I've only heard bad things, hopefully these issues will be cleaned up before I start using it.

2005 Full-Text Search Architecture

2008 Full-Text Search Architecture

Liberation Day - Power To Developers

tarn@tarnbarford.net (tarn) — Thu, 13 Nov 2008 22:40:00 +0000

Tarn Barford

I'm not in anyway formally associated Microsoft other than that I've been primarily using their tools to develop software for the over the last few years, so I was a little excited I was invited to Liberation Day in Sydney after placing 2nd in DevSta. I thought it would be great to see Steve Ballmer, infamous for this famous clip and this. He is, of course, also the Microsoft CEO and was going to announce Azure the new Microsoft cloud computing platform. He was sure to be entertaining anyway.

As I'd never been to Sydney I decided I could spend a couple of days on Bondi Beach with my girlfriend and drop into the conference and while I was there. I'm expecting to be looking for a new job from next year, probably not in Sydney, but I thought it would also be good to see who was there and meet a few people anyway. I was expecting there would be a couple of DevSta judges there too.

There were heaps of people at the event, I wouldn't be surprised if there actually was the 1000 developers the flyer claimed there would be. It was cool, I can't image an event with that many developers in Melbourne. It was streamed live and can be replayed on the Power To Developers site. I watched Steve get up and do his thing, and it was fun, not rock'n'roll, but fun. Unfortunately I wasn't feeling very well, almost feverish, and I had to duck out to find a chemist.

Gianpaolo Carraro's presentation attempted to demonstrate how easy developing cloud solutions for the Azure was with Visual Studios 2008. I thought it was pretty cool despite most of his demos failing in ways he couldn't have imagined. The Azure platform sounded pretty awesome, basically allowing custom .Net assemblies to be uploaded and invoked on Microsoft servers. He also managed to successfully demonstrate the cloud emulator for locally running and debugging cloud applications. I also really like the idea of being able to a write LINQ query joining two data tables from a SQL Server Service in the cloud.

Mike Culver presented the Amazon Web Services cloud solutions to the Vic.NET user group about a year ago and I think he sold their technology and the possibilities better. He showed the architecture of a video compression service that used a message queue service to queue requests and a controller application that can automatically rent, build and deploy servers capable of processing the requests. I thought it was pretty cool. I think Microsoft are a long way behind, but we'll see what the software giant is capable of. I'm certainly keen to get in and give it a go.

I still wasn't feeling well and missed most of Tim Sneath which was a shame, Brodie had seen him in London years ago but said he "knew his stuff". At the networking drinks I chatted with Michael Kordahi which was kind of fun, he'd judged my entry and he is a Silverlight guy. He introduced me to a couple of people, one of which was Jeremi Kelaher who does Strange Devices Podcasts. I also ran into Tatham Oddie on the way out who I'd seen present MVC stuff at REMIX and Vic.NET. I would have liked to have chatted with Andrew Coates who is a great presenter and was also a DevSta judge, but I didn't end up seeing him this time.

DevSta {Challenge 2008} - 2nd Place!

tarn@tarnbarford.net (tarn) — Thu, 23 Oct 2008 13:40:00 +0000

Tarn Barford

The winners for DevSta {Challenge 2008} have been announced: GPS Tag wins Microsoft Devsta Challenge. My Desktop Racer entry came 2nd. I'm naturally disappointed I didn't win, but definitely happy with 2nd. It was heaps of fun entering and I'm glad the stress is over ;-)

Congratulations to all the entrants, particularly the winners Jarred Sarge and Michael Minutillo for their GPS Tag Game. Hopefully we'll compete again next year. Enjoy Vegas boys.

Command Line Arguments

tarn@tarnbarford.net (tarn) — Tue, 21 Oct 2008 11:28:00 +0000

Tarn Barford

I still write heaps of little console applications to do all sorts of things. Most of the console applications I write are tools for tasks like deployment, manipulating data and processing images. Usually they are for my own use, if I find one really useful I might pass it on to the rest of our development team or publish it on this blog.

For these little tools, I prefer using console applications over windows application as I feel they are more useful. Even though its easier to use a tool that has a little GUI, its not more useful if you ever want to automate it.

Ideally I would prefer to write assemblies and use them from Powershell as it would provide the greatest usage flexibility. But it also requires competence scripting with Powershell. I was very excited about Windows Powershell when it was released, but a year on, I still haven't made it the integral part of my development toolset I thought it would be. This is an area I want to improve on, but this post is about command line arguments.

Generally these little non production, internal applications are quite light weight and fit for purpose. I find I often write simple logging and command line processing that is also fit for purpose. This is fine, its usually very simple, quick to write and works. The problem is that I write this same functionality over and over again for each little application.

I was toying with the idea of writing a generic command line parsing class. I figured I could use reflection on a settings class decorated with attributes describing the specific command line syntax. I figured I could even generate the help text using reflection too. While this seemed like a great idea, I am aware of time I've previously spent (wasted?) re-writing code where a perfectly good open source solution is available.

So I did a search and found a couple of existing command line solutions. Not surprisingly, given they were both written in .NET 2.0+, they both used reflection and attributes. I ended up using the Command Line Parser Library from CodePlex. I'm not sure if it does everything, but it does everything I need for 99% of the console applications I write that require command line processing.

Anyway here is all the code from my test application:

public class Options
{
    [Option("r", "run", HelpText = "Run the export", Required = true)]
    public bool Run;

    [Option("n", "Name", HelpText = "Use a specific name")]
    public string Name = String.Empty;

    [HelpOption(HelpText = "Dispaly this help screen.")]
    public string GetUsage()
    {
        HelpText help = new HelpText(new HeadingInfo("Hello WorldExample", "1.0.0"));
        help.Copyright = new CopyrightInfo("sharpthinking", 2008);
        help.AddPreOptionsLine("Usage: Hello --run");
        help.AddOptions(this);
        return help;
    }
}

class Program
{
    static void Main(string[] args)
    {
        Options options = new Options();
        if (Parser.ParseArguments(args, options, Console.Error))
        {
            string name = string.IsNullOrEmpty(options.Name) ? "World" : options.Name;
            Console.WriteLine("Hello {0}!", name);
        }
    }
}

And we get the expected usage from the command line

image

Now I have to make sure I bundle the console applications with an external assembly, but it does make using command line arguments easy.

Functional Programming

tarn@tarnbarford.net (tarn) — Sun, 19 Oct 2008 19:18:00 +0000

Tarn Barford

The Problem, The Opportunity

Brodie and my initiatives in the area of search have been rewarded by management with a two week project to improve search on our sites. In the two weeks we also intend to fix one of the darkest corners of the code base, the Product/Site publishing logic, which is the main reason the search needs revisiting. We couldn't get the two weeks just to improve infrastructure, so we promised sub one second search results and better relevance as a carrot.

The web application powers multiple sites (>100) and has lots of products (>100,000). There needed to be some way of controlling what products belonged to what site. As products can appear on multiple sites and new sites are constantly being added, this is not trivial. The solution implemented is a site publishing rules system. Products can be excluded from a site by Classification, Product or Brand.

So far the logic is pretty simple, but another type of rule was also added. Products, Classifications and Brands can also be made specific to a site (they can actually be made specific to multiple sites). The logic is that a Product, Classification or Brand that is specific to a site will appear on that site, but not on other sites. In my opinion this is kind of crazy, but the thinking is; If you have a very specific product that should only be published on one or two sites, its easier to make it specific to those two sites rather than excluding it from every other site and new sites. Ok, fair enough.

The problem is that this logic is scattered through all our layers down to the database. There is duplication of logic over separate layers causing maintenance problems and it has been identified as a clear performance bottle- neck of any product list retrieval operation.

The scope of this prototype is a quick, robust, maintainable and fully unit tested solution that implements the product site publish logic.

Database

These rules are held in 6 SQL joining tables, which I think is interesting itself. The primary keys are not unique across products, brands and classifications meaning to enforce referential integrity and not allow nulls you do need a few tables. While I think this is a very good argument to have primary keys unique across all your database objects, its not always an option.

We could also have used 3 tables by combining the excluded and specific tables and adding a flag or lookup id. I think less is better, less tables means we can be more flexible with how we manipulate the data in code. Even in T-SQL I think I'd prefer less tables, less joins and more where statements - de- normalized. But that just my opinion in this scenario.

As I move this logic out of the database and into managed code I'll end up abstracting the data to one table anyway, for this prototype it doesn't matter how the rules are created or persisted.

Functional Languages

Now back to the title of this post, I think this problem is really suited to functional programming. I've become more interested in functional programming after listening to a Heading Code pod cast with Matt Podwysocki. My experience with functional programming is limited and I've never found it very exciting. I learnt a bit of Haskel to help a mate with a computer science subject he was doing, but I was never really going to get to into it. At that time I believed all languages were inferior to C++.

As I've become more involved in software development I think its more important to use other languages. There is an inevitability that your language of choice will be relegated to legacy tasks as newer languages emerge better suited modern computing and user experience. C# code making heavy use of generics, extension methods, anonymous delegates and LINQ is just not the same language as the C# that rolled out with .Net 1.0. Dynamic and functional languages are becoming first class citizens.

I'm not actually going to write this in F# for now, but I think the way I've implemented my solution in C# is very functional although I leverage some OO concepts I might not be able to use in a strictly functional language.

Solution Overview

Instead of having all these separate lists of rules I've just created one list which can be queried. It makes heavy use of what Rob Conery described as a Pipes and Filters pattern in the ASP.NET MVC Storefront series. The pipes and filters are implemented as extension methods that add filters and data transformations. A collection of these small and easily testable extension methods can be used to build very powerful data retrieval and processing constructs to clearly describe business logic.

The scope of the prototype is really just a single function that implements the site publishing logic described earlier. It takes a Product, Site and List of Rules as parameters and returns the publish rule.

As soon as I started thinking about a solution to the problem I knew there was one thing I didn't want to do. I could see there was potentially a lot of repeated code for calling logic for products, classifications and brands individually. There is no need as it is the same logic that needs to be applied to each. I wanted to find a more elegant solution. I used an object orientated solution; the Rule class itself is abstract and is inherited by ProductRule, ClassificationRule and BrandRule.

public abstract class Rule
{
    public int SiteId { get; set; }
    public RuleType Type { get; set; }
    public int Value;
    public abstract bool Compare(Product product);
}

The only thing implemented by the specific rule classes is how to bind the Value to a property on the Product class.

public class ClassificationRule : Rule
{
    public override bool Compare(Product product)
    {
        return (Value == product.ClassificationId);
    }
}

public class ProductRule : Rule
{
    public override bool Compare(Product product)
    {
        return (Value == product.ProductId);
    }
}
..

I could have implemented this in a more functional way, but it does seem like a clean solution given I am using C#.

Testing the Bits and Pieces

Testing the extension methods is easy with such simple domain objects. For most of the extension methods I only wrote a couple of small tests like this:

[TestMethod]
public void FromSite_ReturnsSiteRule()
{
    List<Rule> rules = new List<Rule>() { { new ProductRule() { SiteId = 1} } };
    Assert.AreEqual(1, rules.FromSite(1).Count());
}

[TestMethod]
public void FromSite_FiltersSiteRule()
{
    List<Rule> rules = new List<Rule>() { { new ProductRule() { SiteId = 1} } };
    Assert.AreEqual(0, rules.FromSite(2).Count());
}

I used LINQ constructs rather than the trusty delegates I use everywhere else for the extension methods. I like it, its clear and easy to read. And this is my point, its changed the language dramatically.

public static IEnumerable<Rule> FromSite(this IEnumerable<Rule> rules, int SiteId)
{
    return from rule in rules
           where rule.SiteId == SiteId
           select rule;
}

Generic Test Classes

I'm going to start using the the Microsoft Visual Studios Unit Test Framework. It means people who don't use the same testing framework flavour don't get build errors when they download code download and try to build it. As the attributes work with NUnit GUI and TestDriven.NET I'm happy to use it for now. I not sure if there is any mock testing support, but it does allow inheriting templated test methods.

For the same reason I didn't want to write code that called product rules, classification rules and brand rules individually, I don't want to write tests for them individually either.

public class GenericSiteRulesTests<T>
    where T : Rule
{
    [TestMethod]
    public void ProductWithNoRules_ProductIsReturned()
    {
        Assert.IsTrue(RuleLogic.ProcessRules(new Product(), new List<Rule>(), 1));
    }

    [TestMethod]
    public void ProductIdCanBeExcluded_ExcludedProductIsNotReturned()
    {
        List<Rule> rules = new List<Rule>() {{ CreateRule(1,RuleType.Excluded, 0 )}};
        Assert.IsFalse(RuleLogic.ProcessRules(new Product(), rules, 1));
    }

    ..

}

The way this is done I think is pretty cool, I did something similar when I was looking at testing Text Search algorithms recently. All the generic rule tests are written in a generic test class which must be templated to a type the derives from Rule. The class doesn't have a test class attribute. This is because the test classes we want to run are actually classes that inherit from the generic test class.

[TestClass]
public class ClassificationSiteRulesTests
    : GenericSiteRulesTests<ClassificationRule>
{
}


[TestClass]
public class ProductSiteRulesTest
    : GenericSiteRulesTests<ProductRule>
{
}


[TestClass]
public class BrandSiteRulesTest
    : GenericSiteRulesTests<BrandRule>
{
}

These classes do have the test class attributes. Although they are empty, they expose all the templated methods inherited from the generic test class. These methods can be executed by a test runner.

NUnit GUI has running all the tests.

The Logic

Ok, so its not really "clear and easy to read", but it is very compact and concise.

public class RuleLogic
{
    public static bool ProcessRules(Product product,  IEnumerable<Rule> rules, int siteId)
    {
        // return false if there is any exclusion rule that applies to this product

        if (rules.FromSite(siteId)
                 .WithType(RuleType.Excluded)
                 .AppliesTo(product)
                 .Count() > 0) return false;

        // every rule type (ie ProductRule) that any site has a specific rule relating too
        foreach (Type type in rules.WithType(RuleType.Specific)
                                   .AppliesTo(product)
                                   .RuleTypes())
        {
            // this product requires a specific rule of this type
            if (rules.WithType(RuleType.Specific)
                     .WithTarget(type)
                     .FromSite(siteId)
                     .AppliesTo(product)
                     .Count() == 0) return false;
        }

        return true;
    }
}

Conclusion

It all seems fine, all the 33 tests are running and passing, coverage is at 100%.. still I have some strange feeling there is still a critical bug in there, somewhere.

We'll wait and see Monday morning if the project will go ahead and this prototype can be extended into part of a production quality product publishing rules engine. Oh, and the sub 1 second product searching..

Text Searching

tarn@tarnbarford.net (tarn) — Tue, 14 Oct 2008 21:11:00 +0000

Tarn Barford

Introduction

Brodie and I recently started presenting code, prototypes and ideas to the other Devs we work with. This post came about after Brodie presented a search engine prototype. He was interested to find if we could load about 100,000 pages of text to in-memory collections for fast searching. The prototype used WCF and LINQ to create a RESTful server that binds to a HTTP endpoint. I thought it was a really cool prototype, it used less memory than I expected and was many orders a magnitude faster than the incumbent full-text indexed SQL query.

We were discussing how we could make non-cached searches faster and agreed the String.IndexOf() and String.Contains() methods inside the filtering and relevance Linq queries would be the best place to start. I probably wouldn't have done anything more with it but I couldn't stop thinking about how different indexing full-text methods we discussed might perform, particularly a tree base algorithm.

The following week I presented a text search engine interface, some implementation prototypes, a testing framework prototype and the use of the StructureMap to inject concrete implementations of a text search engine interface into Brodies prototype. I'd already taken this way further than I ever intended, but I decided that since I had gone this far I might as well wrap it up in a blog post too.

I started off by creating a search engine interface. I used an interface from the start as I wanted to try mock testing and dependency injection.

public interface ITextSearchEngine
{
    object Setup(string searchText);
    bool Contains(string query);
    int Find(string query);
    List<int> FindAll();
}

The most simple search engine is just using a string and the methods on System.String. It shows how easy it is to implement the ITextSearchEngine interface:

public class SimpleTextSearch : ITextSearchEngine
{
    private string _contents;

    #region ITextSeach Members

    public object Setup(string searchText)
    {
        _contents = searchText;
        return _contents;
    }

    public bool Contains(string searchText)
    {
        return _contents.Contains(searchText);
    }

    public int Find(string query)
    {
        return _contents.IndexOf(query);
    }

    public List<int> FindAll()
    {
        throw new NotImplementedException();
    }

    #endregion
}

Search Engine Test Framework

The test framework is a custom framework that can load types that implement ITextSearchEngine, ITestHarness and ITextDataStore from external assemblies at runtime. ITestHarness instances are created and passed new instances of ITextSearchEngine and ITextDataStore. The test methods in the test harness are executed and the results a saved as XML and also transformed to HTML.

The ITextDataStore was added to make it possible to write tests that are independent of actual data. This is for performance tests where the data will effect the results. I intend to verify this by running a tests against different data.

public interface ITextDataStore
{
    string SearchText();
}

The ITestHarness interface must be implemented by test classes. The Create(..) method allows the test framework to inject ITextSearchEngine and ITextDataStore implementations into the test class.

public interface ITestHarness
{
    void Create(ITextSearchEngine engine, ITextDataStore datastore);
    void Setup();
}

Now we can start implementing some tests before we've even implemented a search engine. To do this we start by creating a base test class:

public class TestHarnessBase : ITestHarness
{
    #region Data Members
    protected ITextSearchEngine _searchEngine;
    protected ITextDataStoreTest _dataStore;
    #endregion

    #region ITestHarness Members
    public void Create(ITextSearchEngine engine, ITextDataStoreTestdatastore)
    {
        _searchEngine = engine;
        _dataStore = datastore;
    }

    public virtual void Setup()
    {
    }

    #endregion
}

The test framework will consider public methods with a known return type and no parameters as test methods.

public class SearchEngineSearchTests : TestHarnessBase
{
    #region Test Methods
    public bool Contains_FindExistingWord()
    {
        _searchEngine.Setup("The quick brown fox jumps over the lazydog");
        return _searchEngine.Contains("fox");
    }

    public bool Contains_NotFindNonExistingWord()
    {
        _searchEngine.Setup("The quick brown fox jumps over the lazydog");
        return !_searchEngine.Contains("cat");
    }

    #endregion
}

Text Searching Algorithms and Indexing

There is lots of documentation on the Boyer-Moore algorithm, I didn't look into it too deeply as its already been implemented in .NET, but also as it is designed to work when is only possible to enumerate through the searchable text. As we have our searchable text in-memory we could leverage random access to the data. Aho-Corasick is the same, but it uses a cool tree to search for multiple words at the same time. It was that tree and later some influence from the Suffix Tree that led me to think I could store the entire searchable text in a tree.

A Text Tree

Consider the string "big bad barry" represented as a tree as below.

This tree is great for exact match string searching; Start with the root node and the first character of the search string. Check if the current node has a child matching the current character. If it does, move to the node and the next character and repeat the process. If it doesn't then the word doesn't exist in the searchable text. If we find a match for every letter, the word exists in the searchable text.

A Text Tree With Indexes

The previous tree is only good for the Contains() method as the tree has no information about where the matches are found or how many times they occur. To add this information to the tree we could add a new node at the end of every word with its index in the original text

We follow the same procedure as before to find words, only this time we know if we have found a complete word (if there is one or more index nodes on the last node we check). We can also find all instances of the word by traversing the rest of the tree and returning all the index nodes found. For example we can see that "ba" is in the string twice by finding all the index nodes after the 'a' node; In this case at characters 4 and 8.

From this tree we can also find entire sentences if our search algorithm is good enough. Basically it just needs to find the words in the sentence individually and then using the indexes and the word lengths work out which words combinations were next to each other in the original text.

The reason this is all possible is the tree has enough information to re-build the original string.

A Text Tree With Indexes and Suffixes

In the previous trees we can only search efficiently for words and prefixes. What if we want to be able to search for substrings? We can do this by also adding all the suffixes of each word to the tree.

Now we can search for words, substrings and sentences and get all the occurrences. That's pretty cool, but this tree is getting pretty massive compared to the original string.

Tree Implementation

I'm not actually going to describe how to implement this tree, its easier just to check the code and step through it if you have too.

The search engines I have implemented are

SimpleTextSearch Using IndexOf() and Contains() methods on System.String

RegexTextSearch Quick and dirty Regex implementation

SimpleTreeTextSearch The basic tree with no index information. Only implements the Contains() method

IndexTreeTextSearch The basic tree with index information. Search algorithm only looks for a single word

ExtendedIndexTextTreeSearch Extends the IndexedTreeTextSearch search algorithm to support multiple word search phrases

SuffixIndexTreeTextSearch Extends the IndexedTreeTextSearch to include suffixes in the tree.

ExtendedSuffixIndexTreeTextSearch Includes suffixes and the search algorithm from above that supports multiple word search phrases.

Some things I didn't do but would look into if I was actually going to use this anywhere:

Hash tables for each nodes children (currently using List<>)
Split text into words more effectively and maybe remove stop words (How to Split a Text into Words might be worth reading)
Case insensitive search algorithm (with case information still in the tree)
Removing casing from searchable text

To be continued...

To be honest, I got distracted away from this by getting into the new Silverlight 2 SDK for my DevSta entry, Desktop Racer. DevStar is an Australian Microsoft Visual Studios development competition and I figured Silverlight 2 is a little more exciting than a text searching algorithm by someone with little to no background text search theory. Never-the-less I've got the ExtendedSuffixIndexTreeTextSearch passing all the functional search test I've written. I've uploaded a copy of the results and the project in its current state.

HTML Functional Test Results

TextSearch VS2008 Project

I haven't yet written any performance tests to establish each engines setup speed, search speed, CPU usage and memory usage. I think this will be very interesting, while I'm pretty confident the tree search methods will be quick, I suspect they will require significantly more memory than storing plain text. I would not be surprised to find that the memory usage and setup times for my implementations of tree searches make them impractical to use for most real world search problems. Regardless, it has been lots of fun just getting the tree search functional but I'd be surprised if I've added any value to one of the most researched topics in computer science.

DevSta {Challenge 2008}

tarn@tarnbarford.net (tarn) — Tue, 07 Oct 2008 11:00:00 +0000

Tarn Barford

200 hours and 8 minutes later, my devsta entry is finally complete and submitted! Over the next week or so I'll write a post about writing it, how it turned out, what I learned and hopefully provide the code for download. For now, here is a brief description and some screen shots from in the game.

"Micro Machines was one of my favourite old school games and I thought it would be cool to race a Micro Machine around a fictional devstas desktop. Who wouldnt want to race around those icons and windows, trashing the desktop with tire marks?

Desktop Racer is a Silverlight 2 game. It has a Micro Machine like zoomed in mode, and also a mode where the whole desktop is visible. Players can play 1 or 3 laps time trails."

The game starts with an intro and some jokes in a command window

I used a cruise control window for the start lights

You can watch the devsta video while playing!

Finally a trashed desktop and the finish flag toaster.

Amazon S3

tarn@tarnbarford.net (tarn) — Tue, 09 Sep 2008 16:58:00 +0000

Tarn Barford

Introduction

As this site is currently hosted on a shared hosting environment with limited bandwidth and disk space we decided we would start hosting some content on Amazon S3.

I first saw S3 when a guy from Amazon presented it at a VIC.NET user group meeting. I thought it sounded pretty cool and signed up for an account as soon as I got home. I quickly found a use for S3 in a web application I was building at the time. I used S3 for all the files users uploaded to the site. I also used it for hosting some web components like static CSS, JavaScript and image files.

This works fine when you just want to reference static content on a web page. To do this you just need to upload the components to S3, make them public and modify the URLs in your HTML output to point to the S3 resource.

I did run into cross-domain browser security restrictions when I tried to host the entire site on S3 and use web services to get dynamic content. In this case I persisted and got dynamic behavior by dynamically inserting dynamic script elements into the DOM. While Google Maps does this very well to dynamically get map tiles, it doesn't work so well for scripts as the browsers blocks the page while a script is downloading. I would defiantly be looking at the alternatives before doing that again. In this case we just want some of the larger downloadable files off the web server and onto S3, which shouldn't be a problem.

Tools

I've always used S3Fox which is a Firefox plug-in to manage files on S3. But when I fired it up this morning it isn't accepting my credentials.

I tried using S3Drive which is a Windows Desktop application. This defiantly wasn't what I wanted. It makes S3 look like a "drive" in Windows that has files and directories. This hides the fact the S3 is actually a flat file structure. It seems to use a database to manage the virtual file structure (like a file allocation table). Anyway this would be really good for backing stuff up as you can treat it like a normal drive and drag and drop entire folders. Its no good for us as our content needs to be accessed by HTTP GET Requests.

I ended up using s3:/// which is an "experimental" Firefox plug-in. It works fine but doesn't seem to have the tools to manage the access control list (ACL) that S3Fox has. Anyway its enough to create a bucket and upload some files (Note: The files were already public and didn't require me allow public access. This is fine for me now, but might be a problem if you were uploading sensitive information you didn't want to share)

Virtual Sub Domains with CNAME records

One of the many things I like about S3 is that you can add a CNAME record on your domain and point it to an S3 bucket on account at amazon. So while you can still download your public content from the amazonaws.com URL:

http://static.sharpthinking.com.au.s3.amazonaws.com/sharpthinking.jpg

You can also download the same resource from a sub domain of your own URL:

http://static.sharpthinking.com.au/sharpthinking.jpg

But as you can see with trusty Firebug the image is still hosted from Amazon servers

Reading and Writing to S3 with .NET

I have an S3 project I used previously to read and write files to S3. I wrote it ages ago, copying large parts of Amazon examples from memory. Although it looks a little shabby, the one and only test still works, so I'm confident its still working OK. I'm sure there a better .NET wrappers around these days if you did a quick Google search.

Here is the source code. You'll notice it still has the namespaces from the original project, but hopefully my S3 credentials have been removed. You'll also need to add a reference to an NUnit assembly to run the test.

BlogEngine.NET, Live Writer and S3

As we use Live Writer to publish content to this blog, I am going to update the BlogEngine.NET Metablog API to upload the media elements to S3 instead of the file system. I've looked into it and it should be pretty straight forward. I'm pretty sure you can already get WordPress plug-ins that do this. I'll post the code when I get it working.

Flex Builder Linux

tarn@tarnbarford.net (tarn) — Sun, 31 Aug 2008 22:36:00 +0000

Tarn Barford

I read in a Linux magazine that Adobe were releasing Flex Builder for Linux. I thought this was really exciting as it was a great chance to use my new Linux box for some development and write a basic Flash application. It regularly annoyed me that I had never developed a Flash application.

Adobe Flex Builder Linux Public Alpha is currently up to alpha 4. It is a plug-in for Eclipse, a very popular, extendable development IDE build on Java. I had to install Eclipse 3.3.2 because earlier version weren't working on my machine and later version have problems with the plug in. I wanted to use OpenJava but I couldn't get Eclipse working on it, so I decided it would just be easier to use Sun Java.

Once I got Eclipse and Flex Builder working it was pretty fun. I was surprised how much of the stuff I learned for Silverlight could be applied to Flex Builder. In fact quite a few things I thought were unique to Silverlight were already features in Flex Builder.

I found ActionScript was quite easy to get into, it looks similar to JavaScript but with extra type information. The intellisense for ActionScript with Eclipse and Flex Builder is really good and really helped.

Anyway here is my first Flash application!

It was great from me to solve the mystery behind Flash files and find it is just source code. In this application there are some ActionScript classes, the main MXML and HTML/JavaScript to host it. Get the source code here.

Well that was pretty fun but there is so much more. I'd like to write an application that interacts with a server and try some of the cool data binding Flex Builder supports.

Using Linux on a Media Box

tarn@tarnbarford.net (tarn) — Mon, 25 Aug 2008 22:24:00 +0000

Tarn Barford

The existing system

We have a moderate to low spec'd computer in our lounge room running Windows XP. We built it about a year ago with spare parts lying round the house. We use the box for:

Sharing files for everyone on our network
Playing media through the TV and stereo system
Hosting Subversion

The box usually doesn't have a monitor, keyboard or mouse attached. As it has no input devices we use VNC server. This works out ok as we have several other desktops and laptops around the house we use to connect to it and control it.

Why change

The box was working fine so there was no real reason to change to Linux except that I really wanted some exposure to it. Eventually I want to try out setting up a LAMP web server and maybe try Struts and JSP.

I knew I needed to get the box doing everything it was doing before or else my house mates would be pushing to revert back to the XP system. I haven't used Linux much, but I figured if I spend some time with it I could probably get it doing all the things the existing box is doing. I wrote a list of the things I need the Linux box to do:

Use the TV as a primary display
Share files with our Microsoft boxes
Boot to an IDE without user input
Host Subversion
Allow remote access
Play media (DVD, CD's, MP3, DivX etc)

Nothing to difficult sounding there you would hope..

Choose a distro

I went into this with fairly limited knowledge of Linux. I'd installed Mandrake years ago as a dual boot on my personal PC but never really spent any time actually using it. I'd also used openSUSE while I was in Ireland getting training with OpenJaws travel software. Recently Brodie tried the Ubuntu live CD at work, but we didn't really get passed the gimmicky UI features.

After doing a bit of reading I still had no idea which one I wanted to use.

Trials and tribulations

I started with Mandriva for no real reason. On the first install it didn't detect my graphics card correctly and crashed horribly every time I went anywhere near the GUI graphic settings. I wasn't ready to start screwing with my xorg.conf yet, so I tried installing it again. I didn't have any luck this time either, I couldn't even finish the install because after I tried testing a graphic mode it returned me to a mode I couldn't read the text. (That mode reminded me of writing little real mode VGA graphics demos in assembly, it was easy to leave the console unreadable if you messed with the graphics mode using the BIOS interrupts and didn't return it to the correct mode, 0x80 I think I remember). I decided to try another distro in hope the install might do a better job detecting my graphics card.

openSUSE was much quicker to install but comes with very few extras. openSUSE did choose the correct open source drivers from my graphics card and installed perfectly. There were tools to configure a dual head display but after a couple of hours reading endless forums, manually editing my xorg.conf file, restarting the X system and unsuccessfully attempting to install the ATI binary drivers, I gave up. It wasn't a success yet but I did learn heaps more about Linux and was confident I could get the TV working if I persisted.

Finally I tried Ubuntu after I read that installing the ATI drivers was easy. Installing the ATI drivers was simple but it didn't immediately solve my problem. I still wasn't getting anything on the TV. After another couple of hours many more "atp-gets" I finally got the TV to clone the primary display.

Success!

Everything else was pretty easy;

I installed the proprietary stuff needed to play proprietary media formats
I used "apt-get" to get most of the additional software I needed which made things pretty easy
I could share a folder using the GUI
I could read my external NTFS drive, but I did have to force mount it.
There was an option in the administration to automatically login a user on startup
Remote access to VNC clients was build-in and I just needed to enable it
Subversion was easy to configure as it used the same command line functionality as the Windows version.
It was quite easy to get the Subversion server automatically starting and I learnt how run scripts on startup

Can't turn water into wine

Everything went so well I was inspired to try using a crappy DVB TV tuner card I bought, hated and never used to watch television. Even with a fast computer and perfect reception it would still start losing frames when the video was panning. What frustrated me even more was I could only use the software that came with it to use it, when I started using Vista I couldn't use it at all.

In about 20 minutes I able to add it, tune it and get some video out it. The only problem was it looked even worse than when it was using its own software on XP. I know there is plenty more I could do to get it working better, I was happy I got it working at all and probably won't use it again.

The response

After two weekends I've finally got the box back where it belongs, doing everything it was before except now running ubuntu. I can still boot into the old XP system if I boot off the secondary hard disk, but I'm happy leaving ubuntu as the default boot and not even adding XP as an option in the boot loader. My house mates are happy too, apparently the VNC is much better that it was with windows but they want a music player that supports play lists.

That's it

I'll report back on how the Linux box handles over time and what I do with it. I've also created a VMWare Linux instance on my personal computer which I'll hopefully use to try out running web servers and maybe get into a little bit of programming.

Its been frustrating at times but over all its been lots of fun getting this Linux box up and running, I've learned lots about Linux and I want to learn more.

Network Checks

tarn@tarnbarford.net (tarn) — Tue, 22 Jul 2008 19:12:00 +0000

Tarn Barford

I work with an IT manager who is very into knowing about how all the servers on the network are performing, and rightfully so. But the process isn't great; he gets an IT person to remote into all the servers daily and manually record information in a spreadsheet. These include things like disk usage, logs, log sizes, databases, web servers, processes, memory and CPU usage.

A while back I wrote a Powershell Script that recorded heaps of the results for him. Unfortunately it didn't take many server changes before the script had more errors than results, I guess he could do stuff with all that extra time. I would fix stuff from time to time, but more often than not I was too busy with things I had to do or things I wanted to do.

The other day I decided it would be cool if I could write all the checks as NUnit tests. That way he could just run NUnit, point it towards the assembly and get cool Red/Green light testing of the network status. I could write test like:

[Test]
public void Server1_DiskUsage_CDrive_LessThan80Percent().

I also found this article: Setting up CruiseControl.NET to be a Continuous Monitoring Server. I haven't read it in detail, but I think its a pretty cool idea.

I don't have time to implement this yet, but I have started writing some of the tests from the Powershell script in C#. I started some code to get the memory usage from a server and I was immediately reminded how annoying it was trying to work out how values in WMI related to the items on the Performance tab in task manager. So marked up an image of the task manager window with the corresponding properties of the WMI.

Performance

See Win32_OperatingSystem_Class on MSDN for all the properties and descriptions

Anyway you might now be asking to see the C# code, well here it is. There is no exception handling as I was running it as an NUnit test which handled them for me.

private OperatingSystemInfo GetOperatingSystemInfo(string serverName)
{
    string wmiObjectName = "Win32_OperatingSystem";
    string connectionString = string.Format("\\\\{0}\\root\\cimv2:{1}",serverName, wmiObjectName);
    ManagementClass os = new ManagementClass(connectionString);

    foreach (ManagementObject obj in os.GetInstances())
    {
        OperatingSystemInfo osInfo = new OperatingSystemInfo();
        osInfo.NumberOfProcesses = (uint)obj.GetPropertyValue("NumberOfProcesses");
        osInfo.TotalVisibleMemorySize = (ulong)obj.GetPropertyValue("TotalVisibleMemorySize");
        osInfo.TotalVirtualMemorySize = (ulong)obj.GetPropertyValue("TotalVirtualMemorySize");
        osInfo.FreeVirtualMemory = (ulong)obj.GetPropertyValue("FreeVirtualMemory");
        osInfo.FreeSpaceInPagingFiles = (ulong)obj.GetPropertyValue("FreeSpaceInPagingFiles");
        osInfo.FreePhysicalMemory = (ulong)obj.GetPropertyValue("FreePhysicalMemory");
        return osInfo;
    }
    return null;
}

public class OperatingSystemInfo
{
    public uint NumberOfProcesses { get; set; }
    public ulong TotalVisibleMemorySize { get; set; }
    public ulong TotalVirtualMemorySize { get; set; }
    public ulong FreeVirtualMemory { get; set; }
    public ulong FreeSpaceInPagingFiles { get; set; }
    public ulong FreePhysicalMemory { get; set; }
}

I hope it helps. I'm getting back to playing with the Microsoft MVC Preview 3 which is really cool and lots of fun. I hope to get a post out about what I like and don't like about it soon.

Microsoft Robotics Studio

tarn@tarnbarford.net (tarn) — Thu, 17 Jul 2008 20:50:00 +0000

Tarn Barford

I finally decided to download Microsoft Robotics Studio (MSRS) and have a look at it. I had no idea what I would get, but I was hoping I could use it to simulate autonomous robot soccer matches and I wanted to see what else it could do. I spent an evening having a play with it, and here are my first impressions.

The SDK

The download was much smaller than I expected (approximately 87Mb), for some reason I thought it was going to be a plug-in for the Visual Studios IDE. It's just a collection a projects and some tools. I haven't yet download the April CTP which must have a bit more; its around 400Mb.

I looked at a few projects and most appear to be based around a Decentralized System Services (DSS) application model, or framework. The application model provides a framework for concurrent distributed services to interact. These could be sensors and actuators, controllers or processors. The services use a light-weight REST messages to communicate and interact. During simulation your services run normally, but instead of getting real images from the camera and really actuating motors you communicate with a simulated environment. The DSS is setup and linked to relevant service assemblies by XML files. There is a visual editor to assist configuring these files.

The simulations uses the AGEIA physics engine and DirectX 3d. I don't know much about the physics engine, except its made by NVidia and it can use dedicated physics hardware if you have it, which I don't. I have a mate who writes low level tests for graphics cards as a job and writes his own physics engine for fun. I'm sure he'll have plenty to tell me about it. Actually - Bryce you know we want you to post on this blog

A visual programming language is provided. Its a visual data flow designer that can connect services that fit into the DSS framework. It seems similar to event based circuit simulation tools I've used, but with a very simple user interface and debugging tools. I did some of the tutorials but the only thing that really kept me interested was that I could send text to the Text to Speech convertor. I don't have a robot that I want to work right now, maybe then I would rate this tool more highly.

The Robots

There is a fairly wide price range of off-the-shelf robots and robot kits that the Microsoft Robotics Studio that have been can be simulated (I found a list of these in this article). I was immediately drawn to a robot described as a "fighting Japanese robot" (for around $1500). When I looked him up on YouTube I was surprised to find that I'd seen him before, but he was dancing! Lego also has a more reasonably priced Robotics kit ($120). There is also the Nao which will apparently be used in RoboCup 2008! (I'm not sure if those Sony dogs compete in a different category, or are obsolete)

I haven't yet decided how much, if any, I am willing to fork out for a robotic toy. Can I write it off as tax deduction?

RoboCup and RoboChamps

The RoboChamps site was disappointing for me, it seems to be Silverlight 2. It caused javascript errors didn't rendering anything for me. I didn't want to spend the evening updating my Silverlight installation so I moved on.

Things started looking up when I found the Microsoft Soccer Simulation download. This download has the soccer simulation framework and a demo project with a two-on-two autonomous robot soccer match setup. The robots aren't humanoid, they're two wheeled things with a kicker and a camera. The robots have only been setup for two tasks; Find Ball and Approach Ball. Hence its not really a great spectacle, but a fantastic starting point.

There is a very basic vision processing method the robots use in the demo. The purpose of the method is simple; it takes a 320x200 array of colours from the camera and determines whether or not the ball is in the image. I had a look at the code and quickly realised I'd need to read up on some vision processing theory to really understand how it was working.

I then looked into an announcement: MSRS Nao Simulation Competition for RoboCup 2008 is now available. The post has everything you need to get started writing code for the humanoid Nao robots and simulating 4x4 or 8x8 soccer competitions. The RoboCup simulations uses the Microsoft Soccer Simulation. While these guys are much cooler than the Microsoft robots, they also take a lot more grunt to simulate and there are more of them on the pitch. My quad core processor and low range graphics didn't simulate them very well. (I've never been into computer games, I spend too much timing working and developing on computers and prefer board games, till now, my low range graphics card has always done enough for me).

Conclusion?

I only scratched the surface but I'm glad I checked it out. I'm wrapped there's a framework I can write C# services for autonomous robotic soccer players. I just need a better graphics card, more time and some mates to contribute.

It seems the framework can only be used to make truly autonomous that have a processor running Windows and the .NET framework. This is a big step from the 8-bit microcontrollers at university. Robots have got cooler since then, but I'd look further into other operating systems, languages and tools before I started developing code for a real robot in MSRS.

If you just want to write some cool bot code and battle you mates, checkout this Scott Hanselman article, Learning Opportunity - .NET Terrarium is back! I haven't tried the original or the new, but I'm looking forward to it.

Finally, if your feeling a little concerned that the rise of the machines is fast approaching, this video won't give you any comfort. But it's also the topic of one of my favourite XKCD cartoons that doesn't involve velociraptors.

Web Development Frameworks

tarn@tarnbarford.net (tarn) — Thu, 17 Jul 2008 10:40:00 +0000

Tarn Barford

I've just started getting into the Microsoft MVC Preview 3. Its cool and I'm having heaps of fun with it. Who thought you could have multiple form tags on a page ;-)

I wrote a post earlier this year after spending a couple of weeks in Ireland with some Java developers. I originally posted it internally at work with the intention of stirring up some .NET developers in my team. Today I decided to check out what I'd written to see if added any value to Microsoft MVC framework I am using now. It doesn't, but I decided to post it publicly anyway.

Its taken a while but you knew it was coming. Having come in contact with some Java development from a .Net world I wanted post some observations and opinions. Ive really only had one day of exposure to server side Java, so Im sure this article wont be completely technically correct and Im happy for people to comment on my errors or oversights.

Firstly Ill start with the background. These boys are using Java SE (Standard Edition I guess) and not EE (Enterprise Edition). My understanding is that Java EE comes with some additional web framework elements, but I stand to be corrected on this.

Java SE at first glance actually looks like classic ASP with Java code. Just to clarify, both a server side languages and not VBScript and Javascript respectively. Both use the <% %> characters to separate the code from the response. Both can just write to the output directly. Both .jsp and .asp files can be called directly by URI and can respond (provided they are hosted on suitable web servers). So right now this looks like I should be comparing Classic ASP to Java, but thats no brainer. Java is a native OO language, platform independent and also has native support for threading; game over classic ASP.

There still is the issue that Java server side code can very quickly degenerate to the spaghetti code we have seen in classic ASP code. And it can, and would if written badly. But to be fair, any language can. In some ways being a native OO language you have the tools to protect you against this. Really its still pretty primitive for a web framework.

The comparison with the ASP.Net framework becomes a much more interesting when you add the Apache Struts framework to Java. Basically the Struts framework is an implementation of the Model View Controller (MVC) pattern. The MVC pattern has is roots in an Apple SmallTalk framework. It wasnt a framework that was used for web applications, but it did have clearly defined model, view and controller components.

MVC is a pattern to implement a system that takes requests and deliver responses. This is what HTTP is all about, all web servers do it.

The MVC pattern consists of three components. There is the Model which is the domain logic (a.k.a business logic or middle tier) and it also includes the data persistence the domain logic works with. There is also the Controller which is responsible for taking the request and routing them to the correct business logic, it then routs the output of the logic to a View. The View selected by the Controller takes the output from the domain logic and presents it to the user. These guys weren't using the standard "View" from Apache struts they used XSLT to transform XML data from the model to HTML.

ASP.Net uses a page life cycle pattern to handle the same problem. We know this well; An .aspx page is requested, the aspx page points to some page logic, which can then calls some domain logic. The page then binds data to objects capable of rendering a view of that data. It seems normal that events on the rendered page are posted back and handled by that same page code. This is vastly different to how the MVC pattern works.

_This has already got to long, so Ill post another blog soon to continue this discussion and hopefully get stuck into the inevitable debates about the merit of both.

Feel free to comment corrections, additional information or opinions.

Tarn

I hope to write some more relevant posts about my recent experiences with the Microsoft MVC framework soon. At the moment I'm having too much fun coding with it, and don't yet have much to say that Rob Connery, Scott Hanselman and Scott Gu haven't covered.

Page summary in Google search results

tarn@tarnbarford.net (tarn) — Mon, 23 Jun 2008 15:46:00 +0000

Tarn Barford

I just realised the summary of some of my pages in Google search results were terrible. So I looked into sites that had good descriptions and discovered Google will use the "description" meta tag for the summary when it's present.

<meta name="description" content="Your Page Summary Here!">

This is easy to do for static pages, and is not the hard to do for dynamic pages. If you use Master Pages its common to define a place holder inside the head tag. This means any dynamic pages can implement a content server control for that place holder and insert a meta tag with relevant information to be displayed in the summary of the Google search results.

You could also give the head tag ID and runat="server" attributes. This means you can find it when your processing the page and dynamically append a meta tag element to its children property.

Minify Javascript

tarn@tarnbarford.net (tarn) — Sun, 22 Jun 2008 22:04:00 +0000

Tarn Barford

One of my favourite books on web development is High Performance Web Sites: Essential Knowledge for Front-End Engineers by Steve Souders. From the authors role as Chief Performance Yahoo! he describes 14 surprisingly simple rules for improving front end performance of web sites. Most professional web developers should know most of the best practices, but might also pick up a tip or two. The rules are justified with quantifiable test results, which is great when you're trying to convince a manager its worth while implementing front end improvements.

There also is 34 categorized rules for best practice on the Yahoo! web site. Yahoo! provides YSlow a plug-in for Firefox which analyses pages and explains rules that would improve front end performance.

This post is about implementing rule 10, Minify JavaScript. I have known about minifying Javascript for a long time but had never implemented it, today I decided to give it a go on a site I run. I had a look at a few minifying tools:

JSMin - Created by Douglas Crockford, JSMin is the most basic of the minifying tools I've seen out there. It has been ported to .Net, but its not really a very sophisticated tool. It removes line breaks, white space and doesn't do any variable renaming. There is a site that you can paste some JavaScript in a compress it with JSMin or Packer to have a look at the output.

Packer - Created by Dean Edwards, Packer uses a different approach to minifying JavaScript. Packer compresses the code into an expression that can be evaluated to its original form by the Javascript eval function. I think this is cool and I'm not influenced by people out there saying you should never use the eval function. Packer is used and the eval function has been around since Javascript 1.0, it will be supported by modern browsers I'm supporting. The two main criticisms of Packer I am a little concerned about are; There is a delay after the script is received by the browser before it evaluates to your actual Javascript code. With that in mind, unless you design the page well, you may intermittently cause Javascript errors by referencing code that doesn't yet exist. The second criticism that while packers compression is very good, it doesn't compress much more when gziped before being sent to the browser. Packer has been ported to .Net, Perl and PHP.

YUI Compressor - Created by Yahoo!, this is what I wanted in a minifying tool. It removes all unnecessary characters and does variable renaming. The YUI Compressor is implemented in Java, there is an open source project on CodePlex to port it to .Net but at the time or writing this, it has only ported the CSS compression algorithm. Julien Lecomte an architect at Yahoo talks about the YUI Compressors performance in his blog.

Dojo ShrinkSafe - Create by the Dojo people, it seems to be doing the same thing as the YUI Compressor. I didn't look very hard, but I didn't find a .Net port for this one either.

There is also Minify a PHP library on Google Code for creating servers that can combine and minify script files on the fly before they are sent to the browser. The project I'm working on is hosted by IIS and S3 so I didn't look into this any further.

I decided to try the Yahoo YUI Compressor and Dojo ShrinkSafe even though there was no .Net port for either. JSMin didn't provide the compression I wanted to make it worth while. I outlined some of my concerns about Packer, and I decided I would just prefer plain compacted Javascript.

Then the was the question of whether its ok to use a Java application in my .Net build process? Java is a free and widely used platform, and in this case the best tools for are built on it. I have the JRE on all the computers I use anyway, so for me its no problem. If you need to compress scripts at runtime in .Net it may be worth looking a bit further into other .Net solutions and ports.

Both the Yahoo! and Dojo compressors were easy to use from the command line:

java -jar yuicompressor-2.3.5.jar scripts\combined.js -o scripts\yui_compressed.js

java -jar custom_rhino.jar -c scripts\combined.js > scripts\dojo_compress.js 2>&1

The only thing I had to do to get them working was change the UFT-8 encoding of some of my script files to ANSI. After that everything on my site worked as is should with both libraries. If I had any problems I was going to give JS Lint a try. It is recommended to use JS Lint to verify your Javascript before compressing it.

After using a tool to combine my uncompressed script files, the final script was 16,042 bytes. Below are the result of using ShrinkSafe and YUI Compressor on that code:

Dojo ShrinkSafe, 11,543 bytes, 71%

YUI Compressor, 10,448 bytes, 65%

So what did I discover? I was really happy with both tools, they both compressed and didn't break my code. Because both can be run from the command line they should be easy to integrate into my build process, provided my build and development machines have Java installed.

At this stage I'm not gziping my scripts, but that has more to do with my applications architecture, the scripts are being served by a storage in the cloud solution. I may look into providing the correct meta data to statically host gziped scripts, but that will be the topic for another post.

As for getting it into my build process I just used the post build tasks hook in Visual Studios. It could just as easily be part of your NAnt script.

I hope this helps the front end performance of your web pages too.

Anonymous Methods - Understanding Scope

tarn@tarnbarford.net (tarn) — Thu, 19 Jun 2008 11:42:00 +0000

Tarn Barford

Anonymous Methods were introduced in .net 2.0. If you haven't used or seen them before I suggest reading some of the many blogs and MSDN articles about them. I'm not going to introduce them here, I'm going to briefly look into how the scope works when they are invoked from outside the scope they were created in.

Below I have a simple example of a class that has a member variable and a method that returns an anonymous method. The only purpose of this little class is to investigate some scope issues.

public delegate string FooDelegate();

public class Foo
{
    public string Bar { get; set; }

    public FooDelegate GetDelegate()
    {
        string bar = Bar;
        return delegate()
            {
                return string.Format("Member: {0}; Local: {1}", Bar, bar);
            };
    }
}

You can see that in this class the anonymous method we are creating references a local variable and a member variable of the class. This is completely legitimate code in .net 2.0, 3.0 and 3.5. Now lets see how it works, below is a simple example.

Foo foo = new Foo();
foo.Bar = "test";

FooDelegate fooDelegate = foo.GetDelegate();

string result = fooDelegate.Invoke();

The result is "Member: test; Local: test"

You can see that when we invoked the anonymous method both the local and member variables evaluated to "test". Once you accept its ok to reference variables from the scope the anonymous method was created in, this behavior makes sense.

But what happens if we change the property after we create the anonymous method? What would we expect the values to be when we invoke the method? If you've used anonymous methods in other languages it might be quite obvious. If you don't know, or want to validate that you do, the example below will do just that.

Foo foo = new Foo();
foo.Bar = "test";

FooDelegate fooDelegate = foo.GetDelegate();
foo.Bar = "update";

string result = fooDelegate.Invoke();

The result is "Member: update; Local: test"

As you can the see the method did use the updated property, but only for the class property. The value of the local variable it was referring to when the anonymous method was created has not been changed.

Normally a local variable would be discarded and later collected by the GC once execution of the block is complete. In this case it's still in the scope of the anonymous method and is therefore not discarded while the anonymous method is still referenced.

The local variable is not updated with the new value as we set the local variable when we created the delegate, and we changed the class property after that. In this case we can't change the value of that variable as we no longer have a reference to it once we leave the GetDelegate() method. Its only reference by the delegate itself.

This example works because strings create a new instance of a string when they are assigned. The example would behave differently if the local variable was a reference to the same object as the class property.

Using the local variable in an anonymous method that is called from outside scope it was created might not be something you ever do in .Net, but its a concept .Net developers should understand. I've haven't been able to think of a simple, relevant, real world .net example for this, but I do use the same concept in some Javascript I write about in another post.

EDIT: This is known as closure. This article I found on the VS2008 Start Page talks about it and discusses the MSIL generated : Lambdas - Know Your Closures

AdRotator: Injecting scripts

tarn@tarnbarford.net (tarn) — Sun, 15 Jun 2008 18:55:00 +0000

Tarn Barford

This article is part of a series of posts about various aspects of writing web controls for ASP.Net using an ad rotator as an example. The AdRotator WebControl Example post has links to related posts and downloads.

The AdRotator needs to insert a class definition script in the output once, even if multiple AdRotator controls are declared on the page. It also needs to insert a script that instantiates that class prototype for every AdRotator control on the page. Fortunately there are static methods on the System.Web.UI.Page object that will do this for us.

Page.ClientScript.RegisterStartupScript(Type type, string key, string script)

The keys here a crucial, there is only one script rendered for each key. If we inject the definition script with a static key it means no matter how many instances control try to register it, only one instance of script will be sent to the output.

To to insert a unique script for every instance of the AdRotator we need to use a unique key. I use the server side controls ClientId property which will be difference for every instance of the control on any page. You could also use a GUID.

This is clearly the weakest post of all the posts about the AdRotator. I just kind of got this working an never really got very inspired to learn anything else about it. I posted it anyway as I felt I did need to cover this topic for completeness of the series. I'll try to update it soon.

AdRotator: Json serialization

tarn@tarnbarford.net (tarn) — Sun, 15 Jun 2008 18:46:00 +0000

Tarn Barford

The AdRotator needs to pass some data to the client side code so I use a Json serializer. I use the same class we use on the server side. The Json serializer converts an instance of it, or in our case a templated list of it, to a sting on JSON we can emit in the output. Json is JavaScript Object Notation and literally describes an object in Javascript that we can use in our client side Javascript code.

The object is very simple and only contains three strings and constructors. I have removed some of the attributes described in other articles as they are not relevant in this context.

public class ImageItem
{
    public ImageItem()
        : this(string.Empty, string.Empty, string.Empty)
    {
    }

    public ImageItem(string linkUrl, string imageUrl, string displayTime)
    {
        LinkUrl = linkUrl;
        ImageUrl = imageUrl;
        DisplayTime = displayTime;
    }

    public string LinkUrl { get; set; }
    public string ImageUrl { get; set; }
    public string DisplayTime { get; set; }
}

The AdRotator has a public property Images that is List of the ImageItem type above.

public List<ImageItem> Images;

During the RenderContents event of the AdRotator we can serialize this list into a string of Json.

JavaScriptSerializer serializer = new JavaScriptSerializer();
string json = serializer.Serialize(Images);

The Json will look something like this.

[{"ImageUrl":"Images/Winter.jpg","LinkUrl":"#","DisplayTime":"1000"},
 {"ImageUrl":"Images/Sunset.jpg","LinkUrl":"#","DisplayTime":"4000"}]

The following script and corresponding output will hopefully demonstrate how we can use the Json on the client side as data.

var imageList = [{"ImageUrl":"Images/Winter.jpg","LinkUrl":"#","DisplayTime":"1000"},
                 {"ImageUrl":"Images/Sunset.jpg","LinkUrl":"#","DisplayTime":"4000"}];

for(var i=0; i<imageList.length; i++)
{
    document.writeln(imageList[0].ImageUrl);
}

Output:

Images/Winter.jpg
Images/Winter.jpg

Well that's all the information I'm going to include in this article. I'd like to go further into how the Json is used in the AdRotator but I've rambled on long enough. I think there is enough here to see that Json is a pretty cool language for communicating data between the server and client side Javascript.

I should also note that the JavaScriptSerializer is marked as depreciated and points towards the DataContractJsonSerializer. I briefly tried to get this working, but I couldn't seem to find the object in my system assemblies. To read more about that, check out DataContractJsonSerializer in .NET 3.5 that discusses using the DataContractJsonSerializer.

I hope you've found the article interesting, if you have you might be interested in reading more article about the AdRotator example or you might want to download and checkout the example project.

AdRotator: Client side code

tarn@tarnbarford.net (tarn) — Sun, 15 Jun 2008 18:23:00 +0000

Tarn Barford

The Prototype

I wanted the client side code to be as object orientated as possible. I implemented the behaviour of our AdRotator as a prototyped Javascript object, equivalent to a class definition in other OO languages. This means every instance of the AdRotator on a page only needs to be an instance of the ad rotator prototype.

I create the anchor and image elements on the fly. I need the client side object to know about these elements to change them when the image changes. I could have created them in the server-side and passed their ids to find them in the DOM, or found them by looking through our containers child controls. In the end creating them on client side was just slightly easier.

The images parameter is an array of objects that have ImageUrl, LinkUrl and DisplayTime properties. I pass it in as Json (Javascript object notion) that I generated on the server using a Json serializer. You'll see the Json later in this article, and I've also added an article about generating Json script on the server side.

You notice the strange way setTimeout is called in the RotateAd function. It is all about Javascript scope and is far to big an issue to go into here. All I will say is that if I had used "this" instead on the "thisObject" which is set to "this" anyway, it might not point to this instance of this class, it might actually point to the document, window or whatever the current "this" actually is when the timeout event occurs.

function AdRotator(id, ads, height, width)
{
   this.id = id;
   this.ads = ads;
   this.index = 0;
   this.container = document.getElementById(id);
   this.anchorElement = document.createElement('a');
   this.imageElement = document.createElement('img');
   this.imageElement.setAttribute('height', height);
   this.imageElement.setAttribute('width', height);
   this.anchorElement.appendChild(this.imageElement);
   this.container.appendChild(this.anchorElement);
   this.RotateAd();
}


AdRotator.prototype.RotateAd = function()
{
    var currentAd = this.NextAd();
    this.imageElement.setAttribute('src', currentAd.ImageUrl);
    this.anchorElement.setAttribute('href', currentAd.LinkUrl);
    var thisObject = this;
    setTimeout(function() { thisObject.RotateAd(); }, currentAd.DisplayTime);
}

AdRotator.prototype.NextAd = function()
{
    var ad = this.ads[this.index];
    this.index ++;
    if (this.index == this.ads.length) this.index = 0;
    return ad;
}

The Instance

For each instance of the ad rotator we simply use this script. You'll immediately notice that the code is filled with place holders. I replace these placed holders on the server-side before I inject it into the output. I'll describe these in more detail and finally I'll include the source of a rendered page to see this as the browser sees.

You might also notice that we are not actually assigning this object to anything. Hopefully because it will always be referenced by a setTimeout event it will always stay in scope. If I ever do have any problem with this I could easily create and array in the definition and push each instance into the array.

(new AdRotator('$ElementId', $Images, $Height, $Width));

The $ElementId placeholder is replaced with server control the server controls ClientId property. This means we can reference the DOM object (a span element) that the server outputs as a container for the control. The ClientId property is very importing when creating elements on the server and interacting with them on the client side. Often elements are not output with the ID of the server control. One of the reasons is that its possible in ASP.Net to have two control with the same ID in different placeholders. Or the same ID in a master page and in the content page. I won't go into detail about how this naming works, but its pretty clear when you view the source of the pages rendered by an ASP.Net server. It is a string as it's used by definition to find an element with the getElementById function.

The $Images place holder is replaced with Json. Effectively a Javascript object describing an object which is a an array of objects containing ImageUrl, LinkUrl and DisplayTime properties. I have written an article about how this Json is generated on the server side.

The $Height and $Width place holders are replaced by the corresponding values on the AdRotator server control.

An example of how this script is actually output by the server should hopefully demonstrates how this client side script actually works, and hopefully how it works for multiple controls.

<script type="text/javascript">
(new AdRotator('ctl00_MainContentPlaceHolder_rotator1', 
               [{"ImageUrl":"Images/Winter.jpg",
                 "LinkUrl":"Winter.aspx",
                 "DisplayTime":"1000"
                },
                {"ImageUrl":"Images/Sunset.jpg",
                 "LinkUrl":"Sunset.aspxd",
                  "DisplayTime":"4000"}],
                200, 200))
</script>

This is just one article in a series describing various aspects of writing ASP.Net server side controls. You can download the example code or assembly and see how it works.

AdRotator: Declaratively setting a collection property

tarn@tarnbarford.net (tarn) — Sun, 15 Jun 2008 18:22:00 +0000

Tarn Barford

I really wanted to make it possible for the control to be added completely declaratively. Much the same way as you can create a list box with list items declaratively.

<asp:ListBox ID="2" runat="server">
    <asp:ListItem>Option1</asp:ListItem>
    <asp:ListItem>Option1</asp:ListItem>
</asp:ListBox>

This actually turned out to be really easy after I found a great example from Microsoft, Web Control Collection Property Example. I'd advise getting straight into that if you are implementing this and need more detail. I will just briefly describe what I had to implement for the AdRotator Control.

Attributes

I regularly set public properties on a web control with the appropriate type converters declaratively. For example where the public properties Height and Width declared as below.

public class AdRotator : WebControl
{
    public new string Width { get; set; }
    public new string Height { get; set; }
    ...
}

They can be set declaratively in ASP.Net web files

<x:AdRotator ID="AdRotator1" runat="server" Height="100" Width="100" />

Children

When it came to adding the child data, I had already implemented a strongly typed public data source property on the AdRotator control, so all I really needed to do to get it working was add attribute meta data to some classes and properties.

On the actual control I had to add meta data about the type of child elements we the control will support.

[DefaultProperty("Images")]
[ParseChildren(true, "Images")]
...
public class AdRotator : WebControl

On the property "Images" I added the following metadata. I didn't need it all, the "Category" for example is used when the object is displayed in a property list control. The attribute classes and CollectionEditor are all in the System.ComponentModel.Design namespace and the UITypeEditor is in System.Drawing.Design namespace. As the collection we are exposing is strongly typed we don't need to extend the CollectionEditor class.

[Category("Behavior")]
[Description("Image URL collection")]
[DesignerSerializationVisibility(DesignerSerializationVisibility.Content)]
[Editor(typeof(CollectionEditor), typeof(UITypeEditor))]
[PersistenceMode(PersistenceMode.InnerDefaultProperty)]
public List<ImageItem> Images

The ImageItem class also need some additional meta data.

[TypeConverter(typeof(ExpandableObjectConverter))]
public ImageItem() : this(string.Empty, string.Empty, string.Empty)
{

    public ImageItem(string linkUrl, string imageUrl, string displayTime)
    {
        LinkUrl = linkUrl;
        ImageUrl = imageUrl;
        DisplayTime = displayTime;
    }

    [Category("Behavior")]
    [DefaultValue("")]
    [Description("Anchor Href Url")]
    [NotifyParentProperty(true)]
    public string LinkUrl { get; set; }

    ...

}

Finally

By the time the RenderContents method is fired, the Images property is fully populated with all the items we declaratively added. When we are declaring the image items we are getting nice intellisense. That's everything we set out to do.

ad_rotator_intellisense_example1

Although I didn't go into much detail here we did get everything I wanted in the AdRotator example. I find this quite interesting as the concept of XML based object declarations was build into the XAML SilverLight and WPF technologies.

AdRotator: Embedding resources in an assembly

tarn@tarnbarford.net (tarn) — Sun, 15 Jun 2008 18:20:00 +0000

Tarn Barford

The AdRotator has several scripts which it emits in the output. I wanted the AdRotator completely encapsulated in a single "dll" file. I choose to embed the scripts in the assembly and read them back at run time. I could have included the scripts in the code as a string but I feel having escaped script declarations in code is more difficult to read and maintain. It also makes it difficult for the IDE to provide intellisense.

Its easy to embed a file included in a project into the projects output assembly. The setting can be found by selecting properties of the file you want embed and selecting "Embedded Resource" as the "Build Action".

embed_resource_example

The file can be read back as a stream by reflecting the assembly and getting the resource by name. Below is an example of reading a resource by name into a string.

private string GetResourceString(string resourceName)
{
    Assembly assembly = GetType().Assembly;
    string resourceFullName = string.Format("{0}.{1}", assembly.GetName().Name, resourceName);
    string resourceString;

    using (Stream stream = assembly.GetManifestResourceStream(resourceFullName))
    {
        TextReader reader = new StreamReader(stream);
        resourceString = reader.ReadToEnd();
        reader.Close();
    }

    return resourceString;
}

As far as the code above goes, if it was a more professional application the method would be protected by a unit tested wrapper with application specific exception handling.

There is a free tool created by the Sells Brothers you can use to view the resources embedded in an assembly that may be useful.

AdRotator WebControl Example

tarn@tarnbarford.net (tarn) — Sun, 15 Jun 2008 18:01:00 +0000

Tarn Barford

I didn't really actually want to write another ad rotator, but I did think it would be a good example control to implement. It has the scope to cover some aspects of writing web controls I wanted to learn more about and discuss. I decided to build an ASP.Net web control that cycles through a list of images on the client side. There are a few areas I want to discuss.

I wanted the control completely encapsulated within an assembly. So all you'd need to use the control would be a copy of the assembly. To do this I embedded some resources in the assembly and read them out at runtime. I discuss this in the article.

AdRotator: Embedding resources in an assembly

I wanted the control to be able to be created completely declaratively, and I thought providing the sort of intellisense you get when using standard ASP.Net controls would be nice.

AdRotator: Declaratively setting a collection property

I wanted the server-side control to inject Javascript into the output. I wanted the control to be smart enough to only output the Javascript prototype, or class definition once, even if there were multiple controls on the page. I wanted the client side code to be as object orientated as possible.

You can download the source code which includes an example website in the solution.

Source Download

I would like to add a post about supporting difference storage mechanisms and using the control, but I have other projects demanding my attention. Hopefully I'll get back to it sometime.

I Hope you've found some of these articles useful or interesting. I'll keep and eye on the comments, contribute to further discussion, and update the posts where errors or omissions are noted.

Cheers

String.Format style parameters

tarn@tarnbarford.net (tarn) — Thu, 05 Jun 2008 14:34:00 +0000

Tarn Barford

Just a reminder that the String.Format parameters aren't magic, we can make methods with these parameters. Here is an example.

public string StringFormat(string format, params object [] parameters)
{
      return string.Format(format, parameters);
}

I actually can't think of any great uses for it now, but I do use it to expose string.Format style parameters on a LogMessage method in a custom logging tool. This makes the code cleaner where I was using string.Format before sending it to a LogMessage method as a string.

Next week, Named Parameters! (well probably not, but if I ever use them.. )

Dynamically inserting dynamic scripts

tarn@tarnbarford.net (tarn) — Tue, 03 Jun 2008 15:40:00 +0000

Tarn Barford

I came across this technique when I was trying to get around the cross-domain restrictions applied by modern browsers to the XMLHttpRequest object. Basically you can't do it.

This got me thinking, how does Google Maps do it? I have Google Map code in pages hosted on my domain. Somehow it's going to the Google server, from the browser, to get images. It can't be using XMLHttpRequest to do it, so how does it work?

Its seems ridiculously simple. You can reference scripts from another domain. You can use javascript to manipulate the DOM and web servers can deliver dynamic content based on a request.

So all we have to do to dynamically request a dynamic script is run some javascript that will insert a script tag with the source attribute pointing to the desired URL in the document head tag. The browser will then request the script from that URL. The server can return a script based on the request parameters provided. When the browser receives the script it will add it to the DOM.

My example here doesn't actually get the scripts off an external server or dynamically generate the script. The example has buttons that insert a script element into the head element when clicked. The script elements source attribute depends on which button is clicked, but both local script files display an alert message when loaded.

Download Demo (> 1k)

To see this example running, just open the folder as a Website in Visual Studios and run it. I'll add a link to a live demo soon.

While this works really well, I will go into a light-weight Javascript framework I wrote that wraps round this concept and provides response, timeout and some error handling events as you would expect in most XMLHttpRequest wrappers.

Client-side onload event and Master pages

tarn@tarnbarford.net (tarn) — Mon, 02 Jun 2008 10:03:00 +0000

Tarn Barford

First blog ever for me, YAY! I'll add to it regularly (yeah right).

Anyways, you've just added some cool JavaScript to your page and you want to call it when your page loads. Usually you'd just add an attribute to your body tag, something like:

<body onload="MyLoadFunction()">

But you find that your body element is defined in your Master page and you can't just add it to the Master as not all the pages that use the master will implement MyLoadFunction(). There are heaps of ways of getting round this problem, but I've found the following solution to be quite elegant.

The solution basically involves creating a method on your Master page to add an onload attribute to the body tag. So in the content page you want to add an onload method you would call something like

Master.SetBodyOnloadAttribute("MyLoadFunction")

To implement this method we need to be able to find the body tag and add the attribute. We can find the body tag if we put id and runat="server" attributes it when we define it:

<body id="masterBody" runat="server">

Then we can implement our SetBodyOnloadAttribute method:

public void SetBodyOnloadAttribute(string value)
{
    HtmlGenericControl bodyControl = FindControl("masterBody") as HtmlGenericControl;
    if (bodyControl != null)
    {
        bodyControl.Attributes.Add("onload", value);
    }
}

So now all you need to do is call the method on the master from your content page and make sure the content page has the JavaScript function to handle the event. Thats it.

Note:

To use specialized methods on Master, you will have to add a Master Type directive in the content aspx page, this will inform Visual Studios to use your implementation of MasterPage. This is added after the Page directive.

<%@ MasterType VirtualPath="~/MyMaster.Master" %>

Hope this helps

Copy to Output Directory

tarn@tarnbarford.net (tarn) — Mon, 02 Jun 2008 09:53:00 +0000

Tarn Barford

Far to often I've had multi-project solutions in Visual Studio not build because of post build copy tasks. Far too often I've seen references to the parent directory of a project to get files. No doubt both these things have a time and place; you might have a Javascript minify task in your post build or be writing an application that needs to access data or configuration from a parent directory.

Most of the time this is not the case and I often see people overlooking the "copy to output directory" property on items in the solution explorer. This property is really useful to solve the common problem of needing resources from your project in the output, and it works in multi-project solutions!

An example scenario would be if you had a solution with an application project, and assembly project and a test assembly project. In this scenario the assembly project has an XSL transform file, it makes sense to have this file with the code in the project. Both the test project and the application project will need to know about this file.

I've actually seen code that references the file in project location, using lots of ".." and "\" in the path to reference the source file. This will cause problems if your application actually modifies the source code. It will cause problems if you restructure your solution. And if you use absolute paths it will prevent the solution from building unless its in a specific location.

I've also seen extensive post build tasks that copy the required files to all the places that might need them. This is means that every new project the references the assembly will needed to be added in the post-build copy steps.

Both these solutions have maintenance problems and are excessively difficult to implement. The best solution here is to use the "copy to output directory" property on the XSL file in the solution explorer. This will mean that when you build your solution it will be copied to the output of all projects in the solution that reference the project.

There is a "copy always" and a "copy if newer" option. As I said earlier there is a time and place for the other methods so it's best to think through what makes sense for your solution.

The Journals of Tarn Barford

debugProxy

debugProxy

Creating Debian virtual machines

Creating Debian virtual machines

Building a Debian 12 base image

Building a Debian 12 base image

Building NixOS qcow2 images with flakes

Building NixOS qcow2 images with flakes

Using libvirtd VNC Remotely

Using libvirtd VNC Remotely

Using libvirtd Remotely

Using libvirtd Remotely

Securing TCP Sockets with TLS

Securing TCP Sockets with TLS

Simple Private Public Key Infrastructure

Simple Private Public Key Infrastructure

Linux One-Liner for Monitoring File Changes

Linux One-Liner for Monitoring File Changes

Static virtual bridge IP addresses with DHCP

Static virtual bridge IP addresses with DHCP

Virtual bridge networking with libvirt

Virtual bridge networking with libvirt

Cheap root servers from Hetzner auctions

Cheap root servers from Hetzner auctions

Techbikers 2016 and World Bicyle Relief

Techbikers 2016 and World Bicyle Relief

Cross-compiling with Rust

Cross-compiling with Rust

Building Linux and BusyBox for ARM and emulating it on QEMU

Building Linux and BusyBox for ARM and emulating it on QEMU

Compiling an arm-unkown-linux-gnueabi toolchain with crosstool-ng

Compiling an arm-unkown-linux-gnueabi toolchain with crosstool-ng

Securing my Raspberry Pi backup server

Securing my Raspberry Pi backup server

How I delivered a million spam emails

How I delivered a million spam emails

What happened

What was sent

Adding security and anti-spam measures

Blacklists

Conclusion

Packaging a Flask Application

Packaging a Flask Application

Running of the Bulls

Running of the Bulls

Chess

Chess

Game is Drawn

New Game

Haste Web Workers

Haste Web Workers

Swipe Keyboard

Swipe Keyboard

LoadingJavascript is Required

Mozilla Persona

Mozilla Persona

Prague to Berlin

Prague to Berlin

PGP/GPG

PGP/GPG

Setting up a pair programming environment on Amazon EC2

Setting up a pair programming environment on Amazon EC2

Connecting to the instance

Update the system

Creating users

Allowing users remote access

All users are not equal

Prevent root login

Install some development tools system wide

Multiuser Screen

Build it and they will come

vim-slime

vim-slime

UHC: Haskell to JavaScript

UHC: Haskell to JavaScript

Basic long polling chat with Manos de Mono

Basic long polling chat with Manos de Mono

Installing Mono

Installing Manos

Loading