Nick

Nick - Authenticating to Alfresco Share using Apache Auth (eg for SSO)

2013-03-13T17:06:36Z

This post carries on from my earlier one on Alfresco Repo SSO, and builds on the information on how the Alfresco WebScript authentication methods work.

At this point, we can hopefully log into the Alfresco repository (Explorer or /alfresco/wcservice) automatically by authenticating to Apache. Our next step, before moving onto SSO in Share, is to also allow header based authentication to Alfresco. We need to do this because of how Share will talk to the Repo - it will know the user's username from Apache, but it won't have their password (and equally won't have their client cert if doing SSL auth). Share needs to be able to tell the repo who you are, and have that trusted. For that, we need to tweak our External Authentication SubSystem.

Before doing that, be aware that by turning on header based auth, anyone who can make http requests to the repo could add in that header and impersonate anyone, so take great care! You should do one of use firewall / auth rules it so that only Share can, use something like an Apache module to strip out the header except from Share, or use SSL + client auth + proxyUserName.

Currently, our external-authentication.properties file looks like:

    external.authentication.defaultAdministratorUserNames=admin,nick
    external.authentication.enabled=true

To support header based auth, we add two more lines, giving us:

    external.authentication.defaultAdministratorUserNames=admin,nick
    external.authentication.enabled=true

    external.authentication.proxyUserName=
    external.authentication.proxyHeader=X-My-Custom-SSO-Header

If you don't specify a proxyHeader, the default is X-Alfresco-Remote-User (that comes from here).

With that in place, from a machine that is allowed to talk to Alfresco with the header, try executing something like:

    $ curl -X GET -L -H "X-My-Custom-SSO-Header: nick" http://localhost:8080/alfresco/ | less

Take a look through the HTML, and check you see something like Logout (nick) and not Login (guest)

Now, we can turn to Share. There is information on this in the wiki and the Enterprise docs, but they're a bit short on explanation to understand why you need to do things.

First, two things we can safely ignore now. One of those is the KeyStore entry. That's used if you want Share to talk to the Repo over SSL with client Auth. You may well want to turn that on for production, but not for getting started. (If you do, you should set the SSL client cert username as your proxyUserName, then the Repo will only check for the proxyHeader HTTP Header for the username for requests from that user). Secondly, we're going to be using a HTTP Header to carry the username, so we don't need the alfrescoCookie connector.

If you don't have one already, create a alfresco/web-extension/share-config-custom.xml file, typically either in a module, or in the Tomcat shared classes. If you're creating a fresh one, use share-config-custom.xml.sample as a basis (it has nearly the config in it you need, along with other bits you won't want now so remove / comment them out). The key bit for us is to add/modify a Remote config section. This is where you define how Share talks to other systems over http/https, especially to the repository (via the alfresco endpoint). Define a Remote section as follows:

   <config evaluator="string-compare" condition="Remote">
        <remote>
            <connector>
               <id>alfrescoHeader</id>
               <name>Alfresco Connector</name>
               <description>Connects to an Alfresco instance using header and cookie-based authentication</description>
               <class>org.alfresco.web.site.servlet.SlingshotAlfrescoConnector</class>
               <userheader>X-My-Custom-SSO-Header</userHeader>
            </connector>

            <endpoint>
                <id>alfresco</id>
                <name>Alfresco - user access</name>
                <description>Access to Alfresco Repository WebScripts that require user authentication</description>
                <connector-id>alfrescoHeader</connector-id>
                <endpoint-url>http://localhost:8080/alfresco/wcs</endpoint-url>
                <identity>user</identity>
                <external-auth>true<</external-auth>
            </endpoint>

        </remote>
    </config>

With that in place, if you go to Share in your browser via the Apache connection - http://local-alfresco/share/ if you followed part 1 - you get an Apache basic auth prompt, then you should be automatically logged into Share with the user details you gave to Alfresco. SSO done!

One other thing to try before we understand how it works (and you lock down the security!), we'll try curl again:

$ curl -X GET -L -H "X-My-Custom-SSO-Header: nick" http://localhost:8080/share/ | less

Look for something near the top showing Alfresco.constants.USERNAME = "nick" and not guest to confirm header based auth worked for Share.

So, what's going on, and how does it all work? At this point, you'll probably want to grab the Spring Surf source code if you want to dig around, as most of the logic isn't in Share itself, but the underlying Surf framework.

Looking at the config, there are a few things to note. In the alfresco endpoint, we have <endpoint-url>http://localhost:8080/alfresco/wcs</endpoint-url> - note that it has changed to the WCServices URL from the regular Services one. As seen in the how the Alfresco WebScript authentication methods work blog post, this is so that we can use the Authentication subsystems via the filters to authenticate, while services just does basic auth + tickets. The next thing is <external-auth>true</external-auth>, which tells Share that we're using External Authentication and it should hide a few bits. AuthenticationUtil exposes this via isExternalAuthentication, and a few bits of Share use that. Finally, the connector-id ties us back to the Header and some Java.

In the connector section of the config, the route taken is that Share finds an endpoint with the id alfresco, then uses that to find the connector to use by the connector-id defined there. For us, that's our new alfrescoHeader one. That includes <userHeader>X-My-Custom-SSO-Header</userHeader> which is the HTTP Header used by Share and the Repo to pass along the username to use. The other thing in there is the class that drives it all, SlingshotAlfrescoConnector.

At this point, I should perhaps point out a gotcha. The underlying External Authentication SubSystem supports external.authentication.userIdPattern to only use part of the Header as the username, but due to ALF-18393 Share doesn't and it all goes a bit wrong... The option isn't (currently!) Share Compatible

Looking some more at SlingshotAlfrescoConnector which we defined above, there's one key method in there, applyRequestHeaders. This is the part where Share is tweaking the headers on any connections it makes to the Repo. In it, it checks to see if you authenticated without a password (i.e. with SSO). If you have, your username is sent to the repo in the header X-Alfresco-Remote-User and if you set a config entry userHeader then with that too as well. There's also setConnectorSession which feeds your custom header down the stack.

So, if you want to send some extra headers to the Repo to prove to it you are really Share, and you don't want to do it with SSL client certs (for which there is an example in share-config-custom.xml.sample), then overriding SlingshotAlfrescoConnector + applyRequestHeaders + specifying that class instead is likely the way to go.

Where does the username come from to pass along? That's within Spring Surf, in AbstractUserFactory inside initialiseUser

The final bit of magic is explaining how specifying our magic header gets passed through Share to the Repo. The solution to that is to be found in SSOAuthenticationFilter. As well as handling Kerberos and NTML challenge stuff, it also does a bit of magic in the case of custom username headers. Specifically, in wrapHeaderAuthenticatedRequest it makes the value of the custom header available as the result of getRemoteUser(), which allows the rest of Surf to know who you are.

Phew, and we're done! You should now have a working Alfresco + Share SSO setup, which can work either through Apache with Apache handling the Authentication, or direct to Tomcat with special headers. We (hopefully) understand how it works, why, and have a testbed. At this point, we can tweak it for our final SSO setup, put in any security around it we need, and go live!

Authenticating to Alfresco using Apache Auth (eg for SSO)

2013-03-11T17:43:21Z

I'm currently setting up SSO (Single Sign On) for an Alfresco + Share install. I know I'm going to need to do a bit of customising, but I've struggled a bit with understanding what's going on with the various SSO tutorials. They give instructions, but not always a lot of explanation, which can be tricky if you know you want to go off piste. So, having reminded myself how the WebScript /services/ + /wcservices/ authentication works, my next step is to get some basic Apache Authentication in place, and have Alfresco trust that.

Why basic Apache Authentication? Well, it's the basis on which things like Alfresco + CAS and Alfresco + Shibboleth work, it can be used for SSL client cert auth, it's fairly easy to setup, I know most of the pieces already (always a bonus!). My aim in this - get Alfresco Explorer and the /wcservices/ WebScripts trusting Apache provided authentication. I'm not looking at Share, that's a follow-on post.

So, what components do we need? Firstly, there's the Alfresco External Authentication Subsystem. Next there's Tomcat AJP-13. There's Apache mod_jk to talk AJP-13. Finally, there's the basic Apache Authentication. Because I just want to test, I'm using a simple file based Apache auth setup. You likely won't use that in production, but it'll get you most of the way to a working SSO setup, and is easy to test with.

Our first thing to do is tell Alfresco that it can trust the remote authentication from Apache. (The process for this is similar to enabling Header based external Authentication). To do this, we need to enable + configure another Authentication SubSystem, specifically the External Auth SubSystem. Create a directory alfresco/extension/subsystems/Authentication/external/external-apache in your usual customisation spot (typically a module or the Tomcat Shared Classes). Into this new directory, create external-authentication.properties, and populate it with something like:

# Which users should be treated as an Admin?
external.authentication.defaultAdministratorUserNames=admin,nick
# Enable the External Authentication
external.authentication.enabled=true

That's not all though, you also need to add this new authentication subsystem to the active list. Somewhere, quite possibly in your alfresco-global.properties file you should have an authentication.chain line. Prepend this with external-apache:external, something like:

# This is what we used to have, LDAP + Built-in
##authentication.chain=ldap-main:ldap,alfrescoNtlm1:alfrescoNtlm

# Try the External Auth if we can, otherwise the previous ones
authentication.chain=external-apache:external,ldap-main:ldap,alfrescoNtlm1:alfrescoNtlm

Restart Alfresco, and check that you can log in as before. You should also see (assuming the default logging settings) something like

INFO  [org.alfresco.repo.management.subsystems.ChildApplicationContextFactory] Starting 'Authentication' subsystem, ID: [Authentication, managed, external-apache]

in your logs to show it was found and used.

Next up, we need to enable AJP-13 in Tomcat, and tell it to trust Apache supplied Authentication. I'm assuming you have two Tomcats, one for Alfresco on port 8080, and one for Share on 8081. If you only have one, skip the Share/8081/8010 bits. Edit [Tomcat Root]/conf/server.xml; for both Tomcats. Around line 90, you should come across an AJP Connector section, which may be commented out. Uncomment it, set the Alfresco one to port 8009 (default), and the share one to 8010. Finally, add the attribute tomcatAuthentication="true" to the Connector. This should give you something on the Alfresco Tomcat looking like:

   <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false" />

And for Share it'll be:

   <!-- Define an AJP 1.3 Connector on port 8010 (Alternate Port) -->
    <Connector port="8010" protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false" />

Restart your tomcats, and ensure they come up without error. Check you can telnet to localhost 8009 and 8010 (there's no welcome banner) to ensure the Tomcats are listening and no local firewalling breaks.

Half way!

If you haven't already, install a copy of Apache (latest 2.4, or perhaps even 2.5/2.6 if that's out by the time you read this!), and make sure you can get the welcome page up. If on a Unix, make sure you have the Apache developer tools package installed too, if you're doing a package install. If you can, install a packaged version of the Tomcat Connectors module. Failing that, grab the source code and build + install the module

There are two parts to configuring up mod_jk, the first is a JK specific file worker.properties, the second is the regular Apache config. For the former, create a new file somewhere sensible (but make sure it's not somewhere that Apache auto-loads config files from, or you'll get errors). FWIW, I created a new directory under /etc/apache2/ of conf.other and created it as tomcat-workers.properties but it'll depend on your distro. Assuming you setup your Tomcat AJP as above, create the file with:

# For communicating with Tomcat via AJP13

# We have two Tomcats, and we'd like status please
worker.list=jkalfresco,jkshare,jkstatus

# Enable the status worker
worker.jkstatus.type=status

# Alfresco is on 8080 (http) / 8009 (AJP)
worker.jkalfresco.type=ajp13
worker.jkalfresco.host=localhost
worker.jkalfresco.port=8009
worker.jkalfresco.connection_pool_timeout=600
worker.jkalfresco.socket_keepalive=1

# Share is on 8081 (http) / 8010 (AJP)
worker.jkshare.type=ajp13
worker.jkshare.host=localhost
worker.jkshare.port=8010
worker.jkshare.connection_pool_timeout=600
worker.jkshare.socket_keepalive=1

Now, define a new vhost to Apache, either by tacking it on the end of your httpd.conf, or putting it in a new file in conf.d, or in sites-enabled, or .... as appropriate for your setup. To start with, go for something like:

# Load the mod_jk module (your path to it may be different)
LoadModule jk_module /usr/lib/apache2/modules/mod_jk.so

# These paths may need changing for your setup too
# Where the file we created above lives
JkWorkersFile /etc/apache2/conf.other/tomcat-workers.properties
# Where to put jk shared memory
JkShmFile     /var/log/apache2/mod_jk.shm
# Where to put jk logs
JkLogFile     /var/log/apache2/mod_jk.log
# Set the jk log level [debug/error/info]
JkLogLevel    info
# Select the timestamp log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "

# Define the vhost that'll expose Tomcat
<VirtualHost *:80>
	ServerAdmin webmaster@localhost
	ServerName local-alfresco

        JkMount /status     jkstatus
        JkMount /alfresco   jkalfresco
        JkMount /alfresco/* jkalfresco
        JkMount /share      jkshare
        JkMount /share/*    jkshare
</VirtualHost>

The ServerName needs to be unique, so either make it your machine name (only vhost), or create a new alias for 127.0.0.1 in your /etc/hosts file and use that. Restart apache, and try visiting [server name]/status to check that mod_jk is loaded and running. Then, visit [server name]/alfresco and check that you get explorer as guest.

Finally, before we consider about how it all fits together, we need to turn on Authentication in Apache, so it has some auth to pass to Tomcat. Full details on this are in the Apache docs, but basically you'll want to use htpasswd to create a digest password file, something like:

  $ # Create the file, add admin
  $ htpasswd -c /etc/apache2/conf.other/alfresco.htpasswd admin admin
  $ # Create another user (don't use these passwords in production!)
  $ htpasswd /etc/apache2/conf.other/alfresco.htpasswd nick password

Now, add to our vhost some auth lines:

        <Location />
          AuthType Basic
          AuthName "Test Alfresco Access Auth"
          AuthBasicProvider file
          AuthUserFile /etc/apache2/conf.other/alfresco.htpasswd
          Require valid-user
        </Location>

Restart Apache, and re-visit [server name]/alfresco . You should be prompted for a password. Specify admin and admin (as given to htpasswd), and you should be taken to Explorer logged in as admin automatically. We're basically there!

At this point, make sure you're only using a test server, have suitable firewalling etc in place. We're just trying to get some basics in place to test with, so we have a working base before making our changes. More is needed to make this production...

So, how does this all work? Apache is doing the authentication, and passing that through to Tomcat. We can see that Tomcat sees the username by creating a small test jsp somewhere in the Alfresco webapp, containing:

User: <%=request.getRemoteUser()%>

Run that, and you'll see the username you gave to the Apache auth come through. That username is passed into the Servlet Request.

What does Alfresco do with it? When we enabled our External Authentication SubSystem, we effectively pulled in the external-filter-context.xml context file, along with partly overriding the default external-filter.properties. The webscriptAuthenticationFilter isn't really doing that much for us, the key bit is the remoteUserMapper. This is configured up to use DefaultRemoteUserMapper

When the request comes in to Alfresco, for either Explorer or /wcservices/ (but not /services/ as that's different), the global Authentication Filter fires. In doFilter it calls getSessionUser which in turn checks for a Remote User Mapper. If one is found (and we've enabled one with our next Authentication SubSystem), getRemoteUser is called. If you look at that method in DefaultRemoteUserMapper, we'll see it checks the Remote User on the request, and returns that. So, no login is needed at the Tomcat / Alfresco layer, because the value from Apache is trusted.

We can also see how we could do http header auth at this point if we wanted, and also see some of the security issues which mean you shouldn't blindly enable this in production...!

So, we can now use our test Apache auth setup to be auto logged-in to Alfresco Explorer and wcservices. In the next part, we'll see how we enable Share auto-login with the same setup, and how that actually works under the hood.

Alfresco WebScripts (Services) and Authentication

2013-03-07T16:10:39Z

I'm currently doing some work on enabling SSO for Alfresco, and I've come across some slightly confused wiki pages and blog posts out there around how webscripts and authentication fits together. As I've done quite a bit of fighting with this in the past, I thought it might be worth trying to summarise what works and why.

Firstly, WebScript URL prefixes. The Alfresco webscripts are available under three pairs of URL prefixes, also known as WebScript runtimes. As detailed on the wiki, the three are:

/alfresco/service/ or /alfresco/s/ - HTTP Auth + Tickets, "apiServlet"

/alfresco/wcservice/ or /alfresco/wcs/ - Explorer Style Auth, "wcapiServlet"

/alfresco/168service/ or /alfresco/168s/ - JSR-168 Portlet Auth, "portalapiServlet"

I'll skip the portlet stuff, as I don't use it, and concentrate on the other too. (Note that all three expose the same set of webscripts, it's just the authentication in front of them that changes)

The first thing to note is that describing /alfresco/service/ as using HTTP Auth misses a few bits out. There are in fact several ways to authenticate to it. One option, most commonly used when debugging and admining, is to use HTTP Basic Auth (but make sure you're using it over SSL in production!). If no other authentication is sent, and the webscript requires it, the webscript framework will prompt you for your username and password with basic auth.

Another option is to use Alfresco Tickets, as detailed in the wiki here. You can get a ticket a few different ways, but posting JSON of the username and password to /alfresco/service/api/login is often the most common. The ticket can be supplied with a request via the alf_ticket URL parameter, or by sending HTTP basic auth with a username of ROLE_TICKET and the ticket as the basic auth password. Another option is to send the URL parameter of guest=true to explicitly request guest access.

If you want to see how the authentication to /alfresco/service/ works, the main class to look at is BasicHttpAuthenticatorFactory

When looking at /alfresco/wcservice/ , most things will describe it as using Explorer authentication. That isn't quite the full story though, if you take a look at web.xml you'll see that there are actually two Authentication Filters that apply to requests through wcservice. The Global Authentication Filter is used (that's the one shared with Explorer), but before that a special WebScript Authentication Filter is also tried.

The latter is passed off to a spring bean webscriptAuthenticationFilter. Not all authentication subsystems implement one of these, so often it'll do nothing. If the authentication filter declines, then it'll fall back to the default Global Authentication Filter, which ends up as the bean globalAuthenticationFilter, and from there the normal Explorer authentication kicks in.

Quite a few of the instructions out there for doing SSO involve specifying some custom share config, to change the alfresco Remote Endpoint to use /wcs . If you see one of these, it will almost always be coupled with changes to your Authentication subsystem(s) to do something with the webscriptAuthenticationFilter. When setting this sort of thing up / debugging, be aware that you'll need to test with /wcs and not /service, and that normally you can't test with Explorer as that doesn't use the same filter!

Overriding Alfresco webscript library ftl files

2013-01-30T19:04:30Z

On the whole, Alfresco is pretty good for customising and overriding, and there's been a lot of work done in 4.2 to make it even easier to customise Share and Surf. Overriding the spring beans for Java backed webscripts can be done with a bit of fiddling. One problem we have hit though is with overriding FTL files, especially library ones.

I'll use the Workflow REST API as an example here, as it's one we needed to change, but it applies to a lot of the webscripts too. A common JSON emitting Alfresco REST API has a FTL file somewhat like:

<#import "workflow.lib.ftl" as workflowLib />
{
   "data":
   <@workflowLib.taskJSON task=workflowTask detailed=true/>
}

It imports a library file, then calls one of the functions in there. If you want to inject a few extra bits into the JSON structure, then your only option is to edit the library file, or clone it. For simple libraries with a single macro, you could just clone the whole thing and customise, but for a library with many macros that gets rather nasty especially when upgrading. FreeMarker macros get replaced when a new one is defined with the same name, much like spring beans, so for our hypothetical macro file:

<#macro taskJSON task detailed=false>
<#escape x as jsonUtils.encodeJSONString(x)>
      {
         "id": "${task.id}",
          ....
      }
</#escape>
</#macro>

<#macro propertiesJSON properties>
<#escape x as jsonUtils.encodeJSONString(x)>
{
<#list properties?keys as key>
   "${key}":
   ...
</#escape>
</#macro>

To change taskJSON to inject some extra bits from the model, we'd have to copy the whole file and customise. However, there is a way that makes things a little less evil, which relies on macros overwriting each other. We take the original file, and copy it to the same directory structure in our module, but with a new name. Instead of workflow.lib.ftl we might instead go for workflow-original.lib.ftl. Next, add a new comment to the top, documenting where it originally came from, what version it is taken from etc. Now, create a new workflow.lib.ftl, and copy into that the whole of the macro we want to override. Add in the extra few JSON clauses. At the top of the file, add some documentation saying what version you copied and pasted from, what file it came from, and what changes you made. Trust me, you'll be thankful you did this when you come to upgrade... Finally, at the top of the file, before your customised macro, add in an include (not import, include) of the renamed original. Your new lib, with the name of the old one, now looks something like:

<#-- Customised File -->
<#-- Original: remote-api/config/alfresco/templates/webscripts/org/alfresco/repository/workflow/workflow.lib.ftl -->
<#-- Alfresco Version: 4.1.1.3 -->
<#-- Modification: Added custom1 and custom2 properties to the JSON -->

<#-- This pulls in all the macros we haven't needed to change -->
<#include "workflow-original.lib.ftl">

<#-- Override and customise the one we do need to change -->
<#macro taskJSON task detailed=false>
<#escape x as jsonUtils.encodeJSONString(x)>
      {
         "id": "${task.id}",
          ....
         "myCustom1": ${task.custom1},
         "myCustom1": ${task.custom2}
      }
</#escape>
</#macro>

(It's probably best to put your custom json at the start or end, to make upgrading easier)

Overriding built-in Java backed WebScripts in Alfresco

2013-01-30T15:27:48Z

Generally, Alfresco makes it fairly easy for you to override the built-in core repository Services. There's this wiki page and this one to get you started, then once you know about Spring you can largely make it happen.

The Alfresco spring context file load order for services etc is controlled by application-context.xml. The last bean defined with a given name wins, so if Alfresco ships with a bean definition for fooService and you define your own one later with the same name, yours will be used. Handily, as shown in that context file, spring context files in /extensions/ and /modules/ are loaded late, so your modules can include override services.

The picture is not quite so rosy for overriding the built-in webscripts. Here, the key context file is web-application-context.xml. This loads application-context.xml which we saw above, and then it loads the web scripts. This means that the built-in webscritps are always loaded after modules, so your module can never override a built in webscript.

However, at the bottom of that file we see the answer - we need to provide a single alfresco/extension/custom-web-context.xml which pulls in our module webscripts. That gets loaded last, so will win!

For my current project, we have added the following file as alfresco/extension/custom-web-context.xml

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'>
<beans>
    <import resource="classpath*:alfresco/module/*/module-webscripts-context.xml" />
</beans>

All of our Repository modules already had a module-context.xml file in alfresco/module/<name>/. Those that need to override webscripts can now also provide a second file in that directory, module-webscripts-context.xml, and any webscripts defined in there can be guaranteed to win and be used.

(The new custom-web-context.xml file can be bundled in a jar, shipped in an existing AMP, packaged as a brand new AMP, or dropped in your classes override directory similar to your repository properties file - it all depends what'll be cleaner for your setup)

Spring in a QName

2013-01-30T15:11:38Z

This post was originally written in September 2010, and was available at http://blogs.alfresco.com/wp/nickb/2010/09/28/spring-in-a-qname/ - I have reposted it here as that blog is no longer online.

Within Alfresco, we make a lot of use of Qualified Names (QNames) for addressing and naming things. Generally, when configuring Alfresco through Spring or properties files, we can use the short form, eg

<bean id="coreBean" class="org.alfresco.some.thing.core"> <property name="typeQName"> <value>cm:description</value> </property> </bean>

Within the bean, the NamespaceResolver is used to turn the friendly, short form (eg cm:description) into the full form (eg {http://www.alfresco.org/model/content/1.0}description).

However, every so often you may find yourself trying to configure something with Spring that no-one ever expected you to be trying to do… In this situation, the string form isn’t accepted by the class, and only a real QName object may be sprung in.

As it turns out, creating a real QName object from within Spring isn’t actually too hard to do. So, in case you ever find yourself needing to do it, the definition will look something like this:

<bean id="coreBean" class="org.alfresco.some.thing.core"> <property name="typeQName"> <value> <bean class="org.alfresco.service.namespace.QName" factory-method="createQName"> <constructor-arg value="http://www.alfresco.org/model/content/1.0" /> <constructor-arg value="description"/> </bean> </value> </property> </bean>

Adding new Apache Tika Plugins to Alfresco (Part 3)

2013-01-30T15:09:04Z

This post was originally written in October 2010, and was available at http://blogs.alfresco.com/wp/nickb/2010/10/27/adding-new-apache-tika-plugins-to-alfresco-part-3/ - I have reposted it here as that blog is no longer online.

In Part 1 we saw what Apache Tika is and does, and in Part 2 we saw what it has brought to Alfresco. Now it’s time to look at adding new Tika Parsers, to support new file formats.

Firstly, why might you want to add a new parser? The most common reason is licensing – all the parsers that ship as standard with Apache Tika are Apache Licensed or similar, along with their dependencies, and so can be freely distributed and included in other projects. However, some file formats only have libraries that are available under GPL or proprietary licenses, and so these can’t be included in the standard Tika distribution.

There is a list of available 3rd party parsers on the Tika 3rd Party Plugins wiki page, currently made up of GPL licensed parsers + dependencies. If your format isn’t listed there, and you want to add it to Tika within Alfresco, then what to do?

Firstly, you need to write / acquire a Tika Parser. Writing a Tika Parser is quite easy, as the 5 minute parser guide explains. There are basically two methods to implement:

Set getSupportedTypes(ParseContext context);

void parse(InputStream stream, ContentHandler handler, Metadata metadata,
           ParseContext context) throws IOException, SAXException, TikaException;

The first allows you to indicate the file types your parser can handle. This is needed when registering the parser with the AutoDetectParser and similar, but isn’t needed if you select the parser explicitly. The second method is the one where you do the real work of outputting the contents and populating the metadata object.

To see this in action, let’s take a look at a simple “Hello World” Tika Parser:

package example;
public class HelloWorldParser implements Parser {
  public Set getSupportedTypes(ParseContext context) {
    Set types = new HashSet();
    types.add(MediaType.parse("hello/world"));
    return types;
  }
  public void parse(InputStream stream, ContentHandler handler,
         Metadata metadata, ParseContext context) throws SAXException {
    XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata);
    xhtml.startDocument();
    xhtml.startElement("h1");
    xhtml.characters("Hello, World!");
    xhtml.endElement("h1");
    xhtml.endDocument();

    metadata.set("hello","world");
    metadata.set("title","Hello World!");
    metadata.set("custom1","Hello, Custom Metadata 1!");
    metadata.set("custom2","Hello, Custom Metadata 2!");
  }
}

Before we can use this in Alfresco, we need to compile this against tika-core.jar (note – you may need to implement the parse method without a ParseContext object if you’re using an older version of Tika), and then wrap our classfile up in a jar. Once our jar is deployed into our application container (eg the shared lib of tomcat), we’re ready to configure it.

For 3rd party parsers which provide the Tika service metadata files, if we don’t want to control the registration in Alfresco then we can simply allow the default Tika-Auto metadata and transformer classes to handle it. In our case, we want to register it explicitly. To do that, we’ll create a new extension spring context file, and populate it:

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'>
<beans>
    <bean id="extracter.MyCustomTika"
          class="org.alfresco.repo.content.metadata.TikaSpringConfiguredMetadataExtracter"
          parent="baseMetadataExtracter" >
        <!-- This is the name of our example parser compiled above -->
        <property name="tikaParserName">
           <value>example.HelloWorldParser</value>
        </property>

         <!-- Use the default mappings from TikaSpringConfiguredMetadataExtracter.properties -->
        <property name="inheritDefaultMapping">
            <value>true</value>
        </property>
        <!-- Map our extra keys to the content model -->
        <property name="mappingProperties">
            <props>
                <prop key="namespace.prefix.cm">http://www.alfresco.org/model/content/1.0</prop>
                <prop key="custom1">cm:description</prop>
                <prop key="custom2">cm:author</prop>
            </props>
        </property>
    </bean>

    <bean id="transformer.MyCustomTika"
          class="org.alfresco.repo.content.transform.TikaSpringConfiguredContentTransformer"
          parent="baseContentTransformer">
        <!-- Same as above -->
        <property name="tikaParserName">
           <value>example.HelloWorldParser</value>
        </property>
    </bean>
</beans>

To test this, we’ll need a node with the special fake mimetype of “hello/world”, which is what our Tika Parser is configured to handle. We can do that with a snippet of JavaScript like this:

var doc = userhome.createFile("hello.world");
doc.content = "This text will largely be ignored";
doc.mimetype = "hello/world";

If we run the above JavaScript, we’ll get a node called “hello.world”. If we run the “extract common metadata fields” action on it, we’ll then see the metadata properties showing through. Then, if we transform it to text/html, then we see a text heading of “Hello, World”. Thus we have verified that our custom Tika parser has been wired into Alfresco, is available for text transformation, and can do metadata extraction including custom keys.

More information on Tika and Alfresco is available on the Alfresco wiki.

Apache Tika powered updates to Alfresco (Part 2)

2013-01-30T15:04:40Z

This post was originally written in September 2010, and was available at http://blogs.alfresco.com/wp/nickb/2010/09/24/apache-tika-powered-updates-to-alfresco-part-2/ - I have reposted it here as that blog is no longer online.

In Part 1, we learnt a little about what Apache Tika is and does. In this part, we’ll see what new features using Tika gives us in Cheetah.

On the metadata side, Tika delivers three important things for us. These are support for a wider range of formats (see below), enhanced ease of adding custom parsers (you can just spring in a bean with the class name of your parser and you’re done), and consistent metadata.

This last one is less of an issue for Alfresco users than for many other users, but is a real issue for extractor developers. Within Alfresco, we always map the raw properties onto ones in the content model, but this is handled at the extractor level. As such, it shouldn’t matter to the user if one document format has a “last author”, another “last editor” and a third “last edited by”, they’ll all turn up in the same property in Alfresco. However, the extractor writer has to know about this, to provide the mapping, and this makes writing an extractor harder, and increases the chance of error.

Within Tika, there is a set list of common metadata keys, and each Tika parser internally maps its properties onto these. As such, when you receive your metadata back from Tika, it all looks the same no matter what file you got it from. If the metadata is a date, then Tika will also take care of converting it to a common format, so you don’t have to worry about parsing a dozen different date representations.

Finally, because the metadata is in a common format, we can more easily map it to the content model. Thus, in Alfresco 3.4, we see most of the common extractors have a wider range of metadata mappings to the content model as standard. One big example of this is in the case of images – EXIF tags are now automatically extracted and mapped onto the content model, and if the image was geotagged, then the location of the image is also mapped onto the content model.

Give it a try – upload a geotagged image to Alfresco share in 3.4, and see all the new metadata that shows up such as the location, camera, focal length and more!

In the past, most text extractors that were used in Alfresco only able to produce plain text. However, all the Tika parsers generate XHTML sax events, and sure are able to produce not only plain text, but also HTML and XHTML. Also, since Tika the XHTML is a true XML document, we can make use of XSLT to chain transformations.

The immediate benefit then is that all plain text content transformers that are powered by Tika can deliver an HTML version at no effort. Thus, HTML versions of PDFs, Word Documents etc can now be requested.

(At the moment, the HTML generated is very clean, but not always all that complex. The Tika community is gradually improving the markup generated to include more meaning, especially semantic information, and Alfresco is pleased to be involved in this effort)

Does being able to generate XHTML help that much? I’d say yes! With the forthcoming WCM Quick Start, we’ll shortly be adding some features around HTML versions of some kinds of uploaded documents. Using Tika, we were able to implement this feature very quickly, allowing us to concentrate the developer time on enhancing Tika. Next up, for some cases we wanted a whole XHTML document, and others we only wanted the body content. Using Tika and the SAX handlers, it’s a one line change to toggle between the whole document, or just the body contents, by picking a different transform handler. Finally, the output is XHTML, so for demo’s we’ve been able to use XSLT and E4X (from within a script action) to effortlessly manipulate the content.

Finally, as mentioned, using Tika delivers us support for a large number of new file formats. The current list of files supported via Tika is:

Audio – wav, riff, midi
DWG (CAD files)
Epub
RSS and ATOM feeds
True Type Fonts
HTML
Image – JPEG, PNG, Gif, TIFF and Bitmap (including EXIF information where found)
iWork (Keynote, Pages etc)
Mbox mail
Microsoft Office – Word, PowerPoint, Excel, Visio, Outlook, Publisher, Works
Microsoft Office OOXML – Word (docx), PowerPoint (pptx), Excel (xlsx)
MP3 (ID3 v1 and v2)
CDF (scientific data)
Open Document Format
PDF
Zip and Tar archives
RDF
Plain Text
FLV Video
XML
Java class files

What’s more, generally it’s just a case of dropping new Tika jars into Alfresco with little/no configuration changes, so we can look forward to easy addition of new formats with each new Alfresco release as the Tika support grows!

In Part 3, we will look at mapping between Tika’s common metadata, and the Alfresco content model.

More information on Tika and Alfresco is available on the Alfresco Wiki. Tika will also be discussed at the Alfresco Developer Conferences in Paris and New York later this year.

Apache Tika and Alfresco – Part 1

2013-01-30T14:58:32Z

This post was originally written in September 2010, and was available at http://blogs.alfresco.com/wp/nickb/2010/09/24/apache-tika-and-alfresco-part-1/ - I have reposted it here as that blog is no longer online.

For the forthcoming Project Cheetah release, there have been a number of improvements to Metadata Extraction and Content Transformations. These improvements have been delivered by using Apache Tika to power many of the standard extractors and transformers.

In this series of blog posts, we’ll be looking at what Apache Tika is and what it does, how it fits into Alfresco, what new features it has delivered, how you can customise how Tika works, and how you can add new Tika parsers to easily support new formats.

The idea for Apache Tika was hatched in 2006, largely from people involved in Apache Lucene, who were struggling to sensibly index all of their documents. The project went through the Apache Incubator, and after a period of time as a Lucene sub-project, in 2010 became it’s own top level Apache project. Tika is used by people indexing content, spidering the web, doing NLP and text processing, as well as with content repositories.

For all these use cases, the problems are largely the same. You start with a number of documents in a variety of formats. You wish to know what they are, and hence which libraries may be useful in processing them. You then want to get some consistent metadata out of them, and possibly a rich textual representation of the content. You also probably wanted all of this yesterday!

(As a side note, Alfresco users have historically been in a more fortunate position than most when faced with these challenges, as the Metadata Extractor and Content Transformation services have handled most of these for you.)

What services does Tika provide then?

Firstly, Tika offers content and language detection. Through this, you can pass Tika a piece of unknown content, and get back information on what kind of file it is (eg pdf, docx), along with what language the text is written in (eg utf-8 english). Within Alfresco we tend to already know this information, so as yet don’t make much use of detection.

Secondly, through the parser system, Tika provides access to the metadata of the document. You can use Tika to find out the last author of a word file, the title of an HTML page, or even the location where a geo-tagged image was taken. In addition, Tika provides a consistent view across the different format’s metadata, mapping internally from document specific to general metadata entries. As such, you don’t need to know if a format uses “last author”, “last editor” or “last edited by”, Tika instead always provides the same information. We’ll see more on using Tika for metadata in part 2.

Thirdly, through the parsers, Tika provides access to the textual content of files. The text is available as plain text, html and xhtml, with the latter offering options for onward transformations through SAX and XSLT to additional representations. This can be used for full text indexing, for web previews, and much more. Again in part 2 we’ll see how this is being used in Alfresco.

Finally, Tika provides access to the embedded resources within files. This could be 2 images embedded in a word document, or an excel spreadsheet held within an powerpoint file, or even half a dozen PDFs contained within a zip file. This is quite a new Tika feature, and we’ll hopefully be making more use of it in the future. For now, it offers the adventurous a consistent way to get at resources inside other files.

In Part 2, we’ll look at the new features and support that Tika delivers to Cheetah.

More information on Tika and Alfresco is available on the Alfresco Wiki. Tika will also be discussed at the Alfresco Developer Conferences in Paris and New York later this year.

Oxford Digital Switchover and DVB-Apps

2011-09-28T14:18:33Z

If you use Linux with the DVB-Apps to watch TV, and you're in Oxford, then as of today (28th September 2011) you'll need to re-tune. (Well, everyone in Oxford has to too!)

One slight snag is that the command line scanning tools (like scan) take an initial tuning file, and the one that ships in Debian / Ubuntu / Hg isn't yet updated.

Armed with the data from ukfree.tv and some trial and error, I've got what looks to be the correct new tuning file. Debian / Ubuntu users will want to place this in /usr/share/dvb/dvb-t/uk-Oxford (other users wherever you want!) and then run a scan. (You could also use w_scan, which iterates each frequency in turn, but scan with a start file is normally much quicker)

# Post-Switchover

# Local, C51, details TBA, not yet transmitting as of Nov 2011
T 713833000 8MHz 2/3 NONE QAM64 2k 1/32 NONE

# PSB1 BBC-A, C53+. Apparently 730.2 but actually looks to be 730.167
T 730167000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

# ArqB (COM6), C55, 746.0
T 746000000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

# PSB3 BBC-B, C57, 256QAM DVB-T2
T 762000000 8MHz 2/3 NONE QAM256 8k 1/32 NONE

# ArqA (COM5), C59-, Apparently 777.8 but actually looks to be 777.833
T 777833000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

# PSB2, D3+4, C60-, Apparently 785.0 but actually looks to be 785.833
T 785833000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

# SDN (COM4), C62, 802.0
T 802000000 8MHz 2/3 NONE QAM64 8k 1/32 NONE

Note - as of 2011-12-08 these changes have been accepted into the upstream Linux Media repo. Depending on the age of your distro / checkout / etc, you may or may not need to update to the above by hand.

Having done that, you should be able to run something like scan -v -a 0 uk-Oxford > new-channels.conf to build your new channels.conf file. The result should be something like this:

1-2-1 Dating:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:14816
301:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:901:951:7168
303:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:19584
4Music:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:101:102:25664
5*:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6673:6674:12928
5 USA:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6689:6690:12992
Absolute Radio:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6082:14688
ADULT Babestn2:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:23872
ADULT Babestn:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27904
ADULT Blue:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24128
ADULT Filth:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15728
ADULT PARTY:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24256
ADULT redhotTV:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15680
ADULT Section:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27584
ADULT Section:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27616
ADULT Section:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15200
ADULT Section:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15424
ADULT smileTV2:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27520
ADULT smileTV3:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:22400
Al Jazeera Eng:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:27712
BBC 6 Music:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:432:18048
BBC 6 Music:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1602:5760
BBC Asian Net.:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:435:18240
BBC Asian Net.:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1902:5952
BBC FOUR:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:16832
BBC FOUR:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:4544
BBC NEWS:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:501:502:4352
BBC ONE:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:101:102:4171
BBC Parliament:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:205:421:17024
BBC Parliament:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:601:602:4736
BBC R1X:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:434:18176
BBC R1X:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1802:5888
BBC R5L:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:430:17920
BBC R5L:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1402:5632
BBC R5SX:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:431:17984
BBC R5SX:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1502:5696
BBC Radio 1:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:436:18496
BBC Radio 1:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1002:6720
BBC Radio 2:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:437:18560
BBC Radio 2:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1102:6784
BBC Radio 3:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:438:18624
BBC Radio 3:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1202:6848
BBC Radio 4:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:439:18688
BBC Radio 4:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1302:6912
BBC Radio 4 Ex:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:433:18112
BBC Radio 4 Ex:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1702:5824
BBC Red Button:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:4416
BBC THREE:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:4288
BBC TWO:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:202:4287
BBC World Sv.:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:440:18304
BBC World Sv.:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:2002:6016
bid tv:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6273:6274:14272
Big Deal:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24000
Capital FM:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6114:14752
CBBC Channel:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:301:302:4608
CBeebies:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:401:16960
CBeebies:730167000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_8K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:401:402:4672
Challenge:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6785:6786:15520
Channel 4+1:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:580:581:8452
Channel 4:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:560:561:8384
Channel 5:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:540:541:8500
Channel Zero:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28224
CITV:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6833:6834:16032
COMMUNITY:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24064
CONNECT:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28352
Create & Craft:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2301:2302:24384
Dave:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:401:402:22272
Dave ja vu:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:23936
Daystar:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:24320
E4+1:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:501:502:22336
E4:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:570:571:8448
ESPN:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6801:6802:16096
Film4:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:701:702:27136
Food Network:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:23040
Gay Rabbit:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:8577
Gems TV:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2401:2402:24448
G.O.L.D.:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15552
Heart:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6098:14720
heat:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1201:26240
Home:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:14976
Ideal World:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:501:502:25920
ITV1 +1:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:600:601:8362
ITV1:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:520:521:8263
ITV2 +1:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6529:6530:15952
ITV2:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:530:531:8325
ITV3:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6881:6882:16048
ITV4:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:601:602:28032
Kerrang!:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1301:26304
Kiss:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1101:26176
Magic:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1801:26624
More 4:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:590:591:8442
Pick TV+1:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:202:22244
Pick TV:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:301:302:22208
Premier Radio:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1601:28160
price-drop tv:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:611:612:22464
Q:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1901:26688
QUEST:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6929:6930:14498
QVC:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:6049:6050:13120
QVC Beauty:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28288
Rabbit:850000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:8642
Really:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2111:2112:23712
Rocks & Co 1:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2311:2312:27840
Russia Today:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:2101:2102:27456
Sky News:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:101:102:22080
Sky Sports 1:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:202:403:20160
Sky Sports 2:721833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:204:411:20224
Sky Text:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28096
Smash Hits!:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1501:27040
SMOOTH RADIO:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1401:26368
STREAM:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28384
talkSPORT:578000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_1_2:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1101:22592
Teletext Hols:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:6354:14784
Television X:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:15232
The Hits Radio:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:1701:26560
TOPUP Anytime1:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:16224
TOPUP Anytime3:713833330:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_2_3:FEC_1_2:QAM_64:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:16288
VISION:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:0:0:28320
VIVA:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:201:202:25728
Yesterday:538000000:INVERSION_AUTO:BANDWIDTH_8_MHZ:FEC_3_4:FEC_3_4:QAM_16:TRANSMISSION_MODE_2K:GUARD_INTERVAL_1_32:HIERARCHY_NONE:301:302:25792

Finally, you'll want to do a full rescan in MythTV, which is a faff... :(

Upcoming Conferences

2011-08-28T13:43:44Z

For anyone who doesn't hawkishly follow my Lanyrd upcoming events page, here's a quick round-up of some of the conferences I'm speaking at / helping with in the near future.

Coming up in just under a fortnight (mental note - must finish my talk...) is Transfer Summit, 7-8 September. Handily located in Keble, so no need to leave Oxford, this conference is back for it's 2nd year covering open source, open development, innovation and collaboration. Last year featured loads of very interesting talks, and this year's looking promising too. If you're interested in open source, open development, academic to business transitions for software, or just a fun conference in Oxford, come join us!

The weekend following Transfer Summit we have BarCampApacheOxford, once again being held in the University Club. This year we're on Sunday 11th September, with a meal the night before. As usual, we'll have a wide range of sessions, including (but not limited to!) ones on the Apache Software Foundation and it's projects. We'll also likely have things on mapping, Django, Arduino, OpenOffice, Paper Aeroplanes and listening to Satellites, so it should be a great mix! If you want to come, sign up on the wiki ASAP, so you can ensure you get one of the spots that includes a free lunch :)

A month later, I'm off to Amsterdam for a few days. On Thursday 13th and Friday 14th October I'll be at GotoCon Amsterdam, mostly going to the great sessions on offer, but also talking as well. There's a good mix of sessions across lots of different areas of software development, and we've an Apache track on there too on the Friday. If you are thinking of coming, you can use the code burc200 for 20% off the registration price.

On Saturday 15th, the day after GotoCon, we're also doing an Apache Hackathon + mini BarCamp in Amsterdam. Hippo are kindly hosting us, and should be a fun chance to get Apache types from Europe together for a chat, some coding, and hopefully also share some knowledge amongst ourselves. It's open to not only Apache committers, but also anyone who contributes code or documentation to Apache (basically anyone on a dev list), sign up on the wiki and come along if you're in Amsterdam that weekend!

Next up is the Alfresco DevCon 2011 USA in San Diego. Not sure yet what I'm speaking on (though I think I'm down for a few sessions...). It's a chance to learn about developing solutions with and on top of Alfresco, in the lovely surroundings of San Diego, so who wouldn't want to go :)

Finally (for now, but possibly not for this year...) is ApacheCon NA 2011 in Vancouver, 7-11 November (plus hopefully a BarCamp on the 12th). We've an amazing set of speakers and talks lined up as usual, and it's looking promising for a great conference. Early bird rates end soon, so sign up!

Fringe 2011

2011-08-27T17:32:19Z

Last weekend I was up at the Edinburgh Fringe, and once again I managed to see quite a few shows, with a few gems amongst them. I tweeted reviews of most of them at the time (where wifi permitted!), but this is aiming to be a slightly more expanded set. Mostly to help with picking shows next year, it must be said, but there is a day or two of the fringe left for other people to use them!

Stupid People Will Be Offended - Free standup by Mikey Mileos at Dropkick Murphy's. We happened across this show when we got to The Vaults early, and nipped over the road for a drink. Only caught the first half of the show, but for free standup it was pretty good. I'd say probably half the jokes worked, which beats some paid-for fringe standup shows I've seen! Could be one to watch for next year, if he can improve over the year. 3*

Sherica - Shred Productions at Paradise in the Vault. Possibly summed up by the line early on: Sorry. Stop ... This isn’t me. I don’t visit prostitutes. For God’s sake – I read the Guardian!. This dark and thought-provoking piece follows the downward spiral after a teacher falls for a sex worker who turns out to be the sister of one of his pupils. Add in some blackmail, a new academy school struggling with the change, and it's a good piece. Not as good as Paper Birds were when they tackled a similar subject with In A Thousand Pieces, but good none the less. 4*

What You Will - Shrewsbury School at C. Their own musical telling of Shakespeare's Twelfth Night worked out really well, with some good songs but still keeping the original a lot. Good lunchtime fun, 4*

Lethal Injection - Five One Productions at C ECA. A compelling, minimalist production about 4 men on death row, and their guard. As their time approaches, we learn of their crimes and how they ended there, all for very different reasons. 4*

Sanctuary - Little Room Productions at Paradise in the Vault. A new opera set just before the apocalypse, as a father and daughter hide. Then others try to seek sanctuary with them. Very very operatic, which sometimes worked, but often got in the way. 3*

Matt Kirshen at Underbelly. Self-deprecating stories from his life, with the odd bit of politics in there as well. Funny, smart and enjoyable comedy. 4*

Wasted Love - One Academy Productions at C. This musical tells the story of a relationship support group, which despite how it may sounds was excellent. Great singing, great storytelling, heartbreak into uplifting redemption then sometimes tumbling back down again. This is what new and different musicals at the fringe should be like, 5*

Principal Parts - Strip Theatre at C soco. Surrealist telling of a group trying to assassinate Archduke Franz Ferdinand in Sarajevo, despite their incompetences. Good fun, 4*

Sunday in the Park with George - One Academy Productions again (twice in an afternoon!) at C. A full 2 hour production of Sondheim's musical about Georges Seurat's painting. A good (but not amazing) production of this good (but not amazing) musical. 4*

Sammy J and Randy: Ricketts Lane at Udderbelly Pasture. On top form again, this time Sammy J and his friend Randy (the foul mouthed purple puppet) are battling tax avoidance amidst their domestic misadventures. Silly tales, word plays, long running gags (can I get a pwappwa new guinea, pwappwa new guinea!), and generally very very very funny. 5*

Belt Up's Outland - Belt Up in C Soco once again. A smaller set of shows to last year, and in a small and intimate setting for this immersive theatre production. This time it's related to Lewis Carroll's worlds and life. Similar to last year's The Boy James, more static and less interactive than some of their have been. Generally good, but one or two bits felt a little too similar to some of their past shows. 4*

Isy Suttie at Pleasance was another funny, well done telling of the tale of her friend's relationship. Cheery comedy, with some good songs in it too, generally feel-good with laughs. 4*

Horne Section was late night music and banter in the Spiegel Tent. Quite a bit of live jazz, with some standup and silly games thrown in, not to mention the crazy but great Jazz/Morris Dancing Crossover version of a Beyonce song! Some good guests too the night we saw it. 4*

Street Dreams - Little Cauliflower at Underbelly. A puppetry show, but not really as you know it. Instead, an amazingly done piece of abstract storytelling, as an old man goes about his day pestered and chased by a variety of everyday objects. Sweet, funny, sad, brilliantly done and engaging. A little hard to describe, true, but an enchanting hour not to be missed! 5*

Which One's Fergal by Garrett Millerick at Just the Tonic at the Caves. A series of strong character sketches, satirical but well done, with a great mix of laughs and cringes. Garrett weaves his way between God's gift to Advertising Execs, a Traffic Warden, a failed motivational Hip-Hop singer and more. Biting but funny, a solid 4*

Flanders and Swan - Tim FitzHigham and Duncan Walsh Atkins at the Pleasance. An excellent hour of Flanders and Swan songs and tales, with just the right amount of sillyness (the music stand french horn anyone?). 4*

Seagull Effect - Idle Motion at Zoo Roxy. The Oxford based group were back again, after last year's great The Vanishing Horizon, this time with a tale of the 1987 hurricane. Once again they had a great mix of theatre, multimedia, and the odd bit of dance and physical theatre. The stories of different people interleave with archive footage and sound, with stories swirling together as the storm approaches. Great stuff, but not quite as good as last year. 4*

Belt Up's 20 minutes to 9 - back to C Soco for another Belt Up production. This one-woman immersive show takes us through recollections, stories and social observations of an aging apparently well-off woman. At times funny, others odd, almost always engaging, but you're never entirely sure quite what's going on, in that way that Belt Up can do well. Great stuff, but not quite as good as they have been before. 4*

Sammy J with Potentially at Underbelly is last but not least. Back again with stories, songs, and silly anecdotes that slowly stitch together to reveal the ridiculous but true linked narrative. Cruise Ships, Possums, a mystery man with a death wish in a red g-string, children's stories and a hell of a lot of laughs fill the hour. On top form again, 5*

Only 19 shows this year, in just over 3 days, but I'm sure we'll get the show count back up again next year!

Streetcar - first experiences

2011-08-13T17:31:34Z

A bit over a month ago, the friend who was letting me borrow their parking space moved out of Oxford, and the cost of renting a space was a bit insane given how little I drive at the moment. So, I sadly sold my car (it's found a good new home down in Devon!), and I decided to get by with just renting cars as needed.

(I had been hoping that CommonWheels would be able to get a space near me, and either adopt my car or get a new one to go there, but alas they couldn't get the council to give them a parking space)

CommonWheels don't have any cars near me, but Streetcar have 3 within a 10 minute walk of my flat, so that seemed a good bet for getting a car when I needed one, especially as they offer rentals as short as an hour, and have a nifty online booking system. I managed to find a signup voucher with £40 of credit (no longer available sadly, but this link will get you £20), and I figured that the £40 of credit would help offset the ~£60 signup fee for the year.

As they said it might take a few days for the license check, I signed up just after selling the car, as I figured I should probably try to be organised and get it all sorted before I actually wanted to book something. As it turned out, the license check was completed very quickly (a quick conference call with streetcar and the DVLA and it was sorted), so I was actually good to go sooner than expected. This did present a slight snag, as the £40 "welcome credit" I'd got turned out to only be valid for 30 days, and not for the year as I'd expected... Luckily the streetcar team are nice and friendly, so when I emailed them about this they extended the validity of the welcome credit by a few more weeks, so it was still valid when I came to want to use the car for the first time.

A little over a month after signing up, I finally had a spare weekend with a vague excuse to do something needing a car, so I booked the nearest car to me for as long as my welcome credit would allow. The car nearest to me is one of their swisher cars, a BMW, so I only got 9.30am - 3pm for my credit, rather than until 4pm if I'd gone for the Golf that's a few minutes further away. Still, I figured a BMW might be fun to drive, and it's nearer! I booked the car online, confirmed with my friends when I'd pick them up, got a friendly confirmation text and read their helpful instructions online about the car.

Roll on to this morning, and it was time to give it a whirl. I got to the car just after half 9, eventually found it (not in it's space as the booking said, but not too far away), figured out where to wave my streetcard to open it (magic box on the dashboard), and unlocked the car. In the glove box was the magic widget as expected, I tapped in my pin and it gave me the keys. Then it was onto the first challenge, checking for damage.

When I do a normal hire car, they normally have a sheet with all the existing damage marked on it (certainly with Hertz, less so with some of the others). It's then a quick walk around the car checking that everything's on there, then you're away. With Streetcar it's not so simple. In the instruction booklet is the damage log, but each bit of damage has it's own entry, and for this car the damage log went over about 5 pages. So, as I walked round the car, at each thing I spotted I had to check each (small) entry over many pages to see if it was mentioned, then move onto the next scratch and start again. It was really rather tedious to not have a summary, but have to check each thing in turn. Then the bad news, there were a couple of scratches not in the log. This meant a call to Streetcar to let them know, on their 0845 number (so instead of the call being free with inclusive minutes, I had to pay 35p/minute to call them, grr...). The call to let them know about the damage was fairly quick, but did have a rather long recorded message first, and as well as getting a damage reference for the log book I was also able to alert them to the fact that the car hadn't been left very clean inside as it was supposed to be. Having written the details in the log, I could finally hop in, adjust the mirrors and be off. I'd say the process took about twice as long from arriving to driving off as it's taken me the last few times I've got a car from Hertz, which is a bit pesky.

The car itself was fine, nice to drive, and after picking up my friends and running some errands we headed south to visit a National Trust property (Basildon Park near Reading, worth a visit!). Around 2pm we headed back, figuring an hour should be fine.

One of the conditions of hiring a streetcar, along with not leaving it a mess, is that you need to return it with at least 1/4 of a tank of fuel. The car had only just had over a quarter of a tank when I collected it, which meant I was the one standing when the music stopped. The cars all come with a fuel card, so all you need to do is swing by a petrol station, fill it up, charge it to the fuel card and you're away. Not too bad, but it does take a bit of time to do, or slightly longer if you're like me and tried to go to a petrol station that wasn't where I thought it was, doh! This meant I was cutting it a bit fine to return the car for 3pm after all, so we tried texting (my passenger rather than me I should add!) to try to extend the booking by 30 minutes. A rather nifty feature I must say, just a shame someone else had booked it for 3pm... At 2.58 when we were still a few minutes off making it to the carpark we rung up to say we'd be late, and were told that they'd let the next person know but we had a few minutes grace. On returning to the carpark at about 3.03 we found someone else had parked in the streetcar space (it has a sign and everything!), so we had to find another space and ring customer services once again to let them know where the car was. Just as we were leaving the carpark we spotted the next people getting to the car, sorry for making you a minute or two late whoever you were...!

Finally at home I checked my account to see the cost. As we'd driven to near Reading and back, we'd clocked up 61 miles, of which only 20 were included in the hire. That means there was an extra charge of £9.43 (it's 23p/mile, as shown on their pricing page). Having filled the tank up I know that's more than the cost of the diesel, but not too much more and they've a business to run. I'd also got a £3.62 "Unauthorized Extension" charge, I guess for being a few minutes late. Total cost was £52.93, or £12.93 after the credit. If I'd gone for the VW Polo that's a 10 minute walk from my flat (rather than the BMW at 5 minutes), it would only have been £41.81 for the 5.5 hours.

Out of interest, I checked with Hertz about how much it'd cost to hire from them. They're just over a 10 minute walk from my flat (just past the Streetcar VW Polo), so if I was to go for the cheapest streetcar they're probably a fair comparison. On a weekend, they're only open on Saturday mornings, so not as convenient. However, to hire a car slightly bigger than a VW Polo from Saturday morning to Monday morning, their website quoted me £43.60 (plus any petrol used). For the same journey I'd just done, that would mean a cost of about £50, but instead of only having the car for 5.5 hours, I'd not have had to worry about being late back, and I'd have the car for all of tomorrow too. Streetcar want £105 for to have their polo for an equivalent 48 hours, that's with 40 miles included, which seems rather bad value. In the week Hertz are open a lot more, and they only want £22.40 for a 24 hour hire (plus petrol). Add in a bit of fuel, and any hire over about 3 hours looks to be cheaper to go with Hertz, which was a surprise to discover. That's not even taking account of the £60/year Streetcar fee.

Overall, the process with Streetcar was ok, but averages out not quite as quick and easy as Hertz, but certainly better than other car hire places I've used. Cost wise, if you just want a car for an hour or two, it's handy and (fairly) quick and better value than a taxi. For me though, any hire over a few hours and I'm better off just going with Hertz for the day, as long as I'm not doing a last minute night or Sunday booking. It's a bit disappointing to discover this, but at least now I know. I'm sure I'll be using my Streetcar membership a lot more this coming year, just only for short trips or last minute things, otherwise you'll find me heading to Hertz for a longer hire, or on the train!

The Microsoft Binary File Format Validator

2011-07-18T21:45:08Z

I first got involved in Apache POI back in November 2003 (or so a check of the archives tell me!), which is almost 8 years ago as a write this... Back then, documentation of the Microsoft Binary File Formats was pretty slim on the ground, and there was quite a bit of reverse engineering needed. If you'd told me then that in 2010 I'd be in Seattle at a Microsoft event on the public documentation on the file formats, I wouldn't have believed yet! Despite this, the Documentation is public (and has been for some time), and at the end of last year I did head out to Seattle for the Office Binary File Format Plugfest.

One of the neat things about the plugfest (beyond learning lots, meeting other people developing similar libraries to read the file formats, and reporting docs bugs in person to their authors) was the Binary File Format Validator Tool. One of the few frustrating things was that I couldn't share the tool with anyone else... Good news though, as it's now out of private beta, and you can go learn about it and grab a copy from here! (It's technically Windows only, but it seems to work just fine under Wine)

Short term, I think one of the biggest uses for the BFF tool for POI will be for those dreaded bug reports of "I've got a file that seems to show a bug, but I can't let you have the file". Hopefully now the user will be able to run the BFF against their input file, and check it's valid. If it's not, then we know it's probably not our fault that it can't be read, and the user can go and speak to whoever wrote the software that generated it. In the case that the input's fine, the BFF can be run on the POI generated output. If this flag's a problem, then it'll hopefully come with the nifty details and docs references to help us fix it. No more "it doesn't work but I don't know why", and instead "option 5 should only be 0x3 when there are fewer than 32 entries, then it has to be 0x7, but POI has 0x3 hard coded". Link to the spec, a link to the problem code, and maybe even a patch too. Well, we can dream.... :)

Longer term, we'll hopefully hook up the BFF to some sort of integration testing, to be run every once in a while, to alert us to when we're generating files with issues. That said, the BFF is stricted than Office is, so we'll need to fix up all the cases where Office lets us get away with something the spec doesn't technically allow, so we're warning free. This may take a little while to work through, so volunteers and patches are very welcome, please come help us!

BarCampApacheSydney

2010-11-16T11:26:28Z

Having had a great time helping to organise a BarCamp in Oxford in June as part of Transfer Summit, another in Atlanta as part of ApacheCon, plus a few at Apache Retreats, it shouldn't be a bit surprise to hear that I'm helping with another in December.

What might be a bit different about this next on is that it'll be in Sydney! Yes, by popular demand, and thanks to several excellent and hard working volunteers on the ground, we're going to be having our first Apache event in Australia!

BarCampApacheSyndey is to be held on Saturday 11th December, at the The Darling Centre at the University of Sydney. In keeping with the Oxford BarCamps, we'll be doing a meal on the Friday night, then a full day's BarCamping on the Saturday.

If you're going to be in (or near!) Syndey around the weekend of Saturday 11th December, you can get more details on the event, plus sign up, on the website: http://barcamp.org/BarCampApacheSydney. See you then!

Upcoming Conferences

2010-09-21T15:00:29Z

This weekend I was at the Apache Retreat in Hursley, near Winchester. As you would have hoped from a weekend-long conference with Tipis and the ground floor of a former stately home, it was a great event. See the photos for yourself! Several people asked me about what other conferences I was doing this year, so I thought I'd do a quick write up.

First up is the Alfresco Developer Conference in Paris, which is the 20th and 21st of October. It looks like I'm only going to be there for the Thursday now (owing to a Microsoft binary formats event), which is a shame as it's looking like a great two days. I'll be talking about two related Alfresco services, the Content Transformation Service and Metadata Extractor Service. I'll be looking at what these services do, how to make better use of them, and how to extend them. Quite a bit of this will be around the new work for 3.4 which uses Apache Tika to deliver support for a large number of new formats.

For more information on my talk, you can watch the video interview I've done as part of the conference promotion. More information on the event is at http://www.alfresco.com/about/events/devcon/

Two weeks later is ApacheCon in Atlanta. ApacheCon runs for the whole week, from Monday 1st to Friday 5th November (which means I can't be in New York for the 2nd Alfresco DevConf, why do these things always have to clash?). It's looking like a great conference, and I'm going to be quite busy there, talking, organising and attending! The Monday and Tuesday see the BarCamp un-conference, which should feature loads of great sessions and discussions on new and existing Apache technologies.

The main sessions run Wednesday - Friday, and here I'm running the "Content Technologies" track. I'm kicking off the track with an overview session, which should give an introduction to a lot of the many and varied content related projects at Apache. Hopefully, it'll also whet the appetite of people for the rest of the track! The track will then continue with talks on various content areas, from the new CMIS standard + Apache Chemistry project which helps you build applications backed by content stores, through PDFBox and POI, to how convert your application to be content driven. The chances are, your application deals with content somewhere, so come along and learn how Apache projects can help you!

Finally, I'm hoping to help with running an Apache BarCamp in Sydney in December. The Apache committers in Sydney are currently in the final stages of booking the venue, so hopefully this can be confirmed and announced shortly...

Fringe 2010

2010-09-05T21:37:02Z

Note - This post has taken a little longer than planned in the writing....

After a couple of days hiking near Glen Affric, I made it to the fringe for the Friday afternoon. My first few ideas for what to see were thwarted by shows having sold out, but by just gone 4pm I'd started my fringe trip for 2010!

Friday

First up was The Leeds Tealights at Underbelly, which was a random picked based on the 4* reviews on the fliers. It was a 4 person sketch show, 3 guys and a girl, which set up the dynamic for a fair few of the skits. Most of the skits were only a few minutes long, so they got through a lot of material during the hour. A couple of them were absolute gems, most were fairly good for most of the time, and only a few fell completely flat... Overall, I'd give it 3*, and it was a fun and lighthearted start to the fringe!

Next up was the stunning Tabu out on Leith Walk. The non-central location allowed them to have a large big top as their venue, setup for standing rather than seating. It was billed as a circus show, but turned out to be so much more. Firstly, there was the live band, providing a musical backdrop to the show, often accompanied on vocals by the performers not doing acrobatics at that moment. Secondly, it wasn't just the (amazing!) acrobatics, the pieces changed and were woven together which each other, and the music, to provide a narrative for the show. It wasn't a static show either, instead the whole audience was moved about between pieces to create the appropriate gaps and spaces for the next set. Finally, the acrobatics themselves - stunning! Very varied, covering most major circus and acrobatic styles, stunningly well executed. It was a wider range of things than Circa (from last year), so not every piece ended up as developed as in that, but with the music and the thematic links between pieces, it was a wonderful performance. 5*

Saturday

My 2nd day at the fringe kicked off with Shakespeare For Breakfast at C. With free croissants, and being one of the few early starting plays, this is a usual favourite of ours. This year source of inspiration was King Lear, and their modern, silly take on it worked quite well. However, it's by no means the best production they've managed in all the years we've seen them... Good start to the day though, 3*
Was a zany version of King Lear, solid 3*, but last year was better

This is Belt Up's third year at the fringe (2 years ago they did The Red Room, and last year The Squat as C Soco). They were back at C Soco again this year, with a series of immersive theatre set in The Room Above. The first of their shows that we saw was The Boy James, which was a very intimate and immersive show. A boy, his older friend/self, the joys of childhood adventures shared, and the loss of growing up. Amazing stuff, an excellent 5*

Next up was another Belt Up show, The Second Star to the Right, carrying on with the JM Barrie theme. It was an almost hypnotic at times performance, with some lovely dance-like parts, as 4 actresses took us through bits of the Peter Pan story. 4*

Roam at Zoo Southside was a piece of modern dance by a company we'd seen in previous years. Unfortunately, this year's show had no real narrative, but that did still leave us with some amazing dancing and great music! The dance was great as ever, 4*

Vanishing Horizon at Zoo was largely back to theatre, though with some almost dance-like physical elements in places. A variety of interwoven stories covered the history of women aviators, family discovery, and the perils of trying to make a radio play... The stories fitted well together, and apart from a few iffy timings in the fictional bits (the historic factual parts seemed fine!), worked well. There was frequent, very impressive use of a large number of suitcases as props, and great physical parts using them to link everything together. Interesting and engaging theatre, 5*

For a lighter note, we opted for Amateur Transplants again this year, and were rewarded as they were on top form! This year's show featured both of them, and focused more on the excellent shorter funny songs, rather than fewer longer works. As ever, it was dark and silly medic humour, but hilariously done! No-one left through being offended either... Back up to their previous 5* standard again.

Finally was Geraldine Quinn (@geraldinequinn), for some musical Australian comedy. The show was about modern pop music, and featured some excellent, cutting songs and dance routines, exposing and ridiculing the faults of the pop industry. 4*

Sunday

Carrying on with Belt Up shows, we began Sunday withBelt Up's Oddssey. This was another excellent immersive piece, in one of their smaller venues, and powerfully done. However, while the re-setting into a dystopian future was good, it did sometimes make it tricky to entirely follow, so largely one for people who already know the original! 4*

Hunchback of Notere Damn at Pleasance wasn't the only production of the story running, but was one of the good ones! Set in a smallish venue, it was a one man show, a powerful portrail of a poignant tale, 4*

More Belt up, this time Belt Up's Metamorphosis, with their take on Kafka's tale. It was interesting staging, but I didn't find it as well translated as last year's "The Trial", so it ended up being that little bit to odd for my tastes... (I realise that Kafka isn't supposed to be normal, but still!) I rate it 3*, but many others in my group gave it 5, take of that what you will!

Over to Pleasance for our first of two boxing shows - Beautiful Burnout. This proved to be an excellent mix of story, monologues, group pieces, physical theatre and some dance-like fights. It did a great job of showing all sides of it, and worked well no matter what your knowledge or feelings on boxing were to start. An excellent 5*

Another fun Australian musical comedy followed, in the form of Sammy J. This was our third time seeing him, and he remained as great as ever. True, you often question his sanity, but you laugh all the same! Songs and stories, probably almost all true, of his more recent past (last year's show did the earlier parts), and very funny. 5*

Putting It Together by Sondheim was a late night almost-musical show. Rather than being a full Sondheim work, it was a collection of his songs around the themes of love and marriage. The songs were done well, but the pick'n'mix feel that went with the lack of any real narrative was the slight snag. 4*

Monday

An early start, and for a full length play - History Boys at Greenside (again one of several productions). As one would have hoped from the reviews from previous years, this was a good story well told. The direction largely worked well, and it was a solid 4* production.

Then for something a bit darker - Blackout at Underbelly. This dark, depressing, excellent piece of physical theatre told of one young man's journey to mindless violence. Based on a real story and real interviews, it was dark but great. 4*

Pluck were back again this year, relocated to the Gilded Balloon. Once again the trio impressed us with some crazy slapstick whilst managing a fairly decent rendition of a classical music concert at the same time! It wasn't quite as good as their previous show, but is was still good fun - 4*

Belt Up's Lorca is Dead was I think their best show of the fringe. Belt Up are always surreal to some extent, but the chance to play the Paris set of surrealists, debating and retelling the life and death of Lorca really lets them into their own. Strange, but great! 5*

More Belt Up followed, with Belt Up's Atrium. More great immersive stuff, weaving in and out of the different fantasies of a dying writer dictating his memoirs. Some great silly costumes too, but I must remember not to be a tiny bit more reserved when sat at the front, lest anyone have to see another one of my 5 second Kafka looks...! Good show though, 5*

Tuesday

Ovid's Metamorphoses at #edfringe, told through the medium of 40s music, a few great bits with puppets, and some fun physical theatre! 4*

Others by Paper Birds, who for the previous two years did the stunningly powerful and amazing In A Thousand Pieces. This year they'd sent letters, and then questionnaires to a number of women they felt to be interesting, and different `(other) to them. The play again featured three actresses, moderately minimal props, but this time was light-hearted for parts. It was made up of the letters sent and received, the responses not received, the actress's ideas of what the responses might be, and the discussions that led from them. Parts showed the similarities between apparently different women, parts the unexpected differences, and parts the terrifying cluelessness of the actresses (I think, well, hope, largely hammed up!). In the end, it was a lighter piece than last year, but ended up not quite so powerful and thought-provoking. Still great stuff though, 4*

Hamlet! The Musical - You'd probably expect this to be dreadful, but it actually turns out to be silly and quite good fun! 3*

NewsRevue were on fairly good form once again. A few of the tory jokes seemed a bit rehashed from the 80s, but the Nick Clegg puppet worked well! Throw in some good characatures, and several Glee related songs, and you have a fun hour. 3*

Finally the last show of the Fringe - Shadow Boxing. This was a one man show, about not only being a boxer, but the personal growth and discovery that lay around it, and with some good twists as it went along. Not as good as Beautiful Burnout, but a solid 4* none the less.

And after 25 shows, that's it! During the course of the fringe, I tweeted short reviews, which seemed to go well (based on in-person feedback from many of my friends, and from twitter comments), so I'll probably do that again next year. The above are hopefully longer and more thought out reviews, though as they vary in being written between 1 hour and 2 weeks after the event, they do vary somewhat...!

Playing BBC HD and ITV HD on a DVBWorld DW2104

2010-06-12T17:07:59Z

Having got fed up with the ropey reception on Freeview, due to ongoing problems with my local transmitter (engineering works, fires, too windy for helicopters, the list of excuses is impressive...), I decided to pick up a DVB-S adapter for my mythtv box.

The snag with having a very small, low power mythtv machine is that all the adapters need to be USB, which does limit the choices of linux compatible DVB-S adapters. However, a bit of browsing of the Linux TV wiki and some patience with ebay searches paid off, and I was able to pick up a DVBWorld HD 2104.

With the help of this site, I was able to get the firmware for it, and it was quickly up and running. I followed this guide to get things running, which largely worked for the SD channels.

However, despite the scan of the Astra 28.2E and Eurobird 1 satellites showing me both BBC HD and ITV HD, I was unable to get either working with MythTV or mplayer. Trying in mplayer I saw:

nick@minimyth:/dev/dvb/adapter1$ mplayer dvb://2@"BBC HD"
MPlayer UNKNOWN-4.4.3 (C) 2000-2010 MPlayer Team
Can't open joystick device /dev/input/js0: No such file or directory
Can't init input joystick
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing dvb://2@BBC HD.
dvb_tune Freq: 10847000
Cache fill: 18.85% (1581056 bytes)   
TS file format detected.
VIDEO MPEG2(pid=5500) AUDIO MPA(pid=5502) NO SUBS (yet)!  PROGRAM N. 0

But nothing played. Using szap myself to capture a bit of the stream myself, and trying with that, I got sound but no picture:

mplayer /tmp/HD
MPlayer UNKNOWN-4.4.3 (C) 2000-2010 MPlayer Team
Can't open joystick device /dev/input/js0: No such file or directory
Can't init input joystick
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing /tmp/HD.
Cache fill:  0.00% (0 bytes)   
TS file format detected.
VIDEO MPEG2(pid=5500) AUDIO MPA(pid=5502) NO SUBS (yet)!  PROGRAM N. 0
MPEG: FATAL: EOF while searching for sequence header.
Video: Cannot read properties.
==========================================================================
Opening audio decoder: [mp3lib] MPEG layer-2, layer-3
AUDIO: 48000 Hz, 2 ch, s16le, 256.0 kbit/16.67% (ratio: 32000->192000)
Selected audio codec: [mp3] afm: mp3lib (mp3lib MPEG layer-2, layer-3)
==========================================================================
AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
Video: no video
Starting playback...

The key thing to spot here is that mplayer thinks it has MPEG2 video. However, both BBC HD and ITV HD are H264 when broadcast over Freesat! After some googling, it turns out that there's something up with how the dvb-s scan program outputs the channel lines. The easy option is to tell mplayer to use a workaround for this, by passing in an extra option to mplayer - -demuxer lavf

Thus, the easy way to play BBC HD on the DVBWorld card (my 2nd DVB adapter), the command is:

mplayer -demuxer lavf dvb://2@"BBC HD"

This largely seems to work fine, though the sound sometimes drifts which needs a quit and restart to work.

However, it is possible to also hack channels.conf to contain the correct details to "just work(TM)". This seems a bit black magic, but you need to run mplayer with "-identify", pick the PMT_PID (via trial and error...), and add this into the channels.conf video entry with a plus. Thus, my channels.conf entries for the HD freesat channels are:

BBC HD:10847:v:0:22000:5500+259:5502:6940
ITV1 HD:10832:h:0:22000:2362+288:2369:10000
ITV1 HD:10935:v:0:22000:513+289:641:3851

By adding in the correct PMT PIDs, I can then just do mplayer dvb://2@"BBC HD" and it picks the correct video and starts playing! Still has sound drifts though, which I think might be slightly worse than with lavf but I've yet to double-blind test...

Next up, time to make MythTV believe the stream is H264 too!

Mentoring and Incubation at the Apache Software Foundation

2010-05-25T14:54:10Z

I'd like to give my personal view on the Apache Incubator, and how I see my role as a mentor of an incubating project with that. This post explains my views, and while it ought to chime nicely with official policy, there might be the odd error, and so this isn't official foundation policy. But first, a bit of history on how I ended up mentoring a project.

I've been using Apache software for a very long time now, and I'd guess I first played with the webserver back in something like 1996 or 1997, which seems a very long time ago.... I'd say I first properly got involved in Apache with POI when I started with Torchbox. A check of the archives shows I first started contributing patches back in 2003. I stayed around on the POI list after that, contributing more patches and helping out with user questions, and then in 2005 everyone had got fed up of committing my patches without changes, and I was granted committership.

In 2006, I attended my first ApacheCon, and had a great time whilst learning lots. In late 2006, I was elected to the Jakarta PMC, and discovered a whole new world as I learnt about PMCs, the board, how the ASF works and so on. This didn't put me off, and in May 2007 I helped POI leave the Jakarta umbrella, becoming PMC chair. At the 2007 ApacheCon, I'd gone to lots of foundation related talks, and began to learn in detail about "The Apache Way", which is partly how I ended up as the new POI chair. Roll forward to 2009 and I found myself giving talks on "The Apache Way", having in the mean time made member. Late last year I also joined the new Apache Community Development project, helping out with mentoring, outreach and so on.

It's been quite a long journey, from my first involements in apache, through my first commits and on to today. It has taken me a long time to learn about "The Apache Way", to really understand what it means, and get to a position where I'm able to try to help others to learn about it. It's really great to be able to stand up in front of an audience, and talk to them about open source, open development, and apache-style open development, and see that moment when someone gets it, understands part of why it's so powerful. Well, I say stand in front, but quite often it could just be sat in a pub, or even on the grass in a circle, enjoying the Irish sunshine!

Since starting at Alfresco, I've also become involved in the Apache Incubator, and I'm currently a mentor of the Apache Chemistry incubating project. Having explained how I got here, I'd like to look at what I think this role of mentor involves, and using Chemistry as an example.

Firstly, what isn't it? Well, I'd say a mentor isn't a committer. Sure, a mentor can also be a committer, and it's great when that's the case. However, I'd see those as two distinct roles, and you may sometimes need to switch hats. It's great if mentors can commit code, since it helps ensure that if there is any pain from the incubation process, one of the mentors quickly feels it too, but I don't think it's in any way required.

What is in the role of mentor then? I'd say the most important, and over-arching role is to teach the podling (incubating project) about The Apache Way. You can't do this in one go though (see above - it took me at least 4 years to get to the point that I was happy enough with my understanding that I could pass it on!), but you need to be trying to pass it on as and when you can. Partly this means watching the mailing lists, and offering insight and advice when things happen. Or don't happen. Possibly especially when the don't happen...

Within the incubator, various things need to happen whilst the project is there, and others before graduation can occur. As a mentor, you need to help everyone understand why they need to do something, as well as helping them do it. For example, the requirement that as much as possible of the project's communications should be on-list, and off-list things should be relayed back. It's no good just saying "no", you need to explain why, explain what gets missed if you don't, explain the benefits of doing it The Apache Way. With Chemistry, the big test for this was with in-person meetup in Munich back in March. Many of the committers were there, but certainly not all. I explained on-list beforehand why it was important to keep the list updated, then spoke again about it at the meeting. During and after the meeting, everyone who couldn't make it seemed very happy with how it had worked out, and how their views had been included. We also now have a record of those discussions to check back on if an architecture question surfaces again in future. Overall, it seemed to work well!

Three things that often cause issues within the incubator are IP clearance, releases and new committers. With all of these, the podling needs to do some work, and then the incubator PMC needs to sign off on this. Firstly in this then, the role of the mentor is to help the podling do the thing right, pointing that at documentation as needed, advising them on their process, and reviewing what they've done. When everything is fine, you then need to put on your IPMC hat, and approve it. Next, you need to prod your friends within the IPMC to ensure that a quorum approves the action, so the podling isn't stuck waiting. Finally, you need to ensure that the podling understands why and how to do it, because after graduation they'll need to do the same thing again, but without the oversight, so they need to be able to get it right themselves :)

Where does that leave us for defining the role of the mentor? Overall, you're there to help the project learn the apache way, both in what to do and why they should be doing that. You should be helping them when problems come up, and trying to spot problems and head them off before they develop. You should be helping the project to do the steps needed to run and graduate (clearance, releases etc), giving them guidance, advice and voting. You should be there to answer questions. If it all works, then the closer graduation gets, the less you'll need to do, as the closer the project will be to running itself happily in The Apache Way!

One thing I should probably point out at this juncture is about in-person vs remote mentoring. You sign up to mentor the whole podling project (while ComDev do have a formal 1:1 mentoring program, the incubator is about mentoring the project), so much of that mentoring needs to be remote, using mailing lists. However, with Chemistry, I found it very helpful to also have a chance to meet and advice many of the project committers in person. Much as the ASF is a global organisation with proceedures and a way of working that handles everyone being disparate and remote, some parts of teaching the Apache Way work best in person. That's partly why I've helped set up the Local Mentors Program (aka take a new contributor out for a drink to help explain the whole thing to them). With that in mind, I'm tempted to recommend that where possible, at least one of the mentors meets at least some of the committers at least once, probably ideally at an Apache BarCamp or conference.

There are many different ways to run a project, be that an open source project or a closed one. I'm a big believer that open development is the right way for many projects, but I'm also fairly well aware of the cases where it may not be the best. This isn't a post about project management and methodologies, though buy me a beer and I'll happily talk at length on the subject! Instead, I want to finish off with my view on why the Apache Incubator exists.

Within the ASF, all projects should be running to The Apache Way. Now, the Apache Way doesn't cover everything, just certain key areas, so each project is free to make their own decisions on how to do many things. It's part of the beauty of the ASF that different projects do try out different things, and share what works well, even if sometimes this leads to a week of discussion on members@.... However, most projects outside of the ASF don't run to the Apache Way. So, for a project to join the ASF, we need to ensure that the licensing is correct, the IP is properly cleared, and that the project runs itself The Apache Way. That's where the Apache Incubator steps in.

So, my view of the incubator is that it's somewhere to do the IP clearance, it's somewhere to sort out licensing and dependencies, but mostly it's a place to learn the Apache Way. Projects come in, they learn, and then the choose to either tweak themselves to fit the Apache Way, or they say "no thanks, that's not for us" and leave to go on their own way. Seems simple enough, doesn't it? :)

Who drives, directs and supports projects within the Apache Software Foundation

2010-05-25T13:30:10Z

Following on from my previous post on CLAs and Release Votes, I thought I'd do another one on the Apache Software Foundation. This time, it's looking at different roles in directing and supporting a project, and how the ASF deals with corporate contributions.

Note - this is a personal view. While I am a member of and PMC chair within the ASF, it's my view, not an official foundation statement. It's based on lots of discussions, talks and mailing list posts, but please do shout if you think I've got something wrong...

When looking at an Apache project, people will often make distinctions between four kinds of people:

Users - anyone who just uses the software, without really interacting with the project beyond that

Contributors - anyone who helps out with answering questions, contributing simple patches or tests cases, often but not always on the project mailing lists

Committers - people able to commit changes to SVN, both their own changes, and those coming in from the community from Contributors

PMC Members - those people tasked with keeping the project on-track, and directed by the ASF board to look after the project on behalf of everyone

It's often said that there is a pyramid of people within an Apache project. At the bottom are very large numbers of users, above them a smaller number of contributors powering the project, then a smaller number of committers making changes to the code, then a smaller number of PMC members overseeing it. If you can get yourself to a The Apache Way talk somewhere (ApacheCon, Apache BarCamp etc), you'll hear a lot more on this, but the important thing is that a project's success is driven by this large base.

Two common questions about an Apache project are: who drives and directs the project, and who keeps it going?

The answer to the 2nd is easy, but perhaps not the one you may think. It's the contributors! Without all the contributors out there, answering the questions of others, helping write examples, providing test cases and the odd patch, evangelising, your project won't grow and maintain itself. This is why you see some projects (eg httpd) rewarding those people who contribute documentation and examples, and not just those who contribute code. Almost all new committers come from the pool of contributors. Most users will hit a problem at some point, and it's through the work of the contributors in blog posts, mailing list answers, documentation or examples that help them solve their issue and carry on. Of course, on every project there are committers doing the same thing, but in most cases they started doing all that before as contributors, and it's the work done as contributors that keeps the project going!

Now, what about who drives the project? On most projects, at least some of the people, and possibly almost all of them will be there in their role as employees of a company. How should committers (and pmc members) handle the need to wear two hats, those of an employee and as a committer to the project?

Within the project, new features are added by whoever does the work. This might seem obvious, but it's worth remembering that there's no magic coding fairy secretly committing while we all sleep - all new features and bug fixes come from someone spending time at a keyboard working on them. In some cases, that's someone working on it in their own time. More often though, it's someone working on company time, usually on a project that is important to their employer.

So, we can see that on many projects, the new features in the project will be those which matter to the companies providing the employees to work on them. But, does that mean that those companies get to set the direction of the project? In some foundations, the answer would be yes, but at Apache, the answer is a little more complex. It's maybe, but only if the community agrees!

How does this work then? Generally, people should announce in advance what they're working on, and seek feedback on it. That could mean opening a JIRA for the new feature, describing it, then later attaching the patch to it. It could be a quick post to the mailing list, followed by a later commit. For a very uncontentious change, it could even be committing the patch and describing it in the commit message, assuming that your project allows commit-then-review for that sort of change. Generally though, for a company-driven change, a committer will announce what the new feature they're planning to work on.

Once the new feature has been described, what then? It could be that everyone feels this is a useful addition, and +1's it. That's the ideal case - your company's needs closely match those of everyone else, and everyone's happy! Much more commonly, people will say things like "that isn't what I'd choose to work on first, but it's useful, so go for it" or "I wouldn't use that, and wouldn't code it, but I can see the value for others, so go for it". These would be more of a +0, hinting that while the change is good for the project, it isn't something that others care deeply about (which is perhaps why no-one has done it before...) In all these cases though, the community have agreed that the change that your employeer needs fits with their needs too.

In a few cases, you'll get someone saying "that's a stupid idea, you should be working on the thing that I need instead!". That's usually the point that you need someone else to step in (so as not to start a flame war), and explain that Apache is a participatory meritocracy. If there's a feature you want, you have three choices - code it yourself, pay someone to code it for you, or inspire someone to code it for you. No-one's under any obligation to write something for you, everything's done by people who want to do so!

Now, what about the case when you describe a new feature that you want to work on (either because your employer wants you to, or even just you want to in your own time), and the community isn't keen? That might be because it will have wide ranging changes, or introduce new dependencies, or will work using a new methodology, or half a dozen other things. Well, firstly, the thing to do is to hold fire on your changes. Instead, you need to make your case, probably on the mailing list, and explain why you feel your change is the right one to do. With any luck, people will suggest alternate ways of doing things without the problems, and you can get the new feature you need whilst also having the community support you.

What happens though if after that discussion, you haven't convinced the community? Generally, that's the time to try it anyway, but not on the main codebase! Let everyone know you're trying it out, and do so on a branch, or in an external SCM, depending on the scope of the change. Try to let the community see what you're up to. Then, when you're done, come back and make your case again. Hopefully now things will be easier, because you can point to real code, and say things like "there had been concern that the new configuration system would slow down the core, but if you try it with this benchmark, you'll see it's actually 5% faster". It could be that when people see the new code, they'll agree, you have a quick vote, commit the change, and everyone's happy :)

And if even having seen the code, the community doesn't agree? Well, sorry, but you'll need to maintain a vendor fork at this point. After all, your company needs it, and the community doesn't want it, so it'll be up to you to maintain. Next time, try to make the case to the community better...

In general though, the project moves in the direction of those with the time to code. When those people are working on company time, the project will tend to move in the direction that that company wants it to. However, that's only the case while the community agrees with the direction, and if the company tries to push something the community doesn't want, that all changes.

Apache - powered by the people who do the work, but directed for and by the community!

CLAs and Release Votes within the Apache Software Foundation

2010-05-23T15:21:46Z

There has been a fair bit of discussion of late, both within the Apache Software Foundation and outside of it, about whether some of the processes and proceedures are a help or a hindrance. With this in mind, I thought I'd write something about how I see things.

Note - this is a personal view. While I am a member of and PMC chair within the ASF, it's my view, not an official foundation statement. It's based on lots of discussions, talks and mailing list posts, but please do shout if you think I've got something wrong...

There are three main things that people seem to moan about around the ASF. The first I won't really dwell on, that of "why can't I use ${latest_tool_of_the_moment}. As the hard-worked ASF infrastructure crew will happily tell you, you can use anything that can be supported, maintained, bug fixed and generally looked after 24/7 by volunteers scattered around the globe. If you want something new, be prepared to help look after it!

The other two main things seem to be around CLAs, and release voting. These two offer some very important protections for everyone - users, developers, committers and the ASF. The need for these are documented at http://www.apache.org/dev/, but as there's a lot to read there, I'll try to give my view on why they matter.

Firstly, a question. What are two of the easiest ways to close down an open source project?

The answers, I feel, are to either sue the release manager, or to sue whoever holds the source code.

For the first, think how long your project would last if the release manager was sued for a very large sum of money, and anyone else who might consider signing up was threatened with a similar lawsuit too? With no releases, one key community member out of action through a lawsuit, and everyone else worried, the answer is alas not all that long. Personally, I think it's great that I'm both allowed (and encouraged!) to work on open source at work, whilst also well enough paid that I can afford to spend part of my free time coding, mentoring etc within open source. However, it does also mean I have enough to loose... And having been an Apache release manager in the past (for Apache POI), I'm very glad that the ASF protected me. But how?

If you do a release of some open source code on your own, and someone has it in for your project, then you could be at risk. One of the key reasons for founding the ASF was to provide protection for this. What you want, should something bad like this to ever happen, is for some ASF lawyers to turn up in court and say "stop suing this poor contributor, sue us if you're going to sue anyone", and have the judge agree. For that to work, the release needs to have been done by the ASF. That's where the release votes and rules come in.

What you're after, therefore, is for the ASF to stand up and say "that's our release". The ASF wants lots of good open source software to be out there and being used, so wants to support your release, but tat the same time, doesn't want to take needless risks. The release requirements are therefore basically that enough people who provide oversight have OK'd the release, and that as far as is possible, all the code in the release is allowed to be. Therefore, we see the requirement that 3 PMC members have +1'd the vote, the source has appropriate headers, notice files and licenses are clear, the build is open and repeatable, and dependencies are all ok. (The PMC, or project management committee, is the ASF board's eyes and ears on the project). Largely above and beyond that, what each project wants to do in terms of releases is up to them. Things like betas, release candidates and numbering all lies within the project, based on how the community wishes things to work.

Now, on to the CLAs, or Contributor License Agreements. There are two of these, the iCLA (for individuals), and the CCLA (for companies). These are actually fairly simple documents, and all relate to the core Apache License v2.0. For an individual, it is basically "I understand what it is to release software under the apache license, and what rights I wave to do so, and I'm happy and able to do so". For a company, it's basically "I understand what it is to release software under the apache license, and what rights I wave to do so, and I'm happy for my employees to do so". These provide the foundation, and the software users (the community) with the protection that someone can't turn around a year later and say "I know I said I was contributing under the apache licenese, but I've changed my mind, and now I'm going to sue you".

Many other foundations are happy to accept contributions under their licenses (be that the ASL, GPL or whatever) without requiring that those making significant contributions formally state (via a CLA) that they understand the license, and are OK with their contributions being under it. However, within the ASF, we like to be very sure on these things (which is part of what allows our software to be used so widely without lots of legal checks being required first). As such, we ask people to file a CLA before they contribute, just to confirm they're happy with how their contributions will be used.

So, we see two key parts of Apache policy, the release process (checks and votes), and the CLAs, are both there to protect the project and the community. They happen to protect the ASF too, but they're mostly about protecting everyone!

As an aside, if much of this is new to you, I can't recommend enough attending a "The Apache Way" talk somewhere. You'll find them at all Apache Cons, at all Apache Bar Camps, and many other things, and are a great way to learn about things like this, both what the proceedures are, and why they exist. It's much of how I learnt these things!

Compiling for your N900 on debian

2010-01-12T19:34:36Z

I've just got my Nokia N900, and while the phone-phone synchronisation pulled my calendar entries and contacts over from my N95 just fine, it didn't do the sms's.

Since it runs linux, the fact that the official software doesn't support it isn't the end of the world. Handily, someone's written a small sms importer program that reads in the sms's from the S60 pc-suite csv export, and loads them onto the phone.

Only snag is the pre-built version didn't support long sms's, so I needed to patch and re-compile. That means needing an arm cross-compiler, the build environment etc.

Handily, that's fairly easy to setup. The best guide I found was http://maemo-sdk.garage.maemo.org/install-debian.html, which got me almost all the way there.

One thing to note is that you need to install the "Nokia Binaries" if you want to do much development, which includes various key system components and dev libraries that aren't open source. http://wiki.forum.nokia.com/index.php/Maemo_5_SDK_installation_for_beginners has what I found to be a slightly easier guide to follow on that.

Now I'm about ready to try my newly compiled smsimporter. A couple of commands that I've found to be helpful are:

sb2 -eR apt-get install [pkg] - installs the given arm native package in the build root

maemo-sdk enter devel - enter a shell suitable for compiling for native arm

Hopefully I'll be knocking up a few more little bits of code on the weekend :)

Nearly everything runs linux

2010-01-12T19:23:49Z

My new phone runs Linux, which is great. So, I thought I'd do a quick roundup of all my computing devices, and see how close I am to all Linux:

Desktop - Debian Stable

New work laptop - Ubuntu Karmic

Dell Mini laptop - Ubuntu Karmic (netbook edition)

Media PC - Ubuntu Karmic (mythbuntu edition)

Nokia N900 - Maemo Linux

2nd wireless access point - tomato (linux)

Apple TV - stripped down OSX, so Unix but not Linux

ADSL + wireless router - not sure, but not apparently linux

Nintendo Wii - not linux

I think that's about it for computing devices, so not bad all in all. When 802.11n adsl routers that run Linux come down in price a bit more, I'll ditch the current cheap'n'chearful (but with a few annoying bugs) no-name router I have for one of them, and I'll almost be there!

New phone!

2010-01-12T19:19:39Z

At long last, vodafone finally had a N900 in stock, so I have a new phone!

In general, it's proving really good. It runs linux, which is making everything so much easier (no need to hack it like you do with S60 - just apt-get install whatever you want!)

So far, I've enabled SSH, installed several extra bits and pieces, and imported almost everything over from my N95. Will shortly attempt to import the SMS's too....

Now I remember why I try to avoid coding in c....

2009-09-05T18:12:22Z

At work, in common with most IT firms, we have a need to securely store a lot of passwords. Quite a while ago we settled on PWMan as the tool to do this with. It has a nice ncurses front end, and uses gpg encrypted xml files under the hood, so there's no worries about getting your data out if you ever want to go to anything else. After posting a few patches, several of us were given commit access, and all was good.

Roll forward to last week, and I'm now the only active maintainer of the project. We haven't had any problems with pwman for a while, but there are one or two features that people have asked for, and I figured it might be good to try to implement a few of them. The biggest thing was search (filtering results on the screen has been in there for an age, but not finding all entries anywhere in the tree). As pwman is written in c (using ncurses and libxml2 directly), there isn't anyone else at work who was comfortable making the changes, so it was either I do them or no-one would. I had a couple of hours spare, and I figured that might be enough. My hunch was 20-30 minutes were it in python, double it as I hadn't worked on the codebase lately, apply my usual "it's java not python, double it" multiplier, and hope that c and java coding times are similar. That gave me 2 hours, which seemed fine.

How wrong was I...

We're now at about 6 hours of coding, and search is almost but not quite fully implemented. Firstly I wrote the search algorithm and structures as I would in python, perl or java, only to remember at the last moment that auto-extending arrays and vanilla c don't mix, and I didn't want to have to pull in something like APR as a dependency at this stage. So, back to the drawing board, and I had to re-write the whole thing (algorithm, data structures etc) using linked lists.

Then, when I compiled and ran the program normally, and did a search it segfaulted. Compiled it without the optimisations, and with the -g flag, and ran it under gdb. No segfault, seemed to be fine. I was very bemused. Then I ran it a bit longer under gdb, and eventually managed to trigger the segfault.

At this point, I was once again reminded of why I don't like c. The bug was a simple, schoolboy error - I'd added a couple of new fields to a structure, but I'd forgotten to null one of them. Not something you need to worry about with java, python or perl, but as any C hacker will tell you, very very important there. I'd remembered for almost all of my new fields, forgotten one, and thus as soon as the memory was re-used for another similar structure (more common at the higher optimisation levels), it all went to pot :(

So, I did a full review of all the structure creating code, spotted a few other places that might need it too and added it explicitly, then all seemed fine. I do now know a lot more about using gdb than I used to, but it's all taken longer than I'd hoped for, and so I still haven't managed to find the time to write the rest of the UI. Hopefully I'll get a chance soon, and then pwman will have nice and shiny search. In the mean time, I'm reminded why lots of people end up deciding that porting non-performance critical code from c to python is worth doing when they want to add moderately complicated features....