While trying to track down some issues, I wished I had Squeak’s PointerExplorer, which makes it easy to examine the references to an object—and figure out who’s causing problems by hanging on to one.

After a quick look at a few existing tools for tracing leaks, it doesn’t appear that there’s currently any way of doing this. So I took some code from Joe Damato, updated it a bit, and implemented two things: weak arrays, and GC.heap_find().

Weak arrays work pretty much as you’d expect:

>> a = []
=> []
>> a.weak = true
=> true
>> 1000.times { a << Object.new }
=> 1000
>> GC.start
=> nil
>> a[200]
=> nil

Like the PointerExplorer, GC.heap_find scans the heap to find references to an object. Since it uses the same algorithms that the garbage collector itself uses, it won’t miss anything. (If Ruby’s mark-and-sweep collector can’t find your object, it’s not going to be kept.)

>> obj = Object.new
=> #<Object:0x1007f6900>
>> a = [obj]
=> [#<Object:0x1007f6900>]
>> h = {:foo => 2, :bar => obj}
=> {:bar=>#<Object:0x1007f6900>, :foo=>2}
>> refs = GC.heap_find(obj)
=> [{:bar=>#<Object:0x1007f6900>, :foo=>2}, [#<Object:0x1007f6900>]]
>> refs[1].object_id
=> 2151609120
>> a.object_id
=> 2151609120

One slight issue is that heap_find can’t return some objects that are traversed by Ruby’s GC, but aren’t first class Ruby objects—things like SCOPEs and VARMAPs. heap_find could indicate their presence somehow, but, since they’re not first class, they can’t be returned directly. This isn’t that big a deal: memory issues here are pretty unlikely, and any present probably indicate a bug in Ruby itself.

Because the arrays returned by heap_find are weak, you can continue to explore the object graph as far as you like, without worrying about them interfering with the results, and leading to confusing additional references.

>> refs.weak?
=> true
>> obj2 = Object.new
=> #<Object:0x1013ec9c0>
>> foo = [1, 2, obj2]
=> [1, 2, #<Object:0x1013ec9c0>]
>> GC.heap_find(obj2)
=> [[1, 2, #<Object:0x1013ec9c0>]]
>> foo.weak = true
=> true
>> GC.heap_find(obj2)
=> []

Here’s the patch for ruby 1.8.7-p174 (shipped with OS X 10.6).

Potentially most powerfully of all, developers should be able create their own plug-ins that run on the server. There should be an agreement between plug-in developers and the webmail provider that creating a plug-in automatically grants a royalty-free perpetual irrevocable worldwide (etc.) license to the provider, and that the source code to any plug-in may be merged into the main product. Though plug-ins have niche appeal, this could be a good source of new features, and a strong competitive advantage. I’d just fix Gmail if I could.

I’d happily pay for any service that got this stuff right.

$ crontab -l
0 * * * *   /Users/patrick/Binaries/clean-downloads
$ cat Binaries/clean-downloads
#!/bin/zsh
find ~/Downloads -maxdepth 1 -amin +720 -exec mv {} ~/.Trash \;

You have twelve hours to do something with whatever it is you downloaded, or it’ll end up in the Trash. I highly recommend it.

• How slowly things change. When we started Auctomatic in mid-2007, eBay was dominant but faltering, PayPal was lumbering along without any innovation, Facebook was amazing everyone (and scaring Google) with their growth, the iPhone was reshaping the cell phone industry, and everyone was talking about Twitter. Two and a half years on, that still sounds like a reasonable assessment of things. With Auctomatic, we felt this tremendous sense of urgency. That may have been good for other reasons, but it was misguided. When building something, I now think you should assume that you’ve plenty of time, and that very little will change in the interim. (This is, admittedly, the kind of rule that works until it doesn’t.)
• Tied to this, the realization that a good product is so much more important than first mover advantage. Obvious, but keeps surprising me. Facebook, the iPhone, Dropbox, Skype, etc., have an amazing array of failed predecessors.
• Succeeding at biz dev just requires endless persistence.
• Internationalization is an underexploited axis—people try to expand on the x-y plane, and ignore this third dimension. We grew our iPhone app revenue by over 200% through internationalization. The biggest competitve advantage we ever had with Auctomatic was supporting the obscure international eBay sites that the big US players ignored. I’m generalizing from pretty limited experience, but if I were a floundering start-up trying to get to cashflow positive, internationalizing is probably the first trick I’d try.
• How dangerous the assumption that you’re going to grow is: “oh, we need to build this this way, because when we’re successful, we’ll need it”. This is often used as an argument against over-engineering for scalability, but it applies more generally. Now, when coding, I try to think: “how can I write this such that if people saw my code, they’d be amazed at how little there is and how little it does”.

So here’s undop. Every time I visit a domain in ~/.bad_sites, it dims the display by 30%, gradually returning it to its previous brightness over the course of 30 seconds. It’s both an annoyance and a reminder. The effect is small enough that I don’t mind it on those occasions when I actually do want to read Twitter or Hacker News or whatever blog, but it’s jarring enough that I now subconsciously avoiding following links to them, and have mostly broken the check-for-new-stuff muscle memory.

Caveats: it only works with Safari on OS X. (Patches with support for other browsers welcome.) I’m guessing it might not be good for display backlights.

$ git-find-removal HEAD~20 'def my_important_function'

I’ve known Daniel since first year in secondary school—he convinced me to learn PHP, my first real (”real”?) programming language. He’s one of the best programmers I’ve ever worked with, and just won an IBM Open Source Award for his work on Yakumo [PDF], an open-source library for using the iPhone a game input device. Despite only just graduating now, a couple of companies have already spotted his win-ness—he interned with the hardcore physics guys at Havok in San Francisco last summer.

App School is lucky to have him, and I think the courses are going to be killer.

See also: Damien, Web 2 Ireland, Irish Times.

Every day, you log into your analytics dashboard, and you see basically the same picture that you saw the day before. It’s left up to you to find out what’s changed. Did your search referrals for some mid-ranking keyword rise hugely? Are you getting referrals from some site that never linked to you before?

Additionally, most packages don’t provide any mechanism for you to integrate your business-specific data with your web site data. Sure, you can sometimes hack this with things like tracking visits to special “checkout” pages, but fundamentally there’s no direct way of integrating off-site data (sales of an iPhone app, say).

I wrote some Emacs-based software to do all of this last year (including charts in the REPL), and have been using it fairly happily. Recently, I’ve started using Mathematica more and more to process log data, and have been very happy with the results.

A few days ago, Google released an API for Google Analytics (largely invisible but enormously successful). I spent a while over the weekend writing a bridge to Mathematica, and playing with the data, particularly that related to my iPhone app (Encyclopedia, which stores a copy of Wikipedia on your phone for offline browsing).

We can start off by doing fairly standard things, like number of visits from each search term by date:

And we can, say, chart the result. (First, we remove the “(not set)” results that the API returns.)

What fraction of this is from search traffic related to the iPhone app? We can select the promising search terms, and compare them with overall search traffic (as far as I know, no boolean logic is possible in the Google Analytics web interface):

Looks like most of the Google traffic is from the app-related keywords. We can also pretty trivially look at the geographical distribution of these visitors. First, let’s load the data:

And group the result by country:

We can now define a simple WorldPlot function, and use it with our data:

(Brighter colours indicate more visits.) Okay, that’s somewhat interesting. But how many sales does the app get per visit? I track my sales in Dabble DB, so I’ll load the data from there. They provide a CSV file of country code to unit sales mappings:

We can pretty easily import this into Mathematica:

Using Mathematica’s CountryData[] function, which knows about country codes (as well as their shapes), we can easily integrate the keyword referrals and the sales data to generate a heat map of purchases per search referral:

The interesting outlier here is Mexico. Though they don’t visit the site much, they do buy the app a lot. (Hundreds of times to date.) I’m still not sure why. The two maps also show that, although Ireland sends quite a bit of traffic to the site, it converts quite poorly—perhaps because my .ie domain causes my search rank there to be artificially inflated.

Okay, ignoring the app for a while, we might also be interested in questions like “what are my most important search terms?”. One way of measuring this is to look at the total time on your site due by each search term. (Google Analytics tells you the average time on site for each term, but that’s not much help in telling you where you should direct your work.)

Let’s load the data:

And define a simple function that computes the total number of seconds on the site driven by each keyword:

We can compute the set of search terms that were used to get to the site:

And sort them by time on site:

Ignoring the first two (corresponding to “(not set)”, which we could of course filter out), it seems that “wiki”, “wikipedia” and “iphone” are the important terms.

We might also be interested to see how keyword usage changes over time. Let’s load two months of keyword data:

And define a simple function to count the fraction of referrals that are from a particular keyword in some given dataset:

We can test it with something like:

Okay, 4% of the site’s visitors came by searching for something containing the word “iPhone”.

Now let’s compare two months of data:

These are the search keywords that showed the biggest increase between October and January. And they correspond quite closely to what you’d expect—default.png files on the iPhone and Back To My Mac are both topics I wrote about in the intervening time period.

Keeping with the spirit of analysing changes, we can look at non-search referrals:

We can drop the direct referrals, group them by source, and select the first referral from each site:

And then we can do something like plot the first referral from each site against the amount of traffic it sent—a chart of buzz, basically:

Next, we can look at how traffic flows around the site. First, we load (landing page, exit page) tuples:

And then hack together a function to generate a directed graph:

Yielding a pretty interesting result:

As a final example, we can, for no good reason, take advantage of the latitude and longitude metrics that the API provides and quickly create a video of visits by hour, showing traffic move across the globe. First, we load the data, and define two helper functions:

We can test it out on one of the hours:

Looks good. Let’s export an image for each hour, which we can then join with QuickTime:

The result is here. (You probably want to have it loop when you play it.)

Mathematica doesn’t get much attention from the programming community (largely because of Wolfram’s pricing, as far as I can tell). But its power is undeniable—I spent about 5 hours writing the Google Analytics interface and generating the above data. I wrote this blog post over lunch. If anyone else is interested in using the Mathematica/Google Analytics interface, let me know in the comments, and I’ll package it up and release it somewhere (it requires modified versions of a few libraries).

Lastly, over the past year, I spent a lot of time talking to my friend Avi about the state of web analytics. He and the guys at Dabble DB decided to do something about it, and it looks like dshbrd will launch soon. From what I’ve seen so far, it looks like it’ll be win.