Patrick Collison » blog

Thrift doesn’t build out-of-the-box on Leopard. To get it to do so, you’ll need to do something like the following:

$ cd thrift-20070917
$ sudo cp -R lib/cpp/aclocal/ /usr/share/aclocal
$ curl collison.ie/code/thrift-time.patch | patch -p 0


Update (06/11): You’ll also need the Boost headers, as Ben points out. If you haven’t installed them before, note that you don’t have to actually build Boost—just copy the boost directory from the boost distribution to /usr/include.

Thrift is a cool project that Facebook open-sourced a few months ago. From their description:

Thrift allows you to define data types and service interfaces in a simple definition file. Taking that file as input, the compiler generates code to be used to easily build RPC clients and servers that communicate seamlessly across programming languages.

It doesn’t try to do too much (serialization of any arbitrary object), but it does more than enough to be useful: it allows reasonably elaborate data-types (like structs and dictionaries) to be transferred; is fast to develop with (no wrapper objects are required); and, with its binary protocol, has very little runtime overhead (Facebook use it for their ad-serving and logging mechanisms). The whitepaper is worth reading.

I recently wrote Smalltalk (Squeak) bindings for Thrift. They’re still a little rough and incompletely tested, but if anyone wants to take them for a spin, I’ve uploaded the patch for Thrift and the supporting Squeak code[1]. To try it out with the standard Thrift tutorial files, generate the Smalltalk code with something like:

$ thrift -r -st tutorial.thrift

Fire up one of the supplied server implementations for the tutorial, file in the generated .st files with Squeak, and then do something like:

calc := CalculatorClient binaryOnHost: 'localhost' port: 9090.
calc addNum1: 10 num2: 15


Which should result in 25 being computed by the server implementation of your choice.

Update (01/11): A more complete patch is now available.

Update (19/11): This code is now in the Thrift SVN repository.

[1] I’ve removed the links since the code is now in Thrift itself.

I’m going to be at the LispVan meet-up later today at Think! on 4512 West 10th. Come along and join in the language wars! (Or at least the beer drinking.) Full details are on Bill Clementson’s blog.

Oliver Steele writes:

“Language mavens wax rhapsodic about the power of higher-level programming — first-class functions, staged programming, AOP, MOPs, and reflection. Tool mavens are skilled at the use of integrated build and debug tools, integrated documentation, code completion, refactoring, and code comprehension. […] This means that the more you invest in language features, the more they benefit you, to the exclusion of tool features — and vice versa. And this is what creates the two camps, with two perspectives on the relative merits of language features and tool support.”

Given the existence given the existence of software like the Genera operating system, the various Smalltalk environments, Emacs, Factor, Mathematica, and the PLT Scheme environment, this argument—that good languages in some way preclude good environments—is empirically baseless.

The exact opposite seems to be true—increasing language power leads to increases in environment power. In a sense, it has to be this way: in a powerful environment, the development environment and the runtime environment are the same thing. The real power in good environments is their ability to interact with a program during any stage of its life—from compile-time to runtime. This sort of power can only be accomplished by having the environment and program coexist as different parts of the same whole, and any language lacking this kind of metacircularity has a low upper bound on the maximum power of its environments.

Giles Bowkett writes, in the context of Ruby, “debugger support is like nail-biting support, or farting-in-public support. Its absence is a feature.”

I strongly disagree. And while Giles’s stance is extreme, the perception is common. Most programmers think of the debugger as a tool they shouldn’t really need, if only they could write proper code. Even the name itself leans this way. The tool in question is one which inspects the state of a running program; to call it a “debugger” is an oddly opinionated and restrictive way of expressing its functionality.

By claiming that “debugger support is for languages that you can’t run tests against gracefully”, Giles himself falls into the trap of seeing the debugger as a mechanism for fixing code. It can certainly be used for that purpose, but it’s like seeing a web browser as a tool for reading the news.

In the simplest case, the debugger is a great way for getting a feel for the running state of a program; how it flows, and how its state varies. Using it to find bugs is a small subset of this class of uses. A more common case (for me, at least) is to use the debugger while reading code. With the debugger, you jump around the running state of the program in parallel with reading the source. You get to see the program on two axes—that on which the program is defined, and that on which the program runs.

All of the information that that can be gleaned from the debugger is of course contained within the code itself (except in disastrous cases!), but to claim that one should therefore not need the debugger is an argument which reductios very fast to absurdum—you might as well read the binary. Using the debugger makes understanding exactly how code works hugely easier and faster.

Personally, my most frequent use of the debugger is to get the program into some running state, and then start writing code in the debugger itself. (This idea will probably seem strange to some people, in the same way that not freeing your memory once did.) With the debugger, I can write code in chunks that are immediately compiled into the executing environment, and which are “tested” in the most direct manner possible—in the environment of the running code. Most of this code is part of a web application, and more often than not, it’s being written while a web request is pending. To claim that tests are an adequate replacement for this kind of use of the debugger is crazy.

To return for a moment to Giles’s claim—that debuggers are bad—I’m curious to know exactly how fully he holds this position. Would he argue that print statements and stack traces, the primordial debugging instruments that Ruby does provide, be disallowed for the same reasons that debuggers are bad? And if not, at what point do debugging tools reach the critical mass of “fart-in-public” proportions?

Lastly, and on a basic level, I think Giles is making a stronger claim than he thinks. His reasoning is apparently that “tests are a replacement for debuggers, and a superior replacement at that”. But his stated claim is that “having a debugger is a very bad thing”. Nowhere in his article does he actually argue for the stronger form. Consider two hypothetical languages, Foo and Bar. Foo is a superset of Bar, but also has feature X. Giles claims that Foo is worse than Bar. To do this, though, not only has to show that X is without any possible use, but also that it’s actively harmful.

On both of these counts, the post fails to present any kind of decent evidence.

Footnote:

The article tries to drive home the point by saying “it’s also like asking why Lisp doesn’t have design patterns.” This is (presumably) a corrupted version of Norvig’s 1998 claim that most (16 of 23) of the Gang of Four design patterns are invisible or simpler in Lisp. Given also that this was shown as part of a long presentation on design patterns in Lisp-like languages, which was created while Norvig worked at Harlequin (a major commercial user of Lisp), I don’t really know what Giles’s claim is based on.

If the statement was just referring to design patterns generally, Richard Gabriel put it well:

“Here’s another true statement about patterns: If there are no such things as patterns in the Lisp world, then there are no statements a master programmer can give as advice for junior programmers, and hence there is no difference at all between an beginning programmer and a master programmer in the Lisp world.”

It’s been a while since I’ve posted something directly programming-related, so here goes. The mailing list that resulted fom the Lightweight Language Workshops at MIT is a gold mine, and something I keep coming back to read.

A lot of the most interesting and fundamental questions in programming language design tend to be debated in echo chambers: programming communities seem naturally insular in the way that any linguistic community is. While the best hackers in any language will of course frequently debate others within their community, it’s pretty rare that interesting discussion happens across language boundaries. The LL1 list was an exception, and featured a bunch of top-notch programmers (Guy Steele, Paul Graham, Avi Bryant, Robby Findler, Shiram Krishnamurthi, Matthias Felleisen, Joe Marshall, Dan Weinreb, Olin Shivers, Scott McKay…). The ensuing traffic covers all of the interesting topics: syntax, macros, environments, typing, language design, functional programming, etc.

Some highlights:

• The Scheme mistakes thread.
• Guy Steele’s Scheme interpreter that’s sufficiently complete that it can run Kent Pitman’s “evil code”.
• An interesting discussion of the power of macros, including neat examples of their power from Steele and Noel Welsh, and rebuttal by Trevor Blackwell of most of the arguments put forward. (That nobody could come up with a powerful example of a use of macros that can’t be easily implemented in a language that lacks them is surprising, to me at least.)
• Steele: “Only later, after our discovery that actors and closures of lambda expressions were implemented by identical mechanisms did we consider extending Scheme to become a useful programming language.”
• Good discussion of file-oriented and image-oriented environments. (Scott McKay was a founder of Symbolics, and Avi Bryant is a well-known Squeak/Smalltalk hacker—Symbolics’s Genera and Squeak are still two of the most impressive programming environments ever created.)
• Questions for a language designer.
• The origin of the “HREF Considered Harmful” phrase, which (by analogy) expresses perfectly the advantage of continuation-based web frameworks. (Someone notices early on how this doesn’t immediately mix well with what-we-now-call AJAX.)
• “I don’t think I intentionally misled Yahoo (too much) on this point, but if I did it was for their own good.”—Paul Graham, on pitching obscure languages to big companies.
• Static typing: “It is merely a tradition.” “[There are] TWO different traditions.” “It is the fact that a quantity may have connections to many places in a program text that makes it valuable to document the expected invariant properties of that quantity; type is one useful kind of invariant.”
• OO flamewars.
• Steele’s candidate for the worst computer science pun in the history of the field: “It seems [functional programmers] believe in the separation of Church and state”.

I often have to resorting to resetting the system clock for debugging, testing, or avoiding stupid restrictions in software. Doing so is messy, since it screws with everything else on the system. After having to do it a lot yesterday, I wrote Time Warp, which is a tiny OS X dynamic library (plus two hackish front-end scripts) that intercepts system calls to gettimeofday (upon which all of the other time-related functions are implemented).

Sample use looks something like:

./time-warp -2M cal
Which outputs a calendar for 2 months ago, or:
./app -14d /Applications/iCal.app
which runs iCal thinking that it’s two weeks ago. You can download it here. Cue joke about using it to save time.

Conventionally, you have a few spare servers at the datacenter for testing purposes. If you’re using OS X, the combination of OpenVPN and Parallels make them pretty redundant.

OpenVPN is something of an overlooked gem—a highly-portable, userspace (uses tun/tap drivers) VPN implementation that works well with NAT, firewalls and routers (unlike IPSec (usually used with L2TP) and PPTP), and uses the OpenSSL PKI for authentication. You can do all kinds of cool stuff like run the server on port 53/udp, which means your tunneled traffic looks like ordinary DNS traffic, and will consequently cause many paid wireless hotspots to allow the packets through. Tunnelblick is a neat client for OS X.

Parallels needs no introduction. While it’s most commonly used to get Windows apps to run on OS X, everything from Linux to OpenSolaris is supported. We have Parallels VMs that duplicate the environment of each machine at the datacenter (these VMs can be shared between different machines, of course). The VMs mean that development can continue unhindered even when disconnected from the internet (something that can otherwise become a problem as the server infrastructure starts acquiring any sort of complexity).

The addition of OpenVPN means that the Parallels VMs can integrate into the server environment as ordinary citizens on the network. (Want to test a new app server configuration without touching your load balancer or database server? No problem.)

The bottom line is that we end up paying less for rack space and bandwidth, and can continue developing when on the CalTrain/plane. Kudos to James Yonan (OpenVPN’s author) and the Parallels team for some sweet software.

In other news, I organised flights back to Ireland over the weekend. Brian Collins, a friend from the UK, is coming to hack with us in Limerick for a week. One of his cool hacks got to the top of reddit some time ago.

Most large CSS designs have a number of properties that are hard-coded in different places—colours, sizes, widths, paths to locations for static files, etc. Changing these values can be tedious. A simple find-and-replace can (a little inelegantly) allow you to do global changes for colours, but changing sizes can be a lot more awkward—you’ll often have values that differ from, but depend on, your “master” values (or different properties that through coincidence have the same value). Here, textual substitution won’t cut it.

To keep sane at Auctomatic, and to make experimentation with major changes in CSS easy, we use Ruby scripts and Makefiles as a preprocessing system to generate all of our CSS.

In the simplest setup, say you have a CSS file called main.css. We rename it to have a .ucss (Unbaked CSS) extension, and gradually integrate Ruby expressions with the existing CSS:

#foo {
  font-size: 10pt;
  margin-left: #{Width + 10}px;
  background-image: url("#{StaticHost}/images/bg.png");
}
#bar {
  float: left;
  width: #{Width}px;
}

This isn’t just slightly better textual substitution. The fact that all of Ruby is available means that relations that were previously only implicit in the CSS can be made explicit in code. We’ve found that once you approach CSS like this, you need surprisingly few hard-coded values; mostly, constants can be readily derived from something else. The syntax is also unobtrusive enough that the files still work with syntax-highlighting in the CSS mode of most editors.

We then use a Ruby script, gen.rb, to evaluate the CSS files in a context that maps variables to their values:

Width = 200
StaticHost = "http://example.com"

puts eval("<<-EOF\n" + File.open(ARGV[0]).read + \
          "\nEOF")

Lastly, we have a simple Makefile:

GEN=gen.rb
RUBY=ruby
SOURCES=main.css common.css test.css

all: $(SOURCES)

%.css: %.ucss $(GEN)
	$(RUBY) $(GEN) $< > $@.out && mv $@.out $@

clean:
	rm -f $(SOURCES) $(SOURCES:=.out)

Now, you can just run make every time your unbaked CSS files are modified.

This can be easily extended to multiple CSS files (all of which may share properties), and has plenty of knock-on benefits (such as that, by factoring out the semantically meaningful data from the dependent data, CSS plays much nicer with source control—changes that are simple don’t generate huge diffs).

There are lots of additional things you can do once you have this set-up in place (e.g. having dependent colour values auto-generated)—that’ll hopefully follow in a future blog post.

(Thanks to ralph on news.yc for feedback.)