fluidinfo

I drove from Oakland up to the O’Reilly Foo camp last Friday. The O’Reilly offices are just outside Sebastopol, CA. I stopped at an ATM and my GPS unit got totally confused. So I took a few turns at random and wound up on Florence Avenue. I drove a couple of hundred meters and started seeing big colorful structures out the front of many houses. They were so good I stopped, got out my camera, and took a whole bunch of pictures.

I talked to a man washing his car in his driveway. He told me that “Patrick” had created all the figures, and installed them on the front lawns. I got the impression that it was all free. Soon after I found the house that was unmistakably Patrick’s and seeing a man loading things into a pickup truck I went up and asked if he was Patrick. It was him and we had a friendly talk (mainly me telling him he was amazing). He gave me a calendar of his work.

Click on the thumbnails below to see bigger versions. There’s even a FC Barcelona structure. As I found out later, lots of people (of course) have seen these sculptures. When I got to Foo, there was one (image above) outside the O’Reilly office. Google for Patrick Amiot or Florence Avenue, Sebastopol and you’ll find much more. And Patrick has his own web site.

OK, this will be a quick one. I’m trying to post occasional videos taken in my neighborhood.

Here you have a typical Catalan scene: a band playing and people dancing Sardanas. You can see this any weekend in front of the cathedral. But this was in the Born and I happened across it on the way home. It’s 60 steps from my front door (yes, I counted).

I don’t really like this music. Like living in Santa Fe and eating Southwest cuisine, I thought it was great at first but that quickly changed. I don’t enjoy the too-reedy quality of the sound and that it’s almost always identical. It’s also really long. But you may go ahead and enjoy it. Be my guest. I really like it when the first person does the initial piping and beats the tiny drum attached to his forearm near the elbow. The dancing starts a couple of minutes into the video. It’s cute.

The church in the background is Santa Maria del Mar, whose stained glass windows and gargoyles are about 4 meters from my balcony.

[This is a bit rambling / repetitive, sorry. I don’t have time to make it shorter, etc.]

Last year at FOWA I had a discussion with Paul Graham about programming and programmers in which we disagreed over the importance of knowing the fundamentals.

By this I mean the importance of knowing things down to the nuts and bolts level, to really understand what’s going on at the lower levels when you’re writing code. I used to think that sort of thing mattered a lot, but now I think it rarely does.

I well remember learning to program in AWK and being acutely aware of how resource intensive “associative arrays” (as we quaintly called them in those days) were, and knowing full well what was going on behind the scenes. I wrote a full Pascal compiler (no lex, no yacc) in the mid-80’s with Keith Rowe. If you haven’t done that, you really can’t appreciate the amount of computation that goes on when you compile a program to an executable. It’s astonishing. I did lots of assembly language programming, starting from age 15 or so, and spent years squeezing code into embedded environments, where a client might call to ask if you couldn’t come up with a way to reduce your executable code by 2 bytes so it would fit in their device.

But you know what? None of those skills really matter any more. Or they matter only very rarely.

The reason is that best practices have been worked out and incorporated into low-level libraries, and for the most part you don’t need to have any awareness at all of how those levels work. In fact it can be detrimental to you to spend years learning all those details if you could instead be learning how to build great things using the low-level libraries as black-box tools.

That’s the way the world moves in general. Successive generations get the accumulated wisdom of earlier generations packaged up for them. We used log tables, slide rules, and our heads, while our kids use calculators with hundreds of built-in functions. We learned to read analog 12-hour clocks, our kids learn to read digital clocks (so much easier!) and may not be able to read an analog clock until later. And it doesn’t matter. We buy a CD player (remember them?) or an iPod, and when it breaks you don’t even consider getting it “fixed” (remember that?). You just go out and buy another one. That’s because it’s cheaper and much faster and easier to just get a new one that has been put together by a machine than it is to have an actual human try to open the thing and figure out how to repair it. You can’t even (easily) open an iPod. And so the people who know how to do these things dwindle in number until there are none left. Like watch makers or the specialist knife sharpeners we have in Barcelona who ride around on motorcycles with their distinctive whistles, calling to people to bring down their blunt knives. And it doesn’t matter, at least from a technical point of view. Their brilliance and knowledge and hard-won experience has been encapsulated and put into machines and higher-level tools, or simply baked into society in smaller, more accurate and easier to digest forms. In computers it goes down into libraries and compilers and hardware. There’s simply no need for anyone to know how, learn how, or to bother, to do those sorts of things any more.

Note that I’m not saying it’s not nice to have your watch repaired by someone with a jeweler’s eyepiece or your knife or scissors sharpened in the street. I’m just noting the general progression by which knowledge inevitably becomes encapsulated.

In my discussion with Paul Graham, he argued that it was still important for tech founders to be great programmers at a low level. I argued that that’s not right. Sure, people like that are good to have around, but I don’t think you need to be that way and as I said I think it can even be detrimental because all that knowledge comes at a price (other knowledge, other experience).

I work with a young guy called Esteve (Hi Esteve!). He’s great at many levels, including the lower ones. He’s also a product of a new generation of programmers. They’re people who grew up only knowing object-oriented programming, only really writing in very high-level languages (not you Esteve! I mean that just in general), who think in those terms, and who instead of spending many years working with nuts and bolts spent the years working with newer high-level tools.

I think people like Esteve have a triple advantage over us dinosaurs. 1) They tend to use more powerful tools; 2) Because they use better tools, they are more comfortable and think more naturally in the terms of the higher-level abstractions their tools present them; and 3) they also have more experience putting those tools and methods to good use.

The experience gap widens at double speed, just as when a single voter changes side; the gap between the two parties increases by two votes. Even when the dinosaur modernizes itself and learns a few new tricks, you’re still way behind because the 25 year-old you’re working with (again, excluding Esteve) has never had to work at the nuts and bolts level. They think with the new paradigms and can put more general and more powerful tools directly into action. They don’t have to think about protocols or timeouts or dynamically resizing buffers or partial reads or memory management or data structures or error propogation. They simply think “Computer, fetch me the contents of that web page!” And most of the time it all just works. When it doesn’t, you can call in a gray-haired repair person or, more likely, just throw the busted tool away and buy another (or just get it free, in the case of Open Source software).

That’s real progress, and to insist that we should make the young suffer through all the stuff we had to learn in order to build all the libraries and compilers etc., that are now available to us all is just wrong. It’s wrong because it goes against the flow of history, because it’s counter-productive, and because it smacks of “I had to suffer through this stuff, walk barefoot to school in the snow, and therefore you must too.”

Some of the above will probably sound a bit abstract, but to me it’s not. I think it’s important to realize and accept. The fact that your kid can’t tie their shoelaces because they have velcro and have never owned a shoe with a lace is probably a good thing. You don’t know how to hunt your own food or start a fire, and it just doesn’t matter. The same goes for programming. The collective brilliance of generations of programmers is now built in to languages like Java, Python and Ruby, and into operating systems, graphics libraries, etc. etc., and it really doesn’t matter a damn if young people who are using those tools don’t have a clue what’s going on at the lower levels (as I said above, that’s probably a good thing). One day very few people will. The knowledge wont be lost. It’s just encapsulated into more modern environments and tools.

I’m writing all this down because I’ve been thinking about it on and off since FOWA, but also because of what I’m working on right now. I’m trying to modify 12K lines of synchronous Python code to use Twisted (an extraordinarily good set of asynchronous networking libraries written by a set of extraordinarily young and gifted programmers). The work is a bit awkward and three times I’ve not known how best to proceed in terms of design. Each time, Esteve has taken a look at the problem and quickly suggested a fairly clean way to tackle it. Desperate to cook up a way to think that he might not be that much smarter than I am, I’m forced into a corner in which I conclude that he has spent more time working with new tools (patterns, OO, a nice language like Python). So he looks at the world in a different way and naturally says “oh, you just do that”. Then I go do the routine work of making his ideas work - which is great by me, I get to learn in the best way, by doing. How nice to hire people who are better than you are.

That’s it. Encapsulation is inevitable. So you either have to embrace it or become a hand-wringing dinosaur moaning about the kids of today and how they no longer know the fundamentals. It’s not as though any of us could survive if we suddenly had to do everything from first principles (hunt, rub sticks together to make fire, etc). So relax. Enjoy it. The young are much better than we are because they grow up with better tools and they spend more time using them. It’s not enough to learn them when you’re older, even if you can do that really fast. You’ll never catch up on the experience front.

But it sure is fun to try.

[Listening just now to the Tiger Lillies song Bankrobber Blues I decided to post another little fictional story I wrote some years ago. The first in the series was Lucky Streak. As a kid I had a head full of criminal schemes and at times was convinced I’d inevitably grow up to be a cat burglar. Mathematician by day, cat burglar by night. It didn’t pan out that way.]

Manhole

I recently became a photographer. Before high-quality digital cameras became available, I had never had an interest. Sure, I liked photos as much as the next person, but it was all so messy. So imprecise. So… analog. I mean why bother? I’m into digital music in a way I could never have been with tapes or vinyl. Why do anything else? Nostalgia? Tradition? Gimme a break. Analog devices with linear access? Puhlease. I keep an eye on things, I have a feel for this stuff. When technology crosses some line, it’s time to buy in. Before that it’s just too messy.

I recently became a criminal. Historically, crime at a distance was always a challenge. Physical events took place, objects were exchanged, people met or talked on the phone. Messy, way too messy. There was always a chain, a trail: someone had met the guy, someone had given someone else a brown paper bag stuffed with cash and a pistol. With time and money, the feds were eventually going to catch you. You couldn’t find good help, couldn’t recruit and train them, had to trust them, were betrayed or sold out, either to the cops or a rival. Whatever. It made no sense, unless you were the mob and you paid everyone off. Or bumped them off. Messy.

The internet changed all that. No government wants to allow anonymous action, but the net was too big. For nearly twenty years I have watched and been amazed that they didn’t do something to stop it. Now it’s too late, far too late. Anonymous action is possible. It’s not straightforward, one must be careful, but it can be done. Tools like mixmaster (anonymous remailer), spamgourmet (similar), the anonymizer (anonymous browsing), ssh tunnels, cheap and disposable machines for rent on the network running simple forwarding servers, paypal for anonymous transfers of cash, public auction sites for posting messages to those who know what to look for, steganography, instant messaging, strong crypto, wide open windows boxes (with broadband connections and fixed IP addresses) just waiting to be commandeered, etc. The tools are all out there and in most cases they’re freely available. With some knowledge, one can carefully build an action network that no-one’s ever going to trace. The whole point of some of these tools is to preserve anonymity. No logs are kept. Subpoenas are useless when there simply are no records.

Just to warm up, I robbed a bank. Not a white-collar electronic robbery. I’m talking blue-collar physical bank robbery. It’s hardly a well-paying profession, but it’s a good lark. In fact, it pays rather badly. It’s also increasingly common and easier, as tellers and other bank staff are instructed to do exactly as told, not to make a fuss, etc. So why rob a bank? Well, why take a dumbass picture of a tree or your pet dog with your new digital camera? Why? Because you have the technology and you want to try it out. Because you can. Because you’ve gone digital and you can take a no-cost snap of whatever you damned well please.

I, physically, robbed a bank. That’s messy, true. But at some point I wanted skin in the game, wanted the whole thing to be real, to prove that it all worked. Plus, I can’t solve everything at once: I didn’t want someone else actually robbing the bank. That way, I’d either never see the money or I’d leave a trail that could be followed. So I planned carefully and I did it myself.

I recruited a dozen people online. That was actually the easy part. There’s any number of bored and broke college kids out there on the net. I made two roles: watchers and actors. I had actors do various silly extroverted tasks and each time had a watcher turn up to verify that the actor showed up. Everyone got different instructions and everyone was told enough and reported enough back to me so that a) actors knew they were being watched and b) I knew the test tasks were in fact getting done. I paid on task completion and verification. An actor would be instructed to do something oddball and very specific, like walk into a 7-eleven at a certain time in the middle of summer, buy six dozen hotdogs and a case of Bud Lite and comment to the cashier that it sure did look like it might snow. I’d schedule a watcher, and tell them to look out for the person who bought six dozen hotdogs and said how it looked like snowing. The watcher would be told to pay special attention to the other purchases and to the shoes of the purchaser. Afterwards, when I heard back from both, I’d ask the watcher what the other purchase was and tell the actor what sort of shoes they were wearing. So the actor knew that I was in the loop and I could verify that the watcher had been there to verify. Everyone had to do their part or I’d know. In this way, I built up the confidence of my players: they knew that the game was for real and that, for whatever reason, if they just did some seemingly random and simple acts from time to time, they’d be paid. I let the actors and watchers take turns at both roles. If they ever asked what the deal was I just told them I was a university psych professor experimenting on social norms.

I admit this took some time. Two months. I had a few dropouts: people who found it too weird, people who tried to cheat by not turning up while claiming they did, people who just dropped out of contact.

When I was ready, I gave all my little helpers an identical task, each at a different bank. Each was to walk into their appointed bank, get in line for a teller, hand the teller a note that said they had a gun and to hand over all the cash from the drawer. Having done that, the actor was to turn and walk straight out of the bank. No watchers, just twelve people playing actors. Everyone dressed the same. I figured several would drop out at this stage, but on the other hand I’d had them doing some pretty silly stuff, pushing the edge, and I knew at least half of them would be up for it. I also know a thing or two about banks. When the note gets to the teller, the first thing they’ll do is trigger the silent alarm. The security guard in the bank will still be clueless, but the bank manager will see it and so will the cops, who are directly connected.

The banks are dotted around midtown Manhattan. The area’s thick with banks and crazy with traffic. Between 11:00 and 11:05 on July 2, the day before the holiday weekend break, the direct-connect police alarm systems of the 5th and 6th precincts go nuts. Cops cars are scrambled and routed to what turns out to be nine simultaneous false-alarm bank robberies. And one real robbery.

I walk into my bank at ten minutes after eleven. I’m dressed in a red shirt, just like my actors who are busy triggering the alarms in a dozen other midtown banks. After a few minutes I reach the teller and hand her a note: “I have a gun. Keep your mouth shut. Pass me all your cash.” After a pause, she does this without a word. That’s the training, the banks learned that one a long time ago. A hero teller or guard taking a bullet winds up costing more than dozens of unsophisticated heists. Plus the banks are fully insured. So teller training is explicit: trigger the silent alarm and do anything the nasty bank robber tells you to.

I sweep the cash into my bag, there couldn’t have been more than $10,000 probably closer to $5,000. I hand her two more notes: “The client at my window has a gun. Keep your mouth shut. Pass all the cash from your drawer to your right.” The other is identical, for the other direction. I indicate that she should pass them the notes, which she does. Handfuls of cash are passed from one to another between five tellers, under the glass to me, where I drop them into my bag. Other customers look on, and I guess they figure something odd is happening, but of course no-one says anything.

Start to finish, I’m there putting money into my bag for less than two minutes. Up until the point the first cash hit my bag, I had a pretty good story: some random person on the internet put me up to it, got me to do all this weird shit, watch others do weird random stuff, paid me through paypal. With the money in my bag though, the game has changed. I know the silent alarm was probably tripped in the first thirty seconds. With your typical mid-morning midtown traffic chaos and the cops already dispatched to at least half a dozen banks apparently being robbed simultaneously, I figure they may not get here for twenty minutes. I’m not counting on the delay any more than I have to, but I figure my margin for error is pretty healthy.

I walk out the front door. Fifteen steps from the bank entrance is a convenient manhole. And look… the cover is already off. The hole is surrounded by four orange cones connected by official-looking Men At Work tape. Of course this is my doing: started and finished a few minutes before entering the bank. From my bag I take a hard hat, some old gloves, and an orange reflective sleeveless vest. Standard issue street crew garb, bought for a few bucks here and there at thrift stores. I pick up the steel tool used for removing manhole covers and drop it into the hole. The rest of my props I leave for the cops. I climb down into the hole and pull the cover across on top of me. Although the covers are heavy, putting one in place is actually a surprisingly easy job when done from below.

The sewers of Manhattan are pretty ugly, despite large-scale efforts to sanitize and modernize them in the 90s. But there’s no need to stay down for long in order to simply disappear. I’ve walked this route before a few times. It’s no big deal. How do I know all this? The plans are on the net: obtained and published online by some urban spelunking group. Along the way I remove the red shirt and put it in my bag. A little over six minutes later I emerge from another manhole in a lane behind a hotel. A couple of suits are standing outside having a smoke. Nothing if not brazen, I bludge a cigarette and a light from one of them. He’s only too happy to help a working man in a hard hat. Show his mates he’s on good terms with blue collar America. With a false mustache, non-prescription glasses, a vest, and the hard hat, there’s not going to be much to go on later, even if these guys do wind up talking to the cops. Besides, as is well known, witnesses are wonderfully unreliable, so the more the merrier as far as I’m concerned. You’ve got to know how these things work and use them to your advantage. I tell the suits I’m on a break from the sewer, got to go to my bank, and off I slouch. Around the first corner I remove the working gear and it goes into my bag too. By my reckoning, the cops wont have even gone down the manhole outside the bank. At most they’ll have a call in for someone to remove the cover. They’ll also know that I’m probably no longer underground.

I walk down into the subway and head home to count my cash ($47,000 and change) and to catch the headline news: Daring daylight robbery! Criminal mastermind! Nine (only nine) banks used to create a distraction while one was well and truly robbed. The police chief is interviewed, indignant: we’ll hunt these men down and put them where they belong. A sophisticated operation like this isn’t done in a vacuum.

Well, good luck chief. For me this is just the beginning, just a proof of concept really. No-one should have to rob banks for a living. That kind of crime doesn’t pay well enough, even if you never get caught. Of course, I have other plans. As the world becomes more digital, less messy, things will only get easier for those who have the balls.

I spent Sept. 2004 to Sept. 2007 as a postdoc in the Zoology Department at the University of Cambridge. We did research into influenza virus using a technique we called Antigenic Cartography.

I don’t want to go into details now or here, but I do want to say that we yesterday published a paper in Science. The paper’s title is The Global Circulation of Seasonal Influenza A (H3N2) Viruses. It digs into how flu viruses circulate around the world and what happens to them in the off season (summer) in temperate zones. This paper was years in the making. And if you consider the data collected by the worldwide influenza surveillance network, it has been decades in the making. As a result the paper has 28 authors, many of whom work at the international flu collaborating centers.

Here’s the official paper in Science. There’s a ton of other coverage, including stories in Science Daily, New Scientist, the LA Times, the Washington Post, National Geographic, Times Online, Reuters, the Associated Press, the Wall Street Journal, and Scientific American. There are plenty more links (currently) available at Google News if you search for influenza.

I’m listed as the #2 author, but it’s really my close friends the first and last authors, Colin Russell and Derek Smith, who did the heavy lifting on making this paper a reality. It’s so nice to see the thing finally published and getting such wide attention.

I was just awakened, at 1:30am, by a series of prolonged and heartbreaking female screams. Beyond any doubt, a bag snatching.

I suppose there are many reasons one might scream in the night, but people who scream in the instants after their bags have been snatched are unmistakable. The screams are desperate, often hysterical. You can tell that the victim is in motion. Quite often you’ll hear several sets of running footsteps. There are the yells of the slightly more calm, urging others to ¡cogenlo! (catch him), stop him. If the thief gets away, as is usual, the screams change to disbelieving wails of despair and broken sobbing. Then silence returns. Bag, money, passport, credit cards, mobile phone, hotel key, personal things - all gone in a flash.

When I first moved to Barcelona we lived in an area where this scene was played out several times a week, occasionally right under our balcony. We lived on the first level up, in an entresuelo. I’d never experienced anything like it in my life. I’d dash downstairs, once with my inline skates on - and those were the days when I was a heckuva skater, to join the chase.

I started compiling my stories and those of others into a page of Barcelona Street Scams. This interest eventually led me to meet Bob Arno and Bambi Vincent (blog). They’re professional thief hunters. I imagine they’re about the best in the world at what they do. As I’ve promised before, I will one day post the story of my ancestral connection to Bob.

Over the years I’ve managed to stop probably 5 robberies. Usually this means chasing until the thief gives up and drops the bag. Then you make your way slowly back to the grateful would-be victim.

But once I did rather more. Below is the email I sent afterwards. There’s no need to point out that it was probably dumb. It’s not as though you stop to think.

-----

From terry Thu Jun  7 01:26:35 +0200 2001
Message-ID: <15134.48171.932903.631448@terry.eatoni.com>
Date: Thu, 7 Jun 2001 01:26:35 +0200
To: dsmith -AT- cs.unm.edu, high -AT- hci.ucsd.edu

today i saw a bag snatching
happened about 20/30 yards in front of me
2 guys on a motorbike
the back guy leans sideways
smooth as can be
takes the handle of a bag from an old well dressed woman

they head off down the side of the church
right next to where i live

the people yell out to the people at the end of the street
looking away from me

i am in motion

sprinting.

i zoom past the robbed
going absolutely flat out
heading to the end of the street
thinking i had no chance at all

but, around the corner
not more than 5 yards
i see the guys on the motorbike
caught behind some walking other people
(there is construction there
which makes it narrower
harder to pass)

this is right on the corner of paseo del borne (our street)
and montcada

and.......................

i fucking tackled them
yeah
over the top
arms spread to get them both at once
guys to the ground
motorbike to the ground
me falling stepping over the top
grazed shin, no more

i wasn't thinking really
just knew i had to stop them
couldn't do it as good as it could have been
and as it was the bike crashed down almost
into some people beside it
who had no clue what the fuck was going on

the guys jumped up
yelled
ripped off their helmets and flung them away
one smacking hard into the wall
and sprinted off
leaving one shoe behind

i was pretty surprised
didn't occur to me that the bike was stolen too

the cops turned up in about a minute flat
there were 30 or 40 people gathered around
talking like crazy
no-one knew what had happened
the robbed people just came around the corner to find a mess
one guy saw it and one woman
the woman acted like my PR agent
telling the entire crowd
over and over that i was a hero

it was great
so funny
i smiled and bowed to them all
like an idiot
hamming it up

the robbed people thrust a 2000 ptas reward into my hands
absolutely insisted that i take it
(we ate it in pizza later)

the cops shrugged it off
called in the stolen bike

it was pretty cool
i could get into being a vigilante

i should have tried to have held one of the guys
but i thought hitting them hard sideways
and knocking their bike over would do it

but, it wasn't their bike

i was smiling afterwards
the most exercise i've had
since beating derek to the office on skates a few weeks back

My good friend Emily used to try to get me to write more. We had to find a way to turn it into a competition to get me to do it. Today I was talking to another friend on Skype and was reminded of the following story I wrote for Emily. So I’m posting it here. Maybe some other enterprising (and less scrupulous?) entrepreneur can implement it. I bet it would work.

By modern web attention span standards, this is really long.

LUCKY STREAK

They say the first million is always the hardest. In my case it was actually pretty easy; it just took me a long time to figure out how to do it. They also say “if you’re so smart, why aren’t you rich?” Although I don’t think anyone ever asked me that, I could never shake the question after I first came across it. It always rankled me somehow. If you’re that damned smart, making a mere million bucks shouldn’t even raise a sweat. Just think of all the rich idiots you’ve met over the years. What do they have that you don’t? Luck?

Anyway, that’s all the past. The computer I’m using right now has a monster 52 inch flat-screen plasma display. I’m listening to some of my favorite music on a $27,000 stereo system. I have wireless everything. There’s a high-speed multi-homed satellite connection to the internet hidden in the turret of my castle, and several high-performance motor cars in my graveled driveway. Did I say castle? Correct. I’m in a 16th century castle on a hill in Spain. It’s mine. So just how smart am I? It’s hard to say with certainty, but I’m one certifiably rich white dude. And I’m still raking it in. Over $3M a month, clear. It’s a one-man operation and I work about one hour a week.

Here’s my story.

The seed was sown when a friend told me about the following possibility. First of all, buy a mailing list of 50 million email addresses. These can be had online for under $100. Then, pick some sporting event (let’s say) and make a prediction at random about the outcome: team X will beat team Y (ignore draws for now, that’s just a detail). Send your prediction, couched in the appropriate language about mystical forces, seeing the future etc, to all 50 million people on your mailing list. After the event takes place, half of the people you mailed will have received a correct prediction from you. Chuck out the 25 million addresses of the people who got the incorrect prediction. Now pick another upcoming event. Make a prediction at random and mail it to your 25 million survivors. Do this ten times. Each time, say a little more about how omniscient you are in your email, about how you can look into the future. Whatever. After ten mailings you’ll be down to 50,000 people who will have all received ten correct predictions from you.

I probably don’t need to mention that at this point you’re going to have about 50,000 new friends. You could probably start a religion. You could probably make some real money.

That’s the setup. The problem my friend posed: what to do with these 50,000 believers? How can you turn them into money, and in such a way that you also stay out of jail? Clearly you could pull a crude one-time scam. For example, offer to send them (for a price, naturally) the device you were using to make your predictions, and then mail them a Magic 8 ball. Solutions like that are obvious, but they’re very unsatisfying. For one, you’re blowing away all you credibility in one go. For two, you’re going to have some fraction of 50,000 people complaining that they’ve been scammed (i.e., you’re going into hiding or you’re perhaps going to jail). And for three, you wont make that much money: maybe a million before you figure your costs, and that’s under the highly unrealistic assumption that you’ll be able to convince all those 50,000 people to simultaneously pay you $20 each. So while the basic idea holds water, it’s not clear where to take it and how to maximize its potential while staying on the right side of the law.

I admitted the idea was good. At first I didn’t see what to make of it, if anything. Then two months later, lying in bed, I realized I knew exactly what to do. I quit my job the next day.

I’ll tell you what I do. But first….. Why would I tell you? Because I’ve learned, though actually I knew before I started it, that it just doesn’t matter if you know. It’s not the first time I’ve spilled the beans either, though it is the most explicit and it’s the first time in writing. It just doesn’t matter. Hardly anyone would believe you if you told them the truth. Not even if you presented them with a copy of this with my signature at the bottom. Some would say “well of course” but the rest, the great majority of my users, would be more inclined to argue passionately with you and even stop speaking to you if you will not admit your error. And nope, I’m not kidding.

First of all, I use a trade secret algorithm which combines a person’s email address with the text of a question and comes up with a yes/no answer. The secret sauce, oh so special, far too secret for mere patent protection, is in fact based on a simple MD5 checksum of the concatenated email address and question text. If the first character of the MD5 checksum is 0 to 7, the answer to the question for the person with that email address is Yes. Otherwise it’s No. The system can generate millions of these predictions in minutes.

To prime the service, I just sent out the predictions to people. My initial list was 35 million (valid, non-bouncing, unique) email addresses, and I didn’t whittle it down at all when people got sent a wrong prediction. Once I’d gone through ten iterations, the plan went into serious action.

The main point is to change the focus of the original idea. In the original, the focus or pretense is that you can predict the future. Some number of people are going to believe you. But you’d better take advantage of that belief as quickly as possible because they’re going to stop believing pretty fast once you go wrong and it begins costing them. If you’re infallible and then you start screwing up regularly, you can kiss it all goodbye pretty quick smart.

The new focus? Luck. Instead of telling people you can predict the future, you tell them that everyone has lucky (and unlucky) streaks and that you’ve come up with a way to detect when people are on a roll. You tell them up front (click here to accept the Terms and Conditions of service) that you explicitly cannot predict the future. That they should not bet on your predictions. That, as everyone knows, lucky streaks always come to an end, etc. You cover your ass here. What I offer is a service that advises people when it looks like they’re on a lucky (or unlucky) streak.

After the initial priming, all 35 million emails got a further email introducing them to lucky-streak.com. The introductory email depended on the number of initial predictions the system got right for the email address in question. The people who got sent ten correct initial predictions received a very different email from the ones who had an average time of it. Those with a very low success rate got an appropriate letter. Naturally, the take up rate for the service was extremely heavily skewed towards those who got many correct predictions, with a clear bump at the end of the distribution for those with zero correct predictions.

I mail out the predictions (this is a per-user configuration option) and make them available online so that users can see their prediction history. See their luck. For those that want it, an email or SMS text alert is sent to their PDA or mobile phone to tell them just when it looks like they’re on a roll or when their lucky streak seems to be over or ending. All this for $1.99 a month.

I have a couple of servers, located in Seeland. It’s an offshore secure site with basically no laws. The server takes credit card payments and transfers are made to accounts at one of several banks. Cayman Islands, Bermuda, Switzerland, Channel Isles, etc. I have about 30 offshore bank accounts in seven principalities. My business is highly welcomed, and absolutely no questions have ever been asked. In my very occasional visits to these banks, I am treated like royalty. I tip.

Users get to define their own concept of lucky streak. On average, people choose about 4.3. That is, if 4 or 5 of my predictions go their way in a row, they consider themselves to be in a lucky phase. If the same number go against them, they consider themselves to be unlucky. When these things happen, I send out an alert. Consequently, something between 1/16th and 1/32nd of my users are feeling lucky at any one time and roughly the same fraction are being a little bit more cautious than normal. But that’s neither here nor there - the service costs the same whether you’re in a lucky phase or not. I don’t pretend to know anything for sure. No guarantees, no extra charges.

So I’m like a horoscope service or a palm reader. Everyone knows the service makes mistakes, but people love it. And I mean they _really_ love it. Even when their friends tell them it’s a fraud and that there must be some trick, even if they explain the trick, the believers refuse to listen. They come back for more. You can sign up for a year’s worth for a mere $15. Hundreds of thousands of people take that option.

Behind all this are a couple of machines with a simple database. Just linux PCs from Dell. Nothing too special. The database holds email addresses, the prediction questions, and the record of predictions for each user. When a user pulls up their history, I simply generate a line of recent predictions (red dot = wrong, green dot = right) and let them click on these to see the underlying question. It’s pretty easy to see when you’re on a roll. Lucky and unlucky streaks are undeniable. Of course, most of the time, most people are not on any kind of streak. But that’s no problem, that’s exactly what people expect. After all, you can’t be lucky all the time.

Legally, I’m pretty well covered. First of all, I’m quite hard to locate. I spent several grand on a $400/hr lawyer in New York, crafting the Terms and Conditions agreement. It’s very explicit. It states that my predictions may have no more than a 50% chance of being correct. It warns that users should use the service for entertainment only, and under no circumstance as the basis for any kind of betting or other decision making. It tells them that the concept of a lucky streak is extremely nebulous and unproven and that even if there is such a thing it may end without any warning and at any moment. Then there are half a dozen paragraphs of disclaimers and acknowledgements that basically amount to a General Release. I’ve received two legal challenges over the last two years and both have quickly been thrown out long before reaching any court as the result of filing for declarative judgments. The Terms and Conditions are pretty ironclad. The average time new users spend between receiving the page and clicking “I Agree”? About two seconds.

Naturally, people do bet on my predictions, and in general, when they receive an alert to tell them they may be on a lucky streak. Not surprisingly, that seems to be the principal reason people use the service. In fact, I typically receive a couple of hundred grateful emails a month offering me some cut of their winnings! For legal reasons, I always politely decline and instead suggest that they make a contribution to the political party of their choice. The government isn’t too likely to shut me down. In fact, I expect such contributions are pretty scarce. In any case, as far as I’ve been able to determine, I fall outside the jurisdiction and legal system of all countries except the one I reside in and residence itself is invariably a mathematical concept. Just by spending perpetual spring and early summers in various locations around the world, following the sun, it’s not at all clear that I’m a resident of any country or where taxes might be due, supposing any are due at all. The internet created some extraordinarily gaping holes that the historical national tax and legal systems are going to have a hard time filling any time soon.

So that’s it. Right now (and I do mean right now) I have exactly 1,348,216 monthly subscribers at $1.99 a month, plus 703,237 subs who paid the $15 in advance for a year of alerts. That works out at a touch over $3.5M per month. I have no staff and almost nothing to do. Sometimes, I even consider going back to my old job. The servers have multiple redundant independent power supplies, hourly tape backup, RAID arrays, etc. They’re locked down with tight firewalls and the security team at Seeland apply patches on the rare occasion any problem is found (only 3 patches have been applied in two years - one to apache and two to ssh). The credit card companies take a small cut. They love me too, and are lobbyists with considerable clout. My monthly hosting, service and traffic charge is about a thousand bucks. It took me four months to get the site together originally (this done in parallel with sending out the initial predictions). The lucky-streak.com domain costs me $15 a year.

My users love me too. They thank me. They believe. They have chat rooms, mailing lists, IRC channels, and even support groups (only in America of course). There’s a hard core that refuse to believe something so accurate could possibly be a scam. I’m giving them something they want. They evangelize. They proselytize. They swear by me. And yes, they occasionally swear at me. Articles about the site have appeared in the international media and in many national and regional papers and magazines. I refuse virtually all interviews when occasional reporters do manage to get hold of me, and I never allow photos. I’m on record in a Rolling Stone interview as saying that of course it’s all random and that people shouldn’t take it seriously. Did that put a dent in my numbers? No way. Quite the opposite: subscriptions jumped sharply following the interview. My devoted users, interviewed by the same magazine, and others, simply refuse to believe that the service could be anything but real. Of course many don’t believe it has much, if any, accuracy in its actual predictions. They all have one thing in common: they all believe in luck. They believe that luck is real, and that lucky and unlucky streaks exist. And who doesn’t believe that? I just offer a service, backed by seemingly solid evidence (my personalized prediction history for the user), to help flag the good and bad times.

I read some book about the importance of brand. There was a discussion of the Harley Davidson company as a great example of brand. It concluded something like this: if you can convince your customers to tattoo your brand name onto their bodies, you know you’ve won the branding game. I may not be there quite yet, but that’s the kind of following the service has. Reason and rationality are simply not a factor. The average person has no head for probability, in fact no concept of it. Their friends can’t convince them to stop buying lottery tickets or to cancel their subscription to lucky-streak.com. In the end though, the friends don’t figure there’s much harm. After all, it’s just a couple of bucks a month.

When I said I could start a religion, I wasn’t kidding. Not too many leave the service, and at least at the moment, for every one that leaves, two new ones join. Unlucky streaks are just as important to people as lucky ones. Lots of people have no intention of betting or doing anything crazy, but they want to know when they should be a little extra careful. It’s a vital aspect of the service that I’m there for you in the good and the bad, always trying to help, always on your side, ready to celebrate or commiserate as the case may be. The predictions can be wrong half the time (and, of course, they are), but the service doesn’t lose people as a result. The subscribers don’t care about the individual predictions. They care about when they’re in a lucky phase. When they’re on a roll. When a prediction is wrong and you send them email them telling them to watch out, that their streak may be over, they’ll reply to say thank you. I make a random prediction and send it out. The prediction is completely wrong. Nevertheless, the recipient sends me a thank-you note. It’s wacky, but that’s the way it works. I figure tattoos are not out of the question.

I know it can’t grow like this forever, but when you think about it, a couple of million people really is a very small fraction of the number of people online. I figure it’s going to settle down at around 5-10 million subscribers. At some point, probably within the next two years, I expect something will go wrong and that I’ll shut the whole thing down and walk away from it. You can’t make $35M a year in a vacuum. There are all sorts of people that are simply not going to let that happen. Still, after two full years of operation, I have over $40M spread across my various bank accounts, so I’m doing OK. It’s probably strictly 100% legal. It’s not as easy to spend, invest, or move the money as I would like, but on the other hand I don’t pay tax. I do my serious shopping by wire transfer, often to other offshore accounts. There are clearly ethical questions, but I simply ignore them. Like they say, every man has his price. They also say that lotteries are just a tax on people who are bad at math. But I’m not that callous. I have a service, and I’ve been quite up-front about what it does, what can be expected, and even how it works. People choose to send me money in exchange for the service. And after all, it’s for a very good cause.

Like all my satisfied clients, I know my run will also someday come to an end.

But for now, I’m really on a roll.

A few more rambling thoughts.

I’m wearing glasses today, for the first time in 6 years. It’s really really weird. They’re “progressive” bi- or tri-focals. It turned out one of my eyes was great for distance, one great for close ups, and both with an astigmatism. The glasses fix everything - provided you look through the right part of the glass, which also implies turning your head more than usual. I’ve tried wearing them around, and it’s very odd. Among the oddities is a huge improvement in depth perception. Everything seems so 3D, especially things at a distance. But the buttons on my Mac UI seem to be popping out of the screen too.

I’d forgotten how many pan-handlers there are in San Diego. I sometimes give money to people, and sometimes quite a lot: $150 once, $80 once, over $20 several times, and I once gave a homeless guy my bicycle to his great surprise. But 99% of the time I say no and keep walking. You can’t give money to everyone. They need it, but I need it too. Once a guy used to follow me and get a couple of dollars every day on my way to Sydney Uni, back in ‘84 or so. I eventually changed routes to avoid him.

I went to Border’s books on 6th and G. I bought Marching Toward Hell: America and Islam After Iraq by Michael Scheuer, The Conscience of a Liberal by Paul Krugman, Thomas Paine’s Rights of Man: A Biography by Christopher Hitchens, and a boxed set of 5 Jigsaw Jones stories by James Preller (to read to the kids).

I read Scheuer’s then-anonymous Imperial Hubris: Why the West is Losing the War on Terror and enjoyed it. I know I own, and think I also read Through Our Enemies’ Eyes: Osama bin Laden, Radical Islam, and the Future of America. He’s certainly no shrinking violet liberal! I learned a lot about Afghanistan reading his analysis. Russell knows ten times more than I do about almost everything, and agrees that Imperial Hubris is good. We also both like Hitchens a lot. Give me someone who thinks clearly, sincerely tries to weigh evidence, writes well, and speaks his mind any day, no matter how controversial their opinions are. The more the better, in fact. And so I enjoy Orwell, Gore Vidal and Robert Hughes.

I keep meaning to go see the Bonobos in the San Diego zoo. I like Bonobos. There’s a good TED video here, though with an annoying voice-over and somewhat manipulative-sensationalist background music. I’ve been here multiple times, and I lived here for nearly a year, but have never made it to the zoo.

I’ve spent 5 nights, soon to be 6, at the 500 West hotel in San Diego. On the left is a sign they have next to the elevator. I love the action figure going down the stairs at least 3 at a time - and it looks like many more. He or she might not have much of a kneecap left in the next frame. It reminds me of the icons drawn for events at the olympic games.

And yes, I have a new camera. A totally unnecessary purchase, but it was cheap (€100) and it’s absolutely tiny, a Canon PowerShot SD1000. The Office Depot didn’t have my first 4 choices in stock. The SD750 seems to be significantly better, and it was the same price. Whatever.

I have about ten things I could blog about today. Hopefully I wont.

I think I’m going to go out and make an impulse purchase a bit later. Can something be an impulse purchase if you blog about it first? I was in an Office Depot store today. Digital cameras are so cheap it’s ridiculous. Then throw in the value of the euro. For $129 I could pick up a 7M pixel Casio Exilim with a 3 inch screen. Why not? My old camera is a bit of a joke. Or there are nice Canon digital Elph cameras for $150. It’s hardly worth thinking about whether to buy one.

I’m getting glasses. Again. I had Lasik surgery in 2002 or so, and it’s been a wonderful 6 years. But my eyes are getting worse. I hated trying on glasses in the store today. I’ll hardly ever wear them I guess, but it’s clear (fuzzy?) to me that I’d be much better off with them.

I like women. They’re so much more interesting than men, not to mention a few other adjectives. I have a whole blog post on that one, but I’ll probably refrain.

I have two related postings: a book I’ll never write, and a Twitter  app I’ll never build. I should write them down. Twitter has so much interesting and valuable information in it. I wish their API was richer so that more things could be built. I hope they’re building some of them.

I still don’t understand why it’s considered valuable to have an API that many people build on, killing your service, if it can’t be easily monetized.

I’m booking yet another US trip. I went Silver on Delta in just 2 months this year, and am about to go Gold. This could be a Platinum year. BUT, I have to stop traveling and plant my ass on my chair in Barcelona and write more code. Have to. Must stop talking.

John Cleese’s speech at Graham Chapman’s funeral service is so moving. Can someone please do that for me?

I read more of Orwell over breakfast. Specifically, the As I Please articles from December 13 and December 20 1946.

A brief excerpt from the latter:

The whole point of Christmas is that it is a debauch—as it was probably long before the birth of Christ was arbitrarily fixed at that date. Children know this very well. From their point of view Christmas is not a day of temperate enjoyment, but of fierce pleasures which they are quite willing to pay for with a certain amount of pain. The awakening at about 4 a.m. to inspect your stockings; the quarrels over toys all through the morning, and the exciting whiffs of mincemeat and sage-and-onions escaping from the kitchen door; the battle with enormous platefuls of turkey, and the pulling of the wishbone; the darkening of the windows and the entry of the flaming plum pudding; the hurry to make sure that everyone has a piece on his plate while the brandy is still alight; the momentary panic when it is rumoured that Baby has swallowed the threepenny bit; the stupor all through the afternoon; the Christmas cake with almond icing an inch thick; the peevishness next morning and the castor oil on December 27th—it is an up-and-down business, by no means all pleasant, but well worth while for the sake of its more dramatic moments.

Teetotallers and vegetarians are always scandalized by this attitude. As they see it, the only rational objective is to avoid pain and to stay alive as long as possible. If you refrain from drinking alcohol, or eating meat, or whatever it is, you may expect to live an extra five years, while if you overeat or overdrink you will pay for it in acute physical pain on the following day.

Wow.

And apropos. I was out drinking red wine and having a great time on Monday night. Then last night I was invited to a dinner and so finally got to spend some time talking to Jeff Jonas after we’d spent the last year missing each other at various places.

Me: I bet you were always the class clown at school.
Jeff: School??? I didn’t go to school.

Point conceded.

We were in a fancy fish restaurant. So I ordered the filet mignon, done rare. It was a large unadorned cube of semi-cold meat. I thought briefly of all the warnings against eating too much red meat, and tucked right in.

I’m still making my way deliberately slowly through Orwell’s essays. I’m halfway through the final volume. There’s something like 2200 pages in total. I probably read just 5 to 10 pages at a time. That means I get to sit down to pleasures like the above hundreds and hundreds of times. Who would you rather share breakfast with?

That’s all for now. It’s somehow wrong to blog so imperfectly and so soon after reading As I Please.

Here’s a great example of terrible (for me at least) UI design.

I was just trying to change a ticket booking at Cheaptickets. Here’s the interface for selecting what you want to change (click to see the full image).

As you can see, I indicated a date/time change on my return flight. When I clicked on the continue button, I got an error message:

An error has occurred while processing this page. Please see detail below. (Message 1500)

Please select flight attributes to change.

I thought there was some problem with Firefox not sending the information that I’d checked. So I tried again. Then I tried clicking a couple of the boxes. Then I tried with Opera. Then I changed machines and tried with IE on a windows box. All of these got me the exact same error.

I looked at the page several times to see if I’d missed something - like a check box to indicate which of the flights to change. I figured Cheaptickets must have an error server side. Then I thought come on, you must be doing something wrong.

Then I figured it out. Can you?

Blogs are all about self-promotion, right? Right.

I’m talking at ETech in the first week of March in San Diego. The talk is at 2pm on Wednesday March 3, and is titled Antigenic Cartography: Visualizing Viral Evolution for Influenza Vaccine Design.

You can find out more about Antigenic Cartography here and here.

Here’s my abstract:

Mankind has been fighting influenza for thousands of years. The 1918 pandemic killed 50-100 million people. Today, influenza kills roughly half a million people each year. Because the virus evolves, it is necessary for vaccines to track its evolution closely in order to remain effective.

Antigenic Cartography is a new computational method that allows a unique visualization of viral evolution. First published in 2004, the technique is now used to aid the WHO in recommending the composition of human influenza vaccines. It is also being applied to the design of pandemic influenza vaccines and to the study of a variety of other infectious diseases.

The rise of Antigenic Cartography is a remarkable story of how recent immunological theory, mathematics, and computer science have combined with decades of virological and medical research and diligent data collection to produce an entirely new tool with immediate practical impact.

This talk will give you food for thought regarding influenza, and move on to explain what Antigenic Cartography is, how it works, and exactly how it is used to aid vaccine strain selection—all in layman’s terms, with no need for a biological or mathematical background.

In case you’re wondering, no, I didn’t go so far as to make the “I’m speaking” image above. I chose it from the conference speaker resources. Self-promotion has its limits.

I’ve been invited to speak at a TTI/Vanguard conference in Atlanta on SMART(ER) DATA on Feb 20/21.

Here’s the abstract.

Representation, Representation, Representation

In this talk I will argue for the importance of information representation. Choice of representation is critical to the design of computational systems. A good choice can simplify or even eliminate problems by reducing or obviating the need for clever algorithms. By making better choices about low-level information representation we can broadly increase the power and flexibility of higher-level applications, and also make them easier to build. In other words, we can more easily make future applications smarter if we are smarter about how we represent the data they manipulate. Despite all this, representation choice is often ignored or taken for granted.

Key trends in our experience with online data are signalled by terms such as mashups, data web, programmable web, read/write web, and collective databases and also by the increasing focus on APIs, inter-operability, transparency, standards, openness, data portability, and data ownership.

To fully realize our hopes for future applications built along these lines, and our interactions with the information they will present to us, we must rethink three important aspects of how we represent information. These are our model of information ownership and control, the distinction between data and metadata, and how we organize information. I will illustrate why these issues are so fundamental and demonstrate how we are addressing them at Fluidinfo.

I seem to be done with Twitter, at least for now. The graphic shows my monthly usage graph (courtesy of tweetstats) - click for the full-sized image.

I do find Twitter valuable, but I don’t want to spend time on it. It’s a bit like TV or video games for me - I quite enjoy those things, but there are almost always better things to do.

I’ll probably subscribe to some form of Twitter alert or digest at some point. I do find it useful to know when people are coming to Barcelona. But I don’t want to monitor Twitter. Like IM, I find it too distracting and waste too much time just going to check if anything’s new.

Etc.

I’ve got my email inbox locked down so tightly that only one email made it through today. That’s down from several hundred a day just a few weeks ago.

All the email that doesn’t make it immediately into my inbox gets filed elsewhere. I deal with it all quickly - either deleting stuff (mailing lists), saving, or replying and then saving.

I’m spending way less time looking at my inbox wondering what I didn’t reply to in a list of a few thousand emails. That’s good. I’m spending less time blogging. I haven’t been on Twitter for ages.

In the productivity corner, I somehow managed (with help) to get a 3 meter whiteboard up here and onto the wall. It’s fantastic. I spend 2+ hours every morning talking with Estéve, drawing circles, lines, trees, and random scrawly notes. Today I sat talking to him in my chair while using my laser pointer (thanks Derek!) to point to things on the whiteboard. Ah, the luxury.

My 2 older kids got Connect 4 for xmas.

I’ve liked Connect 4 for a long time. The first TCP/IP socket programming I ever did was in 1987 and it was code to let two people on the net play Connect 4 against each other, with graphics done using curses code written with Andrew Hensel. Later I wrote a machine opponent that used some form of Alpha-beta pruning and which was popular among a few CS grad students at the University of Waterloo. Amazingly, you can still find traces of my youthful code (and function names!) online. I like/d to think I am/was a pretty good player.

So you can imagine my confidence as I walked into the kid’s room and asked them who wanted to be beaten at Connect 4 by the champion of the world. My friend Russell has a take-no-prisoners attitude towards playing games with his kids. He wouldn’t dream of deliberately letting them win at anything. I let mine win very often, and find it hard to imagine how you could teach a small kid to play (say) chess if you don’t give them a chance. Anyway, tonight I decided I was going to show no mercy and whip them repeatedly at Connect 4.

I was so wrong.

At xmas just a couple of weeks ago I remember explaining the game to Sofia (8), and thinking what a vast gap existed between her understanding of the game and mine. Of course she quickly got the idea, but she had no idea at all of strategy. Lucas (6) came up during the explanation and of course had to be included, which meant an even more painstaking explanation from the champion of the world to his tabula rasa midgets.

Yesterday Ana told me that the kids, Sofia especially, were getting quite good. I smiled a knowing smile, and inside I scoffed.

Tonight I played Sofia in the first game and won fairly quickly. I told them we were going to play winner stays on, and so I then faced Lucas.

And the little bugger beat me. Fair and square he got me good, knew exactly what he was doing, and celebrated like a wild animal as he dropped the winning piece, while I sat there in shock with a huge smile on my face.

When I finally got back into the game I was up against Sofia. She proceeded to beat me too.

Amazing. Great. Funny. Alarming. How is this possible?

It reminds me of when I was about 12. My father was trying to figure out how to connect something with some cables. I took a look and told him what to do. I’ll never forget it. He knew I was right and he looked straight at me and said “how come you’re smarter than I am?” I guess I shrugged, but inside I was thinking “yep”.

Pride before a fall. Multiple falls. And you wouldn’t want it any other way, of course.

Still, they might have waited a few more years before mowing me down.

I just deactivated my Facebook account. This has nothing to do with Robert Scoble’s account being disabled earlier today, I’m just sick of Facebook. It does nothing whatsoever for me, except send messages that can and would otherwise have been sent in email. I don’t want to use a tool that encourages people to send me messages on a website that I then have to go log in to. I don’t want some website to hold my messages. I like them to be searchable with things like grep. I like to organize them my way. I like email. Apart from receiving messages in a totally unattractive way, Facebook is useless for me - just a steady stream of invitations to things I don’t want to attend from people I don’t know, plus a smattering of cream pies, flying sheep, etc. So I’m outta there. I wonder if I’ll manage to survive.

I like customizing my environment. I’ve spent lots and lots of time doing that over the decades.

Some examples: My emacs environment has about 6000 lines of elisp that I’ve written to help me edit. I have over 500 shell scripts in my bin directory (30K lines of code), and certainly hundreds of other scripts around the place to help with other specific tasks. My bash setup is about 2000 lines of shell script.

That’s about 40K lines of code all written just to help me edit and work in the shell.

As a computer user, I’m damned happy I’m a programmer. I don’t think I can imagine what it would be like to be a computer user and not a programmer.

As a non-programmer you’re at the mercy of others. When you run into a problem you don’t have a solution for, you’re either out of luck, you have to spend often huge amounts of time solving it in some contorted semi- or fully-manual way, you have to find someone else’s (likely partial) solution and maybe pay for it, or you ask or pay someone to solve your problem, or you wait for the thing you need to appear in some product, etc. And all the while you’ve got a perfectly good high-speed general-purpose machine sitting right in front of you, likely with all the programming tools you’d need already installed for you…. but you don’t know how to use it!!

How weird is that?

As a programmer when you run into a problem you don’t have a solution for, you can just write your own.

One thing that always surprises me is how little time most other programmers tend to spend customizing their environments. Given 1) that programmers probably spend a large percentage of each day in their editor, in email, and in the shell, 2) that those things can all be programmed (assuming you use emacs :-)), and 3) that programmers usually don’t like repeating themselves, doing unnecessary work or being inefficient, you’d think that programmers would all be spending vast amounts of time getting things set up just so.

FWIW, here’s a description of the email setup I’ve built up over the years.

But first some stats.

I’ve been saving all my incoming and outgoing emails since Sept 19, 1989. I don’t know why I didn’t start earlier - I wish I had. My first 7 years of emailing is lost, almost certainly forever. I’ve sent 125K emails in that time and received 425K. I’ve got all my incoming email split into files by sender, with some overlap, in 6700 files. The total disk usage of all mails is just under 4G. I have 1.1G (compressed) of saved spam. I have 1250 mail aliases in my .mailrc file.

1. I write mail in emacs, of course. Seeing as email is text, why would you use anything but your text editor to compose it? Not being able to use emacs to edit text is a show-stopper for me when it comes to using software products. Don’t try to make me use an inferior editor. Don’t ask me to edit text in my browser.
2. All my outgoing mails get dumped into a single file. I occasionally move these files when they get too big. I keep things this way as it’s then really fast to look at stuff I’ve sent, which I do frequently. I have shell commands called o, oo, ooo etc., to show me the last (second last, etc) of these files (starting at bottom) instantly.
3. I read mail in emacs (using VM). I could do that differently, but email is (usually) text and I want to copy it, paste it, edit it, reply to it, etc. I also use the emacs supercite package, smart paragraph filling, automatic alias expansion, etc. All that has been standard in emacs for at least 10 years, but it’s still not available in tons of “modern” email readers.
4. VM recognizes the 37 email addresses I’ve used over the years as indicating a mail is from me (and so doesn’t put that address in any followup line).
5. I do all my MIME decoding manually. VM knows how to handle most things, I just don’t let it do it until I want it done. That’s mainly a security thing - several years ago I predicted that PDF files would one day be used to trigger buffer overflows, as just happened. I don’t open any attachment of any form from anyone I don’t know (and don’t open them from some people I do know who like to pass along random crap from others).
6. I have VM figure out exactly where each mail should be saved, based on sending email address. So I never have to make a decision about where to save anything.
7. I have 154 virtual folders defined in VM. These let me dynamically make a mail folder based on fairly flexible rules (subject, sender, etc). They’re not folders on disk, but are composed from these on the fly. It’s a great feature of VM, highly useful. E.g., I have friends with multiple email addresses - my friend Emily has used 21 emails addresses in the last 15 years and I can see all her incoming mail in one virtual folder no matter where she sends it from. Virtual folders can be used for much more than that though.
8. I have an emacs function that detects if the person sending me mail also uses VM and, if so, lets me know if their version of VM is newer than mine. That way I don’t have to think about upgrading VM - when a friend does it, emacs tells me automatically.
9. I have VM keys set up to send messages to SpamBayes to teach it that things are spam or ham.
10. I have an emacs hook function that looks at the mail I’m currently looking at in VM and sets my email address accordingly. So if I’m reading mail from Cambridge it sets my address to be my Cambridge one, and similarly for Fluidinfo, for my jon.es domain and a couple of others. That means I pretty much never reply to an email using an address I didn’t want to use on that email. That’s all totally automatic and I never have to think about email identity, except when mailing someone for the first time.
11. VM also does a bunch of other things for me, like add attachments, encrypt and decrypt mail, etc. But that’s all fairly standard now.
12. I use a script I wrote to repeatedly use fetchmail to pull my incoming mails from half a dozen mailboxes.
13. I use grepmail to search for emails. It’s open source, so I was able to speed it up, fix some problems I ran into, and add some enhancements I wanted in versions 4.72 and 4.80.
14. In front of grepmail I run my own mail-to program which knows where I store my outgoing mail, parses command line from and to dates to figure out the relevant files to pass to grepmail, etc.
15. I use cron and some scripts to maintain a list of email addresses I’ve ever received/sent mail from/to (78500 of these) or just received from (40K of these). Cron updates these files nightly, using another program that knows how to pull things that look like emails out of mail files.
16. I have a shell script which looks in the received mail address file to find email addresses. So if I am wondering about what someone’s address from, e.g., Siemens might be, I can run emails-of siemens and see 140 Siemens email addresses. Yes, I used to send a lot of mail to Siemens.
17. I use procmail to filter my incoming mail. With procmail I do a bunch of things:
18. Procmail logs basic info on all my incoming mail to a file.
19. It looks for a special file in my home directory, and if it’s there it forwards mail to my mobile phone.
20. It also looks for mail from me with a special subject, and when, found either creates or removes the above file. This allows me to turn forwarding to my mobile phone on and off when I’m away from my machine.
21. It dumps some known spam addresses for me.
22. With procmail I run incoming mail through a script I wrote that looks at the above file of all known (received) mail addresses. This adds a header to the mail to tell me it’s from a known former sender. Those mails then get favorable treatment as they’re very likely not spam.
23. With procmail I run incoming mail through another program I wrote that looks at the From line and marks the mail as being something I want delivered immediately. If not it gets put aside for later viewing.
24. With procmail I run incoming mail through another program I wrote that looks at the overall MIME structure of the mail and flags it if it looks like image spam (hint: don’t send me a GIF image attachment).
25. Finally, I also use procmail to run incoming mail through both SpamBayes and Spam Assassin.
26. I used to use procmail to auto-reply to anything considered spam (and then auto-drop the many bounces to this). But I turned that off as it was making too many mistakes replying to forged mails from mailing lists.
27. I have a program that cron runs every night which goes through the day’s spam and summarizes the most interesting messages. It typically pulls out 15-20% of my spam into a summary mail which it sends me. The summary is sorted based on the mail address in the To line (my old mail addresses get scored very low). It also identifies common subjects (so I can kill them), and does some checks like tossing emails whose subjects are not composed of at least some recognizable words. This program is pretty severe - all these mails have already been classed as spam by one of the above programs, so this is just a safety check that I haven’t tossed anything I should keep. It generates a piece of emacs lisp for each message it pulls out so I can jump straight to the correct spam folder and message number in case I want to look at something. It also keeps a list of things to watch for that are definitely not spam. With this program in place I never go looking in my spam folders. I can also run this from the shell at any time.
28. I have a program that summarizes the mail I’ve put aside (not for immediate delivery). Cron runs that nightly and mails me the result. I can also run this from the shell at any time.
29. I have a simple program I use to grep for mail aliases in my .mailrc.
30. I have a script which lists my received email files in reverse order of last update. I can pipe the output of that program into xargs grep to quickly search all incoming mail, in new-to-old order (for speed), mentioning any term.
31. I have a script to send unrecorded mail (from the shell). That’s mail that doesn’t have my usual FCC line in it, in case I’m mailing out something large and don’t want a copy of it in my outgoing mail file.
32. I have an emacs function to visit my current outgoing mail folder with backups disabled (the folders are often large and I rarely want to edit them).
33. And I can’t resist pointing out that I wrote the Spamometer in 1997 to do probabilistic spam detection, and set up a Library of Spam (which attracted a hell of a lot of spam). This was 5 years before Paul Graham wrote his famous A Plan for Spam article about doing Bayesian filtering to detect spam. The Spam Assassin is very similar in approach and design.

It’s new year’s day. I never make new year’s resolutions. But today I’ve finally taken a step I’ve been meaning to take for a while, and it happens to be Jan 1st, so there you go.

Over the last 2 months I’ve spent lots of time running around talking to people and not producing any code (or much of anything else).

I’ve also found it increasingly hard to get anything useful done (by useful I almost always mean “code”).

I’m going to try cutting myself off a little more. But I need to be online - to read docs, to receive/send some mail, to test code, etc.

I’ve just made some changes to my email setup. Now all my mail, with about 15 exceptions, will go into a separate file that I’m only going to look at once a day (more likely I’ll write a little program to send me a summary). If you’re one of the lucky 15 your mail will still go straight into my inbox and I’ll see it pretty quickly.

I get about 550-700 mails a day. 300-500 of them are spam and are caught as spam by my filters. But that still leaves hundreds of mails a day that pop up in my mailbox all the time.

Quite a lot of those are from mailing lists and some spam that slips through. Of the rest, from actual people, hardly any need to be read or replied to straight away. So I’m going to file them out of sight and read them once in a while. If I remember.

I’m planning another blog post on my email setup. It’s heavily customized. That’s good and it’s also why I figure I’ll probably never switch to another email setup - the bar is just too high and too personalized.

I’m also planning to blog less, to twitter less, to stop reading RSS streams, etc. If I don’t I’m going to turn into one of those trendy knowledgeable tech people who generate a lot of hot air and not much else. I don’t want to be like that.