Monday, May 18, 2009

A programmer's view of the Universe, part 3: The Death of Richard Dawkins

We're getting close to the end of my blog. After today's entry, I only have three left to write. After that, I'll only blog anonymously or (more likely) not at all.

This is part three of five in my "Programmer's View of the Universe" series. I struggled for a while with how best to introduce the ideas in this installment, and ultimately opted for a short story.

This is a science fiction short story. It's different from many other sci-fi stories in that it is set in the "near future", but it has realistic schedule estimates. So unlike 1984, 2001, The Singularity is Near and all the other sci-fi stories that grossly underestimated their project durations, this one is set 1000 years in the future. I.e., right around the corner.

The story is disrespectful to pretty much everyone in the world. It will create a fantastic shit storm. This is probably a good time to point out that I don't speak for my employer. [Edit: Yay for fiction! Apparently marking something as fiction placates people. Nice to know.]

The story is 18 pages (PDF from Google Docs print preview). That's not unusual for my blog, but I went ahead and published it as a standalone document.

I'd encourage you to enjoy it, but I'm old and embittered enough to know better. You probably shouldn't even read it. Just wait for someone to summarize it for you.

Installments 4 and 5 will not be short stories; they will be regular old blog rants. In them I will further develop these ideas, and I will also attempt to clear up any gross misconceptions about the story, of which there are bound to be many.

My final blog-rant entry is the only one I care about anymore. I've been working on it so hard that my fingers have started to fail. It's been tons of fun, aside from the chronic pain. It's about a neat programming language, and Emacs, and lots of other stuff. I can't wait!

Oh yeah. Here's the link to the story. I've never done a read-only Google Docs link before, so it's probably broken. Or editable. I don't know. We'll see.

This week is going to suck. People are going to be mad. Maybe I should take a vacation and come back when the whining is finished. Can someone email me and let me know when it's all blown over?

Thursday, April 09, 2009

Have you ever legalized marijuana?

Over the holidays I read a neat book called Predictably Irrational: The Hidden Forces That Shape Our Decisions, by Dan Ariely. The book is a fascinating glimpse into several bizarre and unfortunate bugs in our mental software. These bugs cause us to behave in weird but highly predictable ways in a bunch of everyday situations.

For instance, one chapter explains why bringing an uglier version of yourself to a party is guaranteed to get you more attention than other people who are arguably better-looking than you are. I personally do this all the time, except that I'm usually the ugly one. The same principle explains a ploy used by real-estate agents to get you to buy ugly houses.

Another chapter explains the bug that causes you to be a packrat, and shows why you desperately hold on to things you own, even if you know deep down that they would rate lower than pocket lint on eBay.

In any case, well, good book. I'm going to harsh on it a teeny bit here, but it's only one tiny part towards the end, one that actually has little to do with the rest of the research presented in the book. I still highly recommend it. It's only about a 4- or 5-hour read: beyond the reach of most social-network commenters, perhaps, but you can probably handle it just fine.

So: about that harshing. Dan Ariely, who seems like a pretty fascinating guy in his own right, independent of his nifty book, says something that's kinda naïve towards the end. It doesn't seem naïve at all when you first read it. But naïve it is.

Towards the end of the book — and I apologize here, since my copy is on loan to a friend at the moment, and you can't search inside the book on Amazon.com no-thanks to the book's publisher, so I can't double check the exact details — but towards the end, Dan works himself into a minor frenzy over what seems like a neat idea about credit cards.

Credit Card Buckets

Dan's idea is simple and appealing: let's partition credit limits into "buckets". People are always maxing out their credit cards, and it leads to all sorts of financial misery, since the rates are always by definition just epsilon short of legal usury, so most people can never, ever pay down the debt.

Dan's idea is more or less as follows: you divide up your credit card available balance into "buckets", where each bucket represents a type of expense. You might, for instance, have a bucket for rent and utilities, a bucket for alimony, a bucket for chocolate and desserts, a bucket for sports and leisure activities, a bucket for dining out, a bucket for home improvement, a bucket for groceries, and a bucket for discretionary spending, or "misc".

Each bucket would have its own credit limit, and the sum of all the individual limits would be your credit limit. Let's assume you pay off your credit card entirely every month — not typical, but it simplifies the explanation. Each month, you'd have a certain amount of money to spend on each bucket, and you would not be allowed to spend more than the limit for that purchase type.

OK, so that's the way I understood the proposal. There were several pages devoted to it, as I recall, so I may have missed a few nuances here and there, but I'm hoping I've captured the essence of it.

As an aside, I read a short diatribe many years ago by a working mom whose kids were always asking her why they couldn't spend more money on entertainment purchases like video games. She was having trouble getting through to them, so one month she took her paycheck, went to a bank, and got it issued to her in 1-dollar bills. She took the bills home and piled them up on the table in front of her kids, who were amazed at the giant pile of money she had made. She then went through her budget with them, stacking the bills into piles by expense type: this many dollars for rent, this many for utilities, this many for groceries, this many for soccer uniforms, etc. At the end there were only a couple of dollars left, and the kids soberly realized that they needed to wait about 20 years and then start downloading games illegally online.

Anyway, Dan's idea was kind of similar. In order to train consumers not to overspend in a given category, leading to overall overspending, they would be able to opt-in to a program that partitioned their credit, and presumably it would lead to much wiser, more deliberate spending.

It seemed logical enough to me! It sounds similar to calorie-counting: I've found that explicitly keeping track of my calorie consumption each day does wonders for lowering my overall calorie intake. Along the same lines, I am 100% sure that if I had an explicit budget, rather than just a vague gut feel for how much I'm spending, then I would spend less money each month. We don't really have a proverb for this concept, but we do for the opposite: "out of sight, out of mind". If your budget is in plain sight, well then...

Plus the idea of having types in my credit-card accounting was all the more alluring since I'm a programmer, and I "get" the idea of types. Types are great. Dan is effectively suggesting a strongly-typed approach to credit-card spending, so the programmer in me was all for it.

Evil Banks (as if there's any other kind)

Unfortunately, this story has a sad, bitter ending. Normally I would want to add: "like all overly-strong-typing scenarios", but that would just be mean. So I'm not saying it!

Dan goes on to explain that banks are far too evil, or at any rate far too self-serving to implement his incredibly cool idea. He actually took the idea to at least one bank, meeting with their board of directors and presenting the idea. How cool is that?

Dan says the bank executives seemed to like his idea, and indicated that it might be a great way to get new customers. Credit cards are all pretty much the same (i.e., loan sharks), so they need to find ways to differentiate themselves. A nifty credit-bucketing program seemed like something marketing could run with. They all said they'd look into it and see about maybe implementing it.

And then... nothing. They never implemented the idea! Not even a little prototype of it. Nothing.

Dan theorized that profit margins, as always, are the culprit here. Even if banks could potentially sign up more customers on the promise of better spending control, there's a fundamental problem here, which is that credit cards make money for the banks based on spending. If consumers aren't spending as much, the banks won't make as much profit!

Banks make money off credit cards in at least three ways: they charge the merchant a fee at point of sale, they charge you interest on the loan, and they charge you fees such as the overdraft fee when you inadvertently overspend your limit. All those ways require you to make purchases, and the last way actually requires you to overspend your limit — exactly what Dan's idea is trying to prevent!

I suppose a truly evil bank might look at buckets as an opportunity to screw you on fees for each individual bucket. But Dan seems to think that on the whole, the fear of decreased margins — induced by the suddenly more rational consumer spending — is what is preventing banks from implementing his idea.

At the time I was reading the book, I thought, well gosh, I hate banks. In fact, I don't even use a bank — I now use an investment brokerage that has banking services on the side. You don't have to be rich to do this, and it saves you from ever having to walk into a bank again. And if you choose this route, then whenever you walk into a bank you will immediately be struck by what amazing ghettos they are: little brick buildings with little vaults holding your little dollars, little lines to talk to little tellers who provide you with little help... they're awful. They stink. I detest banks; I've found the whole notion loathsome for at least ten years before hating them became globally fashionable a few months ago.

So yeah. Dan had me. The banks are evil. That's why they aren't implementing his idea. Case closed.

The little winged nagging programmer angel on my shoulder

So just like Memento Guy in L.A. Confidential, my mind wouldn't let the case close forever. (Why can I remember Rollo Tomasi but not the actor's name? Oh wait, Guy Pearce. Him.)

For the next couple of weeks, Dan's situation replayed in my head like a bad song. I myself have given presentations to boards of executives in the past, usually presentations that had come to naught, and I felt a certain empathy with him.

As it happens, I also served time in Amazon.com's Customer Service Tools group for four years, leading the group for the latter half of my stay, and I know a thing or two about credit-card processing. Not a whole lot, but definitely a thing or two.

And I'm a programmer. Just like you. (You might not know it yet, but you are. Trust me.)

The programmer part of me started wondering: how would I implement Dan's idea? What would it take to add "bucketization" to credit cards?

And the programmer part of me started to get a sinking feeling in the pit of his... uh, its... stomach. It got the chills. And a fever. At the same time. Why? Because it, by which I mean "a part of me that wishes I could forget it", has been on software projects like that before.

The little nagging voice in my head started enumerating all the things you would need to do, like counting so many sheep. First I imagined I worked at the bank, some poor schmuck of a programmer wearing a suit, working bankers hours and golfing every day at 3pm. So, you know, pros and cons. Then I imagined my boss coming in and saying: "Steve, we gotta implement Buckets. The board just approved it. Make it happen. Yesterday."

Aw, crap. OK, what to do. First, we need a spec. So, like, I ask my boss a few preliminary questions:

  • Can customers control the buckets, or are they fixed?
  • If fixed, how many are there? What are their names?
  • Let's assume for the remaining questions that they are NOT fixed, since a predefined set of buckets would be "insanely stupid" and rejected by customers. So, how many buckets can a customer make? Min and max?
  • Can customers give the buckets names? If not, do they have to use numbers?
  • What characters can they use in the name? What's the maximum length? If we need to truncate the name in a printed statement, how do we truncate it?
  • Can a customer change their buckets mid-month?
  • Can a customer change their buckets between months? What if their balance is nonzero? Can they transfer balance between buckets?
  • Can a customer change the name of a bucket? Do names have to be unique?
  • Exactly how does a customer name a bucket? Online? Over the phone? By snail mail forms? Talking to bank teller? All of the above?
  • Same question for all other configuration settings. How? Where?
  • Do credit-card customer service reps have to know about the buckets? How much do they have to know? (hint: everything) Is there training involved? (hint: yes)
  • Do the customer-service tools have to be redesigned to take into account this bucketization?
  • What about the bank's customer self-service website?
  • What about the phone interactive voice-response tree?
  • What about the software that sends email updates to the customer?
  • What about the software that generates printed billing statements? How exactly does it represent the buckets, the individual spending limits and balances, the carry-overs from month to month, the transfers, the charge-backs, the individual per-bucket fees?
  • What about the help text on the website? What about the terms and conditions? What about the little marketing pamphlets? Should they try to explain all this shit, or just do some hand-waving?
  • Can a customer insert a new bucket into the list? How are the credit limits of the remaining buckets re-allocated? What if adding a new bucket puts one or more of the older buckets over the limit? Do we charge fees? Do we tell the customer they're about to be charged a fee right before they create the bucket? Is it, like, OK/Cancel? Do we send them a follow-up email telling them they just fucked themselves over? What exact wording do we use?
  • Can a customer delete a bucket? What if there's money in it? What if it's overdrawn? How do we represent the overdraft fee in the database? How do we show the deletion event in their bill?
  • Can a customer merge or consolidate buckets?
  • What if a customer has an emergency situation, plenty of limit in other buckets, and they really really need to charge to a couple of buckets, but they want to avoid an overdraft fee? What do they do? Are the buckets mandatory or discretionary?
  • How the hell do we even tell if they're buying "chocolate", anyway? The vendor doesn't tell us the purchase type. How do we know how to charge the right bucket? What if it's ambiguous? What if the buckets overlap? Does the customer need a point-of-sale interface for deciding which bucket to put the charge in? Can they do "separate checks" and split the charge into several buckets?
  • Where are you going? Answer me!
  • WHAT THE EVER-LOVING *FUCK* ARE YOU PEOPLE SMOKING? HUH? HAVE YOU EVEN THOUGHT ABOUT THIS PROJECT FOR MORE THAN A MILLISECOND? THE SPEC FOR THIS PROJECT WILL BE 5,000 PAGES! IT WILL TAKE THOUSANDS OF MAN-YEARS TO IMPLEMENT, AND *NOBODY* WILL UNDERSTAND HOW IT WORKS OR HOW TO USE IT, EVEN IF WE SOMEHOW MANAGE TO LAUNCH IT! IT'S FRIGGING IMPOSSIBLE! IT'S INSANE! __YOU__ ARE INSANE! I QUIT! NO, WAIT, YOU'RE FIRED! ALL OF YOU! AAAAAAAAAUUUGH!

The little nagging white-robed behaloed programmer whispering in my ear was getting pretty goddamned irritating at this point. And it asked a LOT more questions than the ones in the list above, which is merely a representative sample. My stress level began approaching what I might call "Amazon levels", and I don't even work there anymore. Thank God.

But for all its downsides (e.g. as voiced by my brother Mike, who was in the Navy on an aircraft carrier in the Persian Gulf during the 1990s Gulf War, and later worked at Amazon, and declared after four years that Amazon was _way_ more stressful), Amazon did teach me a valuable lesson, namely that YOUR IDEA IS INSANE. It doesn't even matter what it is. It's frigging insane. You are a nut case. Because anything you try to do at Amazon these days involves touching a thousand systems, all of which are processing gazillions of transactions a second, and you want to completely redo the database schema, and you don't even know the answers to these fucking questions, DO YOU?

I suppose I should think of it as a valuable experience. If nothing else, I understand Complexity in a way most people will, mercifully, never have to.

Anyway, I hope I've imparted the basic flavor of my thinking after having been totally bought into Dan's idea. Here's how I (now) envision the days just after Dan's meeting with the bank executives:

Day 1: (executives): Managers, we'd like you to look into this incredible new idea from Dan Ariely. We think it could revolutionize consumer credit-card spending in a way that makes everyone love us and sign up for our services, dramatically increasing both our profit margin and our overall customer satisfaction. And it's an incredibly simple idea!

Day 2: (managers): Programmers, project managers and marketers, we'd like you to flesh out this idea from On High, and give us some time estimates. We all know we only have a budget of about 2 months for ideas from the Board this year, so let's try to make it fit. How long will it take?

Day 3: (grunts): A billion years. We quit. Fuck you.

Day 4: (managers): Executives, we think it will take about 3 years. It's surprisingly hard. We wouldn't be able to do anything else at our current staffing levels. Should we move forward?

Day 5: (executives): Two years? Good lord! For a project that _might_ increase our profit margins and customer satisfaction, but could also cause customers to be so confused that we have to triple our Customer Service headcount? We don't think that sounds... well, reasonable. Although it would be a very interesting experiment, it's simply too expensive for us to attempt. Should we tell Dan? Well... it might be patentable, and we might be able to get around to it someday if there's a sudden glut of programmer expertise, so... maybe we'd better just sit on it for now. Who's up for golf?

In reality I'm sure it went down a little different from that. For instance, they may have had the Day 5 meeting late, and then gone to a strip club instead of a driving range.

But I'm pretty sure that aside from the mundane details, that's exactly how it went down. Because that kind of shit happened at Amazon pretty much every week I was there, for almost seven years. (And astonishingly, we actually managed to launch at least half those crazy ideas, by burning through people like little tea lights. But that's another story. Plus, no bank can execute like Amazon can. Banks just don't have the culture for it. Bless 'em.)

Legalization

So. I'm two glasses of wine into this whine. I'm going to go get a third glass, mark the calories off in my spreadsheet, and then wrap up. If you don't know what's coming by now, then you're pretty stupid, but on the plus side you're an amazingly fast reader, so I'll go through the motions anyway.

<gets third glass>

It doesn't actually matter what your stance is on the legalization of marijuana, for purposes of this little essay. You could be radically opposed to it on religious, moral, or "parental" grounds. Or you could be so radically in favor that you've been laughing hysterically and rubbing your hands together incessantly ever since you started reading this post. If you know what I mean. Or you could be somewhere in between, moderate and yet open-minded.

It doesn't matter.

This blog is about complexity, the bugbear that haunts software developers, program managers, project managers, and all other individuals associated with trying to launch new software projects and services.

Dan Ariely would have made a great VP (that is, Vice President). If you think that legalizing marijuana is a black-and-white, let's just decide it and get the frigging thing legalized once-and-for-all issue, then you too have some VP blood in you.

VPs have what my brother Mike refers to as "Shit's Easy Syndrome".

You know. As in, shit's easy. If it's easy to imagine, then it's easy to implement. Programming is just turning imagination into reality. You can churn through shit as fast as the conscious mind can envision it. Any programmer who can't keep up is an underperformer who needs to be "topgraded" to make room for incredible new college hires who can make it happen, no matter what "it" happens to be, even if they have to work 27 hours a day, which of course they can because by virtue of being new college hires, they have no social lives and no spouses or significant others, and they probably smoke a lot of crack from being in the dorms so they can stay awake for weeks at a time.

That's the kind of programmer we need at our venerable institution. And we are completely anti-slavery, for the record.

Shit's Easy syndrome is, well, pretty easy to acquire. Heck, you don't even have to be a VP. Directors sometimes get it if they stay away from the code for too long.

As for the rest of us, well, we ought to know better. YOU, as a frequenter of reddit and a programmer (wannabe or actual, it doesn't matter), you ought to know better.

Let's ask our little naggy angel: what would it take to legalize marijuana?

I don't know the answer, and I'm certainly no expert, but I've been on enough projects like this to know how to start asking the right questions.

What exactly do you mean... "legalization"?

So... one minor, teeny-weeny almost insigificant caveat before I continue: I have smoked marijuana (and inhaled it, deeply) on more occasions than I can count. And yet I'm almost undoubtedly smarter than your kid that you're so goddamned worried about. I skipped three grades (3rd, 7th and 8th), entered high school at age 11 and graduated at age 14, took A.P. courses, had stellar SAT scores, was a U.S. Navy nuclear reactor operator, went to the University of Washington and earned a Computer Science degree, worked at major corporations like Amazon.com and Google for many years as a senior staff engineer and/or senior development manager, and now I'm an internationally famous blogger.

I don't usually dwell on that, but today it's relevant. It's relevant because I've smoked a LOT of pot, and I dare you to prove that it has impaired me in any scientifically detectable way. We would debate, and you would lose; nevertheless I double-dog dare you.

So, well, sure... from that perspective, yeah, I'm in favor of legalization. The laws are stupid. Legalize it, already. For cryin' out loud. Jeez.

However, from a programmer's perspective (and keep in mind that I was also an Engineering Project Lead at Geoworks for 3 years, a Technical Program Manager at Amazon for a year, a Senior Development Manager at Amazon for about 5 years, and now I'm a plain-vanilla programmer with 3.5 years at Google, so I've done it all), the idea gives me the chills. And a fever.

Because laws are pretty much like programs. You have to specify their behavior at almost (not quite, but almost) the same level of detail, using a language that's almost as crappy as our programming languages today — English. Or whatever your native language is: it sucks too. If you don't believe me, ask a lawyer. Or try to write a technical spec in your native tongue that the programmers don't ultimately poke full of holes.

Aw, don't try. Don't even bother. Just trust me on this one. Today's natural languages are completely unsuitable for specificity, and "legalese", as much as we all love to ridicule it, is our collective best effort to permit being logical, specific, and unambiguous.

I have more respect for The Average Reddit Commenter than I have for, well, the average commenter in any other forum, period, assuming that "period" is stevey-legalese for "except for LTU, news.ycombinator and their ilk, mumble mumble."

But the Average Reddit Commenter has gone too far. Everyone these days, when debating the merits and demerits of marijuana legalization, seems to have completely overlooked the fact that it's HARD. It's a project of vast, nearly unimaginable complexity.

Think about it. What kinds of laws do we have about alcohol and tobacco? Is it just one law each, saying "it's legal" or "it's illegal?" Of course not, and you're insulted that I asked such an obviously rhetorical question, yet intrigued by my line of reasoning. Admit it! How is marijuana similar to alcohol? How is it different? How is it similar and different to tobacco?

Let's let the little angel ask a few preliminary questions, just to see where it takes us, shall we?

  • Is it legal to drink alcohol in a TV commercial? No? OK, what about marijuana, then? Can you smoke it in a commercial? Can you SHOW it? Can you talk about it? Can you show marijuana smoke at a party, without anyone actually being seen smoking it? Can you recommend its use to children under the age of 9? What exactly are the laws going to be around advertising and marijuana?
  • Do we let everyone out of prison who was incarcerated for possession and/or sale of marijuana? If not, then what do we tell them when they start rioting? If so, what do we do with them? Do we subsidize halfway houses? Do we give them their pot back? How much pot, exactly, do they need to have possessed in order to effect their judicial reversal and subsequent amnesty? A bud? An ounce? A cargo ship full?
  • Is it legal to sell, or just possess? If the latter, then how do we integrate the illegality of selling it into the advertising campaigns that tell us it's legal to own it?
  • If it's legal to sell it, WHO can sell it? Who can they sell it to? Where can they sell it? Where can they purchase it? Are we simply going to relax all the border laws, all the policies, all the local, state and federal laws and statutes that govern how we prioritize policing it? All at once? Is there a grandfather clause? On what _exact_ date, GMT, does it become legal, and what happens to pending litigation at that time?
  • Are we going to license it? Like state alcohol liquor licenses, of which there are a fixed number? What department does the licensing? How do you regulate it? Who inspects the premises looking for license violations, and how often? What, exactly, are they looking for?
  • Is it OK to smoke marijuana at home? At work? In a restaurant? In a designated Pot Bar? On the street? Can you pull out a seventeen-foot-long water bong and take a big hit in the middle of a shopping mall, and ask everyone near you to take a hit with you, since it's totally awesome skunkweed that you, like, can't get in the local vending machine? If it's not OK, then why not?
  • Can you drive when you're stoned? What's the legal blood-THC level? Is it state-regulated or federal-regulated? For that matter, what is the jurisdiction for ALL marijuana-related laws? Can states override federal rulings? Provinces? Counties? Cities? Homeowners associations?
  • What exactly is the Coast Guard supposed to do now? Can illegal drug smugglers just land and start selling on the docks? Are consumers supposed to buy their marijuana on the street? What happens to the existing supply-chain operations? How are they taxed? Who oversees it?
  • Can you smoke marijuana on airplanes? Can airplanes offer it to their customers in-flight? Is it regulated in-flight more like tobacco (don't get the smoke in other peoples' faces) or alcohol (imbibe as you will, as long as you don't "appear intoxicated"?) What about marijuana brownies? Are you allowed to eat it in areas where you're not allowed to smoke it?
  • Can an airplane captain smoke pot? A ship captain? A train conductor? The driver of a car? An attendee at a Broadway musical? A politician in a legislative session? What is the comprehensive list of occupations, positions and scenarios in which smoking pot is legal? What about eating pot? What about holding it? What about holding a pot plant? What about the seeds?
  • Speaking of the seeds, are there different laws governing distribution, sale and possession of seeds vs. plants vs. buds vs. joints? If so, why? If not, why not?
  • What laws govern the transportation of marijuana in any form into or out of countries where it is still illegal? What policies are states able to enact? Is it OK under any circumstances for a person to go to jail over the possession or use of marijuana? If so, what are those circumstances?
  • Are there any laws governing the use of marijuana by atheletes? U.S. military personnel? Government employees? Government contractors? U.S. ambassadors, in title or in spirit? What are our extradition laws? What do we do about citizens who are subject to the death penalty in countries like Singapore for the possession of sufficient quantities of what we now consider to be legal substances?
  • What about derivatives? Are the laws the same for hashish? How do we tell the difference? What if someone engineers a super-powerful plant? How do the new laws extend to a potential spectrum of new drugs similar to THC?
  • For driving and operating machinery, do we have legal definitions that are equivalent of blood-alcohol percentage, and if so, what are these definitions? How do we establish them? How do we figure out what is actually dangerous? How do we test for these levels? When they are established, do we we put up signs on all roadways? Do we update the Driver's Education materials? How do we communicate this change to the public?
  • How does legalization impact our public health education programs? Do they have to immediately retract all campaigning, advertising and distributed literature that mentions marijuana? How does legalization interact with the "Say no to drugs" programs? Do we need extra education to differentiate between a drug that is now legal (but wasn't before) and drugs that are still illegal? What's our story here? What about other drugs that are even less addictive and/or less intrusive than marijuana?
  • Monsanto is eventually going to sue the living shit out of someone for using genetically-engineered pot seeds. Can they sue individuals with a single plant in their windowsill? (answer: yes) Will Oprah step in and help that beleaguered individual? (answer: we'll see!)
I'm not an expert, and in fact I've gone to extra-special effort to avoid all possibility of being accused of having researched this subject. I know NOTHING about it.

But the questions, they're bugging me. How the hell do we implement all this? Sure, it's "legal" in Amsterdam, or so they say. I've never been there, and I suspect their laws are way too vague for the overly-litigious United States of America.

I hope it's obvious that we can't say "it's just like tobacco" (it's not) or "it's just like alcohol" (it's not), or (God help us) "it's just like doing alcohol and tobacco together, so take the intersection of their laws".

Marijuana, whether you like the idea of legalizing it or not, is a project. It requires an implementation, and the implementation is a lot like that of a software project. The US federal government is analogous to "The Company", and the states are analogous to "The Teams" that comprise the company. Some of them have free time; some do not. Some of them agree with the overall goal; some do not. And every single miniscule little detail has to be worked out, and written up, and voted on, and approved, and then specified, and implemented, and enforced.

And there will be bugs. And loopholes. And unexpected interactions. The best-laid plans will go awry.

People will die. It's a certainty. Some people are going to die as a direct consequence of legalization of marijuana. I don't like it, and you don't like it, and most of us would probably argue that it shouldn't hold up legislation or legalization indefinitely... but we have to take it into account. Because if it's YOU who dies, smashed to death on the iceberg by your skunkweed-stoned ship captain, you're going to be REALLY pissed off. I guarantee it.

Shit is NOT easy. Remember that. Shit is NOT easy. If you think it's easy, then you are being naïve. You are being a future VP. Don't be that way.

Try to think about how you would implement it. Yourself. If your boss came to you and said: "Make it happen. Yesterday." Have you ever legalized marijuana?

I haven't. But I wouldn't want to be the people who decide how to legalize it. Their asses are majorly on the line. Even more than ours were at Amazon.

My advice: give it some time. Hell, give _Obama_ some time. Whether you still like him or not, he's not a frigging King, he's a President. He can't make stuff happen overnight by waving his magical sceptre. He just can't. I don't know what you were thinking, but "overnight" is a pipe dream, and "a few months" is definitely "overnight", in presidential terms.

Moral of the story: Shit is not easy. Stuff takes time. Months. Years. Decades. It's OK! You'll still be here. Count your calories. Exercise. And you'll still be here to see it all happen.

Patience. It's a wonderful thing. I can't wait!

Thursday, March 12, 2009

Story Time

So I've got all these fancy blog posts planned. More than planned, actually — they're well underway. But it's also been a busy couple of months, so nothing's really ready yet.

To make my schedule even worse, I kind of sort of got myself a little bit addicted to the writings of this one blogger. Normally I can't frigging stand blogs, including my own. Everyone always asks me what blogs I read, and I reply: "Um... do books count?" Which of course is greeted with blank stares, since nobody seems to read books anymore these days. Such a shame.

Anyway, recently I tried another fling with reddit. I'm always getting addicted to reddit for a little while, and then coming off it cold-turkey for a month or two while I reassess whether I want to be spending my life reading pun threads and upmodded lolcat pictures and conspiracy theories.

Not to mention the comments, which can individually sometimes be quite cool, but in aggregate only make sense when you read them aloud with the voice of the Comic Book Guy from the Simpsons: "I upvoted you for the appropriate uses of 'its' and 'their' in the link title, but downvoted you because your link actually appeared on a little-known German social networking site several hours ago. I feel it is important that you understand that this is not a zero-vote of abstention, but rather a single upvote and a single downvote cancelling one another out." If you read the comments with CBG's voice, a lot of them make a whole lot more sense.

No matter how many times I quit, in the end I always wind up sneaking a peek at the reddit home page in a moment of weakness, and before long I'm hooked again, following the adventures of memes about narwhals and how is babby formed and all this other stuff I have zero context for, but occasionally it's hysterically funny. I'm guessing this is more or less what heroin is like. Some days I don't even shower.

In any case, my latest reddit flirtation ultimately led to a secondary and more severe addiction to the writings of some dude who goes by the name "davesecretary". Yeah, him. He's a fucking genius. I started reading his old re-posted stories, got hooked and basically blacked out and lost about three days where nobody knew where the hell I was. Eventually I made my way to his stories about his trip to China. At that point I became actively envious of someone for I think the second or maybe third time in my adult life. I'm just not the jealous type normally, but damn that guy can write. He has a real gift.

So now I'm flat-out jealous. I'm not going to make a secret of it, either, and pretend he's started some trend and I'm just a bigger light jumping on the bandwagon or some lame shit like that. No, he writes way better than I do, and I'm just going to copy him blatantly, like Terry Brooks rewriting Lord of the Rings line-by-line badly as Shitstorm of Shannara.

I had actually already been thinking of publishing some true stories. In fact before I stumbled on davesecretary's tales, I had written down the story below about my brother Dave taking out the garbage, and I had been planning more. But Dave Secretary's stories were like the Great Pyramids next to the mud pueblo of my own ambition. He's like a force of nature; stories just surround the guy like gnats, and he spews out the funniest, most interesting possible versions of them imaginable.

So while I work on my more technically ambitious blogs-in-progress, I thought I'd kick off a series of light reading. It's all true stories. None of it is anywhere near as cool as trying to find a box bigger than Kyle's goddamn cereal box, and nothing in them is anywhere near as funny as the line "Iodine, Dave." But at least these are my own stories, and they're kinda fun to write down once you get going on them. I probably have about a hundred stories like the ones here. I'll start with nine random stories that managed to get themselves written first, in no particular order.

I hope you like at least one or two of them. If not, well, I'll read your complaints out loud as CBG, so we'll be even.




So this one time I'm when in my early 20s, a fragile time during which I'm as fat as a walrus, I'm spending way too much time thinking about the name of the Star Wars character "Dash Rendar". He's some random Star Wars dude, except he's not from one of the actual movies. He's only in the aftermarket books or games or toys or whatever.

I have no idea where I'd heard the name "Dash Rendar", but I can tell you I absolutely loathed it. Lucas names are always a little hard to believe, but this one was just too much. I guess when you're that fat it's easy to get irritated by little things. At that point in my life I was genuinely annoyed that the famous one in the family hadn't been little Pete Rendar or Floyd Rendar or whoever else with a normal name. No. It just had to be frigging Dash.

I'm telling you all this so you can understand my frame of mind for what happened next.

I'm coming up the elevator from the underground garage because I'm too obese to take the stairs, and I'm in a surly mood on account of this stupidly-named character Dash Rendar. I'm trying to understand basic human nature here: how could anyone, even a nine year old, suspend disbelief for such an idiotic name?

I'm all alone in the elevator, so on the spur of the moment I decide to pretend to be a nine year old and see if it makes any difference. So I make this muscle pose and say in my most ominous nine-year-old voice: "I am DASH RENDAR!"

Right then the elevator doors open and fucking Dash Rendar walks in. I am not making this up. If you lined up ten thousand guys and asked a hundred people to point to the one whose name in real life was most probably Dash Rendar, they'd all point to this dude.

He of course witnessed my little announcement, pose and all, and all I could do was stand there slack-jawed, trying to think of a clever and succinct way to say: "I was just sort of trying to understand what kind of idiot would call himself Dash Rendar, and itjustsortofrequiredmetoawfuck." Dash was staring at me with a look like he'd just stepped in dog shit, except when he lifted up his foot he'd found me there instead. I was embarrassed enough to wish I would die spontaneously, but not quite enough for it to actually happen. So instead I just walked out of the elevator and never came back there again. Ever.

Afterwards I definitely established some new rules about what kinds of things I'll try in elevators.




My brother Dave used to work as a waiter at Applebees. This was back about 2 years after he'd graduated high school as a varsity football player, after which he'd settled into a life of comfort and no small quantity of pizza, and he'd blimped up to about 260 pounds. But it happened so fast that he wasn't entirely in touch with how much he actually weighed.

One day before the restaurant opened they were having the usual employee meeting in the back room, where everyone stood in a circle while the manager ran them through the day's specials and whatever else waiters need to know for the day. Dave was sweating and getting a little fidgety, so he found a chair and sat down.

The next thing that happened was a loud BANG, and everyone looked over at Dave, who was sitting on the floor with his mouth open in a big round 'O'. He had completely flattened the chair. It had once been a strong industrial solid metal chair, but now all four of its legs were sticking in different directions like a baby deer crushed under a UPS truck.

Everyone was a little stunned, and the boss felt like he needed to say something to break the awkward silence, so he said delicately: "Gee, Dave, looks like it might be time to start cutting back a bit, eh?"

Dave said later (after losing like 100 pounds) that at the time he didn't know what the FUCK the boss was talking about, and all he could think was: This chair is DEFECTIVE!




So that reminds me of this time I was watching the Seattle Seahawks play at their (at the time) brand-new stadium. We had sweet seats on the 50 yard line, in the second row back, and we were watching a pretty awesome game between the Seahawks and I think it was the Bears or the Giants. I forget which, but it was definitely one of the two. Probably the Bears, so let's go with that.

Anyway, the row of seats in front of me was occupied by some Bears fans. There were about twelve of them, and every single one looked to weigh well over four hundred pounds. It was seriously the fattest row of people I've ever seen. They were squeezed up against each other in this tangle of arms and legs because they couldn't fit in their assigned seats that had been designed for ordinary three-hundred pound fatass football fans.

Every time the Bears had a good play, they would all stand up and roar "YeaaaaAAAAH!!!!", and then they'd all sit down at the same time, on account of their arms being interlocked due to the aforementioned fatitude. They did this little act over and over for the first half of the game: leaping up, bellowing fiercely, then crashing down again as a unit.

Finally I think just after half-time the Bears executed an especially good defensive play, and they all stood up and screamed wildly as usual. But this time when they crashed their asses down, they ripped the bleacher right out of the fucking concrete.

There was this horrible thundering tearing noise like an earthquake, and the whole row of whales spilled forward onto their faces, with their twelve giant asses sticking up at the rest of us. It looked like a bomb had gone off. There were like twenty iron girders that had been ripped right out of the concrete. To say it was the most wonderful thing I'd ever witnessed would be a gross understatement. I think even now, ten years later, it might still be my favorite whale ass sticking up scene in my whole memory.

To make it all even more sublimely beautiful, our beer splashed all over them in slow-motion because the cup-holders on the backs of their seats were thrown forward with the rest of the wreckage.

Anyway, after a few seconds of general shock and hilarity, the ushers ran over to see if the herd were all OK. They had finally managed to extricate themselves and wipe most of the beer off, and they all stood up and started screaming: "This bleacher is DEFECTIVE!"




Once my family all went to this Chinese restaurant in Seattle's Chinatown. It was one of my favorites, and on this particular occasion I was there with my at-the-time girlfriend Stephanie. Stephanie just happened to be mainland Chinese, from Beijing. But unlike everyone else I've known from Beijing, who by and large seem like pretty laid back people, Stephanie was unusually sensitive to race and culture issues, and she kept a keen eye out for any perceivable slights or offenses. (Everyone else said she was just plain nuts, but I'm trying to be charitable here.)

Steph asks my brother Mike what he's ordering. He ponders the menu a while and answers: "Fly lice." Stephanie instantly blows up at him: "What do you mean FLY LICE!!!1!!?? You're insulting our culture? We have 5000 years of Chinese culture and you Caucasians are always insulting us! I can't believe you made fun of it and called it fly lice! That's very rude! That's not how we pronounce it! I can't believe I'm so offended!" and so on in that vein. Everyone else in the family, including me, suddenly becomes tremendously preoccupied with trying to figure out how to do origami with our chopstick wrappers.

Mike listens patiently, since let's just say this sort of outburst isn't wholly unfamiliar territory when Stephanie's around, and after she finally simmers down a little he says gently: "Uh... ok. Sorry."

So the (Chinese) waitress comes around and is taking our orders, and when she gets to Mike, Stephanie starts hyperventilating a little in case she'll have sudden need of some extra screaming oxygen. Mike says in his blandest and most American possible voice: "I'd like to have the fried rice, please."

The waitress nods happily and says "FLY LICE" really loud and writes it down on her pad. Mike, always an excellent poker player, manages to keep his face pretty straight, but I can tell his eyes are sparkling just a little. Meanwhile Stephanie's eyes have grown to the size of large saucers, and she hisses loudly: "She said FLY LICE!!!!!! Hee hee hee hee hee hee HEE HEE HEE HEE HEE!" Mike just shrugs, like, "hey man, I just want to not get yelled at anymore."

We don't call it fly lice now, though, since you just never know.




When I was growing up the holidays were always crazy because I had over 20 aunts and uncles, and seventy or eighty first cousins, and of course we had to do a massive holiday family party at my grandparents' teeny house every year. Every family would bring a few friends, so there would be well in excess of 100 people running around in pure holiday chaos for a day.

We used to do this secret-santa gift exchange where everyone would put their name in a hat a few weeks in advance, and then we'd all draw names, and that was the person you had to get a gift for. It wasn't split into adults and kids or anything like that.

With an extended family that big, it's kinda hard to keep track of everyone, especially the new additions. It was also kinda hard to know what people actually wanted, and some of my uncles weren't too big on preparation. Uncle Harold was pretty much the worst at it. We loved him to pieces, but he just couldn't get the hang of the holiday party or the secret santa exchange.

To give you the basic Uncle Harold flavor, one year he didn't show up because on the way to the party he decided to stop in a bar down the street, and he stayed there drinking bloody marys until midnight. Another year he was tasked with bringing the turkey, and he showed up at dinnertime with a frozen-solid turkey. And one year his secret santa target was Aunt Celie, who was infamously religious and pretty much nothing else, just religious, so Harold, totally at a loss, got her a picture of Jesus and taped fifty bucks to the back of it. All the adults were pissed off and all the kids thought it was pretty much the funniest goddamn thing in history. Either way, Uncle Harold just couldn't win at Christmas.

But this one year he really outdid himself. I think I was about 11 or 12 years old, just old enough that I was no longer running around screaming like the next generation of cousins. So I got a chance to sit back and watch the party unfold. I'm sitting there watching Uncle Harold give his secret santa gift to one of my cousins, one of Aunt Diana's kids. For some reason Harold looks embarrassed.

My dad had 11 brothers and sisters, all married with like 6 or 8 kids each, and there was no way any of us could really keep track. Usually we'd get in touch with an immediate family member and try to get some direction, but Harold, as usual, just winged it. I don't remember which cousin it was, but last time Harold had seen him, he was a baby, so Harold, thinking it through with his usual clarity, got him a jar of Gerber baby food.

Unfortunately it turns out it's been like six years since Harold had last seen him. So my cousin opens up his present and I hear him saying to my aunt Diana: "Mom. Uncle Harold got me baby food." Diana was busy with two other crying kids, and she's not feeling particularly merciful, so she says: "Well go tell him thank you."

So my cousin walks over to the tree where Harold is sitting, and he says: "Thank you for the baby food, Uncle Harold."




This story is more grim than funny, but it's a story. With sequels, no less!

In college I lived in a big four-bedroom apartment with 3 roommates for about a year. It wasn't air-conditioned and during summer it was hotter than hell, and we all kept our windows open most of the time.

I was just starting to drift off to sleep one night, laying on my back, when I heard this loud clicking noise. It sounded like snapping a ball-point pen out of its plastic cap, and the clicks were coming at roughly one per second in irregular bursts.

The clicking noise was coming from between my eyes. Not, like, a foot away, or an inch away, but from directly between my eyes, near the bridge of my nose.

I remember thinking: "That's odd. Whatever could be making that loud clicking noise coming from between my eyes? I've never heard anything like that before."

Part of me was falling asleep, and was thinking "mufbmlflfsleeep go sleep go sleep mblfjust don't worry abouuuuuut it", or something like that.

But there was this other little part of me, a tiny voice, that was busy thinking out loud to its tiny self: "It could be a spider."

My falling-asleep part woke up a little at that point, and pondered it for a few dozen clicks. Then it replied: "No. No, it's not a spider, because that would just be TOO horrible to contemplate. So that's not what it is."

And the tiny voice was like, "Well you come up with a non-spider explanation then. You can't just ignore loud clicking noises coming from BETWEEN YOUR EYES, dude."

Falling-asleep voice: "Spiders don't make noise. So there."

Tiny voice: "Small ones don't. Medium-sized ones don't. But we don't really know what BIG spiders do, now, do we? I bet a really big one could make a noise just like that."

At this point, opinion-wise, I'd say I was coming down solidly on the side of falling-asleep voice, but neither the clicking nor the tiny voice would shut up, so I rolled over onto my side so I could think about it from a different angle.

As I rolled over I felt something huge jump off my face. It landed on my arm and started dragging itself along with what felt like a dozen scrabbly legs, and of course I was like WAAAAAAAAAAAH!!!! and I jumped out of bed faster than any human being has ever done before, and turned on the light.

Falling-asleep voice and tiny voice were both, like, "Oh, shit!" Sitting there next to my pillow was the biggest fucking wolf spider I've ever seen in my life. It had a good four inch legspan, and it was very specifically glaring at me. There was no mistaking its expression. It was really pissed off. At me.

So I do the only thing I know how to do in Large Spider situations, which is to scream for help. "AAAH!!! A FUCKING SPIDER CRAWLED ON MY FACE!" My roommates' doors all started opening and they came out in their bath robes, all keenly interested to see the spider that had crawled on my face.

Jacob is first to arrive. He takes one look at this massive glaring spider and says: "Holy shit dude, that thing's as big as a horse!" and leaves as fast as he came in.

Dave shows up next. "Holy crap, that thing crawled on your face? You must be psychologically damaged!" Dave opts to stand and goggle at the giant spider, from a safe distance.

Then Eric comes in and says: "Oh, it's just a hobo spider. I'll take care of it." He walks into the bathroom and grabs ONE square of toilet paper. Not a whole handful, no, just one teeny slice, like he's going to wipe the spider's butt with it or something, and he goes and grabs the spider with it. We can clearly see the spider's legs waving around outside the toilet paper, and Eric says: "I'll just put it outside your window."

I of course was having none of that crap. "How the hell do you think it got on my face in the first place? Flush it! Now now now! Flush!" Eric, clearly disappointed, flushed the thing, and I'm certain that it now lives in the Seattle public sewer system, feeding on rats and stray dogs and plotting its revenge on me.

My actions that day clearly angered the Spider God, because over the next few years I had several more spider-on-face incidents, at least one of which was way more dramatic than this one. It's a bit traumatic to tell both stories at once, though, so I'll wait a bit on that one.




When my brother Dave was around 14, our family lived in a house in Southern California. It was kinda rainy at the time, which is sort of unusual for where we lived. On this particular California winter day it was Dave's turn to take out the trash.

Our city-issue trash bin was out in the carport, this sort of concrete alley next to the house where our big Dodge van was parked. The trash bin was on the other side of the van, next to a six-foot brick wall separating us from the neighbor.

Dave grabs the trash bag from the kitchen and heads outside. He walks over to the trash bin, opens the lid and sees that it's completely full. There's a plastic garbage bag right at the top, and the lid just barely closes over it. Dang.

It's starting to rain pretty hard, and Dave just wants to get his bag into the can and get back inside, so he figures he'll do the old Human Trash Compactor trick. He puts down his bag, climbs up on the brick wall, aims for the bag on top of the bin, and jumps.

Usually this works pretty well, right? Your plummeting body weight smushes the bag down just far enough to make room for another bag, and you hop out unharmed.

Unfortunately for Dave, it turns out that this time the bin was completely empty except for the bag on top. That one bag was full of trash, but it was so bulky that it hadn't fallen down into the bin.

So Dave jumps off the wall, plunges all the way into the bin, and the lid slams shut on top of him. His momentum makes the bin tip over, and it wedges itself solidly between the van and the wall, trapping Dave inside with the garbage.

Dave's no more claustrophobic than the next person, but the whole thing caught him a little off guard. In his mind's eye he'd been envisioning some sort of heroic plunge that would compress the garbage about a foot or so, followed by a dashing bounce onto his feet on the driveway. Instead, in just under one second he'd materialized sideways in a dark stinky fallen-over garbage can getting pounded by rain, and the lid wouldn't open.

So he did the first thing that came naturally to him, which was to panic and kick and scream and thrash and flail and try to claw his way out. Pretty much what you or I would have done.

However, he was really stuck in there pretty good, and nobody could hear him because of the rain, so it took about ten minutes of violent side-to-side heaving before he finally rolled the can out from between the van and the wall.

So we're all sitting in our nice, comfy living room, watching TV. Dave has gone to take the garbage out. Ten or fifteen minutes later, after we've all totally forgotten about him, the front door bangs open and Dave barges in, wild-eyed and soaking wet and yelling at us at the top of his lungs. He's completely covered in garbage: coffee grounds, smears of leftover food, pieces of dirty paper, part of a banana peel on his shoe, brown slime all over his head and arms and legs and clothes, and he's screaming: "DIDN'T ANY OF YOU HEAR ME? I WAS STUCK IN THE GARBAGE CAN!"




Back when I started working at Amazon.com in 1998, the company was in this little building in downtown Seattle in kind of a bad part of town. I mean it wasn't terrible, but there were definitely some issues. There was a needle exchange across the street, which was cool and all, but a fair number of the drug users did their thing in the alley behind the building we were in. So you didn't really want to walk out the back door, or you'd run the risk of stepping on some dude with a syringe dangling out of his arm.

The nearest place to eat was the Scaryaki joint across the street, next to the needle exchange. It was this teriyaki place that we all called the Scaryaki, since even though the food was pretty good almost nobody ever went there because it was scary. Usually we'd walk a couple blocks past it, which took us to Pike Place Market, which is also scary in its own way but is much less overtly threatening. Plus there's more culinary variety.

One day my friend Jacob and I left the 'zon premises to eat at Matt's in the Market, which was this really awesome hole-in-the-wall joint that was Zagat rated and nobody knew about it and it was delicious.

On the way back to work, we're crossing the street in front of the Scaryaki, and I can't help but notice Jacob's doing an unusual amount of looking at my ass and my legs. We're both straight, and as a consequence we usually try to avoid staring at each other's asses as a general rule, so I'm a little annoyed. But he's definitely checking me out with waaaay too much interest.

I finally give him the Glare, and he says: "Where'd you get those pants from?"

I shrug. "I don't know."

Jacob is now really keenly eying my pants, which are these dark blue cotton slacks I've been wearing to work on and off for at least a few months now. He starts being more demanding. "No, really, where'd you get those pants from?"

I'm like, "Dude, I don't know! Stop looking at me!"

Jacob suddenly announces, in a really loud voice that everyone within a block or so can hear: "You're wearing my pants!" A couple of people in the vicinity, including some druggies and some coworkers, perk up with some interest as the argument unfolds.

Me: "No I'm not! What are you talking about?" <walking faster>

Jacob: <running to catch up, pointing at my ass> "Those are my pants! Where'd you get them from? Those are mine!"

Me: "I don't know where I got them! I just had them! They're mine, OK? Leave me alone!" I'm now running full bore for the doors, since I want to get back to work and out of this sudden surreal nightmare.

And all the while I'm thinking to myself: "Where DID I get these pants? I never buy blue slacks. I found them in my closet one day, so they MUST be my pants. They fit pretty well, even though they're a little long. So they're mine! I must just not remember buying them!"

We burst into the lobby, where everyone in the frigging company is on their way back from lunch, and Jacob is running after me yelling: "I recognize those pants you're wearing! Those are MY PANTS!" There was chaos for a while. It was pretty messed up.

The funny thing is, it turned out they actually were his pants. He and our mutual friend Jeff and I had all gone out to El Gaucho earlier that year. It's the awesomeest steakhouse in Seattle, arguably in the whole country, and we all dressed up after work one day to eat there. I drove, and they threw their work clothes into the trunk of my car, which was almost completely filled with random crap. When they got their clothes out later, Jacob didn't see his pants or something, and I must have thrown them in my laundry when I finally cleaned my car, months later. One day I started wearing the pants without really thinking too deeply about their origins, and the rest was history.

He probably could have picked a better way to tell me, though.




I remember when I was 23 years old, my dad decided to have one of those Dad to Son talks. He'd clearly thought seriously about it, because he sat me down and gave me one of those This Is Important looks. He said: "You know how sometimes they lose your file?"

"Uh... what?"

"You know, like when you call up to make a dentist appointment, and then when you get there they have no record of the conversation? Or you set up an account with the cable company, or whatever, and they can't find it later?"

"Er... yeah, sure. That happens to everyone sometimes."

My dad is like, "Nope. It happens to us way more often. And when it happens, tell them to look under 'W'."

"What? 'W'?"

"Yep. Ask them to look under Wegge instead of Yegge. 9 times out of 10 they'll find it."

"WTF?"

"Yep. It's weird. But when you spell our name, Y-e-g-g-e, a lot of people write or type 'W' instead of 'Y'."

I'm thinking, "You waited until I'm 23 years old to tell me this?" But I'm also thinking: "Damn, are people really that stupid? And if so, how the hell didn't I notice this myself?"

"Uh, thanks Dad. I'll keep an eye out!"

So I watch. And listen. And sure as shit, he's absolutely right. The percentage is pretty high, like maybe 10% to 20% of the time. Someone (in person or on the phone) will ask me to spell my name, and I'll say 'Y', and they'll enter 'W'. A lot of the time I'll actually be in a position to watch them as they do it -- I'll be looking over the rental-car counter or whatever, and when you're looking at a keyboard upside-down from the side, you can see the 'W' as they type it.

I have no idea how many years it took my dad to figure this out, but he's a pretty perceptive dude, and he was 43 when he told me. So we're talking about half a lifetime of watching people fuck up, and eventually realizing there's a pattern to it. Bravo.

Nowadays I'll say "Yellow Echo Golf Golf Echo" if I'm on the phone, since it slows them down just enough to think about what they're typing. In person, when I can see what they're typing, I still say "Y", because I'm always dying to know if they're one of Those People. If they get it wrong, I'm like, "No, 'Y', not 'W'", and they always say: "Gosh, I have no idea why I did that!"

Me neither. But I think it's because when you pronounce the letter Y, it starts with a W-sound, as in "WHY"? Sometimes it even seems as if the slower and more deliberately I say 'Y', the more likely they are to get it wrong, because my lips started off with this big 'W' sound.

Anyway, the best one ever was at a Jiffy Lube. The Lube Supervisor Dude was asking me for my personal information and writing it all down on a form on his clipboard. He apparently felt he was better qualified to write my personal data on the form than I was, and to be fair, he had pretty good handwriting.

He asked me to spell my name, and I said: "Y". He wrote "W". So far, so good. I really didn't want these fuckers to have my personal information just because they gave me an oil change, anyway.

I said "e", and he wrote "i". Wow, this was new.

I said "g, g" and he wrote "jj". Cool!

I started to get kind of excited to see if he'd get every letter wrong. I said very carefully: "e", and he wrote "i", completing my last name as "Wijji". It was Jiffily Lubriciously Awesome. I told him "THAT'S EXACTLY RIGHT!", like he'd just won on Who Wants to be a Millionaire, and he gave me this big shit-eating grin with several missing teeth. It was nice to make him feel happy.

But check it out: I told this story to my friend Gayle, whose last name has a bunch of doubled letters in it, including ending in "nn". She said that she'd found empirically that in order to get people to write "nn" at the end of her name, she had to say as many as four or five N's. She's gotten into the habit of spelling it with "enn enn enn enn enn" at the end so they'll write two of them.

Go figure.




OK, last story for today.

I've been a snowboarder for almost 20 years, but waaaay back in the old days I actually used to ski. I learned in my late teens, and I spent 2 or 3 seasons skiing, mostly using Navy rental equipment. I have all sorts of Navy stories to tell, but they'll have to wait for another day. Suffice it to say the Navy provided the rental equipment for this story.

This was back when I was at the US Navy S5G Nuclear Reactor Prototype training plant in the middle of Idaho. "S" was for "Submarine", 5 was the reactor model number, and "G" was for General Electric, who had made the reactor. Hmmm, maybe I should tell some of my reactor stories someday... they're definitely interesting. Anyway, the reactor was at a Department of Energy site in the desert in the middle of Idaho, with a 60-mile bus ride from the nearest city, Idaho Falls. I'll leave it to your imagination as to why they located the experimental and training reactor facilities out in the middle of a desert, but your first three guesses are probably all more or less correct.

Anyway, the downside to the program was that we were in Idaho, where the fun thing for residents to do is follow potato trucks in their cars, trying to hit dislodged potatoes. That's pretty much the pinnacle of entertainment in that fugging dustbin wasteland. The upside was that there were some really nice ski resorts nearby, so during winter we got to do a lot of skiing.

This one time we were either at Jackson Hole or Grand Targhee. Both of them are in Wyoming, but only about 2 hours from town, so we usually went to one of them. The way the Navy Nuke program works during the prototype training phase is that you work 7 days on, 1 day off, 7 days on, 2 days off, 7 days on, 4 days off. This is actually a pretty doable schedule, and the once-a-month 4-day weekend was always something you planned in advance, since that's a lot of days for screwing around.

A lot of my friends were skiers, and snowboarding was still pretty new back then. There was one dude, I think his name was Lundberg (everyone goes by last names in the Navy), who was learning to snowboard, and he pretty much spent all his time falling on his ass. So I decided to ski. The Navy pretty much takes care of all your needs, including ski rentals, so I went to the local Navy ski rental place and got hooked up with some rental skis, boots and poles.

Unfortunately this one time, which was probably no more than my seventh or eighth trip ever, the rental guy adjusted my bindings too loose. Bindings are the things on your skis that hold your boots in place, and you don't want them to be too tight or you could break your legs in a bad crash. You want them to be tight enough to hold during an aggressive turn, but loose enough to pop out in a crash. And mine were way too loose.

We ski for a few hours, and I've warmed up a little, so I want to work on my turns instead of snowplowing down steep stuff like a sissy. So I follow my friends to a black diamond. When I was skiing I would actually follow my friends pretty often, which was completely idiotic because they've been skiing since they were in the womb, and they look like frigging Olympians, and they're always dragging me off to slopes I have no business being on. But for some reason I still did it.

This slope is a pretty steep black -- nothing for me today on my snowboard, but back then it was way out of my league. But it doesn't hurt that much to fall, so I went ahead and tried a real turn. Of course as luck would have it my binding popped out and I went flying straight down the slope on my face. It was so steep that I actually started picking up speed, and I wasn't experienced enough to know how to stop myself, so I just started sliding faster and faster downhill, face-first on my stomach.

I remember being vaguely aware of people stopping and looking at me and pointing as I accelerated by them, but after a hundred feet or so I had picked up enough speed that I couldn't really do anything except this emit gurgling screaming noise. The snow was bunching up in my face and going in my eyes and up my nose and down my throat, not to mention down my shirt and pants, but all this plowing wasn't really slowing me down, so I wound up sliding about three hundred yards on my face. I didn't have any really good ski gear at the time, though. I was actually wearing baggy jeans, if you can believe it. So when I near the bottom stopped I was caked in snow and ice.

Someone was nice enough to bring me my ski from way up the slope, and I hobbled into the lodge to grab a beer. I was sitting there and everyone was laughing and having a great time, but I was FREEZING. I went and sat by the big fire, but it wasn't helping. I started shivering badly. Finally I stood up and lifted up my shirt and about ten pounds of snow fell out, which sort of explained it.

My brilliant friends told me that I should go back up for another run before we left for the day. They said you stay warm on the slopes, which is more or less correct if you're actually skiing and not being a human snowplow. I really wanted to warm up so I went back up for one more run.

At the top I took a cat-track between slopes -- just a road, basically. The cat-track was running along the side of a steep roped-off drop-off with a bunch of trees. The cat track headed for the drop-off, then made a sharp turn to the left. When I got to the turn, I turned too hard, and my right binding snapped open and my ski shot out from under me, and of course I wiped out.

I'm laying there on my back, thinking "man, I'm just not sure about this whole skiing thing", and an older guy skis up next to me and asks me if I'm OK. I'm still laying on my back, with my right leg bent awkwardly under me, trying to catch my wind. I tell him I lost my ski, and that I think it went over the edge. I ask him if he can see it.

He skis over to the ledge. I lay there, sun shining on my face, snow on my back, freezing my balls off, just kind of trying to enjoy the moment as best I can while he assesses the situation. I look over and he's at the ledge, scanning the mountain. A bird flies by. Some time elapses -- probably no more than 30 seconds, but it seemed like eternity.

I finally ask the guy, "Hey, can you see my ski?" He looks at me, then back down the mountain. Then he takes a deep breath as he decides how best to tell me. Finally he speaks.

"Yep. She's still on the run."

Saturday, December 27, 2008

A programmer's view of the Universe, part 2: Mario Kart

This is the second installment of a little series of discussions. They're not much more than that, just discussions. And I hope I'm inviting discussion rather than quenching it. But I'll be honest — the goal of this series is to pound a stake through the heart of a certain way of thinking about the world that has become quite popular. If my series fails in that regard, I hope it may still provide some entertainment value.

Part one, The fish, was about a twisty line and a fish's examination of it. Today we move to a twisty plane.

Embedded Systems

There are many kinds of computer programs, and many ways to categorize them. One of the broadest and most interesting program categories is embedded systems. These systems are the centerpiece of today's little chat.

Embedded systems are a bit tricky to define because they come in so many shapes and sizes. Loosely speaking, an embedded system is a little world of its own: an ecosystem with its own rules and its own behavior. So an embedded system need not even be a computer program: a fish tank is also a kind of embedded system.

We call them embedded systems because they exist within the context of a host system. The host system provides the machinery that allows the embedded system to exist, and to do whatever it is that the embedded system likes to do.

For fish tanks, the host system is the tank itself, which you may purchase from a pet store. A tank has walls for holding the water in, filters and pumps for keeping the water clean, lights for keeping the fish and plants alive a little longer, and access holes for putting your hand through to clean the tank. There's not much to it, really. The embedded system is everything inside: the water, the plants, the rocks, the fish, and the little treasure chest with bubbles coming out.

For computer games, another popular kind of embedded system, the host system is the computer that runs the game: a PC, a game console, a phone, anything that can make a game exist for a while so that you may play it.

Programmers have been building embedded systems for many decades: a whole lifetime now. It is a well-studied, well-understood, well-engineered subject. Gamers understand many of the core principles intuitively; programmers, even more so. But in order to apply all that knowledge outside the domain of embedded systems, we will need some new names for the core ideas.

The most important name is the One-Way Wall. I do not have a better name for it. It is the most important concept in embedded systems. In lieu of a good name, I will explain it to you, and then the name will stand for the thing you now understand. It's the best I could come up with.

But first let's dive into an embedded system and see what this wall looks like from the inside.

Mario Kart

I will assume you've played Mario Kart, or you've at least watched someone else play it. Mario Kart is the ultimate racing game in terms of sacrificing realism for fun. It bears so little resemblance to reality that it's a wonder it tickles our senses at all. The Karts skid around with impossible coefficients of friction, righting themselves from any wrong position, and generally making a mockery of the last four centuries of advances in physics.

It's really fun.

Mario Kart, like all games, has a boundary around the edge of the playing area. In Mario Kart you bump into it more often than in most other games, which is part of the reason I chose it to be our Eponymous Hero. If you want to win a race, you will need to become quite good at accelerating around corners, which means you will spend a fair amount of time bumping up against an invisible wall.

You know the wall I'm talking about, right? It's invisible, so you can see right past it to the terrain beyond. But the wall is there, and you are not permitted to venture beyond it.

In slower-paced games, when you arrive at the invisible map boundary, you will sometimes be told by the game: "You can't go that way. Turn back!" And since that is your only option, you comply. These invisible boundaries are non-negotiable.

In other games, you may stop on contact with the boundary, or perhaps bounce off. But the boundary is always there.

Imagine Mario and Luigi chatting about the you-can't-go-that-way wall. Their conversation might go something like this:

Mario: "Luigi, my brother!"

Luigi: "Maaaarioooooo!"

Mario: "Yes, Luigi. I am a here. Tell me my brother, why is it that every a time I a spin around the cactus in the third a bend of the Desert a Track, I get a stuck and have to start accelerating from nooooothing?"

Luigi: "Whaaaaa?"

Mario: "Brother, the Desert a Track! It's Number a Three! You know the big a bend, where you have to slow down? I am always a forgetting to slow, and I just a stop. Just a like that!"

Luigi: "I don't know, Brother. That Bowser, he is always a squishing me right before I hit that turn, and I am a flat like a pan a cake for a looooong a time."

Mario: "What about that a time two races ago, where Wario hit you and you a spun around and you a headed for the hill, and you got a stuck and wailed for me?"

Luigi: "Ah, that Wario! I will get him next a time!"

Mario: "Why did you not just turn around then?"

Luigi: "My Kart, it did not a move, no matter how I wailed."

Mario: "That is a what I am a speaking of, Brother. Your Kart moves in all other places, but if you head for the hills, it just a stops a suddenly!"

Luigi: "I a hate a stopping a suddenly."

Mario: "Yes, Brother. So do I. But why can we not traverse that a part of the hill? What is on the other a side?"

Luigi: "I think it is Track 4, Brother. They do not want you to wind up in the lake."

Mario: "Whaaaa? Who is this a 'they'? And why can we a not see the lake from a Track a 3? I think there is a nothing there, Brother. Just a more hills."

Luigi: "No, my brother. I think it is a Track a 4. Or maybe Track a 2. There must be a something there."

Mario: "Perhaps you are a right, my brother. We should get a back to a racing now. We can talk a more about a this after the next a race."

Luigi: "Yes, brother. I will a get that Wario this a time!"

Well-formed nonsensical questions

In our Highly Realistic Dialog, Mario is wondering about the Invisible Boundary at the edge of the track. He asks his brother the seemingly obvious question: "What is on the other side?"

As a gamer, if you pause to consider the question at all, the answer seems to be: "Nothing I care about." The invisible (or sometimes visible) Boundary seems just like any other wall. It is designed to keep you in where you're at, or out of where you're not, depending on your point of view.

But the gamer's view is that the boundary does have another side. You have no way to get there, but it exists. For maximum gameplay immersion a game universe needs to appear consistent. Thus, when you peer through the wall it appears that the other side is just more of the same.

To an embedded systems programmer, Mario's question is complete nonsense, like a fish asking the temperature of the water on the other side of the glass. There is no water on the other side, and for that matter a fish can't ask questions, so the question itself is based on nonsense premises.

From a perspective within the Mario Kart universe, the "other side" of the Invisible Boundary is... a kind of nothingness. The embedded world quite literally ends at the boundary. The level designers usually try to make it appear as if the current world continues on, but this is an illusion. They discontinue the model's polygons beyond the line of sight. Put another way, the water stops at the edge of the tank.

The nothingness beyond the Invisible Boundary of an embedded system is much deeper and more significant than simply not having objects there. In that nothingness there is no programmatic substrate upon which to place objects. If you were to ask: "What lies between Mario's head and Luigi's head," the answer may well be "nothing", since no game objects may overlap the line between their heads at the particular time you ask the question. But that "nothing" is qualitatively different from the "nothing" on the other side of the Invisible Boundary. Between Mario and Luigi there is a space – a place in their universe where objects and forces and logic apply, even if they are Mario Kart's twisted versions of physics and logic. That universe ends abruptly at the surface of the boundary.

The question "What's on the other side" is well-formed in a strictly mathematical sense. You could project a line from Mario to the nearest boundary, and ask the more precise question: "If Mario is at distance D from nearest boundary B, what overlaps the point P obtained by extending the Mario-boundary line L to a distance D beyond B?" ("Whaaaaa?")

The new formulation of the question is more well-formed, but it is every bit as semantically nonsensical.

What's really on the other side

In the context of the Mario Kart universe, the system boundary truly only has on side, and Mario and Luigi are on that side. From their perspective, we can't meaningfully ask the question "What's on the other side?" However, there is a semantically significant interpretation of "the other side" of that invisible boundary. To get to this better answer we must leave Mario's universe.

From the perspective of an embedded systems programmer, the entire Mario Kart universe is data structures laid out in the memory space of some machine. There is a program — the game engine — that interprets the data in a way that makes it appear to be a universe that is similar to ours in various intuitively appealing ways, yet different from our universe in various fun ways.

It is very unlikely that the polygons and other level data are laid out in strictly contiguous order in the host machine's memory space. It is more likely that they are spread out, with gaps, like dominoes spilled on a tile floor.

The question "What's on the other side?", when viewed from the perspective of a systems programmer, might be phrased more precisely and meaningfully as follows: "What occupies the memory spaces adjacent to the memory used by the Track Three Desert Level?"

This is the same as Mario's question, but we had to step outside the Mario Kart universe in order to ask the question in a way that made sense. The Mario Kart universe, like most game universes and in fact most embedded systems, is designed to appear complete. There is apparently "stuff" beyond the boundary; you just can't go that way.

When we step up into the host system and ask the analogous question about the "other side", both the question and the potential answers become much more complex: many orders of magnitude more complex, in fact. Fortunately, due to the Mario Kart system being so simple, increasing the complexity still gives us a vaguely accessible problem.

Let's try to cook up an answer to this new, more complex question regarding the contents of the program memory space on the "other side" of the invisible wall.

In terms of atomic program units (say, bits or bytes), the amount of memory used by a game like Mario Kart is actually high enough to defy our sense of scale. A game with hundreds of megabytes or gigabytes of runtime data has billions of discrete elements, which is too many for us to keep track of individually. One of the key jobs of a programmer is to organize their code so that the elements can be managed at human-CPU scale: up to seven "things" at a time. But this organization can't mask the fact that billions of individual elements exist, each with its own role to play in supporting the embedded system.

Hence, even for a game as "simple" as Mario Kart, we're stuck imagining how it works internally. Even the programmers who built the game have only a dim and incomplete mental model of the game. It's like building a skyscraper, or a city: you build it a piece at a time, and it's rarely fruitful to try to picture everything inside of it simultaneously.

Anything that goes wrong or is out of place in the program could take days to track down with state of the art tools. That's just how it is. We're not able to comprehend large problems all at once; we can only attempt it in small increments.

Bearing this necessarily incomplete understanding in mind, what exactly would we find in the machine memory between the elements of Mario and Luigi's track mini-universe?

The answer is familiar to programmers and perhaps surprising to non-programmers. The answer is: almost anything. It could be elements of a different program, or random garbage not being interpreted by any program, or supplemental data for the Mario Kart universe, such as language translation strings. Or Luigi could be right: it could be Track 4, pre-cached for the next race. Perhaps not exactly the way he's imagining it, but... close.

Moving beyond the Wall

Our little visualization of the host system's memory raises another natural question: what would happen if you "zapped" Luigi's Kart across the boundary?

This question isn't entirely as nonsensical as "what's on the other side?" With the proper programming tools, you might be able to observe changes in the machine memory as Luigi's Kart moves, and these changes might follow an observable pattern that leads to a relevant prediction of sorts.

As just one possibility, Kart motion might be represented as shifting a Kart memory structure along the sequential indexes of an in-memory array. This arrangement is unlikely for performance reasons, but it's certainly possible, and should serve to illustrate the point. If you were to find that moving Luigi's Kart ten meters (in the scale of Luigi's track universe) resulted in a structure motion of 1000 memory addresses, then you might make the reasonable prediction that in another ten meters his representation would move another 1000 addresses.

This might well put him beyond the end of the physical memory array. In most real-world scenarios this would be a bug, and would result in some sort of nasty response from the machine, such as a game crash. Or in a more user-friendly environment, the game engine might simply prevent his motion in that direction and move him back into his universe. This might put his Kart in an uncomfortable position, but it will at least be a real position according to the logic of the Mario Kart universe. At least the Kart won't have disappeared.

However, it is infinitesimally possible that Luigi's Kart could be moved into another physical machine process in the host system — say, another instance of Mario Kart running on a time-sharing operating system or virtual machine — in such a way that Luigi and his cart physically disappear from the old universe (a process address space) and appear in the new universe (another process address space).

Even if this infinitely remote possibility were to occur, chances are high that the sudden appearance of Luigi and his Kart would wreak local or even global havoc in the new universe. He might get lucky and crush Bowser into a pan-a cake, but more likely he would wind up stuck in a stone wall or a hill, unable to move. Or even more likely, his Kart's memory structure would be copied into a non-game area, such as the translation string section, and Luigi's sudden mysterious appearance might manifest as nothing more than garbled speech from one of the other characters.

There are many possibilities, too many to imagine. The most important takeaway here is that no matter how unlikely Luigi's safe intra-universe migration may be, it is possible. With some external help from the host system, it would even be straightforward: perhaps no more than a few days work from a skilled programmer.

In real-world programs sharing adjacent address spaces, it's improbable (but possible!) that migrating data from one process to another in a randomly chosen destination spot would have semantically meaningful results in the destination address space.

To put it in simpler terms: under just the right circumstances, Luigi could teleport to the other side, and wind up in a different world.

Ghosts in the machine

An embedded system is a little environment. My Betta in the previous installment of this little series lived in a very simple embedded system. Its most obvious difference from my own environment is that the tank was filled with water, while my room was filled with air. The differences from the host system can yield different behavior and different rules. In the fish tank, the rules are only a little different — for example, most things are more buoyant inside the tank. In a virtual embedded system, the rules and behavior might be completely different from those of the host system. It all depends on how the embedded system was designed to work.

Every embedded system has a containing surface, which might be called its frontier. The frontier is one-sided in the sense that the rules of the embedded system may simply stop at the boundary: there is no "other side". In some embedded systems (such as a Euclidean space embedded in a non-Euclidean space), even the supposedly intuitive notion of extending a line across the boundary only has meaning on one side, the "inside" of the frontier.

If there is a formal mathematical term for this One-Sided Frontier idea, I've yet to come across it, and I've spent quite some time looking. If you have any suggestions, let me know. The closest I could find are the spaces that are undefined in noncontinuous functions, such as the Tangent function for a value of 90 or 270 degrees. If you ask a question f(x) at these values of x, the answer is something like "You can't go that way."

So the other side of the frontier is undefined. This statement has a deep, almost philosophical meaning to programmers. It does not mean "nothing"; no, no no! It means anything. It is a modern-day Wand of Wonder, a Bag of Tricks, a Horn of Plenty, a Box of Chocolates. You never know what you're going to get.

If a computer program's code inadvertently reaches into undefined, then it has stepped across a mystical wall into another universe, and anything could happen. It's as if Gremlins have taken over the machine. Your PC speaker might beep erratically or continuously. Your video array might burst into random fireworks. Your printer might fire up and print out mysterious symbols. Your program may even continue to operate normally, but with the addition of unexplainable and unrepeatable phenomena.

I have seen all these things happen. All C/C++/assembly programmers have seen bugs like this — program bugs with "crazy" behavior. The bugs are so spooky, and so hard to track down, that computers are now designed to be gatekeepers at the Wall, and when you try to step across they say STOP! It's much better to be blocked immediately than to let your program wander into undefined, where ghosts may take hold of your data in ways that may even cause your legal department to take notice. Stranger things have happened, as they say.

This computerized gatekeeping is commonly called "protected mode". The computer checks every addressing instruction, and any time a program tries to access the memory area outside its own, things halt immediately. When you see the message "segmentation fault", or a blue screen of death, or some other sign that a fatal, unrecoverable program error has occurred, it is almost always because someone or something in the embedded system tried to escape.

Holes in the Wall

From the perspective of a computer game, the system frontier is relatively uninteresting. It's not much different from any other obstacle.

However, in other embedded systems these frontiers are almost mystical in nature. They provide endless fascination for computer scientists working in the domain of programming languages, which are yet another kind of embedded system.

To see how it works, let's consider the situation in Mario Kart, which is simpler.

In Mario Kart, most of the racers are computerized opponents, or AIs. A few of them, usually at least one, are controlled by people, using steering wheels or other physical controllers. Sometimes (e.g. in simulation mode) all the opponents may be computerized.

In order for people to interact with the embedded system, there must be some way to send information back and forth across the system frontier.

Coming from the Mario Kart side, we typically have video and sound signals. The embedded system generates these signals and sends them to your TV or device.

Coming from the people side, we typically have motion input: which way to move the Karts. The signals begin as your physical motions: buttons you press, or in newer controllers, the direction you tilt the controller. They are sent from the host system to the embedded system and they generally produce observable effects in the embedded system. Like magic!

Mario Kart is especially interesting because the external camera is physically present within the game. You can see it in the instant replays of your races: a little guy with a camera, floating behind you on a little cloud. The game designers have arranged things so that you can almost never see him while racing, because he's always flying behind your head, along your line of sight.

But he's there. And in fact that camera is always there, in all video games. It's just that the Mario Kart designers have chosen to reify him as a cute little turtle guy with a camera, floating behind you on a cloud.

The external camera is a one-way hole in our one-way wall: it sends data out of the embedded system. For all intents and purposes it also sends the sound data, since the sounds are usually scaled as a function of distance from the camera.

During a race, there's a lot of stuff going on in the embedded system, and there's a lot of stuff going on in the host system. But they constrain their communication via mostly-invisible channels, and these channels are restricted in the kinds of communication they may convey. Your controller can send motion inputs, but (at least today) it can't send video data. And (at least today) the characters in the game can't control your arms and legs, the way you can control theirs.

Hopefully now you should have a mental picture of this magical wall between an embedded system and the Great Undefined Beyond on the "other side" of its system frontier. The wall may have deliberate holes in it: channels, really. Information may flow across these channels in predefined ways. And the channels are almost always invisible to occupants of the embedded system.

Reflections

I've spent a lot of effort telling you about a rather strange, twisty kind of surface: the frontier of an embedded system. This frontier exists for all embedded systems. It may have "holes" (data channels) in it. The number and nature of these channels is entirely up to the designers of the embedded system. Programmers sometimes call these channels "reflection" or "metaprogramming".

The holes in the frontier may or may not be detectable within the embedded system itself. They may only be detectable within the host system. This, again, is up to the designers. For most of the embedded systems I know of, the channels are "well defined", in the form of application programming interfaces offered to either the embedded system or the host system for communicating across the frontier.

But you need a channel for this kind of communication. Setting it up is usually not cheap. And setting up meta-reflection (in other words, being able to "see" the channel from within the embedded system or host system) is even more work.

So most of the time, channels through the embedded system frontier are completely invisible and undetectable from inside the embedded system. They're quite real, and information flows either one way or both ways, but they cannot be detected from within the embedded system, and their behavior is intrinsically non-repeatable via experimentation.

When data comes across these invisible channels, stuff "just happens" in the embedded system, with no clear indicator nor explanation as to why.

In our discussion so far, I have intentionally blurred the distinction between the host system (such as a fish tank or a game console device) and the host system's host system (such as your bedroom or living room). But you've probably noticed by now that all host systems are embedded in some larger system. This is just the way things work. The fish tank is in your bedroom, which is a system embedded in a house, which is a system embedded in a neighborhood, embedded in a county, a nation, a continent, a planet, a solar system, a galaxy, a universe.

It's perhaps not as clear in the case of fish tanks, but host systems often overlap and even cooperate. A city is composed of many interleaved subsystems. So is your body. It's not always a simple containment relationship. Systems are made of, and communicate with, other systems.

But one way or another, all systems are embedded systems.

There is no reason to assume that our Universe works any differently. Our Universe is a system; that much should be self-evident. It has boundaries; astronomers and astrophysicists have recently even determined that the boundary appears to be a dodecahedron.

We are already painfully aware of the question "what happened before the Big Bang, if in fact the Big Bang occurred in the way all the evidence suggests", and its inherent nonsensicality. What happened before the beginning of Time? What lies beyond the end of the Universe?

Programmers already know intuitively the answer to these questions. The answer is: undefined is there. Undefined is not the same as "nothing", no indeed. It's anything. And we can't go that way.

This has ramifications for the way we think about things today.

I believe I will have more to say about this soon. Right now I think I will go play Mario Kart: a game as fun as any I think I've ever played.

Thursday, December 25, 2008

Fable II: Arguably Better than Getting Your Head Crapped On

I finished Fallout 3 maybe six or eight weeks ago, and it was hands-down one of the best games I've ever played. A game like that gets you in the mood for more gaming, so I thought to myself: "Hey, I should plop down $160 for Fable II!"

Actually that's not exactly what I thought, but it's what happened. I bought the game for $60, fired it up, got up to the part in the intro where a bird craps on your head (yes, this is how it starts), and it locked up hard. Reset the XBox, tried again, and this time got as far as some guy selling snake oil gadgets before it locked up again. Snake oil, indeed.

I tried playing for about an hour, with the game crashing every 3 to 5 minutes, and I finally went online to read about how it kills XBoxes and it's the Game of Death and blah blah blah, all interesting but not especially helpful. Eventually I stumbled across discussions of the "install to hard drive" option. Nobody actually said how to do it, so it took another hour of digging to deduce that you need to purchase a $100 wireless network adapter (or 100 feet of network cable, I guess). So I shut it down for the night, waited for the stores to open, forked over the $100, and installed the game to the hard drive. To Lionhead's credit the game never crashed again, making it significantly more stable than Oblivion or Fallout 3.

I tried hard to like Fable 2. I didn't even need to like it $160. I would have settled for a $60 value. I vaguely remember liking Fable 1, although I can't remember anything about the game except for one neat scene where you had to escort two NPCs through a dark valley. One of the NPCs has been bitten by a Balverine (a werewolf), and the two argue the whole trip about whether he's going to turn. It's a funny conversation and the scene has a funny ending. Other than that, I just have vague recollections of shooting birds on the roof of some guild, and needing to get a 6-foot handlebar moustache for some side quest. The rest of it is basically a blank. But I had set some flag to the effect that "I liked it," and I wanted to like the sequel too.

Unfortunately, with a few noteworthy exceptions that I'll call out in the "Highlights" below, the game is entirely forgettable. It's already fading from memory as we speak. It wasn't as bad as some people make it out to be. It's playable for a couple of days, and it has its fun moments. But it's not a very good game, and it's definitely not a very memorable game. This is sad, considering the amount of effort that went into its development.

The no-spoiler synopsis of Fable 2 is that it's a bad Zelda clone. You can smell the desperation; there are dozens and dozens of direct rip-offs from the Zelda franchise. Heck, I wouldn't have minded a half-decent Zelda clone; they're some of the best games of all time. But Fable 2 misses the mark by a mile. The humor is juvenile bordering on imbecilic, the hints are hamfisted, the areas are small and cramped, the minigames are lackluster, the music is virtually nonexistent, and the story pacing is rushed and breathless. It's a cargo-cult copy of Zelda that winds up having no identifiable soul: forgettable across the board.

I've given up every piece of Microsoft software and hardware I own except for the XBox, which I had been holding onto just for Fable 2. Now that it's come and gone like a bird crapping on my head, I'm giving up. No more XBox or PC games for me. Ever.

Hence, Fable 2 cost me $160. I hope you got it for cheaper than that.

Anyway, here's a quick rundown of the lowlights and highlights of the game, as I see them. Enjoy!

Lowlights

1) Humor. Fable 2 tries hard to incorporate humor into the game — too hard. The writers use the trusty old "stopped clock" approach to humor, in which they inundate you with jokes, and 1 out of every 43,200 of them is funny. Amazingly, this perseverance leads to 3 or 4 genuinely funny ones, mostly near the entrance to the Crucible (arena). But by the time you get there, you've already tuned out all attempts at humor and have probably even tried killing yourself a few times. So they may fail to register.

2) Theresa. The game features an old lady who watches everything you do and talks at you constantly. This starts in the very beginning of the game and lasts until the very end, with no option to turn her off. Your character can't so much as take a crap without Theresa piping in with helpful advice on which hand to use. "That is ancient paper. Be cautious." She uses some magical form of communication system that only breaks down in the fog — probably shortwave radio — and there's no way to turn the fugging thing off.

I really hated Theresa.

3) Expressions. You can't talk to people. Instead, the game gives you a series of increasingly repugnant forms of nonverbal communication. Initially you're limited to belching, farting, giving people the finger and making lewd pelvic thrusting motions, but as you rise in fame Theresa informs you that you've earned the right to use the "kiss my ass" expression. I am not making this up. I tried to avoid using expressions altogether, but the game forces you to do it once in a while. Made me want to take a shower.

4) Retinal blindness. Fable 2 is nauseatingly saturated. They just don't know how to lay off the paint gun. There are a few OK-ish-ly tasteful areas, such as the big trees in Bower Lake, but most of the game is a frightully garish mix of lime greens, oranges, purples, reds, blues, and general oil-spill iridescence. It makes you color-blind fast, even if you didn't start that way. Finding anything onscreen is like trying to spot where someone threw up on a Matisse.

5) Linearity: the game is unrelentingly linear for the first hour or two (a _long_ time), after which it settles into, well, linearity. The gameplay occasionally approaches the smashing-through-lines-of-baddies feel of Gauntlet Legends, which I liked, but mostly it makes you feel like a rat running a big maze, following a neverending golden trail of cheese.

A major contributor to the linear feel, even after the game opens up, is the plethora of tiny little fences and obstacles that you can't hop. It makes it really hard to know where you can walk, and it feels like you're constantly bumping into things, because, well, you are. So the game is linear at all resolutions: high (the plotline), medium (most of the level designs) and low (the path designs). Linearity can cramp even the best of games — Kingdom Hearts comes to mind. It's just a bad way to design things. And Fable's linearity felt especially suffocating after just having finished Fallout 3, which is immense and wide-open.

6) Controls. It's been a long time since I played a game whose controls were so accident-prone. Normally a game's controls take some getting used to, and then it's like driving a car. In Fable 2, even after days of play, I'd still be trying to hop a fence and wind up shooting the front door off a mansion, blowing boards everywhere and scaring the shit out of the villagers.

Hell, even when I was trying to buy my final sword (this was a $50k sword I'd been looking for all day), I tried to bring up the "buy sword" menu for the blacksmith, and I accidentally wound up casting a massive Inferno spell, causing him to literally run screaming across town. It was weeks of in-game time before I saw him again. God dammit. They really should have had different controls in safe zones.

7) Elephantine mammary glands. I don't know what planet these guys have been living on, but giant udders fell out of fashion at least two or three decades ago. Every single woman in Fable 2 had knockers significantly bigger than her head. It reminded me of my trip to Paris, where every statue of a woman is bare-breasted, presumably so that you can tell it's a woman — a practice which unfortunately suggests that there's really no other way to tell. Dumb French statue-making assholes.

I mean, the people in Fable 1 were ugly — the main character worst of all. They all had this "I'm a programmer who never gets outdoors" look, and I expected (and got) no better from Fable 2. But I was really disappointed that every female in the game was a fugging dairy farm. I mean, someone with some taste and maturity should have a talk with these asshats, and explain to them what women actually look like. Or they should pick up a frigging Victoria's Secret catalog or watch a goddamn Target commercial or something. Jesus.

The milk jug thing... it was really just too much. I have zero respect for those jerk-offs at Lionhead.

8) Ass-kissing. This was probably the most serious problem with the game. It was a disease in Fable 1 that went malignant in Fable 2. Whoever designed these games was apparently neglected as a child, because the gameplay revolves around gaining "renown". Lionhead's hopelessly adolescent view of "renown" is that villagers should follow you around and say things like "yay!" and "hurrah!" It's even worse than I'm making it sound.

They spent so much time coding this crap that they forgot to code pushing into the game: you can't push people out of your way. So as soon as you wander into a dead-end alley you're fucked: a bunch of people will crowd in after you asking for autographs and offering you gifts and all this sickening bullshit.

To the game's credit, and I count this as a highlight, if you pull out your six-barreled rifle, take the safety off, and aim right at their heads, it clears everyone out pretty fast. You can imagine how desperate I was by the time I tried that approach. But they coded it correctly, bless 'em.

9) Too Easy. The game just wasn't hard, period. There were no hard fights. I never died. I don't even know what happens when you die in Fable 2. I used a couple of Resurrection Phials, but only because I had become so lazy in combat that I didn't care anymore. This was a serious flaw in the game: it essentially removed the element of fear, which was the only emotion (other than disgust) that the game had a chance of evoking.

10) Demon Doors. Oh man, oh man. These were probably the low point of the whole game. They made me want to puke. I would run past them as fast as I could so I didn't have to listen to their inane drivel. This was some of the worst game writing I've ever seen. I just don't want to talk about it.

11) Misguided innovation. They really should have stuck with copying Zelda, and Kingdom Hearts, and Gauntlet: Legends, and all the other games they copied blindly, and badly. Because whenever they introduced something entirely new, it almost invariably sucked. Examples? OK. Sure. Since you asked, and all.

How about the "innovation" that when you eat nearly any food (and it only takes a few bites), you bloat up to the size of Orson Welles, and the only way to get rid of it is to eat celery. No amount of exercise will make any difference, but eating a few bites of celery makes it go away. Innovative!

Innovation: you can purchase almost any property in the game. Is this realistic? No. In reality, not everything would be for sale (and especially not posted on the front doors). Real-estate transactions wouldn't be instantaneous. You would need the owner present to buy something. Etc, etc. So given that this feature makes the game less realistic, what purpose does it serve? Is it fun? No. Buying real estate is about as fun as attending insurance seminars, so I don't know what the hell they were thinking. It could have been fun in the right setting, with suitable other human participants, in a Parker-Bros. Monopoly kind of way. Maybe. But slapping it on the side of an RPG and calling it innovation? It boggles the mind.

And what about the the busywork jobs (blacksmith, woodcutter, bartender) for making gold? Um, dudes -- busywork jobs only exist in MMORPGs to limit per-player CPU usage. They're not fun. "Innovatively" bringing them into a single-player game was just flat-out brain damaged.

12) Nonexistent target audience. What age group is the target market for this game? If you enumerate the possibilities, you arrive at the inescapable conclusion that the game was either (a) created by imbeciles, or it was (b) created for imbeciles, or possibly (c) all of the above.

It's presumably not intended for kids, or you wouldn't be finding condoms in treasure chests, soliciting and obtaining sex from male and female prostitutes of all shapes and sizes, performing pelvic thrusts to solve quests, and so on.

It's not for adults either, or you wouldn't be bombarded with the constant barrage of scatological humor, beginning with the bird shitting on your head, continuing with warnings about "extending the fart command and messing it up", and going pretty much straight downhill from there.

Is it intended for teenagers, then? Poooossibly, but (a) that ignores their primary demographic, which is 30-year-olds, and (b) I don't know any teenagers that are that stupid, nor so hard-up for attention that they need AI villagers to yell "hurray!" whenever you pass them, even if you're in a graveyard at midnight.

Dipshits. This game was designed by dipshits. The coding was great, the artwork was great, the sound effects were great; the details were for the most part rock-solid. But the creative direction was just inexcusably bad.

Highlights

OK, I've been pretty tough on the game so far. Fable 2 did actually have a few genuine highlights worth calling out. You could even argue that these highlights make the game almost worth playing, in spite of all the crap you have to suffer through in order to get to them.

1) Banshees. Fable 2's banshees are, in a word, awesome. I've been racking my brain trying to think of a VG monster as cool as these banshees in any game I've ever played. I'm coming up with a few ties, but nothing that beats them. The YouTube videos don't come close to doing them justice. Fable II is worth playing just to get to Wraithmarsh.

The only real problem with the banshees is that since none of the combat is challenging (see Lowlight #9), they're nowhere near as scary as they could have been. But they're amazingly stylish. I'd call them an innovation, but I'm inclined to believe Lionhead stole their basic design from some other game, given all the other copying they've done. (The Fable 2 Trolls, for instance, are about as Zelda-clone-esque as you can get without inviting a lawsuit.)

2) Lucien's speech, where he addresses the recruits. Really great speech. Riveting and convincing. Amazing how Microsoft-run studios that are so consistently bad at humor are so good at creating convincingly evil speeches about taking over the world.

Actually the whole centerpiece drama in the tower was very nicely done. I have to give them credit for that part of the game: it was exceptional by any standard. It basically saved the game from being a total loss.

3) Hammer. She's cool. Great voice acting, surprisingly good scripting, neat character, lots of depth. One of the better-realized VG supporting characters I've seen in many years.

4) The dog. Apparently there was a lot of hype about the dog. Or so I hear, after actually having played the game. Whatever the hype, the reality is that it's a very believable dog. I especially liked how it would run ahead of you -- I've seen pets that follow you, but the dog would often anticipate your direction and run ahead, kinda turned back towards you like "c'mon! let's go!" I encountered no glitches with the dog; the coding was rock solid. Overall it was, well, very... doggy. And what more could you ask for in a dog, really?

As a tribute to the believability of the dog, I'll offer a minor spoiler. (Skip ahead if you don't want a spoiler!) At the end of the main storyline, you are granted one wish. Your choices are: (a) get all the people who died back, (b) get your dog back, or (c) get a bunch of money. What I really wanted was a sort of amalgam of the 3 choices: I wanted my money back for this dog of a game. But when push came to shove, I picked the dog. I kinda missed him.

5) Architecture. Overall the architecture was really nice. The only somewhat dubious exception was Bowerstone, which looks almost exactly like Euro Disney. I kept expecting Tigger to come waltzing around, cursing in French under his breath, just like he did on my real-life trip to Euro Disney a few years back.

Other than the Euro Disney influence, which I could take or leave, I thought the architecture was nice throughout the game. I liked the waterfront town of Bloodstone. I liked the manors in Oakfield. I liked the gypsy wagons. I liked the vendor carts. I loved pretty much every creepy structure in Wraithmarsh. The overall look of the game was beautiful, once you got past the color-saturation problem, and the architecture was a huge contributor.

6) Fight music. Unlike in Fable 1, most of the music in Fable 2 is forgettable background/atmosphere music. They didn't get Danny Elfman this time around, and it shows. The theme for Bower Lake is nice as far as it goes, which is exactly 2 chords over and over and over. But it's still OK. The rest of the music didn't leave any sort of impression on me at all, except for the fight music, which almost made up for everything. It was very good. There were at least two fight themes and both of them were cool. If only the rest of the music had been... present. It was like it wasn't even there. It phoned in its performance.

Folks at Dorkhead studios: Zelda's music is one of the top five reasons for its success as a franchise. Same goes for Mario and Kingdom Hearts and Final Fantasy. Their music is always great, and it's always in your face. The music isn't muttering or mumbling; it's shouting. And they can get away with it because it's always great. Even when it's bad or annoying, which is rare, the music still anchors each place and event in the game in your memory, in a way that only music can. You guys really screwed the pooch on this one.

7) Mixed-tactic fighting. They did a great job of setting things up so that you could use melee, ranged weapons and magic effectively in combat. It was refreshing to be able to switch styles in mid-fight: you could use your sword to kill everything near you, then start blasting everything ten feet or more distant with your rifle. Or you could clear a little space and cast a time-slowing spell, and then just start zinging around whaling on bad guys. The combat was never hard, but it was on the whole fairly satisfying.

The downside of ultra-convenient access to melee, range weapons and spells was that you could effortlessly use them all simultaneously while trying to buy vegetables from a produce stall in the main market. I really wish they'd made it just a teeny bit harder to cast spells in public areas.

8) Well, damn. I can't think of a Highlight #8. I thought of some more lowlights, though: long area loads, unresponsive controls during "scenes", only a handful of available spells, months of coding/design effort wasted on useless features like "groin shots" and tatoos...

Oh, and the lack of control over when quest scenes actually unfold — they trigger from proximity to the relevant NPC rather than interacting with the NPC because, oh, that's right, you can't interact with anyone except to fart on them or give them the finger. Oops! So you're always accidentally wandering into a dungeon that triggers some quest, and there's no way out except to back entirely out of that phase of the quest, which may involve losing hours of your time, all because you walked through the wrong door. Damn that pissed me off.

And how the hell do you sleep in an Inn? I never managed to figure it out. I'd wind up spending $10k for some hovel just to get a frigging bed to sleep in. It was amazingly bad UI design, if there even is a way to do it. If not, then their helpful tutorial message lied to me at least a dozen times.

Argh. Well, this highlights section is going downhill in a hurry, so I think I'll end it here.

Better than a crap on the head?

Maybe, maybe. But compared to Fallout 3, Fable 2 pretty much sucked. It had a couple of nice features, but they were drowned in an ocean of painfully adolescent design. Such a shame.

I've tried to be fair here. I don't mean to discourage you from playing the game, since for all I know there's nothing better out there right now.

If you do decide to play it, I hope I've set your expectations very low. That way, well, who knows? You might actually have some fun with it.

But if you open even one of those Demon Doors I'll lose all respect for you.

Sunday, November 16, 2008

Ejacs: a JavaScript interpreter for Emacs

So! I have all these cool things I want to write about, but I broke my thumbnail. Can you tell that's a long story?

See, this summer I got excited about playing guitar again. I usually switch between all-guitar and all-piano every other year or so. This summer I dusted off the guitars and learned a bunch of pieces, and even composed one. I was prepping for — among other things — a multimedia blog entry. It was going to have a YouTube video, and a detailed discussion of a wacky yet powerful music programming language you've probably heard of but never used, and generally just be really cool.

And then it all came crashing down when I busted my thumbnail off. And I mean off — it broke off at least a quarter inch below where the nail and skin meet. Ick. I just accidentally jabbed my steering wheel, and that was that.

I remember reading an interview with some dude who said he had punched a shark in the nose. He said it was like punching a steering wheel. So now I know more or less what it's like to punch a shark in the nose, I guess. There's always an upside!

Anyway, that was going to be my magnum opus (literally: Op. 1) for the year, but it fell through for now. I'll have to revisit the idea next year. My thumbnail's back, but it's been at least 2 months since I touched my guitar, so I'll have to start over again.

Work has been extraordinarily busy, what with having to collect all these Nuka-Cola Quantum bottles and so on. I'm sure you can imagine. So I haven't had much time to blog lately.

But I do like to publish at least once a month, whether or not anyone actually cares. It's been about a month, or it feels that way anyway, and all I have to show for it is this box of Blamco Mac and Cheese.

So I'm cheating this month.

You know how on Halloween how you walk around in your costume holding your little bag and you say "trick or treat", and every once in a while some asshole does a trick instead of dumping half a pound of candy into your bag? And then he has to explain to all the dumbfounded and unhappy kids that "Trick or Treat" means that a trick is perfectly legal according to the semantics of logical-OR, and the kids remember that a-hole for the rest of their childhoods and avoid his house next year?

Yeah.

So I'm doing a trick this time. Hee. It's actually kind of fun when you're on the giving end.

My trick is this: in lieu of saying anything meaningful or contemporarily relevant, I'm writing about something I did over a year ago. And there isn't much to say, so this really will be short.

Ejacs

Around a year ago, I wrote a blog called Stevey's Boring Status Update, mostly in response to wild rumors that I'd been fired from Google. Not so. Not yet, anyway.

In that blog I mentioned I was working nights part-time (among other things) on a JavaScript interpreter for Emacs, written entirely in Emacs Lisp. I also said I didn't have a name for it. A commenter named Andrew Barry suggested that I should not call it Ejacs, and the name stuck.

Ejacs is a port of Narcissus. Narcissus is a JavaScript interpreter written in JavaScript, by Brendan Eich, who by pure unexpected coincidence is also the inventor of JavaScript. Narcissus is fairly small, so I thought it would be fun to port it to Emacs Lisp.

It turns out Narcissus is fundamentally incomplete. It cheats. It's that trick guy on Halloween. Narcissus has a working parser and evaluator, but for its runtime it calls JavaScript. So it's kind of like saying you're building a car by starting from scratch, using absolutely nothing except for a working car.

This meant I wound up having to write my own Ecma-262 runtime, so that the evaluator would have something to chew on. In particular, the Ecma-262 runtime consists of all the built-in properties, functions and objects: Object, Function, Array, String, Math, Date, RegExp, Boolean, Infinity, NaN, parseInt, encodeURIComponent, etc. A whole bunch of stuff.

I basically did this by reading the ECMA-262 specification and translating their algorithms into Emacs-Lisp. That spec is lousy for learning JavaScript, but it's absolutely indispensable if you're trying to implement JavaScript.

I didn't know Emacs-Lisp all that well before I started, but boy howdy, I know it now.

Emacs actually has a pretty huge runtime of its own — bigger than you would ever, ever expect given its mundane title of "text editor". Emacs has arbitrary-precision mathematics, deep Unicode support, rich Date and Calendar support, and an extensive, fairly complete operating system interface. So a lot of the porting time was just digging through Emacs documentation (also extensive) looking for the Emacs version of whatever it was I was porting. That was nice.

I had big plans for Ejacs. I was going to make it a full-featured, high-performance JavaScript interpreter, with all the Emacs internals surfaced as JavaScript native host objects, so you could write Emacs customizations using object-oriented programming. It was totally going to be the "meow" part of the cat.

And then I broke my thumbnail.

Actually, what happened was js2-mode.

js2-mode

After I got the interpreter working, I was at this crossroads. There were two big use cases: a JavaScript editing mode, or a JavaScript Emacs development mode. Both were going to be a lot of work.

It turns out you really want the editing mode first, if possible, so that when you're doing all your JavaScript programming you have a decent environment for it. So I picked the editing mode.

Unfortunately I found the Ejacs parser wasn't full-featured enough for my editing needs, since at the time I was working on my Rhino-based Rails clone and writing tons of JavaScript 1.7 code on the JVM.

I spent a little time trying to beef up the parser, then realized it would be a lot faster to just rewrite it by porting Mozilla Rhino's parser, which is (only) about 2500 lines of Java code. Ejacs is something like 12,000 lines of Emacs-Lisp code, all told, so that didn't seem like a big deal.

So I jumped in, only to find that while the parser is 2500 lines of code, the scanner is another 2000 lines of code, and there's another 500 or so lines of support code in other files. So I was really looking at porting 5000 lines of Java code.

Moreover, the parse tree Rhino builds is basically completely incompatible with the Ejacs parse tree. It was richer and more complex, and needed more complicated structures to represent it.

So after I'd ported the Rhino parse tree, what I really had was a different code base. I went ahead and finished up the editing mode, or at least enough to make it barely workable (another 5000 lines of code), and launched it. It was a surprisingly big effort.

And it left poor Ejacs lying unused in the basement.

So today, faced with nothing to write about, I figured I'd dust off Ejacs, launch it with lots of fanfare, and then you'd hardly notice that I cheated you. Right?

You're not coming to my house next year. I can tell already.

Anyway, here's the code: http://code.google.com/p/ejacs/

There's a README and a Wiki and installation instructions and stuff. I can't remember how to put the code in SVN, and I'm having trouble finding it on the code.google.com site. As soon as I figure it out I'll also make it available via SVN.

Emacs Lisp vs. JavaScript

In the interests of having something resembling original worthwhile content today, I'll do a little comparison of Emacs Lisp and JavaScript. I know a lot about both languages now, and a few folks mentioned that a comparison would be potentially interesting.

Especially since I think JavaScript is a better language.

So... the best way to compare programming languages is by analogy to cars. Lisp is a whole family of languages, and can be broken down approximately as follows:
  • Scheme is an exotic sports car. Fast. Manual transmission. No radio.
  • Emacs Lisp is a 1984 Subaru GL 4WD: "the car that's always in front of you."
  • Common Lisp is Howl's Moving Castle.

This succinct yet completely accurate synopsis shows that all Lisps have their attractions, and yet each also has a niche. You can choose a Lisp for the busy person, a Lisp for someone without much time, or a Lisp for the dedicated hobbyist, and you'll find that no matter which one you choose, it's missing the library you need.

Emacs Lisp can get the job done. No question. It's a car, and it moves. It's better than walking. But it pretty much combines the elegance of Common Lisp with the industrial strength of Scheme, without hitting either of them, if you catch my drift.

Anyway, here's the comparison. Here's why I think JavaScript is a better language than Emacs Lisp.

Problem #1: Momentum

A recurring theme is that Elisp and JavaScript both will both exhibit a particular problem, and there are specific near-term plans to fix it in JavaScript, but no long-term plans to fix it in Elisp.

It's easier to resign yourself to a workaround when you know it's temporary. If you know the language is going to be enhanced, you can even design your code to accommodate the enhancements more easily when they appear.

People are working on improving JavaScript. It's not happening quite as fast as I'd hoped earlier this year, but it's still happening. As far as I know, Emacs Lisp is "finished" in the sense that no further evolution to the language is deemed necessary by the Emacs development team.

Problem #2: No encapsulation

Every symbol in Emacs-Lisp is in the global namespace. There is rudimentary support for hand-rolled namespaces using obarrays, but there's no equivalent to Common Lisp's in-package, making obarrays effectively useless as a tool for code isolation.

The only effective workaround for this problem is to prefix every symbol with the package name. This practice has become so entrenched in Emacs-Lisp programming that many packages (e.g. apropos and the elp elisp profiler) rely on the convention for proper operation.

The main adverse consequence of this problem in practice is program verbosity; it makes Emacs-Lisp more difficult to read and write than Common Lisp or Scheme. It can also have a non-negligible impact on performance, especially of interpreted code, as the prefix characters can approach 5% to 10% of total program size in some cases.

The problems run slightly deeper than simple verbosity. Without namespaces you have no real encapsulation facility: there is no convenient way to make a "package-private" variable or function. In practice there's little problem with program integrity; it's hard for an external package to change a "private" variable inadvertently in the presence of symbol prefixes. However, it makes it annoyingly difficult for users of the package to discern the "important" top-level configuration variables and program entry points from the unimportant ones. Elisp attempts a few conventions here, but it's a far cry from real encapsulation support.

JavaScript also lacks namespaces. They're being added in ES/Harmony, but in the meantime, browser JavaScript code typically uses the same name-prefixing practice as Emacs-Lisp.

However, JavaScript has lexical closures, which provide a mechanism for creating private names. One common encapsulation idiom in browser JavaScript is to wrap a code unit in an anonymous lambda, so that all the functions in the code unit become nested functions that close lexically over the top-level names in the anonymous lambda. This trick is nowhere near as effective in Emacs-Lisp, for several reasons:
  • elisp is not lexically scoped and has no closures
  • elisp nested defuns are still entered into the global namespace
  • CL's `flet' and `labels' are only weakly supported, via macros, and they frequently confuse the debugger, indenter, and other tools.

Some elisp code (e.g. much of the code in cc-engine) prefers to work around the namespace problem by using enormous functions that can be thousands of lines long, since let-bound variables are slightly better encapsulated. Even this is broken by elisp's dynamic scope:
(defun foo ()
(setq x 7))

(defun bar ()
(let ((x 6))
(foo)
x)) ; you would expect x to still be 6 here

(bar)
7 ; d'oh!

So let-bound variables in elisp can still be destroyed by your callee: a dangerous situation at best.

Emacs is basically one big program soup. There's almost no encapsulation to speak of, and it hurts.

Problem #3: No delegation

One of the big advantages to object-oriented programming is that there is both syntactic support and runtime support for automatic delegation to a "supertype". You can specialize a type and delegate some of the functionality to the base type. Call it virtual methods or prototype inheritance or whatever you like; most successful languages support some notion of automatic delegation.

Emacs Lisp is a lot like ANSI C: it gives you arrays, structs and functions. You don't get pointers, but you do get garbage collection and good support for linked lists, so it's roughly a wash.

For any sufficiently large program, you need delegation. In Ejacs I wound up having to implement my own virtual method tables, because JavaScript objects inherit from Object (and in some cases, Function, which inherits from Object).

Writing your own virtual method dispatch is just not something you should have to do in 2008.

Problem #4: Properties

I wrote about this at length in a recent blog post, The Universal Design Pattern. JavaScript is fundamentally a properties-based language, and it's really nice to be able to just slap named properties on things when you need a place to store data.

Emacs Lisp only offers properties in the form of simple plists – linked lists where the odd entries are names and the even entries are values. Symbols have plists, and symbols operate a little bit like very lightweight Java classes (in that they're in the global namespace), but that only gets you so far. If you want the full JavaScript implementation of the Properties Pattern, you'll have to write a lot of code.

And so I did. Your implementation choice for object property lists has a huge impact on runtime performance. Emacs has hashtables, but they're heavyweight: if you try to instantiate thousands of them it slows Emacs to a crawl. So they're no good for the default Object property list. Emacs also has associative arrays (alists), but their performance is O(n), making them no good for objects with more than maybe 30 or 40 properties.

I wound up writing a hybrid model, where the storage starts with lightweight alists, and as you add properties to an object instance, it crosses a threshold (I set it to 50, which seemed to be about right from profiling), it copies the properties into a hashtable. This had a dramatic increase in performance, but it was a lot of work.

I experimented with using a splay tree. I implemented Sleater and Tarjan's splay tree algorithm in elisp; Ejacs comes with a standalone splay-tree.el that you can use in your programs if you like. I was hoping that its LRU cache-like properties would help, but I never found a use case where it was faster than my alist/hashtable hybrid, so it's not currently used for anything.

And then in the end, after I was done with my implementation, it was a library (at least from the Emacs-Lisp side of the house). It wasn't an object system for Lisp. It's only really usable inside the JavaScript interpreter, where it has syntactic support.

You really want syntactic support. Sure, people have ported subsets of CLOS to Emacs Lisp, but I've always found them a bit clunky. And even in CLOS it's hard to implement the Properties Pattern. You don't get it by default. CLOS has lots of support for compile-time slots and virtual dispatch, but very little support for dynamic properties. It's not terribly hard to build in, but that's my point: for something that fundamental, you don't want to have to build it.

Problem #5: No polymorphic toString

One of the great strengths of JavaScript is the toSource extension. I don't know if they support it over in IE-land; I haven't been a tourist there in a very long time. But in real versions of JavaScript, every object can serialize itself to source, which can then be eval'ed back to construct the original object.

This is even true for functions! A function in JavaScript can print its own source code. This is an amazingly powerful feature.

In Emacs Lisp, some objects have first-class print representations. Lists and vectors do, for instance:

(let ((my-list '()))
(push 1 my-list)
(push 2 my-list)
(push 3 my-list)
my-list)
(3 2 1)

(let ((v (make-vector 3 nil)))
(aset v 0 1)
(aset v 1 2)
(aset v 2 "three")
v)
[1 2 "three"]


But in Emacs Lisp, many built-in types (notably hashtables and functions) do NOT have a way to serialize back as source code. This is a serious omission.

Also, trying to print a sufficiently large tree made entirely of defstructs will crash Emacs, which caused me a lot of grief until I migrated my parse tree to use a mixture of defstructs and lists. Note that simply typing the name of a defstruct, or passing over it ephemerally in the debugger, will cause Emacs to try to print it, and crash. Fun.

The problem of polymorphic debug-printing (or text-serialization) is, I think, a byproduct of Emacs not being object-oriented. If you want a debug dump of a data structure, you write a function to do it. But Emacs provides a half-assed solution: it debug-prints lists very nicely, even detecting cycles and using the #-syntax for representing graph structures (as does SpiderMonkey/JavaScript). But it has no useful debugging representation for hashtables, functions, buffers or other built-in structures, and there's no way to install your own custom printer so that the debugger and other subsystems will use it.

So it sucks. Printing data structures in Emacs just sucks.

The situation in Ecma-262-compliant JavaScript really isn't that much better, although you can at least install your own toString on the built-ins. But any competent "server-side" JavaScript implementation (i.e. one designed for writing real apps, rather than securely scripting browser pages) has a way to define your own non-enumerable properties, so you can usually override the default behavior for things like toString and toSource.

And all else being equal, at least JavaScript functions print themselves.

Emacs advantages: Macros and S-expressions

Pound for pound, Emacs Lisp seems roughly as expressive as JavaScript or Java for writing everyday code. It shouldn't be that way. Emacs Lisp ought to be more succinct because it's Lisp, but it's incredibly verbose because of the namespace problem, and it's also verbose to the extent that you want to use the properties pattern without worrying about alist or hashtable performance.

Elisp does have a few places where it shines, though. One of them is the cl (Common Lisp emulation) package, which provides a whole bunch of goodies that make Elisp actually usable for real work. Defstruct and the loop macro are especially noteworthy standouts.

Some programmers are still operating under the (ancient? legacy?) assumption that the cl package is somehow deprecated or distasteful or something. They're just being silly; don't listen to them. Practicality should be the ONLY consideration.

The cl package wouldn't have been possible without macros. JavaScript has no macros, so even though it has better support for lambdas, closures, and even (in some versions) continuations, there are still copy/paste compression problems you can't solve in JavaScript.

Emacs Lisp has defmacro, which makes up for a LOT of its deficiencies. However, it really only has one flavor. Ideally, at the very least, it should support reader macros. The Emacs documentation says they were left out because they felt it wasn't worth it. Who are they to make the call? It's the users who need them. Implementer convenience is a pretty lame metric for deciding whether to support a feature, especially after 20 years of people asking for it.

Elisp is s-expression based, which is a mixed bag. It has some advantages, no question. However, it fares poorly in two very common domains: object property access, and algebraic expressions.

JavaScript is NOT s-expression based (or it wouldn't be a successful language, many would argue), but it does offer some of the benefits of s-expressions. JSON is one such benefit. JavaScript's declarative object literals (or as a Lisp person would say, "syntax for hashes") and arrays provide a powerful mechanism for designing and walking your own declarative data structures.

JavaScript also has all the usual (which is to say, expected) support for algebraic operators. And unlike Java, JavaScript even got the precedence right, so it's not full of redundant parentheses.

Overall Comparison

In the end, it comes down to personal choice. I've now written at least 30,000 lines of serious code in both Emacs Lisp and JavaScript, which pales next to the 750,000 or so lines of Java I've crapped out, and doesn't even compare to the amount of C, Python, assembly language or other stuff I've written.

But 30,000 lines is a pretty good hunk of code for getting to know a language. Especially if you're writing an interpreter for one language in another language: you wind up knowing both better than you ever wanted to know them.

And I prefer JavaScript over Emacs Lisp.

That said, I suspect I would probably prefer Clojure over Rhino, if I ever get a chance to sit down with the durn thing and use it, so it's not so much "JavaScript vs. Lisp" as it is vs. Emacs Lisp.

I would love to see Emacs Lisp get reader macros, closures, some namespace support, and the ability to install your own print functions. This reasonably small set of features would be a huge step in usability.

However, for the nonce I'm focusing on JavaScript. I've found that JavaScript is a language that smart people like. It's weird, but I keep meeting really really smart people, folks who (unlike me) are actually intelligent, and they like JavaScript. They're always a little defensive about it, and almost a little embarrassed to admit it. But they think of it as an elegant, powerful, slightly flawed but quite enjoyable little language.

I tell ya: if you're a programming language, it's a very good thing to have smart people liking you.

It doesn't make me smart, but I kinda like it too. Even though there's (still) a lot of hype these days about Java, and people tootling on about how Java's going to be the next big Web language... I just don't see it happening. There are too many smart people out there who like JavaScript.

So enjoy the interpreter. Ejacs is just a toy, but I think it also shows a kind of promise. Scripting Emacs using JavaScript (if anyone ever actually implements it) could be really interesting. It could open up the world's most powerful, advanced editing environment to millions of people. Neat.

In the meantime, it doesn't actually do squat except interpret EcmaScript in a little isolated console, so don't get your hopes up.

Reminder — here's the Ejacs URL: http://code.google.com/p/ejacs - enjoy!

And with that, I'm off to find some Nuka-Cola Quantum. I just wish those bastards hadn't capped me at level 20.

Tuesday, October 28, 2008

A programmer's view of the Universe, part 1: The fish

I write a column for computer programmers called "Stevey's Blog Rants." It's basically a magazine column — I publish to it about once a month. The average length of my articles is about 12 pages, although they can range anywhere from 4 to 40 pages, depending on how I'm feeling. But for precedent, don't think blogs: think of Reader's Digest. The blog format sets the wrong expectations.

Hence, some people complain that my articles are too long. Others complain that I have not given my arguments sufficient exposition, and that my articles are in fact too short on detail to warrant any credibility. This is a lose-lose situation for me, but I keep at it nonetheless because I enjoy writing. Even if nobody were to read my blog, the act of writing things down helps me think more clearly, and it's engaging in the same way that solving a Sudoku puzzle is engaging.

You should try it yourself. All it takes is a little practice.

My blog topics vary widely, and sometimes I even venture outside the realm of programming. Programming is where I'm most comfortable, and it's also where people seem to ascribe to me some level of punditry: I'm not necessarily right, but even my greatest detractors grudgingly admit that I'm entitled to an opinion, by virtue of my having spent twenty years hacking day and night without any sign of wanting to give it up and turn into a pointy-haired manager.

Even though I love both programming and to a lesser extent writing about it, there are also lots of non-programming topics I'd like to write about. Being a career programmer gives you an interesting perspective on issues not directly related to programming. You start to see parallels. So maybe I'll branch out some more and see how it goes.

The programmer's view

The first thing you notice as a programmer is that it trains you — forces you, really — to think in a disciplined way about complex logic problems. It also gives you a big booster shot of confidence around problem-solving in general. Junior programmers tend to have very high opinions of themselves; I was no exception.

In time, though, programming eventually humbles you, because it shows you the limits of your reasoning ability in ways that few other activities can match. Eventually every programmer becomes entangled in a system that is overwhelming in its complexity. As we grow in our abilities as programmers we learn to tackle increasingly complex systems. But every human programmer has limits, and some systems are just too hard to grapple with.

When this happens, we usually don't blame ourselves, nor think any less of ourselves. Instead we claim that it's someone else's fault, and it just needs a rewrite to help manage the complexity. In many cases this is even true.

Over time, our worldwide computer-programming community has discovered or invented better and better ways ways to organize programs and systems. We've managed to increase their functionality while keeping the complexity under control.

But even with such controls in place, systems can occasionally get out of hand. And sometimes they even need to be abandoned altogether, like a dog that's gone rabid. No matter how much time and love you've put into such systems, there's no fixing them.

Abandoning a system is a time of grieving for the folks who've worked on it. Such systems are like family.

And there's a disturbing lesson at the tail end of such experiences. The scary thing is that it's very easy, as a programmer standing at the precipice of complexity, to envision systems that are orders of magnitude more complex, millions of times more complex, even unimaginably more complex.

In the end, programming shows us how small we are.

The fish's view

Long ago, I used to have a Siamese fighting fish, also known as a Betta splendens, or simply a "betta". You can buy these fish at almost any pet store. I kept my betta, who was a deep vibrant red, in a pretty little 15-gallon tank decorated with a resplendence of real freshwater plants. And for a while I think my betta was happy there.

Like many Americans, I went through a phase in which I kept and ultimately killed many, many tropical fish. I didn't kill them intentionally; I wanted them to live and thrive. But keeping them alive for long is a challenge when you don't live in the tropics. So they might live for a few months or maybe a year, but they would always die prematurely. It was sad, and eventually I could no longer bear it, so I stopped keeping them.

Of all my fish, my betta left the biggest impression on me. The betta is a remarkable fish in several ways. For one thing, bettas are physically beautiful, and when they are at full display, their fins expand, peacock-like, into a fluid rose shape that is undeniably dramatic.

Bettas are also remarkable because they fight. They do not fight to establish a pecking order, as other fish do; they fight to kill. The males display their fins and then fight whenever they see another male betta, or even their own reflection, so they have to be kept alone and away from mirrors.

But bettas, I think, are most remarkable for their intelligence. Of all of the hundreds of tropical fish I kept, only bettas displayed anything resembling intellectual curiosity.

This really makes bettas some of the saddest stories in the tropical fish industry. Like other hobbyist fishes, they are stolen from their natural habitat and shipped overseas, or at best farmed in unsavory conditions. But unlike most other fish, bettas are also dyed to enhance their color. They are generally housed in tiny fist-sized bowls because of their ability to breathe air when necessary. And they are bred to express their fighting genes, and are often made to fight by their owners. Whereas other fish are kidnapped and sold, bettas are abused.

But worst of all, I believe their high intelligence endows them with greater capacity for suffering than other fish species. They can suffer physically and emotionally, but as we will see shortly, they can also suffer intellectually.

So bettas are a sad story.

My betta

Here is the specific sad story of my betta, the fish that left such an impression on me.

I had taken to lying on my bed and watching my betta for an hour or longer. The betta was the sole occupant of the tank in my bedroom. I had filled the tank with plants and copious natural light, so the effect was calming and serene. At times I almost envied the betta for the nice home I'd made for him.

One day, after the betta had been in his new home for several weeks, I found him exploring. It was a most unusual exploration, and one that I will never forget.

For the first few weeks, the betta explored the way you would expect any reasonably intelligent fish to go about the task. For the first few days he swam around to every nook and cranny of the tank, to make sure he had the lay of the land. Then for a few more days he experimented with staying put in different locations to see how he liked their feel.

Just like people, most fish will soon find a spot or a path they like best, and they'll stay in that spot or on that path for the rest of their lives.

But my betta was different. After his initial explorations he became restless. I'm no Fish Whisperer, but I could tell that he was restless. You would have thought so too. The betta started spending most of his time looking out of the tank, examining my bedroom. And he was clearly looking at specific things in the bedroom, not just "out there" in general. He would periodically swim around looking mildly agitated. He was acting like he wanted out.

I did everything I could to placate him. I experimented with different fish foods. I changed the water weekly and monitored it carefully to keep its temperature and pH within acceptable ranges. I added more lights. I added more plants. I rearranged the plants. In desperation, I even added a little castle.

Every time I tried something new, it would pique his interest for a little while. But in time, and faster each time than before, he would revert to his state of restlessness.

I'd never seen quite this behavior in other fish, so already he was demonstrating what seemed to be above average intelligence.

And then one day I found him engaged in an exploration that was altogether new.

He wasn't exploring the tank. He'd already investigated its topology for weeks. This time, he was exploring the nature of the tank. That's what caught my attention, and not just for that day, but for the rest of my life.

There was a twenty-inch vine in the tank that extended from the lower left back corner to the upper right front corner, along the diagonal of the main volume of the tank. The vine belonged to one of the many plants I'd put in there in the hopes of making it feel more like the Mekong river basin and less like a plexiglass tank in Seattle, Washington.

The betta had his nose on the vine. He was floating just above it, twitching his fins slightly to stay in place, and he was keeping his eyes as close to the vine as possible while keeping it in focus: about half a centimeter to a centimeter. And he was traversing the vine.

With the tiniest of motions, he was propelling himself along the vine towards the lower back right corner, keeping it under close scrutiny at all times. This excursion, from the halfway point to the end of the vine, took him perhaps three minutes. He was taking his time.

When he got to the end of the vine, he remained rooted in place while he inspected the 3-inch-radius spherical volume at the end of the vine, which was truncated in three dimensions by the walls and floor of the tank. He spent about three or four minutes doing this inspection, evidently making sure the vine really did terminate in the corner, and did not escape the tank.

After he had thoroughly scrutinized everything in the betta-sized vicinity of the vine's end, he turned back to the vine, nose pressed close, and began working his way along the vine in the other direction.

At this point I sat down to watch, because if he was doing what I thought he might be doing, then... I didn't know what to think. I wanted to see it for myself.

Over the next seven to ten minutes, he crept along the vine, never losing sight of it nor getting further than a centimeter from it, until he reached the upper-right front corner of the tank. He then proceeded to repeat his inspection of the volume at vine's end, ensuring himself that the vine terminated in the tank rather than protruding beyond the wall.

But what if he had missed something?

Sure enough, he turned and looked down the length of the vine for a time. And then he put his nose back on the vine and began again his long descent to the other end.

He did this for five days.

By the second day my amazement had turned to concern, and by the third day I felt utterly helpless. Here was an intelligent prisoner, my captive, exploring the mechanics of his prison with a thoroughness that only the imprisoned can afford, looking for an escape with deathly tenacity.

But while purchasing my betta had been easy enough, returning him to his real home would be unthinkably difficult, and probably unsuccessful even if I'd tried. Returning him to the fish store seemed like a dead end; he could easily wind up worse off than he was now.

So I concluded that there was nothing I could do. As he inspected the vine, I bit my nails, and timed passed in silence.

After the fifth day he gave up. And then he did something that I still don't understand, even though I've heard about this kind of thing before, and even though I personally saw him do it: he died of unhappiness.

It only took him a few days. He refused his food, he stopped moving, and to all external appearances he had become ill. But I knew better.

The lesson

Whenever I find myself struggling against the tide of massive system complexity, I think of my betta. He had a big heart, a small brain, and a small range of sensory input. I watched him use them all as methodically as any programmer to reason his way through to a soundness proof of the inescapability of his prison.

We like to think of ourselves as being pretty smart. Admit it. We do. But in the grand scheme of things we're intellectually little better off than that fish. We can easily find problems so complex that reasoning about them can take days or weeks of microscopic scrutiny, like my fish swimming along his vine.

And we can just as easily envision problems thousands or millions of times more complex: problems beyond the reasoning abilities of any person, any group of people, or even our entire species.

This has ramifications for the way we think about things today.

I believe I will have more to say about this soon. Right now I need to go mourn my fish, whose soul shone as brightly as that of anyone I've known.