Amen, Brother.

I Want A New Platform

One that won’t go away (under load), one that doesn’t keep me up all night (worrying about scaling), one that won’t make me sleep all day (when I should be adding features). If you have tried to build an Internet site or application recently that needs to work for thousands or tens of thousands of concurrent users you may share this desire.

Why is this still so hard? Why do we find ourselves worrying about locating experts in the dark art of database performance tuning? Why are we spending time haggling with Rackspace over the price for another set of servers or racking our own servers at Equinix? Why are we writing our own user management (from scratch)?

Because we are using tools that simply were not made for the job. Relational databases have gotten faster and better, but their fundamental construct of data stored in rows and neatly parceled out across tables (which frequently need to be joined) does not really match up with either rapid development using some notion of objects or with scaling horizontally using commodity hardware. Web servers started out by simply providing static content and were then forced into running applications with the result of not doing either particularly well. Yes, people have been writing new ones to address that, but even those are fundamentally designed to work on a single machine. The second you go beyond one machine you need separate load balancers, reverse proxies, caches and all sorts of other paraphernalia just to make stuff work together. Worse yet, the web server, the application and the database are connected to each other via thin straws that were bolted on after the fact with important information either not passed at all or only painfully (e.g., information about which user is requesting a particular set of data).

What about the growing cast of frameworks, like Ruby on Rails, CakePHP, Django, etc? Don’t they provide a solution to these issues? Not really. These frameworks are for the most part about enabling rapid development. They do great on the add features rapidly front but often make matters worse on the scaling front. The reason is that they layer more stuff on the very tools that are wrong for the job to begin with.

What then is needed? A platform that is created from the ground up to enable modern web sites and applications. What would such a platform look like? It would be hosted and (nearly) infinitely scaleable. It would provide object storage that’s as simple as saying “here’s an object, store it” (you get back a handle, ideally, if you want a human readable, search engine optimized one). Later on, retrieval should be something like – “here’s a handle, give me back the object” (with full user level access control baked in). Stored stuff should be easily indexed so that one could say “give me back all the handles for objects that match this pattern” (and to which the user has access). The same should work for media: “Here’s a picture, store it for me with this metadata” and “Find all the pictures for me tagged x.”

Aside from storage there are other useful services the new platform should come with, since essentially every modern web site / application needs them, such user authentication, authorization and access control. Flexible processing of pretty URLs. Easy creation and maintenance of page templates. Ability to send emails and process bounces. Handling of RSS feeds (inbound and outbound). Support for mobile access and possibly even voice capabilities.

Code would run inside the platform (this is what Marc Andreessen calls a Level 3 platform) but it would not be cut off from the outside world. It would have full access to other services that live on the net via web services. So, for instance, there would be no need for the platform to have its own payment service.

Let me try to head off some potential objections.

Can such a platform be built at all? Won’t there be problems with distributed transactions (or some other technical objection)? Nobody said it would be easy (besides, if it were it wouldn’t be half the fun or represent a big opportunity). But there is no fundamental impossibility here, especially in a hosted environment and keeping in mind that the goal is not to create a platform on which anything can scale automagically, just a large set of web sites and applications that are currently hand crafted. For instance, the object storage magic does not have to work for arbitrarily complex objects. My guess is that the vast majority of sites and applications can live with (fairly) shallow objects based on just a few underlying types.

Isn’t this just a fancy way of saying you want a hosted object-oriented database and didn’t those fail miserably as software products? Just because stuff failed in the past doesn’t mean it won’t work now, especially when circumstances change (the current ugly picture in the credit markets is a reminder that the opposite is equally true).

Won’t folks be completely locked into this platform? We are not really worried about being locked into MySQL, because we feel that if push came to shove, if we really had to, we could switch to say Postgres or SQL Server or Oracle. This of course is a lot harder to do in practice because in the quest for performance we often resort to using custom SQL extensions that are specific to our current DB. But it does point to standards as an important way to mitigate lock in and anybody providing a platform like this would do well by working with existing standards where available and promoting new ones where not (e.g. an object storage standard). A more dramatic way to mitigate lock in might be to offer an open source “community” version of the platform and compete based on hosting the most reliable version.

Hasn’t Amazon has already built this? Isn’t this S3 and EC2? S3 and EC2 and the upcoming database service are steps in the right direction. But they are not nearly as easy to program as the “new” platform (in particular EC2 is really just high speed provisioning of traditional servers). Amazon could, however, use their pieces to offer something like the new platform.

I am sure there are other objections (look forward to the comments), but those aside, where might this new platform come from? Marc Andreessen lists companies who are working on it now, including his own Ning, Salesforce and Amazon. I would not be surprised if folks at Google were also thinking about offering something. Based on their recent announcements, Salesforce seems furthest along in offering a new platform but they have made many idiosyncratic choices which result in high lock in (not to mention that they run an applications business which might compete). Despite the big guns already working on this, I believe there is still an opportunity for someone new to build the new platform. It’s the kind of disruptive change that requires throwing overboard a lot of existing assumptions and conventional wisdom and trying something radically different. It will also require maniacal focus and independence from the needs and influences of existing operations. Sounds like a startup to me.

Can You Guess Which Organization This Is?

Can you imagine working for a company that has a little more than 500 employees and has the following statistics:

• 29 have been accused of spousal abuse
• 7 have been arrested for fraud
• 19 have been accused of writing bad checks
• 117 have directly or indirectly bankrupted at least 2 businesses
• 3 have done time for assault
• 71 cannot get a credit card due to bad credit
• 14 have been arrested on drug-related charges
• 8 have been arrested for shoplifting
• 21 are currently defendants in lawsuits
• 84 have been arrested for drunk driving in the last year

Can you guess which organization this is?

Give up yet?

It’s the 535 members of the United States Congress. The same group that cranks out hundreds of new laws each year designed to keep the rest of us in line.

Anthony Bourdain’s Overrated Menu

“When the water sommelier comes over, I reach for my gun.”

Anthony Bourdain’s menu of overrated, trendy items.

Circular Reasoning Works Because

Cultural Map Of The World

This map reflects the fact that a large number of basic values are closely correlated; they can be depicted in just major two dimensions of cross-cultural variation.

The Science Of Gangsta Rap

BEEP BEEP. Honk if you are horny.

(view the full thread at The Something Awful Forums)

Ball Four By Jim Bouton

Ball Four is a good and important book to own. If one mark of a good book is its ability to provoke reactions, and often contradictory reactions, then Ball Four is a fine book.

What really makes the book a worthwhile read is the way that it reflects a time of momentous change in baseball. I’m not sure if younger fans can truly appreciate how rapidly things were changing in baseball back then. America experienced a broader social revolution throughout 1967 and 1968, but it really took until 1969 for it to work fully its way into baseball. If you look at the baseball cards from 1967 and 1968, they’re bland grey items whereupon the players all look like crew-cutted astronauts. Come 1970, 1971, and beyond, everything is different: sideburns, Afros, wild psychadelic colors. Ball Four came out at the leading edge of these changes, and it captured festering tensions between baseball’s old guard and a skeptical generation of young players.

Ball Four is widely hailed as a great classic. It’s not quite as pathbreaking as its reputation, however. For one thing, it had a predecessor earlier in the decade, The Long Season, by Jim Brosnan, another “kiss and tell” book written by an active player. Brosnan’s previous book is better written and more insightful. Ball Four created more of a sensation, but mostly because it was slummier — it revels a bit more in the drinking and carousing than does the previous book. Because of this, Ball Four upset the baseball establishment a bit more, and it titillated young readers a bit more. Other aspects of the book were equally shocking (back then, anyway): for example, the portrayal of many authority figures — coaches, managers, and baseball executives — as dunderheads. This was an anti-establishment book in many respects.

Baseball was changing on all fronts, and these changes are well reflected in this book. Bouton pitches for the Seattle Pilots, in their first and last year of their existence, and the first season of the newly created league divisions. You also read of the attempts of baseball players to create a union, and the divisions among players this caused. You’ve also got the new turf parks, such as the Houston Astrodome where Bouton finishes up his season. And there are all of the social changes: the sexual revolution, the hashing out of racial issues, and perhaps first and foremost, the generation gap.

Bouton captures all this and more. Having said all that, my enjoyment of this book is limited by the fact that Bouton’s own perspective is often arrogant and intolerant, in much the same way that he derides the older coaches and managers as being. You get the clear sense while reading him that the 1960s generational wars were caused not only by an older generation stuck in its ways, but equally by a younger generation that assumed it was automatically right and that they had nothing to learn from anybody. For example, Bouton persistently quotes his managers and coaches only to show how stupid they are. Now, there is such a thing as stupidity among the old, but all rebellious kids usually think that the older generation has missed a beat. Sometimes they’re right, and sometimes not. Bouton’s always convinced he’s right, but there’s little reason to believe he always is.

A typical battle between Bouton and his pitching coach Sal Maglie concerns Bouton’s attempt to survive on the knuckleball. Sal Maglie gets on his case about it, and discourages him from throwing the knuckler exclusively. Bouton is convinced that his other stuff is basically gone, and the only way he’s going to hang on is if he relies on the knuckler, and he wants to be left alone to concentrate on that pitch.

Let’s just examine this from both sides for a moment to get a sense of whether Bouton’s contempt for Maglie is justified.

When Bouton came up, he was a very successful pitcher with the New York Yankees. But although he had a reputation for having a young, live arm, the stats show that he was never really an overpowering pitcher. In 1964, he struck out only 120 men in 271 innings, while winning 18 games. Historically, pitchers don’t get by on finesse like that for very long. It’s not at all surprising that a few years later, Bouton no longer had hard enough stuff to get major league hitters out. Bouton’s self-assessment in Ball Four seems to be justified: he probably doesn’t have a good enough fastball or slider by 1969 to make it as a major league pitcher, and unless he gets the knuckler to work, that’s it for him.

Now let’s look on the other side. If there were ever a guy who knew something about such a situation, it was Sal Maglie. Maglie’s emergence as a good major league pitcher was delayed by his “outlaw” years in the Mexican league. Like Bouton, Maglie wasn’t overpowering — in his finest year he struck out only 146 in 298 innings. But unlike Bouton, Maglie was very successful in his 30s. When Maglie tells Bouton that a key ingredient for success of an older pitcher is not walking too many hitters, he’s onto something. Bouton gave up 12 gopher balls in only 92 innings with Seattle in 1969, and if you’re that vulnerable to the gopher, you’ve got to keep the walks down. Most importantly, Maglie had accomplished what Bouton was trying to do — not with overpowering physical gifts, but by assessing his own situation accurately; there was something to learn there.

The point is not that Bouton is right or Maglie is right, but that there are two good perspectives here, and if Bouton weren’t so full of himself, he might be able to pick up what good things Maglie had to offer him, combining them with his own valid insights. Instead, Bouton spends the whole book making fun of Maglie and anyone else in a position of authority, refusing to learn anything from anyone older.

Bouton is a hero to everyone who has ever been fed up with their teachers, their boss, or “the establishment” at large, because many readers find it cathartic to read someone’s rantings against stodgy authority figures. But in the final analysis, Bouton isn’t necessarily all that brighter or more insightful than those he critiques: he’s just as closed-minded, he just has a different opinion. He’s not Galileo; he’s not even Bill James. He’s just a guy speaking his mind, always candidly, often rudely, and only sometimes with a valid point.

Ball Four is a worthy read because there’s no other book quite like it; Bouton is always brutally honest about his feelings, and he conveys the full flavor of a turbulent era in baseball history. The book was considered sensational at the time, but it’s not such a prurient interest anymore: now it’s useful mostly to convey what all the fussing and fighting was about back then.

Greater Internet Dickwad Theory