All articles, tagged with “the biz never sleeps”

how you can tell that amazon put IMDB on autopilot a decade ago

And while I’m snarking about Fortune 500 companies unable to pull their style out of 1997, allow me to quickly demonstrate that Amazon hasn’t paid anyone to look at IMDB in many, many years:

Daddy, what’s a newsgroup?”  “Well son, it’s better to regret something you have done than something you haven’t done.  And if you see your mom…”

why it’s good to regularly audit your boilerplate

I was going to cook up a nice long stemwinder about my misadventures on American Airlines this weekend, and about how I really should have learned by now that any money “saved” by not flying JetBlue, Virgin or Southwest is a false economy, but nevermind all that; the intertubes are clogged with angry airplane consumer stories, and will only get moreso as the dread holiday season comes upon us.

Instead, I’m going to reprint a single sentence from AA’s online check-in confirmation email.  Pay close attention:

“There are Adobe Reader versions for MS Windows, MS Windows NT, OS/2, Macintosh, and Unix platforms.”

American Airlines: boldly supporting 1997’s state of the art… in 2009.

(For the curious or confused: The products known as “Windows 2000”, “Windows XP”, “Windows Vista” and “Windows 7” are all lineal descendants of Windows NT, which was Microsoft’s next-generation OS product from 1993.  There hasn’t been a version of Windows that was not based on NT since the death of “Windows ME” in 2001.  “OS/2” was a competitor to Windows NT that was marketed by IBM in the mid-90s: you haven’t heard of it because it was bought by, statistically speaking, nobody.)

here, let me save you 200 dollars

Long story short here:

If you own an iPhone, Apple’s warranty will often be very liberally applied by Genius Bar employees if you’re nice to them…

…unless the submersion sensor has been tripped.

What submersion sensor, Nathan?”

Well, funny you should ask.  There are actually two submersion sensors inside an iPhone: one just inside the iphone jack port (the wide narrow thing on the bottom that you plug the USB cable into), and one inside the headphone jack.  To see them, you need a magnifying glass and a flashlight, much like your doctor uses to look into your ear canal.  The sensors are just little strips of moisture-sensitive paper: if they get wet they turn bright red.  (Pictures here.)

Here’s the catch: you don’t need to drop your iphone into a toilet bowl (or fishtank, or river) in order to trip the sensor.  In fact, you don’t need to submerse it at all.  Just a few drops of water that creep in at an unlucky angle are enough to do it.  If that happens, your warranty is null and void, and no amount of sweet-talking will change it: Apple will refuse the return of the phone, and they will charge you two hundred dollars for a refurbished replacement.

You can buy a little silicone insert for the iphone jack for $6 from various websites.  You may wish to consider doing so.

Why yes, I do speak from highly annoyed experience here, why do you ask?

things I was not expecting to do today at work


1 — costar in a music video with Amanda Palmer:

Had I known, I might have shaved this morning.

Better pictures forthcoming, I hope. Also forthcoming: a whole frickin’ music video, on youtube.

bring back ma bell

9:10am — get an SMS message on my Treo, explaining that there is a new Critical Firmware Update for my model Treo on Verizon Wireless, and that I must install it immediately in order for billing to continue to function directly. Oh god. The last time I tried to do an firmware update on my Treo, it took most of a day. Fuck this noise: I already knew I was going to get an iPhone, may as well bite the bullet.

Arrive Apple Store Palo Alto 11:10am. Get in line. A few minutes later, a perky young woman in an orange t-shirt comes up, asks me what model iPhone I want, hands me a ticket for that model, and explains that I can wait in line (estimated 45 to 90 minutes) or come back later. I ask if I’ll still have to wait in line if I come back later: in fact I will. Fine, I will wait: I’ve got a zipcar until 1:30pm, and I’m parked in a 2-hour space.

11:55am — arrive at the head of the line. Not bad, not bad. A chipper young man takes my ticket, leads me into the store and starts asking me about what kind of plan I want and do I already have an AT&T account? I’ve got my Verizon account info in hand and know exactly which plan I’m signing up for: he’s happy, this is going to go quickly.

12:30pm — four different attempts to port my old cell phone number have failed. Each attempt requires starting the entire process over from scratch, including running my credit card through for a credit check and having me manually input my SSN into the chipper young man’s handheld. Each attempt to run my credit card through the handheld takes 5 minutes of frantically swiping in hopes that this time it will actually read the magnetic strip. Finally after the fourth try, the now less-chipper young man suggests that maybe we can set up my account with a new number, and then I can walk into an AT&T store and have them do the port there. He says he’s sorry about this, and that usually he recommend as little contact with AT&T as possible. I assent.

12:40pm — the now strongly aggravated young man goes in search of a new handheld unit, as his one is no longer even pretending to be willing to read any of my credit cards.

12:50pm — the first attempt to register the account with a new number has failed. He calls AT&T, tells them the magic words, and they tell him that it’s because he entered my billing address as “Box 12345” instead of “P.O. Box 12345” and that the system doesn’t support this. He relates this information to me with an expression of incredulity. A few moments later, he gets cut off from AT&T and has to re-dial them. As he is attempting to re-establish contact with someone with a clue at AT&T support the session on his handheld expires and the unit resets.

1:05pm — I am starting to get a little nervous about my borrowed car and its parking space

1:20pm — After two more tries, with an AT&T rep on the phone with him, he succeeds in activating a number! I’m done, right?

1:21pm — No, apparently the phone has to be connected to a computer running iTunes, right now, in the store, and activated. I call zipcar and let them know that I’ll be returning the car late.

1:31pm — I leave the store, clutching my new phone and a receipt.

1:45pm — at work, at my desk, I realize that they gave me someone else’s receipt, unless my name has recently changed to “Matthias Vaska” without my knowledge. Call Apple Store.

1:55pm — finally make my way through the voicemail tree and hold queue. Am informed by a different chipper young man that they can’t email me my receipt, but if I come back to the store any time in the next few days they can reprint it. Great.

2:00pm — Figuring that I may as well get all of my time-wastage done at once, call AT&T, am transferred to the porting department.

2:25pm — Am told the following:

1. In order to port my old 718 (New York City) area-code number, I will need to have the operating area of the phone temporarily changed back to new york city.
2. To do this, they will need a NYC address to associate with the account, even if it’s never used for billing.
3. Then they will need to issue a new SIM card with a temporary NYC number and install it into the phone.
4. Once the new SIM card is installed, THEN they can port my Verizon number.
5. …and since this requires physically changing SIM cards, the only way to do this is to personally walk the phone into an AT&T corporate store.

Best retail experience EVAR!

I’m sure more hilarity will ensue once I actually make it to an AT&T store. Stay tuned!

nyc contract sysadmin?

Hey, are any of my NYC-area geek friends interested in a 2-week contract sysadmin gig? The client is a friend and former boss of mine, and they need someone ASAP. Drop me a line and I’ll put you in touch.

today’s unsolicited and slightly decontextualized advice

If you ever get the chance to acquire founders’ stock in a company, take it.

“chance” implicitly including hopefully obvious baseline requirements like “can still pay for food and rent”

macs, radeons, blank/black boot screens and you

The following is 100% geek-speak, and I’m posting here because it’s the solution to a problem that bedeviled me for several days, and as far as I can tell the method is not to be found anywhere via the expected search terms. Hopefully this post will fix that.

Late last year, for the first time in about a decade, I bought a completely new-in-the-box computer: a Mac Pro. (A purchase that was, thankfully, paid for by people who were not me — an amusing story for another time.) It is, hands-down, the best computer I have ever owned. But there was, of course, one small crazy-making problem:

When connected to my Dell LCD monitor (a 2001fp: still quite the respectable little beast, and connected via DVI, so you’d think that the computer and monitor would be able to figure this stuff out), the first-stage boot screen (dark grey apple, light grey background, spinner) did not display. The screen stayed entirely black until it got to the second-stage screen (“Welcome to OSX”, blue background, blue progress meter) and the login window. Sometimes, if the moon was in the right phase, the Dell would display a “can not display this mode” error, but usually it would just stay black.

Now this would have merely been an aesthetic oddity, except that the video mode that the monitor wouldn’t display is the same video mode used for several actually important things, ie:

1. verbose boot: all black
2. single-user boot: black on black
3. Boot Camp selection screen: none more black

The last one, as you can imagine, is a bit of a downer, since the system will happily pause endlessly mid-boot waiting for you to pick a boot volume, which is hard to do when you can’t see anything. (I got pretty good at doing this by keyboard and intuition, but it’s not a skill I’m happy to have picked up.)

Digging around on google suggested that the problem was endemic on older Dell LCD monitors, and more likely to happen to Mac Pros with the ATI Radeon graphics card installed, but I wouldn’t claim either as a 100% certainty.

Well, it took me a while, but I finally sussed it out. The fix for this is documented precisely nowhere on Apple’s site, and the only hints I found of it were in the archives of a bunch of sites dedicated to hacking OSX to run on non-Apple hardware. Even better, not one but two things were wrong, and it took me a while to tease them both out.

First: the DVI cable that came with the Dell monitor, and which I was still using many years later, was a “single-link” DVI cable. This, apparently, is a no-no: the first stage boot screens require a dual-link cable to display on this monitor. Why? I have no idea.

Second, the video card and the monitor were failing to handshake properly. It turns out that you can manually specify the resolution and refresh mode of the first-stage boot process in the /Library/Preferences/SystemConfiguration/ file, like so:

<?xml version=”1.0” encoding=”UTF-8”?>
<!DOCTYPE plist PUBLIC “-//Apple Computer//DTD PLIST 1.0//EN” “”>
<plist version=”1.0”>
     <key>Kernel Flags</key>
     <key>Boot Graphics</key>
     <key>Graphics Mode</key>
Pick a resolution and refresh rate that you’re confident your monitor can support, save the file (you’ll have to be root to edit it, of course) and reboot. Et voila, the boot screen is back.

Obvious warning: playing with system boot parameters is a great way to turn your computer into a large, expensive, electricity-sucking brick. Take a full backup before you attempt any such surgery, and please don’t email me for support.

qmail considered harmful in default configuration

First, a slightly belated update: the spam flood to my server has been stopped in its tracks. Email and web services are back to normal, for localized values of ‘normal.’

Now, a small statistic, followed by a rant. This is high geekery; everyone not interested in hearing me gripe about mail servers should probably skip it.

Since this Thursday afternoon (when I finalized the new configuration), there have been 23,540 attempt to deliver mail to my mail server that were not pre-blocked by the or antispam blacklists. Of those attempts, 19,166 of them were attempts to deliver mail to addresses which do not exist. Either they were “dictionary spam” (in which the spammer goes through a list of common first and last names, and attempts to deliver spam to every possible, or “bounceback spam”, in which the spammer invents a fake address at someone else’s domain and forges that as the sender of the spam, so that any complaints or bounces go to the forged address instead of the spammer.

For those of you without a pocket calculator in the audience, that means that 81% of the emails sent to my mail server were to addresses that never existed.

This would merely be cause for hilarity, except that a design flaw in qmail, for years my mail server of choice, turned it into a complete debacle.

You see, the stock distribution of qmail does not validate the user portion of an email address until after the mail is accepted. In plainer English, that means that if you send mail to, and I’m running qmail on’s mail server, qmail will happily accept that mail even if there is no such user as “bob”. Even if that mail is spam. Even if that mail contains a 300k image file. Even if that mail is spam and contains a 300k image file. And then, once qmail finally realizes that there is no Bob, qmail generates a new piece of email, a bounce message destined for the original sender, letting them know that the message wasn’t delivered, so not only have we spent time, disk and network bandwidth accepting a message that we never wanted in the first place, but we’re then going to spend MORE resources sending out an alert, quite probably to a sender address that was forged by a spammer in the first place.

Multiply that by thousands to millions of bogus messages a day and you have a problem. Add on top the CPU and disk utilization required by any minimally-responsible set of spam- and virus-filtering tools (spamassassin, crm114, clamav, etc) and you have a disaster. I run a very small mail server — a handful of domains, less than 20 users, less than 10 mailing lists — and the increased system and network load caused by accepting all of that bogus mail essentially took me off the air for a week, and came perilously close to blowing my ISP’s network quota and thus costing me quite a bit of money.

This was a defensible design decision in 1996, when you couldn’t sneeze without uncovering a new buffer overrun in sendmail, and massive email spam was a distant thundercloud on the horizon. Today? Not so much.

Luckily, this is a problem that has been solved a few times now. The qmail-validrcptto patch is probably the simplest way for most qmail administrators; if you’ve got some horribly abstruse setup going, it might be easier to use the qmail-spp plugin patch and write your own validator for the RCPT stage. But whatever you do, you should fix the problem: if you’re running a stock qmail or netqmail right now, you are acting as a spam amplifier whether you realize it or not.

fearful symmetry

Well, it’s finally happened. The sheer amount of spam that my mail server gets has reached the point where not only is the spam itself nearly overwhelming my network bandwidth, but all of the clever anti-spam crap I run on the server is crushing the CPU.

I’m working on a New Clever Trick to deal with this, but mail to and from my domains (,, etc) is going to be kinda crappily slow for the next few days at least, as will access to any of the websites hosted there.

Even better, Gmail has apparently decided that my server is a spam source, so all mail from my users to gmail accounts is getting tagged as spam. Luckily, I can personally harrasss the anti-spam team there, so that might get resolved sooner.

Remember: kneecapping spammers is the holiday gift that keeps on giving.