isnoop.net blog

This morning I received a call from a gent with a Boston accent. He indicated that he represents a firm that is displeased with some data I’m using on isnoop.net. According to the caller, my theater listings page is using his client’s intellectual property and I’m not properly licensed to do so. The lawyer seemed nice enough. Perhaps I should have kept him on the phone longer so he could tick up some more billable hours…

Like some other things I’ve developed, theater listings was a simple service I wrote for myself to clean up an otherwise cluttered interface and make the data available in my favorite feed reader. Over the years, many people have written with questions and thanks regarding the page. Thank you to everyone who used the service. I hope you might find some of my other tools just as useful.

As of now, the theater listings page is closed. If you still want this information in your web browser, check out Google’s movie listings service. For you feed reader junkies, Yahoo Pipes is widely known as a useful service for turning any web page into an RSS feed.

I’ll investigate the possibility of re-sourcing the data, but don’t get your hopes up. Also, for those who are already firing up their email clients to ask me for the source code, hold your horses. I’ve been working up a post on ethical screen scraping and now I can finally share it without being hypocritical. I won’t share the source, but look forward to an interesting and useful guide to capturing and reusing data on the web, including some advice that should help prevent you from getting your own C&D.

Google Voice is a very interesting service. If you were one of the people (like myself) that got an account on GrandCentral.com before they were bought out by Google, you are now eligible to be part of the Google Voice beta.

It offers a lot of interesting services such as visual voicemail, speech to text, VOIP, free long distance, and many others. However, in order use most of these, you need to use the phone number Google assigns you. Google can’t be your voicemail provider unless all of your calls are routed through them first.

So are you going to hide your current cell phone number and tell all of your friends and family to call your GV number instead? Unlikely.
I believe it is much more likely that Google is actually moving to become a telephone service provider themselves. That way, you just transfer your phone number to Google and they give you all of the great features of GV along with it. However, in order to participate in LNP (the FCC program that enables users to transfer phone numbers between providers), they must become a wireless carrier.

I know it sounds unbelievable. I am somewhat skeptical myself. It seems like quite a stretch for them to actually get into voice service. After all, couldn’t Google just partner closely with the existing providers and integrate their GV directly into your existing plan? Unfortunately, cellular service providers would probably never play ball with Google this way. GV bundles free long distance VOIP, SMS, and (quite possibly) unlimited airtime.

Many people were skeptical when a search engine company was rumored to be branching into email. There was even more surprise as the rumors of a Google phone came true. Now that they have their own cell phone OS and a fantastic web integration platform, it is not inconceivable that they will take the next step and start leasing tower space.

Google is out to eat the telco’s lunch.

My Google maps making, RSS feed slinging, universal package tracker has moved to greener pastures. Boxoh.com is your new go-to place for tracking UPS, FedEx, USPS, DHL, and Airborne packages.

Backstory: In 2006, I posted a handy new utility I’d cobbled together which was a mashup between package tracking for for multiple services. It quickly became by far the most popular page on this site, with more than 1.4 million tracking requests last month. It gets more than three times the traffic of my movie theater RSS generator and four times the traffic of another spinoff site, FeedSifter, a simple RSS/Atom feed filter.

If you are familiar with MediaTemple’s GridServer service, you’ll know that using up all 1000 GPUs (server work units) for the past several months is not a good thing. Those cycles weren’t just going to waste on poorly written scripts, either. Each hit to the tracker consumed an average of 0.0002 GPU (WordPress uses 8 to 16 times that with each hit). It wasn’t always this way, though. Check back soon for an upcoming post on how I managed to cut down the CPU usage of the package tracker by 90% with some intelligent code analysis and a creative caching solution.

Boxoh.com is now hosted on a screaming VPS server with plenty of spare power. I’m taking full advantage of APC caching and several other behind-the-scenes tweaks one can only get a grip on when they are running a dedicated server.

Thanks to all of the people who have made the service so popular!

Also, thanks to Juplex for a fast and friendly site design!

Search Google for real estate. As of a recent Google update, you should see Google’s newest search integration feature. (If not, try refreshing. This appears to be in a testing phase right now.)

This new feature seems to go head-to-head with the most basic of features (mapping properties for sale) offered by real estate mapping guru site zillow.com and to a lesser degree redfin.com.

Once again, I have irritated the giant that is Google. It seems that I just can’t get on their good side. I just got this email from Scott in Google’s AdSense team:

Hello Ian,

While reviewing your account, we noticed that you are currently displaying Google ads in a manner that is not compliant with our policies. For instance, we found violations of AdSense policies on pages such as dontclickmyads.com.

Publishers are not permitted to bring unnatural attention to the ads on their sites in any way. We found that the site’s URL and page content directs unnatural attention to these ads.

As a result, we have disabled ad serving to the site.

Your AdSense account remains active. However, we strongly suggest that you take the time to review our program policies (https://www.google.com/adsense/policies) to ensure that all of your remaining pages are in compliance.

Please note that we may disable your account if further violations are found in the future.

Sincerely,

Scott
The Google AdSense Team

As it turns out, the website that I made in jest and then promptly forgot about after a prior AdSense policy reminder has once again gotten me in hot (or at least luke warm) water with Google.

I’m not out to fleece Google by funneling mass traffic to that one-joke site. I’m not even out to make a particularly bold statement about being kept down by the Google man. It sure would be nice if they got the joke, though. I doubt anyone made or lost much money off of the 108 visits I’ve had so far this month. Especially so considering two of the top four user agents were googlebot.

Oh well, I’ll just switch to AdBrite. It doesn’t matter that I’ll earn less per click. A lower percentage of zero doesn’t affect my bottom line much.

Just for the record, I do not condone any clicking of my ads. Don’t click them, don’t click near them, don’t look at them, and don’t think about them. I don’t even have ads!

So there.

I recently learned that my Gmail invite spooler was mentioned in a couple of books. The first is Google Search & Rescue for Dummies (pictured below) and the second is the French version of Google Hacks (as seen in Google Book Search).

Unfortunately for these publishers, the spooler was shut down just as the books started rolling off the presses. Oh well, I honestly appreciate their mention!

Judging by the lack of people yammering about this, I assume Google only recently rolled it out. When I first signed up for the Google Hosted Mail beta test in February, they were missing the ability to configure an email catch-all account. This was the one thing keeping me from making the switch.

Upon spotting the new feature in the admin panel, I copied all of my isnoop.net user accounts and added all seven of Google’s MX servers to my DNS. I immediately began receiving email to my fancy new mail hosting. The entire process took fifteen minutes.

I’d write up a review of the service, but if you’ve ever used Gmail, you know just how it works. The only apparent addition for hosted mail is the admin interface. Like many Google labs interfaces, this one is stark and contains just what you need to get the job done.

My only wish would be a means of copying all of the mail out of my original Gmail account into my primary Google Hosted Mail account. However, I wouldn’t hold it against them if they never gave me that ability. They’ve already done much more than I could have expected from a search engine company.

After last month’s Google AdWords Policy Reminder, I’ve decided to make it perfectly clear that you are by no means authorized to click my ads. You should avoid them at all costs, even to your own peril.

In case this isn’t clear, I have repurposed my magnetic words script to serve as entertainment on a new site dedicated to reminding you that you should not click my ads. Check it out:

http://dontclickmyads.com

If you’re a website owner, you might be wondering how to make the biggest buck for your bang online. Selling text ads is a proven technique of making money from a site. Two of the most popular ad-serving services are AdBrite and Google’s AdSense.

I still get quite a bit of traffic to the Gmail spooler, so I decided to split test the two ad services. Each was randomly displayed with the a 50% chance of showing up. I reformatted both services to look similar. Here are my results (click for un-squished version):

First, you’ll notice that Google’s AdSense got less traffic overall. My guess is that is because more people have blocked the AdSense script. Despite this, Google took in more clicks overall. I assume this is because people trust the “Ads by Google” text and the ads were more relevant.

The real surprise comes with the earnings. I had AdBrite ads running on the page during the height of the spooler’s popularity, and all I can do is wonder if that earnings differential would have held up under much greater traffic.

Nine months after closing down the Gmail Invite Spooler, the page remains one of the most popular landing pages on my site. Over the past several months, this page has averaged around 2,500 unique visitors a day. I’ll explain the arc of this wonderful service, but first I’d like to make one thing very clear:

Sorry, I do not have any Gmail invites. Please don’t ask me for Gmail invites. I am truly sorry that I cannot provide you with any. Please go to Google Mail for more information on how you may get your own account.

You may obtain an account without an invite these days. All you need is a cell phone.

Background
In 2004, Gmail was a very hot commodity. Since April 1st of that year, people were clamoring to get in on the exclusive beta of pre-IPO Google’s hottest new offering. In late summer, by the time I decided to write the spooler, Gmail invites were no longer selling for $100 or more on eBay, but there was a large amount of clutter on the internet with people asking for or offering invites.

The forums and blogs that I visited were littered with chatter about trading invites, but the givers and seekers didn’t seem to be coming together efficiently. It wasn’t uncommon to see multiple posts back to back asking for and offering invites.

My wife may not always like it, but when I see a problem, my mind immediately gets to work on a solution. This was one problem that I knew I could make a simple fix for in a matter of hours.

The First Incarnation
Throughout its lifetime, the basic workings of the page remained the same: People with gmail invites would send them to a specific email address. The spooler would then read those emails and store the invites in a database. Site visitors could come and claim available invites on a first-come, first-served basis. There was no backordering of invite requests. When demand exceeded supply, one had to wait until someone else donated some.

Originally the spooler was made solely for the use of the people on the forums I noticed suffered most from inefficient offers and requests. It was a very simple system that was only workable on a small scale, but I assumed it would only ever see a few hundred hits.

On the first day that I had the spooler open, I received 2,592 Gmail invites. The second day saw 4,574 more coming in. By the end of the second day, I had over 3,000 unclaimed invites.

It didn’t take long for word of the “magical free Gmail site” to leak out to the general internet. Within a few days, demand exceeded supply and I had to implement controls on the page to prevent people from refreshing constantly while waiting for a new invite to come in. I also got the first of many lessons in writing code with scaling in mind as I divorced the mailbox checking from page loading.

Ups and Downs
After a month of running the service, the average inbound invites per day dropped below 1,000 for the first time. It seemed that most of the people who had extra invites on hand had heard about the service and donated all they were willing; Google was not giving out new invites on a regular basis at that time. The inbound invites continued to decline through most of December 2004 until they hit a low around 50.

All this time, demand for invites remained strong. I recorded as many as 100,000 visitors and over a million hits per day. On December 20th, the drought was over as Google started to give Gmail users about five fresh invites each day. The average day saw around 2,500 new invites, but they were still being snapped up as soon as they came in. I implemented more restraints to prevent abuse and further streamlined my code in order to keep my server load at a reasonable level. During this time, my web statistics began to break down because Webalizer couldn’t process all of the data without choking.

Way, Way Up
On February 2nd, 2005, Google decided to open the flood gates. They began giving out around 100 new invites per day to Gmail users. My service experienced demand increases like I’d never seen before. For the first time, I was forced to benchmark my code and decide which methods to use based on how many milliseconds they took.

For only the second time, supply was greater than demand. Anyone wanting a Gmail invite could get one through my service without any delay. Unique visitors increased, but hits dropped way down since users had no need to refresh frequently to see if new invites had arrived.

Way, Way Down
Monday June 6th, 2005 was the day I received an email from Stephanie Hannon, Gmail’s Product Manager. Later that day, I had a conference call with Stephanie and her superior regarding my service. They felt that services like mine had become a threat to the quality of Gmail. Their reasons for making the service invite-only were many:

• Limit new subscribers
• Heighten demand and curiosity
• Limit accessibility of accounts to potential abusers

The last reason was the one that made them care about my site. Spammers and abusers have a higher threshold of entry without the spooler. Despite the fact that I think Google should do more on their part to prevent automated account creation and duplication, they do have more random people gaining access to invites through a service like mine.

In short, Google felt as if too many spammers and abusers were getting invites that they obtained from me and saw this as a threat.

Why I Pulled the Plug
I’ve received a few thousand emails asking for invites, complaining about how “unfair” this is, or asking for source code. In the early days after pulling the plug, I would respond to every request with an individually written response explaining the situation. This generated many replies suggesting I just re-open the system in defiance.

Aside from the fact that I really don’t wish to burn any bridges with Google (heck, maybe they’d forget all this and hire me if I ever applied), I have good technical reasons for not re-opening the spooler: My service relied on people with Gmail accounts constantly inviting the now blocked email address gmail@isnoop.net.

Google is no dummy. They know full well that they must track the email addresses that the invites are sent to. They can (and did) automatically invalidate every invite sent to my site. All 1,240,162 invites I had left over the day I shut the service down instantly became duds. To continue the service, I would have to change the method of catching new invites to one substantially more inconvenient for the donor.

In the end, insistence on keeping the spooler open would have certainly summoned the massive lawyering machine deep within the “don’t be evil” company and I don’t think reasonable person wants that fight.

Fast Forward to Today
The former Gmail Invite Spooler page is now a brief testament to what was once the most popular Gmail invite spooler on the internet.

The bulk of the current 2,500 visitors per day come from non-English speaking blog sites that haven’t yet gotten the message that the page is closed. While the rest of isnoop.net has a 66% US visitor rate, the spooler is only 17% US traffic; it holds the #1 slot by less than one percent.

Almost all of the dozens of emails and stray blog posts requesting invites ask the same thing (in broken English). I saw the need for folks who didn’t speak my native language to get the full story, so I wrote a simple script to help them out. This has helped reduce the confused request flow, but it has also crimped the last of my dwindling AdSense revenue. Oh well. I ran this site before it ever brought me a penny and I’ll continue to do so for as long as I have the energy.

I turn down all requests for the source code for the spooler. If Google doesn’t want me starting fires in their back yard, I’m certainly not going to give away my matches to all of the other neighborhood kids.

I have considered revamping the spooler for use with other invite-only services, but I’ve yet to see one of great enough popularity and of proper nature to justify the effort. I refuse to open up such a thing for a community-based website on the principal that it breaks the “six-degreesâ€? network they’re trying to build up by bringing in random people with no association to the inviter.

Media Coverage
Aside from a number of blogs and forums that mentioned the service, these are the print media references I am aware of:

Book: Google Search & Rescue for Dummies – 2005
Text

Book: Google Hacks – 2005
Text

Popular Science Magazine – June 2005
Close-up
Full page

The Mercury News (San Jose) – May 23, 2005
Online version

PC World – April 13, 2005
Online version

Sydney Morning Herald – April 9, 2005
Close-up
Full page
Online version

Are you seriously still reading this?
This post covers almost all of the points I regularly discuss with folks who have questions about the service. I hope this overly long post has satisfied your curiosity.