Dreaming Of Beetles

Update: 12/07 – I’m out of invites. Sorry, if you didn’t get one.

Somehow I stumbled into almost 40 invites. I’m going to give away 30 (have about 20 left).

For those that don’t know what this is, check out the Google Wave site for thrilling and exciting content. You can also watch the Developer Preview and shake your laptop in the air when it’s over.

Ok, the giveaway.. erm.. the Great Google Wave Invite Giveaway. The rules are:

• There are no rules.
  Seriously. I don’t feel like making people jump through hoops, promote my site, promote my Twitter account, or any other degrading thing (but watch me contradict myself below).
• Winners must follow me on Twitter.
  I’ll follow you back. This is necessary so we can DM account info.
• I’ll give a couple invites away each day.
  Yeah, yeah. So I’m milking it a little, so what.
• The Golden Rule: Leave a comment.
  You must use your real email address, this is where I’ll send the Wave invite. Your email will not be publicly displayed.

Follow me on Twitter!

Popularity: unranked [?]

Update: Robert X. Cringely’s Op-Ed is out. Not at all what I expected, but that’s what I like about RXC. After watching a couple hours of Google IO ‘09, I finally realized the deal with Chrome OS. Learn more about Native Client and it will all make sense – Google IS revving up for a fight for the Desktop OS and NaCl is how they’ll do it. And I wish the “Tech” blogs would understand the difference between Linux, GNU/Linux, and a Distro. It’s almost embarrassing to read. Disregard my blathering below.

I must have read over 50 articles on this whole Chrome OS thing and I’m still baffled. Here are some of the better articles I’ve read on the subject:

• Putting What Little We Actually Know About Chrome OS Into Context (Daring Fireball)
• Let’s all take a deep breath and get some perspective (Fake Steve Jobs)
• Android, Chrome OS Relationship Confusing Everyone, Including Google (Gizmodo)

To see a more complete list of stuff I’ve read, take a look at my Recently Read Articles. I’m still waiting on what Robert X. Cringely is going to say (he loves to take his conclusions to the logical extreme), but he’s holding out for an Op-Ed piece in the New York Times.

Some interesting theories:

• Chrome OS is a way of sticking it to the man. By giving netbook/smartbook OEM’s another Windows alternative to use as a bargaining chip to lower the Windows tax.
• Chrome OS is meant as a dual-boot solution. It will be installed alongside any other operating system to give you a way to boot and jump on Gmail within 10 seconds.
• Chrome OS is meant to be a distraction for whatever Microsoft is set to announce on Monday.

What I think is going on, and what is hinted at in the Gizmodo link above, is that Google is using Chrome OS as FUD against itself. Android is creeping into a market for which Google never intended, the netbook/smartbook realm, and in order to prevent Android from seriously forking, Google is saying “We will have an OS for that space, don’t go there!” I’m not going to blather on about this because, as I said, I’m confused.

One other thought, any netbook/smartbook that Chrome OS goes on will be FREE. Not just the operating system, but the hardware – subsidized by Google and a minimal contract with a carrier.

image: Google

Popularity: unranked [?]

Don’t Be Evil

The famous motto attributed to Paul Buchheit for a company that seems to no longer have any choice in the matter. Once you go public, you serve new masters, no matter what you write in the prospectus. Google is not above the market. Nobody is.

Over this past week or so, we’ve seen plenty of evil. Enough to really start to scare me. Here are some examples:

1) Wolfram|Alpha
Sergey Brin interned for Wolfram Research and as a friend of Stephen, was able to see the “computational knowledge engine” months before it was released. Soon after Alpha’s launch, we see Google Squared. This looks a little “cobbled-together-over-two-months” to me, no?

2) Microsoft Bing
Bing has a neato feature that allows you to explore your search in more depth with a dynamic left-hand sidebar. This mostly works with the four verticals they are attacking out the gate – Travel, Shopping, Health, Local. Google preempted this with their own left-hand sidebar allowing you to narrow down by media, time, etc. Again, a bit of a hack-job if you ask me – these features pretty much already existed. Microsoft slams this Google feature in their Bing promo video (along with some other zingers).

Strike two against Bing is Google Wave. Announced almost simultaneously. Who got more press?

3) Google Wave
This is a slam against FriendFeed which was created by none other than Paul Buchheit. Actually, it’s a slam against Twitter, Tumblr, Facebook, and a host of others, but I see it most resembling Friendfeed. Why is this evil? Well, it isn’t really…. competition is good and all… but CHROME IS OPTIMIZED FOR WAVE. That is evil.

4) Yahoo Announcements
These were a bit abstract and lame so I don’t blame the press for ignoring them, but let’s see about Google. Google’s “Searchology” happened the day after Yahoo with similar announcements: user intent, microformats, and mobile search. Check out coverage of Google’s event and Yahoo’s event for the fine details.

Popularity: unranked [?]

If you are fed up with Gmail and want to pull all your mail, here is how you do it. This technique was used on over 30 mail accounts so I’m sure it will work for you.

The problem of exporting your mail from Gmail is not a trivial one. From discussions by Opera Software’s lead QA for Opera Mail’s posting on Gmail’s Buggy IMAP Implementation to Matt Cutts’ posting on How to back up your Gmail on Linux in four easy steps to LifeHacker’s posting on Back up Gmail on Linux with Getmail to Wired’s wiki entry on Make a Local Backup Of Your Gmail Account, it seems that there is no single definitive source on how to pull your mail and retain your labels.

So here is what I’ve done to solve this problem:

1. Use getmail – this has been the best archiver I’ve run across. There are other applications – isync, OfflineIMAP, Fetchmail, etc. – that probably do a decent job, but getmail is still the best in my view. There are other hacks – use Mail.app to synch the Gmail IMAP directory, then convert emlx to maildir; same for Thunderbird and mbox; etc – but we wanted something a little more straightforward – Occam’s razor, right?
2. Install getmail – On my dev machine, I used macports (port install python25; port install getmail) to install the latest getmail which had dependencies on Python 2.5. After this was done, I set up the getmailrc config file and fired off an attempt using SimpleIMAPSSLRetriever… which failed due to a lack of SSL in the newly installed Python. I had to go back and install Readline (port install py25-readline), then install SSL for Python (port install py25-socket-ssl).
3. Patch Python – There is a malloc bug in imaplib when fetching large documents using SSL. So open up imaplib.py from your Python lib dir (in my case /opt/local/lib/python2.5/) and replace:

   data = self.sslobj.read(size-read)

   with

   data = self.sslobj.read(min(size-read, 16384))

   to maintain a 15MB memory block if necessary.

4. Configure getmail – Now that most of the fun is taken care of, we need to set up a configuration file for getmail (~/.getmail/getmailrc) and create the proper local destination. First the getmailrc file:

   [retriever]
   type = SimpleIMAPSSLRetriever
   server = imap.gmail.com
   mailboxes = ("[Gmail]/Starred",)
   username = username@yourdomain.com
   password = xxx
   
   [destination]
   type = Maildir
   path = ~/Maildir/
   
   [options]
   verbose = 2
   message_log = ~/.getmail/gmail.log

   First of all, we are using IMAP to retrieve mail as POP has a limit of 99 documents per access and that would take forever.

   Second, we are using the Maildir format for the destination so we need to make sure the target directories have been created (~/Maildir/cur, ~/Maildir/new, ~/Maildir/tmp).

   Third, we need to specify a mailbox or mailboxes to download or the INBOX will be the default.

   Fourth, we need a trailing comma on the list of mailboxes to download due to a parsing error in getmail (actually the mailboxes option needs to be a tuple, but the trailing comma negates that).

   Fifth, we need to know the syntax of Gmail’s internal IMAP structure to pull down discrete folders. Non-label folders (Starred, Sent Mail, Drafts, etc.) are accessed with “[Gmail]/Starred” (as in the above config) and labels are accessed directly. For example, the label “Important Project” would have this in the config:

   mailboxes = ("Important Project",)

5. Download your Gmail – For every folder/label I had within Gmail, I downloaded to a separate folder so I could import into dovecot IMAP without hassle. This entailed changing the mailboxes option in getmailrc, running getmail, renaming Maildir to label/directory name, rinsing, repeating.
6. Retain Times – Because maildir uses the modification time of every file to determine the sent date, all emails pulled by the above method will basically lose their sense of time. The below PHP script will restore the modification times:

/* VARS ***********************************************************/
$box = '';
$stem = SITE_DIR.'Maildir/'.$box.'/new/';
/******************************************************************/

$dir_contents = scandir($stem);
foreach($dir_contents as $item) {
  if(!ListFind('.,..,.DS_Store',$item)) {
    $file = $stem.$item;
    $content = file_get_contents($file);
    $date = extractText($content,"nDate: ","n");
    $utime = strtotime($date);
    $converted = date('YmdHi.s',$utime);
    shell_exec('touch -mt '.$converted.' "'.$file.'"');
  }
}

function extractText($content,$start,$end) {
  if(strripos($content,$start)===false) { return false; }
  $startpoint = strripos($content,$start)+strlen($start);
  $endpoint = strripos($content,$end,$startpoint);
  $length = $endpoint - $startpoint;
  return trim(substr($content,$startpoint,$length));
}

function ListDeleteAt($inList, $inPosition, $inDelim = ',') {
  $aryList = _listFuncs_PrepListAsArray($inList, $inDelim);
  array_splice($aryList, $inPosition-1, 1);
  $outList = join($inDelim, $aryList);
  return $outList;
}

function _listFuncs_PrepListAsArray($inList, $inDelim) {
  $inList = trim($inList);
  $inList = preg_replace('/^' . preg_quote($inDelim, '/') . '+/', '', $inList);
  $inList = preg_replace('/' . preg_quote($inDelim, '/') . '+$/', '', $inList);
  $outArray = preg_split('/' . preg_quote($inDelim, '/') . '+/', $inList);
  if(sizeof($outArray) == 1 && $outArray[0] == '') {
    $outArray = array();
  }
  return $outArray;
}

photo: chris ivarson

This is a reprint of a post I originally made at http://www.propertymaps.com/blog. I felt it was relevant to the current Gmail posts so am reprinting with slight modifications.

Popularity: unranked [?]

Gmail Account Lockdown

While everyone is up in arms about the recent Gmail outage, I was up in arms about my Account Lockdown (pictured above). Mail.app has been acting up on me again lately (I have a post about Mail.app), so I decided I would clear out my junk in Gmail. I had about 150,000 messages in there and wanted to get rid of about 90% of it so I started mass deleting via IMAP.

Before long, I started noticing that nothing was actually happening. Mail.app was moving the messages to Trash, but when I would pop back over to my “All Mail” folder, the same number of messages were there. WTF? So I decide to log in to the web interface to check things out (and to check out the new button styles) and lo and behold, my account was locked down.

The lockdown was over “suspicious” behavior and I guess in my case that behavior was “deleting large amounts of email”. But I was doing it via IMAP, not POP. So off to the troubleshooting page which was of no help whatsoever, just telling me the lockdown would be in effect for 24 hours. Great.

This, the recent outage, and disappearing email has me slightly worried.

Popularity: unranked [?]

Chromium on OS X

Update: Chromium build 12558 is now available.

Since Mike Pinkerton published some screenshots of Chromium running on OS X, there seems to be a little uproar in the Mac community about when we will see the final version. I decided to investigate a tad and built my own version for OS X. My version, rev 9780, seems to do a bit better than Mike’s because clicking on links work. This seems far from complete though, so if you enjoy lots of unexpected quits, this app is for you.

There is another app that is built with Chromium, called TestShell. It is a version of Chrome with basically no interface – this is the version the Mac developers have been using to port the backend code over. TestShell is quite a bit more stable than Chromium, but you don’t get any of the features. What you are seeing on Pinkerton’s site are the first versions of the Mac look and feel being added.

Some interesting things I noticed in the Chromium source code (I didn’t spend too much time looking):

• There is an iPhone directory with resources and nib files
• The version of WebKit being used is almost in sync with the WebKit trunk
• There is some Mozilla code in here, but not as much as I would have thought

Other findings:

• The Google BuildBot waterfall page is amazing – work is being done on Chromium at a rapid clip.
• There are OS X Chromium torrents popping up of build 9750. People are having tons of problems with it.

Ok, on to benchmarks:

The updated Chromium build whomps WebKit with a 691 millisecond score. The above benchmarks are for the older version of Chromium.

Go To Downloads Page

Because this project is being developed so rapidly, I will start doing nightlies. Give me a few days to set that up.

Popularity: unranked [?]

Update: Mike Pinkerton, Mac Chrome developer, is writing some interesting tidbits on his blog.

Without being involved with Google or Chromium in any way, these are just stabs in the dark. I would love to hear from those in the know about what is going on.

As I see it, there are three fundamental reasons for the delay:

1) Priorities
Google knows the fastest way to gain market share is to bundle the browser at the OEM stage with the dominant operating system. To do this, they need a compelling case for why the browser should be bundled. This led to several things:

• An extremely aggressive development cycle (I’m not on the dev channel, but know that they are pushing weekly builds).
• One of the fastest Google products to pass through beta (OEM’s are somewhat reluctant to pre-load beta software).

This raises the advertising questions:

So can’t Google just use it’s advertising space to drive adoption?
Yes, but it generally doesn’t work. Chrome made a brief showing on the Google frontpage, breaking the rule of 28 but we know this doesn’t drive much traffic. Visitors to Google are usually looking to search for something, not download a new browser. Evidence of this can be see in ReadWriteWeb’s analysis of the traffic the HTC G1 phone got from a Google link.

Also, you do see adsense ads for Chrome (instead of Firefox) popping up around IE or Firefox related articles. But I don’t think these are driving too much traffic.

2) Technology
I’m sure Chrome is packed chock full o’ technology that I don’t bring up here, but this should be a start:

WinHTTP
With the preview release of Chrome 2.0, we saw Google dump WinHTTP in favor of its own codebase. This is great and all, but leads me to believe that the original codebase was developed in a quick and dirty style in 20% time with MS tools. More on this can be read in this thread where people have combed the codebase and saw how dirty it is.

Sandboxing
This is another area where MS rears its ugly head. According to Nicolas Sylvain, Chrome developer:

The Mac and Linux version of Google Chrome are still in development. They are not ready yet.

We haven’t decided the implementation details of the sandbox on these platforms, but we clearly want something equivalent.

October 2, 2008 3:48 PM

This was in a comment made on this blog post titled “A new approach to browser security: the Google Chrome Sandbox“. It seems this is one of the main bottlenecks.

V8
I thought V8 was the culprit for the longest time until I saw this article – “Building and compiling V8 on Mac OS X“.

3) Look and Feel
These is a third and rather weak argument for the delay. Back to the “Platforms and Priorities” post, Amanda Walker, Software Engineer tells us:

One overriding goal we have had from the start has been to build the best browser we can. When it comes to Mac and Linux versions, this means that our goal is not to just “port” a Windows application to these other platforms–rather, our goal is to deliver Chromium’s innovative, Google-style user interface without rough edges on any of them.

Yes, making Chrome feel native to the Mac is important, but does that take six months? I’m not buying it.

image: Google

Popularity: unranked [?]

I’ve been sitting on this one for a while as I read countless reports on “Knol vs. Wikipedia”, “Why [Knol|Wikipedia] is dead”, “Knol has failed”, “Why Didn’t Google Kill Knol?”, etc.

I’m going to keep this one short as a) I see that a lot of people already have already come to conclusion I present below, b) I have a raging cold and want to get back to bed. Check out the comments on “Why Has Knol Survived Google’s Ophan-Killing Spree?“, btw I haven’t even read the article, just the comments.

So here is the deal with Knol:

1) Knol allows you to select New Yorker comics to insert into your Knol for free, WTF? If you are a Seinfeld fan, you’ll appreciate the comedy behing how cryptic these comics are. Well, it turns out Udi Manber is a huge fan. Who is Udi Manber? He is the Google’s VP of Engineering, Core Search.

2) Knol is Udi Manber’s 20%-time project. Udi Manber controls the levers operating Google’s empire, this is important stuff. What Manber wants, Manber gets so this project will never get killed (unless Udi kills it).

3) Udi Manber has a small little goal for Google – “human understanding and intent”. How is this done? Semantics and the semantic web. Manber has been involved in search for a long time and has published many papers and written many applications to harness the power of words/computing to make a better search experience.

4) So the point of Knol? Provide a corpus of structured knowledge to experiment with Manber’s search technologies.

So Knol is not about competing with Wikipedia, it is not about making money for authors or Google via adsense. Google couldn’t care less about these things, they (Manber) want to improve search. Once you can crack the “human intent” nut, search is yours for the taking.

image: Google

Popularity: unranked [?]