Dreaming Of Beetles

Update: And we have an answer – Google Maps Ditches Tele Atlas in Favor of Street View Cars and Crowdsourcing

I have no idea. But the title is catchy, and I’m hoping to learn the answer by a) writing out my thoughts and b) you. The transaction went down in July but was only uncovered recently, causing a day long brouhaha on the blog circuit, and now it’s been forgotten. As someone with a little knowledge in this area, I’m a tad more than intrigued. I’m just going to dive into a couple theories. They all focus on why Apple would move away from Google (I’m not even sure that is their intent):

1. Current Data Limitations
   Google places restrictions on their data use. A clear example of this is Jobs’ “BYOM”, Bring Your Own Map, statement about turn-by-turn directions. Google will not allow their data to be used for such functionality.
2. Cost Reduction
   Tiles aren’t free. Even to important and huge clients. Google went from a “per transaction” payment model to a “per tile” payment model and the entreprise level pricing isn’t exactly cheap. With the amount of Apple’s usage and even at a substantial discount, this cost is still probably in the tens to hundreds of millions per year.
3. More Control
   Apple is a control freak. No need to argue this. Being reliant on an increasingly competing company can’t feel good for anyone, especially Apple. The threat of Google barring Apple from their maps or even significantly altering the usage deal has DOJ written all over. This isn’t going to happen and Apple isn’t worried. Google does have free reign to make their maps look like crap if it affects all customers which leads into the next point…
4. Google Adding Data
   Just recently Google started adding advertisements PLUS user-generated content into iPhone apps. Wait, I thought one of the benefits of the enterprise license was to have the ability to remove ads. And wait, does this only affect iPhone apps? I think Google will come to their senses and flip off that UGC, unless you want to see it. The ad part is a bit odd and doesn’t jive with point 2 and 3 above.
5. More Control, Redux
   Google doesn’t own the tiles, they are licensed. Tele Atlas provides the map information, and other services provide the satellite imagery (Digital Globe, GeoEye, USGS, TerraMetrics, and the list goes on). There are some interesting things to note here.
   Tele Atlas provides data to many vendors including Yahoo and Microsoft and probably Placebase. Tele Atlas provides the map information, not the tiles, so each vendor can make them look however they want through Tele Atlas’ proprietary API. The United States is a 5GB or so text file. Apple’s designers could make their maps look better than everything else on the market, without a doubt.
   The satellite data comes mainly from Digital Globe, but once you start zooming in, watch the copyright info on the lower right of the map – the data comes from many, many sources. Apple can easily strike deals with these same sources. Digital Globe is happy to point out their non-exclusivity with Google. But on the other hand Google seems to be in bed with GeoEye, launching a satellite together and all.
   Other data such as traffic and street view is proprietary to Google.
6. Data Layers
   Placebase offers many layers of statistical data for their paid API and have won awards for their PolicyMap website. But if you look close enough, this is just tract data freely available from the US Census Bureau. Gathering and overlaying this data is a trivial task. In one of my previous companies, we built a much, much richer data set covering many more areas than Placebase offers. This was a hurculean task, but as a small startup, we pulled it off. So I think access to the data layers is moot here as it adds little value.

So what’s this all about then? My guess is that is has to do with Augmented Reality/Extended Mapping capabilites. This could be especially useful on the mythical tablet and could be even more useful for the collection of future Census Data. Not only in the US, but the rest of the world. It’s a little late for the 2010 Census though, so this idea is a tad suspicious.

I welcome all opinions, corrections, and comments on this as I really want to get to the bottom of Why Apple Bought Placebase.

Popularity: unranked [?]

I wasn’t aware of Clang and the transition to Low Level Virtual Machine compiler infrastructure. I knew about LLVM, but never saw a roadmap of Apple’s transition until I read the always excellent Ars Technica OS X Review. I’m just now setting up my development environment and was very tempted to use Clang to compile everything. At the moment, I’m pressed for time so I don’t want anything to go wrong. The 64-bit Apache/PHP tutorial is working great so far under Snow Leopard (just find the more recent version of the projects and tools).

If you want to set up Clang as your default compiler, throw this in your /etc/bashrc:

export CC=clang
export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments
export PATH=/Developer/usr/bin:$PATH

Anyway, I will eventually get around to recompiling with Clang and will report on my results. The first project will be Shiretoko and/or Namoroka.

Thanks for the tip Ajay Kapal.

You should follow me on twitter here.

Popularity: unranked [?]

I’m not sure when, but most likely when Leopard was released, applications started storing their cache files as sqlite databases (usually named Cache.db). For example, Safari has its cache at:

~/Library/Caches/com.apple.Safari/Cache.db

Apps that haven’t caught up yet are still using the less-efficient .cache files. Though not as efficient, these files are easier to access, just toss it onto BBEdit and you can see the contents. Try doing this with a 100+MB .db file and prepare to wait.

There probably is a GUI app to extract data from .db cache files, but I’m too lazy for that. OS X has everything you need already built in so fire up Terminal.app (I’ve been playing with Visor lately) and dig into your cache:

# cd ~/Library/Caches/com.apple.Safari/
# sqlite3 Cache.db

You’ll be in the sqlite interactive mode:

sqlite> select * from cfurl_cache_response;
sqlite> select receiver_data from cfurl_cache_blob_data where entry_ID = [1234];

To output the data to a file use the following:

sqlite> .output test.html
sqlite> select receiver_data from cfurl_cache_blob_data where entry_ID = [1234];
sqlite> .exit

That should do it. Any questions? Leave a comment.

Popularity: unranked [?]

First off, I’m going to start posting in a new format – a “squirt” (screw you zune). I have thoughts to convey that are too long for a tweet and too short for a full entry. I’ll put all these squirts into a single category.

Here are some things about Safari that are chapping my hide:

• With the newest version of WebKit, it is actually slower in the SunSpider JavaScript benchmark. Before, I was running a custom build of Safari 4 with WebKit and was doing over 100 ms better.
• It breaks Mail.app if you are using the GrowlMail Bundle.
• Nitro is the name of the new nebulous engine. How much of this is Squirrelfish and how much is optimization to WebKit?
• 1Password doesn’t work. This is an InputManager so doesn’t really count as it’s an unsupported hack.
• The blue loading bar is gone and the stop/reload within the URL bar is not intuitive. It’s also harder to see which pages are loading at a glance when clicking through tabs.
• The bookmark button is cemented onto the URL bar. I don’t use bookmarks. Go away!
• The trying-to-be-awesome-bar isn’t. It only matches the beginning of what you are typing. For example, in Firefox, I can type “alpha” and it will find the page I want. In Safari, I have to type “secure.xxxxxxx.com/alp” before it finds it.
• The search bar is jaring how it plops down like a ton of bricks. This animation is so not Apple.
• I have a hunch Safari is renicing itself somehow. Opera is not nearly as responsive when Safari 4 is running. Is this just me?
• Reboot to install a browser? Grrr.

Don’t get me wrong, there is plenty of nice eye candy, especially when you load up the browser for the first time. I’ve seen a lot of the “150 New Features” before as I’ve been compiling this version for a while now. Of those 150 features, there are probably 20 that are new and most I don’t really care about.

Anyway, WebKit has been and will continue to be my default browser (as I write this in Shiretoko).

image: apple

Popularity: unranked [?]

Update: If the Envelope index trick doesn’t work for you, disable any bundles you have by moving them to the “Bundles (Disabled)” folder within ~/Library/Mail/. GrowlMail and Safari 4 Public Beta do not get along so you will have to disable this one.

I’m running two plugins (bundles) for Mail.app – GrowlMail and WideMail – and have experience zero problems. Once in a while, I get Mail.app in a frenzied state where it crashes if I breathe on it wrong. The console usually gives me nothing interesting, except maybe a message such as:

Mail[10935]: deleting invalid message from outbox

So I’m stuck with a flaky app. The usual cause of this is an inconsistent state in the mail index resulting from various things like trying to delete tens of thousand of server error messages from Gmail IMAP, or force quitting during a delete, or some other dumb act on my part.

To fix this, move “Envelope Index” from ~/Library/Mail/ and launch Mail. You’ll get an import dialog, hit “Continue” and wait. I have 322,984 messages to import so it will take a little more than a half an hour. Once done, you’ll usually see the corrupted message somewhere, usually in your spam or trash. Delete it (this step isn’t actually necessary for the fix).

Once you verify things are working, go ahead and delete the old “Envelope Index”, the import process created a new one for you.

image: Apple

Popularity: unranked [?]

Update: Mike Pinkerton, Mac Chrome developer, is writing some interesting tidbits on his blog.

Without being involved with Google or Chromium in any way, these are just stabs in the dark. I would love to hear from those in the know about what is going on.

As I see it, there are three fundamental reasons for the delay:

1) Priorities
Google knows the fastest way to gain market share is to bundle the browser at the OEM stage with the dominant operating system. To do this, they need a compelling case for why the browser should be bundled. This led to several things:

• An extremely aggressive development cycle (I’m not on the dev channel, but know that they are pushing weekly builds).
• One of the fastest Google products to pass through beta (OEM’s are somewhat reluctant to pre-load beta software).

This raises the advertising questions:

So can’t Google just use it’s advertising space to drive adoption?
Yes, but it generally doesn’t work. Chrome made a brief showing on the Google frontpage, breaking the rule of 28 but we know this doesn’t drive much traffic. Visitors to Google are usually looking to search for something, not download a new browser. Evidence of this can be see in ReadWriteWeb’s analysis of the traffic the HTC G1 phone got from a Google link.

Also, you do see adsense ads for Chrome (instead of Firefox) popping up around IE or Firefox related articles. But I don’t think these are driving too much traffic.

2) Technology
I’m sure Chrome is packed chock full o’ technology that I don’t bring up here, but this should be a start:

WinHTTP
With the preview release of Chrome 2.0, we saw Google dump WinHTTP in favor of its own codebase. This is great and all, but leads me to believe that the original codebase was developed in a quick and dirty style in 20% time with MS tools. More on this can be read in this thread where people have combed the codebase and saw how dirty it is.

Sandboxing
This is another area where MS rears its ugly head. According to Nicolas Sylvain, Chrome developer:

The Mac and Linux version of Google Chrome are still in development. They are not ready yet.

We haven’t decided the implementation details of the sandbox on these platforms, but we clearly want something equivalent.

October 2, 2008 3:48 PM

This was in a comment made on this blog post titled “A new approach to browser security: the Google Chrome Sandbox“. It seems this is one of the main bottlenecks.

V8
I thought V8 was the culprit for the longest time until I saw this article – “Building and compiling V8 on Mac OS X“.

3) Look and Feel
These is a third and rather weak argument for the delay. Back to the “Platforms and Priorities” post, Amanda Walker, Software Engineer tells us:

One overriding goal we have had from the start has been to build the best browser we can. When it comes to Mac and Linux versions, this means that our goal is not to just “port” a Windows application to these other platforms–rather, our goal is to deliver Chromium’s innovative, Google-style user interface without rough edges on any of them.

Yes, making Chrome feel native to the Mac is important, but does that take six months? I’m not buying it.

image: Google

Popularity: unranked [?]

Update: The below installation still works on Snow Leopard with PHP 5.3. You can use the latest and greatest of each library/project below and it will still work. However, there is one gotcha. There is a bug somewhere (PHP blames Apple and vice versa) when compiling PHP. To get around this, run the following on the command line before the ./configure string:

# export EXTRA_LIBS=-lresolv;

I will update this with a new tutorial once I compile everything with Clang.

Warning: This installation was not done on a virgin machine. Everything has been tested and works on my heavily hacked system.

Ok. We’re going to compile Apache with PHP on OSX 10.5.6. To make things interesting, we’re going to do a 64-bit install. If you don’t know if your machine is 64-bit or not, follow this rule: Core Duo == 32 bit, Core 2 Duo == 64 bit. I haven’t found a way to reliably display the chip architecture from the OS X command line, but on most GNU/Linux variants, you can use lshw. On the mac, you can try `uname -p`, `arch`, `system_profiler SPHardwareDataType` but all of this will make you think you are running 32 bit. As the man page for `arch` states:

The most common use is to select the 32-bit architecture on a 64-bit processor, even if a 64-bit architecture is available.

So everything is defaulted to 32-bit. What we need to do is set some environment variables so gcc knows the architecture we want to compile against. Another thing, I’m going to do all of this as root (back off unix weenies – I know what sudo is and use it regularly, I’m not here to argue about this). You can do this with the following (the # represents the prompt):

# MACOSX_DEPLOYMENT_TARGET=10.5
# CFLAGS="-arch x86_64 -g -Os -pipe -no-cpp-precomp"
# CCFLAGS="-arch x86_64 -g -Os -pipe"
# CXXFLAGS="-arch x86_64 -g -Os -pipe"
# LDFLAGS="-arch x86_64 -bind_at_load"
# export CFLAGS CXXFLAGS LDFLAGS CCFLAGS MACOSX_DEPLOYMENT_TARGET

I’m also going to ratchet up the complexity a bit because I need a GD library, IMAP, iconv, and OpenSSL. So here goes:

1) Grab All The Tarballs
Here are the links to the tarballs needed. Most of these you can grab with `curl -O` or `curl -O -L` to follow a redirect:

• Apache – http://mirror.olnevhost.net/pub/apache/httpd/httpd-2.2.11.tar.gz
• PHP – http://us3.php.net/get/php-5.2.8.tar.gz/from/this/mirror
• IMAP – ftp://ftp.cac.washington.edu/mail/imap.tar.gz
• libiconv – http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.12.tar.gz
• OpenSSL – http://www.openssl.org/source/openssl-0.9.8j.tar.gz
• FreeType – http://savannah.inetbridge.net/freetype/freetype-2.3.8.tar.gz
• libjpeg – http://freshmeat.net/redir/libjpeg/5665/url_tgz/jpegsrc.v6b.tar.gz
• libpng (with config) – ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-1.2.34.tar.gz
• t1lib – ftp://sunsite.unc.edu/pub/Linux/libs/graphics/t1lib-5.1.2.tar.gz

I’m not going full bore with the GD Library here, I just want to create some basic graphs, manipulate images, etc. so I’m not getting XBM or WBMP. I’m going to download all this stuff into /usr/local/src/.

2) Compile The GD Libraries
First we’ll start with the GD libraries as these are pretty straight forward and shouldn’t give you any problems. When on the `make` step of each of these libraries, you’ll see the familiar “-arch x86_64″, this means we’re doing it right.

FreeType

# tar -xzvf freetype-2.3.8.tar.gz
# cd freetype-2.3.8
# ./configure
# make
# make install

libjpeg

# tar -xzvf jpegsrc.v6b.tar.gz
# cd jpeg-6b
# ./configure
# make
# make install

libpng

# tar -xzvf libpng-1.2.34.tar.gz
# libpng-1.2.34
# ./configure
# make
# make install

t1lib
You will need to modify the “configure” file after you untar the file. So open “configure” and modify line 8656 to read:

archive_cmds='$CC -arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe -dynamiclib $allow_undefined_flag -o $lib $libobjs $deplibs $compiler_flags -install_name $rpath/$soname $verstring'

The architecture types must be explicitly stated for configure to work properly. There is an alternate method of modifying the resulting Makefile, but I won’t go into that. Commands are:

# tar -xzvf t1lib-5.1.2.tar.gz
# cd t1lib-5.1.2
# vi configure (make changes above)
# ./configure
# make without_doc
# make install

Some of these installs might give you lip about certain directories not being found. Make sure you have the following directories on your system:

/usr/local/include
/usr/local/bin
/usr/local/lib
/usr/local/man/man1

3) Compile OpenSSL
Since many components rely on this. We want to get it compiled and in place fairly early.

# tar -xzvf openssl-0.9.8j.tar.gz
# cd openssl-0.9.8j
# ./config --prefix=/usr/local/ssl
# make
# make install

4) Compile IMAP
This is pretty easy, but some extra steps need to be taken because make install doesn’t seem to exist:

# tar -xzvf imap.tar.gz
# cd imap-2007e
# make oxp
# mkdir /usr/local/imap
# cp -pr c-client /usr/local/imap/

5) Compile libicon
A very useful library that doesn’t seem to get much love. This will convert between character sets (UTF-8, etc.). Maybe dealing with Japanese characters for so long gave me this appreciation.

# tar -xzvf libiconv-1.12.tar.gz
# cd libiconv-1.12
# ./configure
# make
# make install

6) Compile Apache
This assumes you already have MySQL on your machine. The 64-bit package installer at http://dev.mysql.com/downloads/ should be all you need to install this. I would stick with 5.0.x even though 5.1 is GA (Monty of MySQL fame has some choice words about this). Even though I love compiling my own apps, MySQL does such a good job with their installer, I’ve left it to them for the past couple years.

# tar -xzvf httpd-2.2.11.tar.gz
# cd httpd-2.2.11
# './configure' '--prefix=/usr/local/apache2.2.11' '--with-included-apr' '--enable-module=most' '--enable-shared=max' '--enable-headers' '--enable-rewrite'
# make
# make install

A couple things to note:

• You normally don’t have to explicitly state which apr to use, your system will figure this out. I had a version installed by darwin ports for subversion and the compiler wanted to use that one and would choke (this is Versions‘ fault).
• You don’t need the enable-headers and enable-rewrite. I like to do funky things like mod_rewrite for pretty URLs and mod_proxy for Tomcat. Choice is yours.
• From httpd-2.2.9, the configure options got all out of whack with slight syntax modifications to some important flags. To normalize this, you need to run `./buildconf` in the src directory before configuring. Seems like things are back to normal for 2.2.11.
• In my config string, I spell out which version of the software I’m using – apache2.2.11 – then later symlink /usr/local/apache2 to /usr/local/apache2.2.11. This is useful for running multiple versions.

7) Compile PHP
The final step. Here is where you’ll be able to see if you’ve done everything above correctly.

# tar -xzvf php-5.2.8.tar.gz
# cd php-5.2.8
# './configure' '--prefix=/usr/local/php' '--with-apxs2=/usr/local/apache2.2.11/bin/apxs' '--with-zlib=/usr' '--with-mysql=/usr/local/mysql/' '--with-mysqli=/usr/local/mysql/bin/mysql_config' '--with-gd' '--with-png-dir=/usr/local/php' '--with-jpeg-dir=/usr/local/php' '--with-freetype-dir=/usr/local/php' '--with-t1lib=/usr/local/php' '--with-xmlrpc' '--with-pear' '--enable-mbstring' '--enable-cli' '--with-curl=/usr/local/' '--enable-soap' '--with-openssl=/usr/local/ssl/'
# make
# make install

Some things to note:

• You may not need all these compile options. Pick and choose what you feel you need.
• I compiled my own version of curl, but you can leave this out of the config string if you are ok with the default OSX curl. I’m a huge fan and power user of curl. Maybe someday I’ll write about what that means.

Other Configuration Stuff:

• Change user/group to www:www in conf/httpd.conf
• I’m not going to get into it too much, but make sure you compare your old httpd.conf file (/etc/apache2/httpd.conf) with the new one. There are several Apple specific directives in there to keep your webserver safe.
• Make sure to add support for mod_php:

<IfModule mod_php5.c>
AddType application/x-httpd-php .php
AddType application/x-httpd-php-source .phps
</IfModule>

• If you want this new version of Apache to start automatically at startup, make sure it is enabled in sharing, then do this:

# cd /usr/sbin/
# mv apachectl apachectl.bak
# ln -s /usr/local/apache2/bin/apachectl apachectl

Now fire up apache and load a page with <? phpinfo(); ?> to see that everything has installed correctly.

I’m more than happy to answer any questions you have about this process, so fire away!

Warning: This installation was not done on a virgin machine. Everything has been tested and works on my heavily hacked system.

Popularity: unranked [?]

This one caught me by surprise this morning as I was forwarding a message in OS X Mail. It seems if you have a threaded message (at least two levels deep) and you click within a nested level, a new delete marker comes up. Clicking the “x” on the upper left of the box will wipe out the nested threads. Never saw this before and have to say I’m pleasantly surprised.

The reason I’m blogging about this is to prove a small point. Whereas Windows seems to be out to get you, working against you at every step; OS X works with you and tries to help at every step. When asked why I use OS X, this is what I try to explain and this is just a small example of that in action.

A couple years ago I was talking to an executive in a fortune 500 company about this. The other day, I found out that I had converted him with this logic and he is now a die hard Mac fan.

Anyway, I have a couple big posts waiting in the wings that need a bit more research. Hopefully I’ll get one of those up today.

Popularity: unranked [?]

I have to admit that I’m an Apple fanboy. I’m not one of the switcher fanboys, but one of the longhaul fanboys. My first exposure to a computer (well, after the teletype my father brought home) was our family’s Apple IIe and my first programming was done in Apple basic. I had an Apple sticker on my bedroom door when I was 10.

When I first saw a Macintosh, I was blown away. I knew right there and then, the WIMP (window, icon, menu, pointing) interface was the future. I got my hands on one in the computer lab and was infected.

I’ve never owned a Windows box. My brother got one when I was in high school and this is what I used to check out the Internet while Netscape was still in beta. I hated it.

I went off to college and gained quite a bit of experience with Unix (and VMS). Indiana University is one of the top most progressive schools with technology so often times we were forced to turn in our homework via ftp on Unix. I was intrigued. I guess it was the days of Adventure and Zork and Apple Basic that made the command line something comfortable, not frightening. So I became as profficient as possible with the command line, but not gaining root hampered me a bit.

When I got seriously into computing, I bought a PowerBook 5300ce, one of Apple’s most infamous products. I suffered through all its problems, but it worked well enough for me to learn. And that is what I did. During this time, I wasn’t really a fanboy though, I didn’t know who Michael Spindler was or who Gil Amelio was and I didn’t really care. OS 7.5 and all its elegance was what mattered.

Skip ahead a few years, and I’m riding high on OS 8/8.6. I started following Apple religiously – their stock price, the hardware, the software updates, third party support, etc. I was very happy to see them buy Next rather than Be and kind of laughed when I heard that Steve would be a special advisor to Gil. I knew this arrangement wouldn’t last very long and it didn’t.

I’m not going to recount the story of Apple’s re-emergence from beleagured computer manufacturer to dominant media company, everyone knows that story. It’s the stuff of legend.

I’m happy I was able to see Steve Jobs speak twice, but kind of bummed I never got to meet him. I haven’t met Gates either, but met Ballmer and that was no fun.

While Steve enjoyed the limelight and kept the inner workings of Apple top secret, he is not an idiot. He brought in the highest caliber people possible and instilled in them the Apple spirit and these are the people that run the day to day of Apple. With Steve out of the picture, I’m sure the pipeline is not going to dry up – just read the story of Jony Ive coming up with the 2nd gen iMac. And Jony Ive is a pre-Jobs fixture at Apple. (I got to see him in the first Apple store in Tokyo).

For a while Steve was CEO of both Apple and Pixar, how could they BOTH do so well. Surely it wasn’t Steve micromanaging 24/7, it was his brilliant ideas like putting the Pixar bathrooms in the middle of the building so everyone crossed paths eventually.

I’m not worried about a post-Jobs Apple and will remain dedicated to the best OS and software ever produced (hardware is a different story). So when I start going Apple Fanboy in my posts, I’ll point people back here for reference. I’ll try to keep my Apple slant out of my writing though.

Popularity: unranked [?]

Checking out the Wordpress iPhone app to post this. Many have given this app horrible reviews. While it doesn’t have every feature, it is quite solid and seems to perform it’s main function well. I applaud Automattic for open sourcing this app even before the NDA was lifted. Also, happy 25th Matt!

Well, that crash kind of sucked and that one. Maybe this app does have some problems. I’m going to dig into the code a bit.

Popularity: unranked [?]