I received a cryptic message from Eric Patterson, one of our Software Engineers today. He pointed me to a javascript error report that he noticed while trolling through them and eating lunch.

It looked like this:

javascript_bug_report_-_no_help

(click for full size)

Eric was pretty excited. “Look at the User Agent!” he said. I didn’t see the big deal. Mozilla/5.0 (compatible; Googlebot/2.1). So? I thought.

I expressed my inability to see the excitement and he reminded me that this was a javascript error report! This javascript error was caused by a search engine bot. The same bot that for years could not execute javascript.

 

Javascript_bug_report

(click for full size)

Could it be?

Eric did some quick digging and he found a couple of other interesting things.

  1. About 5% of the sites we manage had similar googlebot javascript errors.
  2. All the javascript errors were found in areas which would need execution of the document ready event and DOM-manipulating javascript code.
  3. We dug into Google Analytics for those sites, and found an interesting “Mozilla Compatible” entry. It hit our sites on regular intervals, stayed for 1 second, and averaged 1.01 pages per visit. It seems to point to a Googlebot triggering the Analytics javascript.
  4. We did IP address lookup on the errors and it resolved to Mountain View, CA.

 

So, the question is, what does this mean? Can we assume that our javascript content will be indexed by Google? Can Google actually interact with the javascript on the site?

We’ll be keeping an eye on this as it develops, but we’d love to hear your experience. Are you seeing Googlebot executing the javascript on your site or in your Analytics?

UPDATE

After data mining the server’s log files we found some interesting information in relation to the activity of Googlebot/2.1.

Housed in one our site’s external javascript file is an ajax call to get news articles. We found the Googlebot executing that javascript ajax call with GET and POST requests shown here:

GET xxx/GetEventsAndNews – 66.249.73.121 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)
POST xxx/GetEventsAndNews – 66.249.73.121 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)

An ip lookup of 66.249.73.121 verifies it is from Google’s Mountain View based headquarters.

We saw something even more intriguing though; a GET request for an image file from the same ip and query string.

GET /assets/xxx.jpg – 66.249.73.121 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html)

Why is this interesting? Because the only way the Googlebot would have known about this was from the returned json data from their ajax call of the GetEventsAndNews function.

This evidence seems to tell us that either a Google employee is masking their desktop user agent string and browsing the web, OR the Googlebot is not only executing javascript, but issuing POST requests and interpreting the returned data!