Arun Gupta, Miles to go ...

Google Developer Day Report

Follow up from here. It's a very long entry, so feel free to skip around, look at the pictures or read notes from sections of your choice.

I reached San Jose McEnery Convention Center around 9:15am and the crowd started building closer to 9:30am.

The keynote hall opened at 9:30am. A video with different mahsups (primarily using Google Maps) was shown until the keynote started (around 10 minutes late). Here are some of the websites that were shown in the video: map.movin-360.com (seems like a virtual tour of Berlin city), prague360.com (name says it), dudewheresmyusedcar.com (looking for a used car near a zip code), wepoco.com/maps/observations.php (making weather and climate information useful), arena.tv/gastronomie/barfinder.jsp (finding a bar in Germany), cleancruising.com.au (Australian cruising site, another Map mashup), turnhere.com/sfbayarea (local Bay area guide), fishingmap.jp, bimrace.com (tracks your progress in race, shows elevation for a race and animates).

Anyway, what follows is notes/impressions from the different sessions I attended today.

Opening Keynote

The Opening Keynote was given by Jeff Huber, Vice President Engineering.

Jeff explained the rationale behind conducting the event at 10 cities simultaneously that instead of bringing the developers to conference, take the conference to developers. There is approximately 1500 in Bay area, 5000 over all, 160 sessions over all.

housingmaps.com digged into the APIs hidden in JavaScript which motivated Google to create the Maps API. There are over 50,000 sites using these APIs all over the web.

Evolution of development over last several years: Start from open-source building blocks, Linux + Apache + MySQL + [Ruby+PHP+Java+Python]. Set of Web services as building blocks, for example Amazon S3 for storage.

Google's Developer Products are centered around three areas

• Integrate (Ability to take Google services to create your applications)
• Reach (How do I get distribution of my products, traffic to my site)
• Build (Building next gen web based applications)

Jeff talked about how Google provide infrastructure to build some of the products like [jogplanner, apartmentfinder, moviemaps. coolphotoeditor, syncingservice, certifiedusedcars, ok-i'm-kidding].google.com

Integrate (Google APIs + Mashup Editor)
GData APIs - Single RESTful Web service model, uses  APP and RSS for reading/writing to Google's application. For example picnik (online photo editing), spanningsync (syncing mechanism to ical and isync on Mac), Honda Certified used cars (Google Base GData APIs, setting custom search).

Announced: Google Mashup Editor - Experimental product, online application to create mashups. When I went online to sign up, then I got the following message:

The demo created a mashup of CA state parks, feed from external source, fed to Google Maps, and then link with Google Base for camping gear in 3 lines of code + HTML wrappers. Finally publish on googlemashups.com or igoogle.com.

Reach (Gadget + Mapplets)
Google's network is the largest network, 500 million users per month. Gadgets API. Very open model, 100,000 gadgets have been created. Todo List Gadget had 6.7 million page views in last week, Expedia Fare Calendar Gadget (updates every time you are online with the latest roundtrip fares) had 4.7 million page views in last week.

10:35am: Seems like they turned on the Air Conditioner and it's too loud!!

Announcement: Google Mapplets - Combined Gadgets and Maps API.

Orbitz mapplet, Weatherbug. The cool thing is you can show data from different sources such as Hotel and Weather on the same map. Combined Orbitz and WeatherBug with Google Maps - this combination is more useful than any of the individual sites. Called as "enabling a mashups of mashups".

Build (GWT + Gears)

Announcement: Google Gears - offline access for web apps, cross-browser, cross-platform, BSD-license. A way by which developers can add off line capability to your applications in a standard manner. Browser extension to allow web apps to work offline - Firefox, IE, Safari + Mac, Linux, Windows. Launching as open source project.

This is an evolutionary approach where end users continue using their application as they are using today w/o worrying about network connection, Developers continue building the application using their existing skills and code base, simply adapt to Gears API.

Also launching Google Reader Online - Working with Adobe, Opera and Firefox for industry collaboration. Kevin Lynch from Adobe showed a Flex/Apollo application with very similar online and offline experience.

Sergey Brin (co-founder of Google) made a cameo appearance. He talked about how the "person" involved in maintaining the Internet needs to be out-of-the-loop for it to truly self-sustaining and how Mapplets are leading towards that. And of course, payback to community which created such a fantastic ecosystem.
 

Google Maps API Introduction

The complete presentation is available at doug.ricket.com/gdd2007. Doug Ricket, the speaker, used the preso from this link.

API is all about writing little bit of JavaScript and do cool things. Here are some examples of how Google Maps API have been used:

• onNYturf.com - Subway map of NY city on Google maps (classic API set - some data put on map, useful to users)
• trulia.com - Real estate search
• Panoramio - Upload your photos and tag them with a specific geographic location
• wikimapia.org - People are adding content all over the world to communicate (entirely user generated content)

The trend is towards more collaborative.

Load maps API package (indicated by "file=api". There used to be a limit to the number of times this API key can be used in a day. But this limit is now removed. However if 5 million users are expected for the key, then contact Google to ensure sufficient server capacity.

"v=2" indicate that there is a big change in the API and required changes in the application code. In reality, you'll get the latest 2.x release, but behind the scenes versions are getting incremented and you get the latest version.

2.s is a more stable release for enterprise customers, solid and no bugs. You can also specify an exact version.

The fact that you can drag the map is the "big feature". Usability studies indicate that the cursor need to be changed to hand instead of mouse pointer (at least 2.51) - 2.50 you've pointer, 2.51 has a hand.

Because exact version numbers can be specified, almost infinite backward compatibility is available.

Key is registered for your specific website. For the public API, completely free but your site also need to be freely available. The basic API set has to be open to public. If you want to run it inside corporate firewall, then you need to use Enterprise and you get support from the Google team.

Follow along the presentation for all the detailed examples.

In Google Maps terms, InfoWindow is what is called as Balloon, Bubble or Pop-up. Marker is the red balloon, Title option to Marker is the tooltip.

Overlays is anything that can go over the map, for example Marker, Polylines, Polygons, Image and others. Can also create a custom overlay, basically any HTML can be converted into an overlay. There are few overlays that are available as default and then forums where people have shared their overlays.

Q. How far can you zoom in a particular level ?
A. Access to whatever is available to Google maps, in the area of 22 zoom levels. Certain level that is available all over the world. No API to get that level. The documentation says the default level available at all levels, but it's likely 18.

Q. When Maps are not available, can people provide their own tiles
A. Yes.

Icons: Any icon (local or remote) can be used on the map. After 200 markers (on Firefox) and 100 on IE shows performance degradation.

Asynchronous HTTP: Static data in your JavaScript file. Or separate the data from the code. GDownloadUrl downloads XML file, parse the data (GXml.parse) and create a marker (addOverlay(new GMarker(..))). If it's huge amount of markup, then there could be a performance hit.

Geocoder: Translating in address into lat/long and mapping it.

Full documentation: www.google.com/apis/maps/documentation

Q. Will Google add the ability to print satellite views of the maps
A. Not right now but working on it.

Q. Do you have Routing APIs
A. Come to the advanced session.

Q. Is there a way to highlight marker
A. Yes through JavaScript, use GMarker.setImage and highlight the marker that user mouseover.

Q. Can the Geocoder return the quality of geocode
A. It just returns null, check the documentation.

Q. Can you display a title for each marker without mouse over
A. Markers use an image, open source publicly create Overlay called TextMarker would serve that purpose, or create your own in 20-30 lines.

Q. Can Flash be embedded in Info windows
A. Put any HTML in the Info Window.

Q. Displaying Raster data on the map like a bitmap
A. Come to Advanced session. Take an image, put it down on the map anywhere and then add overlays. Or an API where you can specify tiles for different zoom levels.

"Go Forth and Create Mashups" was the key message.

The multi-cuisine (Asian, Mexican, Italian ...) lunch gave a feeling of what Googlers might be eating every day.

A Look Behind the Scenes at Google

Jeff Dean, Google Fellow gave this talk explaining "Building a Computer System for the Worlds' Information". The talk was all about behind the scenes look at Google systems infrastructure.

Every Google service sees continuing growth in computational needs = More queries + More data + More results.

The goal is to create very large scale, high performance computing infrastructure => Hardware + Software systems to make it easy to build products. Focus on price/performance and ease of use.

The Hardware Design Philosophy is:

• Buy as many commodity class PCs (Prefer low-end server/PC-class designs)
• Larger number of unreliable computers than a big unreliable computer because reliability can be handled in software
• Rather parallelize and get latency down
• "Ultra-reliable" hardware makes programmers lazy
  • Fault-tolerance to be built into software

The following set of pictures show how Google's infrastructure evolved over the years:

The three main components in Google's infrastructure are:

• Google File Systems (GFS)
• MapReduce
• BigTable - Simple database doesn't support relational database but lets you spread data over multiple machines.

Why GFS ? Unique file system requirements Huge read/write bandwidth Reliability over thousands of nodes Most operating on large data blocks Need efficient distributed operations Have control over apps, libraries and operating systems

MapReduce - a programming model and library to simplify large-scale computations on large clusters.

Map phase extract relevant information from each record of the input, Reduce phase collects data together and produces final output.

MapReduce is

• Fast (locality optimization, optimized sorter, lots of tuning work)
• Robust (handles machine failure, bad records)
• Easy to Use (little boilerplate, supports many formats)
• Scalable (can easily add more machines to handler more data or reduce the run-time)
• Widely applicable (can solve a broad range of problems)

Lots of products within Google use it.

BigTable - A large-scale storage system for semi-structured data

• Higher level API that a raw file system
  • Somewhat like a database, but not as full-featured
• Useful for structured/semi-structured data
  • URLs
    • Contents, crawl metadata, links, anchors, pagerank ...
  • Per-user data
    • User preference settings, recent queries/search results ...
  • Geographic Data
    • Physical entities, roads, satellite imagery ...
• Scales to large amount of data

Allows to do low-level storage optimization help performance significantly

• Distributed multi-level map
• Fault-tolerant, persistent
  • Uses GFS underlying
• Scalable
  • Thousands of servers
  • Terabytes of in-memory data
  • Petabytes of disk-based data
  • Millions of reads/writes per second, efficient scans
• Self-managing
  • Servers can be added/removed dynamically, adjust to load imbalance

Lot of internal pre-processing such as satellite imagery. Largest BigTable manages ~3000TB of data spread over several thousand machines (larger cells planned)

Used by Google Print, My Search History, Orkut, Crawling/Indexing pipeline

Q. What flavors of Linux ?
A. Some using 2.4 and variants of 2.6

Q. Do you do destructive testing ?
A. Definitely yes. A battery of tests, take a data center offline, much more load than expected and see if it degrades gracefully and much more.

Q. Do you make an attempt to measure that interesting data grows than raw data ?
A. Raw data requires computation to make interesting data. Not looked at different class of data.

Q. Will we see an open source version of GFS ?
A. There have been some discussions about it which would make difficult to open source it. There is an open source structure (Hudop?) that created a mimic of MapReduce.

Q. Do you modify the Linux kernel ?
A. Do modify as problems are discovered, lot of them are contributed back to the community.

Q. Few 1000 thousand servers, what policy is about recycling and green computing ?
A. Generally donate after outlived it's reasonable life and try to repair the machine instead of throwing it away.

Mashups the Easy Way

First mashup - housingmaps.com. Paul did it before the Maps API.

Whole bunch of things before you can make/release the mashup.

Three parts to create a Mashup:

• Feeds
• User Interface
  • Common set of Ajax UI elements (cross-browser compatible)
  • Simple declaration of Ajax components (similar to using HTML tags)
  • Flexible way of using UI controls programmatically
• Infrastructure
  • Lots of machines and serving capacity
  • Hosting or serve infrastructure with no setup up
  • Simple data store modeled after a feed
  • Ability to sandbox application for testing
  • One click deployment of application

Experimental project built by a small team in NY

• An Ajax application that lets you built other Ajax applications
• JavaScript library
• Hosting container at googlemashups.com.

Feeds -> UI -> Test -> Publish -> Distribute

Mashups and Simple Applications made easy.

Google Desktop Gadgets - Access, Share and Personalize Information

The speaker started with a survey and the results were:

• 80% use gadgets on desktop
• 50% use Google gadgets
• 5% develop gadgets

2007 - The Year of the Gadget ?

Specialized mini-application that runs in a web page or on the desktop. There are two types of widgets:

• Universal gadget
  • Minimal web app
  • No downloads necessary
  • Needs online connection
  • HTML + JavaScript
• Desktop Gadget
  • Runs on the desktop
  • Advanced functionality
  • Works online/offline
  • XML + JavaScript

The talk focused on desktop gadgets.

Why develop desktop gadget ? Instantly reach millions of users, Easily distribute content, Increase traffic to your sites.

Gadget is a zip archive with .gg extension and the following structure:

An easier way to develop a gadget from scratch - Google Desktop Gadget Designer - allows you to easily experiment with WYSIWYG interface. It then showed a simple demo to create the gadget using Designer.

Advanced APIs - Add Flash, Graphics, Drag/Drop, Transparency, Download streams or Internet but following are the interesting ones:

• Communication API - send and receive data using the Google Talk framework. For example, multi-user games, social networking gadgets.
• Query API - find related information by searching for a specific term in the user index. For example, music or photo collection organizers. Searches the local desktop and prepare a list.
• Event API - Provide contextual information based on the users actions. For example, For web pages containing an address, show a map of the place.

Useful links:

• Desktop Google SDK
• Desktop Gadgets Developer Group
• FAQ / Knowledge Base
• Desktop Gadget Articles
• Google Desktop Blog
• Open source Desktop Gadgets

Next steps:

• Submit your gadget to Google to reach more users.
• The gadget will appear in the online gadget gallery and in the 'Add gadgets' panel on the users machine.
• Gadgets are automatically targeted and recommended to users who are most likely to use them.
• Populare gadgets will give you the chance to get into Google Desktop Hall of Fame

Key Take-Aways

• 2007 - Year of the Gadget
• Desktop Gadget benefits:
  • Easy to develop and install (Only JavaScript + XML required and WYSIWYG Designer)
  • Easy to use, low resource requirements
  • Powerful APIs offer opportunities for feature-rich designs
• Through the Google Desktop Gadget Gallery developers can:
  • Reach millions of users around the world
  • Target your gadget to users most likely to see it
  • Distribute your content and increase traffic to your website
  • Become famous as a top gadget developer

Custom Search Engine - Search in Your Hands

Ramanathan Guha started Custom Search Engine at Google. Google Custom Search extends Open Publishing to search.

Why Custom Search ?

• Anyone, anywhere can create a substantially better search experience for their chosen topic
• No more complexity that creating a webpage
• The usual incentives apply: fame, fortune, pride ...

What Custom Search provides ?

• Harness the power of Google search
• Put search results on your website
• Customize the look & feel
• Invite friends
• Make money

Allows producing a custom search on demand.

That's the end of sessions. All in all, it was a great crash course for different Google offerings.

Technorati: googledeveloperday gdd gdd07 gdd07us

Posted by Arun Gupta in web2.0  |  Comments[4]