How to maximize use of Google Bots for your site

There are about 10 million web pages on the index which need to be indexed by any search engine. Each search engine uses a spider to index the pages. The spider used by Google is known as Google bot and in today’s article, I would share some of its internal working procedures and how it affects indexing of your site. Google bot is commonly used as a single term while in fact google bots are of two types; deepbot and fresh bot. Deepbot is the google spider that is meant to deep crawl of your site and index as many pages and assess internal structure of your site; that is to know about site map of your site. In fact if you keep the site map ready or keep the structure and navigation of your site simple, you are helping this bot to index maximum data from your site. Each page indexed by google has a time stamp on it. Fresh bot is meant to index the fresh or modified pages of your site so as to keep the content updated on the web.

These bot decide about their periodicity of visit to any site depending upon the frequency the content is changed. Thus more frequently any content is changed or updated, higher would be the visits of these bots. A good an efficient site map would help deep bot to crawl the site better and about the links it was unable to crawl so as to attempt their crawl in subsequent visit. In case you are not having customized site map for your site, you may have Google site for your site. These would help you guide the pages Google bot was unable to crawl.
Another bot from Google is mediabot which is used to analyse adsense pages and determine which would be most relevant ads for your page. As a webmaster you would be interested in maximum clicks and click rate does increase in case more relevant ads are delivered on your page. If you allow media bot by simply entering the following code into your robots.txt file; User-agent: Mediapartners-Google*
Disallow; it would ensure that the MediaBot is able to place relevant Adsense ads on your site.
You must be knowing that ads may still appear on the page if the MediaBot has not yet visited, but in this case the ads chosen will be based on the overall theme of the site or the public service announcements are displayed instead, which give nothing. It must be added here that extent of the role that media bot plays in delivering targeted ads is a subject matter of debate and many sites do get relevant ad despite no specific configuration in robot file, but that is because of overall content of the site.
There is also a controversy that pages with adsense would be indexed more frequently but Google denies any such advantage given to adword subscribers and there's no boost at all in rankings if you're in AdSense or Google News.
However ir does promote a slightly higher degree of spider activity and Google uses its crawl cache for use by its other spiders. Similarly, by using Google Analytics on your site you can expect additional spider activity.
However such spidering has a limited utility in helping indexing of your site and having quality inbound links, quality content, and frequent updates helps the site like nothing else.
Spider used by Google to index images in ImageBot and it crawls the Web for images to place in Google's image search. If your site has large number of images that you would like to get indexed, it would be helpful if you use proper filename, surrounding text, alt text, and page title for those images. This can get you some extra traffic from Google. However such traffic has low sales conversion rate but eats away your bandwidth.
Another important bot from Google is adsbot but it is helpful only to those using Google adwords. It analyzes the content of advertising landing pages and determines the Quality score that Google assigns to your ads. This quality score coupled with the amount you bid helps determine the position of your ads. Therefore, ads with a high quality score can rank higher even if other advertisers are paying more than you.
This is one of Google's many efforts to ensure that they are delivering the best results to their users.
Can you still block being spidered? Of course, but it will lower your overall Adwords quality score, which could end up lowering the positioning of your ads. If possible, it is best to give AdsBot complete access to your site.
Today's Google bots are becoming more advanced all the time. However, nothing beats relevant, quality, updated content. Deliver that and the search engines will eat it up.
At the end , I must add that while knowing the way spiders work does help you get better indexing but nothing would beat the unique, updated and frequent content.