Apple yesterday gave a few interesting details on something it's calling "Applebot", the company's in-house web crawler that is used to help power services like Siri and Spotlight on iOS and OS X platforms

 Apple Confirms 'Applebot' as In-House Web Crawler for Services Like Siri and Spotlight