What are crawlers / robots / spiders?

A spider is a program run by a search engine to build a summary of a website’s content (content index). It creates a text-based summary of content and an address (URL) for each webpage.

When a person searches, the keyword(s) they enter are compared with the available website content indexes. Due to the large number of webpages indexed, direct text-only-matching is rare, rather search engines use sophisticated logics (algorithms) to rank potential matches. For example, the underlying information hierarchy of a webpage (semantic markup) may be factored into the ranking a webpage is assigned.