Revise Computingrevisecomputing.co.uk
Free student revisionSee questionsFor schoolsSchool enquiry
Free revisionSee questionsFor schoolsSchool enquiry
Knowledge/A Level/H446/01 Computer systems/Web Technologies/Search engine indexing

Search engine indexing

1.3.4 Web Technologies

Practise exam questions

Web Technologies — Note 2 of 4

Candidates should understand how and why search engine results are indexed.

What you need to know

Search engines cannot search the whole web every time a query is made, so they build an index in advance. Programs called crawlers visit pages and record their content, so that results can be found almost instantly when a user searches.

What to know

  • The index stores keywords (and where they appear) along with the URLs of the pages that contain them, similar to the index at the back of a book.
  • Information such as page titles, headings, meta tags and the position of keywords is used to help judge relevance.
  • Crawlers revisit pages regularly so the index stays up to date as pages change.
  • Website owners can use a robots.txt file to ask crawlers not to index certain pages.

Key terms

Search engine indexing

The process of collecting, parsing and storing information about web pages so that relevant pages can be found quickly when a search is made.

Web crawlerspider

A program that visits web pages, follows the links on each page to find new pages, and passes page content back to be indexed.

Exam technique

  • Tip:Explain WHY indexing is needed: searching the whole web for every query would be far too slow, so a prepared index is searched instead.

Ready to test yourself?

Practise 1.3.4 questions
← Previous noteHTML, CSS and JavaScript1.3.4 Web TechnologiesNext note →PageRank algorithm1.3.4 Web Technologies

Back to Web Technologies

Web Technologies

Note 2 of 4

  1. 1HTML, CSS and JavaScript
  2. 2Search engine indexing
  3. 3PageRank algorithm
  4. 4Server and client side processing
Topic overview