Network Discovery

The robots.txt file, when it exists, is stored at the root of a website. It contains a list of the resources of the site that are not supposed to be indexed by search engine spiders.
By convention, robots read robots.txt before indexing a website.
Its content may therefore be of interest to us.
Plus d'info :