The robots.txt file, when it exists, is stored at the root of a website. It contains a list of the resources of the site that are not supposed to be indexed by search engine spiders.
By convention, robots read robots.txt before indexing a website.
Its content may therefore be of interest to us.

http://10.10.10.8/robots.txt