Robots.Txt File, situated in an internet site's root directory, guides search engine crawlers on what to index and what to ignore. It performs an important r It is a proprietary metric and isn't used by search engines like google and yahoo like Google within their algorithms. In its place