| comp., MS | Robots.txt file | file Robots.txt (" A file that informs search engines about the pages in a Web site that the owner wants to exclude from, or allow for, indexing. The file contains "allow" rules and "disallow" rules to indicate which portions of a Web site should, or should not be, indexed. If a Web site has subdomains, each subdomain must have its own Robots.txt file.") |