What is Robots.txt File ?

on

Robots.txt File:

  • Gives instruction to crawler that which page have to index or which page not to index
  • It is used to restrict the crawler
Robots.tx
Robots.txt


Basic Format

User-agent: [user-agent name]
Disallow: [URL string not to be crawled]
By using this file we can restrict the web crawler the crawl admin page of our website 

























































Comments











Post a Comment