Robots.txt generator tool is a web crawler command txt file that tell crawler who is allow to be crawled. In simple word website owner is giving permission to specific search engine crawler to crawl the site.
Robots.txt Generator
Build a robots.txt file with granular, per-bot crawl rules
Default rule for all crawlersApplies to “User-agent: *” — anything not overridden below
Not honored by Google, but respected by some other crawlers.