robots.txt generation — user-agent groups with allow/disallow/crawl-delay, plus sitemap + host directives.
Optional
robots.txt generation — user-agent groups with allow/disallow/crawl-delay, plus sitemap + host directives.