Advanced Robots.txt Generator

Create customized robots.txt rules for multiple search engine crawlers

Crawler #1

User-agent (Crawler) All crawlers (*) Googlebot (Google) Googlebot-Image (Google Images) Googlebot-News (Google News) Googlebot-Video (Google Video) Bingbot (Bing) Slurp (Yahoo) DuckDuckBot (DuckDuckGo) Baiduspider (Baidu) YandexBot (Yandex) Applebot (Apple) Facebook crawler Twitter crawler LinkedIn crawler Pinterest crawler Custom crawler

Disallow Rules

Disallow /admin/ directory

Disallow /cgi-bin/ directory

Disallow /temp/ directory

Disallow /images/ directory

Disallow /scripts/ directory

Custom path:

Allow Rules (optional)

Allow /public/images/ directory

Allow /css/ directory

Allow /js/ directory

Custom path:

Crawl-delay (seconds, optional)

Sitemap URL (optional)

Your robots.txt file:

How to use your robots.txt file:

• Copy the generated content above
• Create a new text file named "robots.txt"
• Paste the content into this file
• Upload the file to the root directory of your website (e.g., https://example.com/robots.txt)
• Test your robots.txt file using Google Search Console or other webmaster tools

About Crawlers:

• * - Applies to all crawlers
• Googlebot - Google's main crawler
• Googlebot-Image - Google's image crawler
• Bingbot - Microsoft Bing's crawler
• Slurp - Yahoo's crawler
• Applebot - Apple's crawler for Siri and Spotlight
• Social media crawlers - Control how social platforms access your content