Robots.txt File Generator Tool: Free Robots.txt File Generator

Every website owner needs to use a robots.txt generator tool to get a customized robots.txt file for their site, which is essential for indexing your site properly on the Google search engine and facilitating the crawling of search engine spiders.

How to Use Robots.txt Code Generator Tool

  1. Select the hosting type of your site: Blogger or WordPress
  2. Enter your site URL with the prefix https:// in the provided field.
  3. Click Generate, and the robots.txt file will start processing.
  4. Copy the generated code to your site in: Blogger settings, enable custom Robots.txt content, and paste the code there.
  5. Now, go to Robots.txt Testing Tool and paste the same code there and click Submit.

Generate Robots.txt File for Blogger
Enter your site URL with the prefix ‘https://’

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: User-Agent: * Allow: /wp-content/uploads/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-content/plugin Disallow: /wp-admin/ Disallow: /blog/page/ Disallow: /search/ Sitemap:

Benefits of Creating a Custom Robots.txt File

By creating a robots.txt file for your site, you'll be able to instruct the Google search engine about all the pages you want to display as search results, as well as to specify the pages you want to prevent crawling and indexing to.

You might want to block certain pages from your site and disallow search engine spiders from accessing them.

Add Disallow: /yourpage.html to your robots.txt file and send a notification to update the new code, make sure to change yourpage.html with the link of the page(s) you want to prevent indexing before sending.

Full Explanation of robots.txt Commands

User-agent: Mediapartners-Google: This is the code to allow Google AdSense spiders to crawl your site and has a fixed value and cannot be modified.

Disallow: This command is used to prevent specific pages from your site from appearing as results in Google search, for example: search terms used by visitors to search for pages within your site, so we add Disallow: /search to prevent indexing search result links.

Allow: / Means allowing indexing. After adding Disallow to block access to a specific page, we use Allow afterwards to allow crawling for the rest of the site pages and indexing them.

Sitemap: The sitemap file containing a list of the topics published on your site, the same code must be sent and installed on Google Search Console and the robots.txt file alike to ensure search engine spiders understand your site's content correctly.