ArticleBloggerSEOWordPress

Create Perfect Robots.txt for Blogger Wordpress boost Google SERP

Understand how to implement the best robots.txt file to your WordPress or Blogger Blogspot blog.

Perfect robots.txt for blogger WordPress CMS
Perfect robots.txt for blogger WordPress CMS
 Google Webmaster tool robots.txt file check for blogger and WordPress
Google Webmaster tool robots.txt file check for blogger and WordPress

What is the function of the robots.txt file?

In a website or blog, robots.txt file tells search engine bots about the crawlable area of that blog or website. This file contain robots name i.e. name of the search engine bot that crawls your website and index it in their web search result. And define which section to be crawl and which not.

Usually, we use command for all search engine crawling bot as we want to index our pages throughout the web.

Should we disallow search engine to crawl some pages? (for not to index in search results)

For pages in which we can control or implement noindex tag, that should be allowed to crawl. And pages that can’t be edited in any way or we are not able to use noindex should be disallowed.

Most important pages and sections that must be indexed.

You must index your homepage, pages, and posts. Because it contain the work that you want to the public.

If you want to make some content online but not public means only viewers with the link can only access that, in such scenario, you’ve to put URLs under a category and that category should be blocked by robots.txt

Finally robots.txt file for blogger

User-agent: Mediapartners-Google
Disallow: 
User-agent: *
Allow: /
Sitemap: https://site.blogspot.com/sitemap.xml
Sitemap: https://site.blogspot.com/sitemap-pages.xml

Please keep in mind that, for above robots.txt you’ve to add the noindex meta tag to search, label, archive pages. by simply edit your theme. Paste code provided below just after <html> tag.

<b:if cond='!data:view.isHomepage and !data:view.isSingleItem'>
<meta content='noindex,follow' name='robots'/>
</b:if>

How this robots.txt is perfect or best for the blogger?

This robots.txt file regulates search engine bot and guides it, what to crawl in your website and what to not. And when the bot of any search engine reaches in the archive, tags, search pages meta tag declare that to noindex and thus only required pages will index only. This is how it is best robots.txt file, SEO point of view.

How to Edit robots.txt file in Blogger

And robots.txt for WordPress.

User-agent: *
Disallow: /wp-admin/
Allow: /

You can add more sections in the Disallow section if you want to, e.g. for pages under “blog” use Disallow: /blog/, this will block bot to crawl the section.

robots.txt file that allows the bot to crawl the whole website and fetch pages you want to index. This needs some changes in your website meta tags for SEO purposes. To change meta tags in WordPress You’ve to install Yoast SEO Plugin. With this plugin, you can mark noindex category, tags and error pages. So that junk will not index to google. Instead of Yoast, you can also use the code given below. You just have to edit the header.php file of your theme, just below <html> in header.php paste code given below.

<?php if (! is_home() && ! is_single() && ! is_page() ||  is_paged()  ) {?>
    <meta name="robots" content="noindex,follow">
<?php } ?>

How this robots.txt file is best for WordPress?

This robots.txt file regulates search engine bot to crawl your website. And when the bot of any search engine reaches the section like archive, category, tags, search pages, in such case meta tag declare that to noindex and thus only required pages will only index. In some cases, in WordPress, you can index category pages so you can make changes accordingly. If your blogging website is common then in that case you can use provided guidelines. This is how it is best robots.txt file, SEO point of view.

I hope this article will help you to understand robots.txt file and its working. If you like this article or face any issue, doubt please comment below.

Related Articles

2 Comments

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top button