Home Article Blogger SEO Let Google crawl your whole webpage.

Blogger SEO Let Google crawl your whole webpage.

Are you confused which pages to be index by Google and which to be not?

Blogger SEO role of Google page indexing: Blogger now gives you more control to its features than earlier. As blogger launch new theme panel supporting new themes.

Blogger SEO index noindex tool. A smart way

Understand Google Indexing

Before starting you’ve to understand how it works.

When google bot visit your blog and crawl your webpage. It follow some rules that are defined by you in robots.txt file of your website. When you create a blogger blog this text file auto generated along with that. You can check that by visiting http://www.yourblog.com/robots.txt or https://xyz.blogspot.com/robots.txt

As we already discussed this file contain rules for bot crawl. Try visiting your robots.txt file

User-agent: Mediapartners-Google 
Disallow: 
User-agent: * 
Disallow: /search 
Allow: / 
Sitemap: https://www.xyz.blogspot.com/sitemap.xml 

What is the mean of each line ? Understand it here

If we talk about first line User-agent: Mediapartners-Google . Here User-agent: means a bot that crwal your website. And Mediapartners-Google is the type of that bot or here it is Google Adsense Ads bot spider. This line declares that bot is Google Ads Bot and that can visit page are, now pages allowed and disallowed declared in next line/s. The next line is Disallow: declared that no page is Disallow and all pages can be crwal. This is for the first bot declared in the robots.txt file.

Now here is the next line that declared User-agent: * means bots or spiders of all search engine. next line Disallow: /search means that all pages that are similar structure to that of https://xyz.blogspot.com/search are not allowed to crawl. Next line : Allow: / states that all other pages are allowed.

Here now Sitemap: https://www.xyz.blogspot.com/sitemap.xml, declared XML sitemap that is used by Google or all other search engines for indexing of your website or webpage.

Changes to be made on robots.txt

Now I hope you understand all robots.txt file functions. So some changes to be made for better search engine crawling and index your web pages.

  • Go to blogger.com and sign in.
  • Dashboard > setting > Search Preference.
  • Find Custom robots.txt and enable it.
  • Paste the code given below. Note: Change the website name with yours.
User-agent: *

Allow: /

Sitemap: https://www.xyz.com/sitemap.xml
Sitemap: https://www.xyz.com/sitemap-pages.xml

Here you submitted both sitemaps for pages and posts. And also, Google Spider or bot can crawl your whole website. But for SEO, keep this in mind that you’ve to submit only post and pages for Search Engine Index. Why? Read this here.

To boost SEO and avoid junk to index please follow this practice.

  1. Go to Theme
  2. Now click on Edit HTML
  3. Find <b:include data='blog' name='all-head-content'/>

Now, below this line <b:include data='blog' name='all-head-content'/>, past the code given below.

<b:if cond='!data:view.isHomepage and !data:view.isSingleItem'>
 <meta content='noindex,follow' name='robots'/>
<else/>
 <meta content='max-snippet:-1, max-image-preview:large, max-video-preview:-1' name='robots'/>
</b:if>

What happens with this code?

  • Boost SEO as only juice will index to google or other search engines.
  • Resolve webmaster error “blogger blocked by robots.txt” for all pages.
  • the crawling channel will vast.
  • The bot or spider can visit all pages of your website/ webpage. That result in more pages to index.

You can download our blogger themes, Best SEO, Speed and User Friendly.

I hope you like this article, If any question regarding any issue please comment below. Thank you.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.