Are you confused which pages to be index by Google and which to be not?
Blogger SEO role of Google page indexing: Blogger now gives you more control to its features than earlier. As blogger launch new theme panel supporting new themes.
Understand Google Indexing
Before starting you’ve to understand how it works.
When google bot visit your blog and crawl your webpage. It follow some rules that are defined by you in robots.txt file of your website. When you create a blogger blog this text file auto generated along with that. You can check that by visiting http://www.yourblog.com/robots.txt or https://xyz.blogspot.com/robots.txt
As we already discussed this file contain rules for bot crawl. Try visiting your robots.txt file
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: https://www.xyz.blogspot.com/sitemap.xml
What is the mean of each line ? Understand it here
If we talk about first line
User-agent: Mediapartners-Google . Here
User-agent: means a bot that crwal your website. And
Mediapartners-Google is the type of that bot or here it is Google Adsense Ads bot spider. This line declares that bot is Google Ads Bot and that can visit page are, now pages allowed and disallowed declared in next line/s. The next line is
Disallow: declared that no page is Disallow and all pages can be crwal. This is for the first bot declared in the robots.txt file.
Now here is the next line that declared
User-agent: * means bots or spiders of all search engine. next line
Disallow: /search means that all pages that are similar structure to that of https://xyz.blogspot.com/search are not allowed to crawl. Next line :
Allow: / states that all other pages are allowed.
Sitemap: https://www.xyz.blogspot.com/sitemap.xml, declared XML sitemap that is used by Google or all other search engines for indexing of your website or webpage.
Changes to be made on robots.txt
Now I hope you understand all robots.txt file functions. So some changes to be made for better search engine crawling and index your web pages.
- Go to blogger.com and sign in.
- Dashboard > setting > Search Preference.
- Find Custom robots.txt and enable it.
- Paste the code given below. Note: Change the website name with yours.
User-agent: * Allow: / Sitemap: https://www.xyz.com/sitemap.xml Sitemap: https://www.xyz.com/sitemap-pages.xml
Here you submitted both sitemaps for pages and posts. And also, Google Spider or bot can crawl your whole website. But for SEO, keep this in mind that you’ve to submit only post and pages for Search Engine Index. Why? Read this here.
To boost SEO and avoid junk to index please follow this practice.
- Go to Theme
- Now click on Edit HTML
<b:include data='blog' name='all-head-content'/>
Now, below this line
<b:include data='blog' name='all-head-content'/>, past the code given below.
<b:if cond='!data:view.isHomepage and !data:view.isSingleItem'> <meta content='noindex,follow' name='robots'/> <else/> <meta content='max-snippet:-1, max-image-preview:large, max-video-preview:-1' name='robots'/> </b:if>
What happens with this code?
- Boost SEO as only juice will index to google or other search engines.
- Resolve webmaster error “blogger blocked by robots.txt” for all pages.
- the crawling channel will vast.
- The bot or spider can visit all pages of your website/ webpage. That result in more pages to index.
You can download our blogger themes, Best SEO, Speed and User Friendly.
I hope you like this article, If any question regarding any issue please comment below. Thank you.