Google analytics is free web service from Google to track web visits and statistics of your web site. It will give excellant reports on your site usage, web site content and visitors in charts and graphs.
Google WebMaster tools are free web service from Google for webmasters. Basically it helps webmasters to check indexing status and optimize visibility of their websites.
using these tools webmaster can,
Check and set the crawl rate, and view statistics about how Googlebot accesses a particular site
Generate and check a robots.txt file
List internal and external pages that link to the site
See what keyword searches on Google led to the site being listed in the SERPs, and the click through rates of such listings
View statistics about how Google indexes the site, and if it found any errors while doing it
Set a preferred domain (e.g. prefer "google-page-ranking.blogspot.com" over "www.google-page-ranking.blogspot.com"), which determines how the site url is displayed in SERPs.
Showing posts with label robots.txt file. Show all posts
Showing posts with label robots.txt file. Show all posts
What is the role of Robots.txt file in SEO?
The purpose of robots.txt file is to tell the search engine crawlers not to index the folders or files that you don't want to see in Search engines.
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file at all.
Robots are often used by search engines to categorize and archive web sites. Also known as "Robot Exclusion Standard" and "Robots Exclusion Protocol".
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.
For websites with multiple subdomains, each subdomain must have its own robots.txt file. i.e) If you have robots.txt file with domain.com , and no robots.txt file with subdomain.domain.com, the rules that would apply for domain.com will not apply to subdomain.domain.com.
Below Syntax is used to allow all files in the website
User-agent: *
Allow: /
This is to restrict all files
User-agent: *
Disallow: /
To selectively restrict folders
User-agent: *
Disallow: /tmp/
Disallow: /pesonal/
To selectively restrict specific file
User-agent: *
Disallow: /personal/mybankdetails.html
Some crawlers support a Sitemap directive, i.e) allowing multiple Sitemaps in the same robots.txt file.
Example-
Sitemap: http://www.domain.com/sitemaps/sitemap.xml
Sitemap: http://www.domain.com/news/newsitemaps/newssitemap.xml
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file at all.
Robots are often used by search engines to categorize and archive web sites. Also known as "Robot Exclusion Standard" and "Robots Exclusion Protocol".
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.
For websites with multiple subdomains, each subdomain must have its own robots.txt file. i.e) If you have robots.txt file with domain.com , and no robots.txt file with subdomain.domain.com, the rules that would apply for domain.com will not apply to subdomain.domain.com.
Below Syntax is used to allow all files in the website
User-agent: *
Allow: /
This is to restrict all files
User-agent: *
Disallow: /
To selectively restrict folders
User-agent: *
Disallow: /tmp/
Disallow: /pesonal/
To selectively restrict specific file
User-agent: *
Disallow: /personal/mybankdetails.html
Some crawlers support a Sitemap directive, i.e) allowing multiple Sitemaps in the same robots.txt file.
Example-
Sitemap: http://www.domain.com/sitemaps/sitemap.xml
Sitemap: http://www.domain.com/news/newsitemaps/newssitemap.xml
Subscribe to:
Posts (Atom)