What is a robot txt file?
This is a small text file of your website’s public folder. We call this file robot txt also known as the robots exclusion protocol or standard. Basically, it has instructions to tell search engine spiders not to crawl some pages or some sections of your website.
Now, we will see some details of the robot txt file because this file plays an important role of the SEO of a WordPress site.
Table of content
- What does robot txt file do?
- Where do I put the robot txt file?
- How do add robot.txt file to your site?
- How to edit robot txt for SEO
- How do I submit robots txt to Google?
Where do I put the robot txt file?
You can place robot txt file inside the root for your web folder. Search engines can access it www.techiecontent.com/robot.txt
A search engine can read the contents of this file so that it can crawl your site according to the instructions of the robot txt file.
You can edit the robot txt file to give instructions to search engine how to crawl your website.
This is super easy to edit the file but lots of web site owners do not know about this.
Please read the full article and you will understand what we are talking about.
How do add robot.txt file to your site?
There are two ways to add robot.txt file to your web server
use a FTP tool like FileZilla and connect to your server
You can use the File Manager from your cPanel
How to edit robot txt for SEO
Now you have the question: what should be in robot txt file?
This is the basic syntax of the standard robot txt file and this is the ideal robots txt for your site too. So you can use this as your sample robot txt file.
Robot txt user agent
You can specify the user agent or Search engine bot. Normally we allow all the search engine to crawl the web site.
User-agent: *
The * means that any search engine can crawl the entire website
Robot txt allow and disallow path
You can specify the path here to block being accessed by the search engine.
Disallow: /wp-admin
Disallow syntax tells Bot not to visit the /wp-admin folder.
If you are running your website on WordPress you do not want Bot to crawl your entire pages of the website.
You can use the robot txt file to block the wp-admin folder where you have the pages related to the admin side of the website.
In other words, search engines should not crawl the pages which you are not going to show to the public.
Can you edit robots.txt file of WordPress site?
Yes, you can edit the robots.txt file easily on your WordPress site. If you have the Yoast premium plugin you can do this effortlessly.
Add sitemap to robot txt
You can add your sitemap to the robot txt file using the following code.
sitemap: https://www.techiecontent.com/sitemap.xmlAlthough you do not add your sitemap to robots txt file search engine boot will look it in your web server.
How to test the robot file?
You can find the Robots Testing Tool in your google webmasters tool. Once you open the link you can select the domain name to run your test.
You can click the test button at the bottom of the screen
If the “Test” text changes to “Allowed,” that means your robots txt is valid.
Once you test the contents of the file you can edit the file and upload the new version to the root directory of your website.