This is a small text file of your website public folder. We call this file as robot txt also known as the robots exclusion protocol or standard. Basically it has instructions to tell search engine spiders to not to crawl some pages or some sections of your website.
Now, we will see some details of the robot txt file.
You can place robot txt file inside the root for your web folder. Search engines can access it www.techiecontent.com/robot.txt
A search engine can read the contents of this file so that it can crawl your site according to the instructions of the robot txt file.
You can edit the robot txt file to give instructions to search engine how to crawl your website.
This is super easy to edit the file but lots of web site owners do not know about this.
Please read the full article and you will understand what we are talking about.
There are two ways to add robot.txt file to your web server
use a FTP tool like FileZilla and connect to your server
You can use the File Manager from your cPanel
Now you have the question: what should be in robot txt file?
This is the basic syntax of the standard robot txt file and this is the ideal robots txt for your site too. So you can use this as your sample robot txt file.
You can specify the user agent or Search engine bot. Normally we allow all the search engine to crawl the web site.
The * means that any search engine can crawl the entire website
You can specify the path here to block being accessed by the search engine.
Disallow syntax tells Bot not to visit the /wp-admin folder.
If you are running your website on WordPress you do not want Bot to crawl your entire pages of the website.
You can use the robot txt file to block the wp-admin folder where you have the pages related to the admin side of the website.
In other words, search engines should not crawl the pages which you are not going to show to the public.
You can add your site map to the robot txt file using the following code.
Although you do not add your sitemap to robots txt file search engine boot will look it in your web server.
You can find the Robots Testing Tool in your google webmasters tool. Once you open the link you can select the domain name to run your test.
You can click the test button at the bottom of the screen
If the “Test” text changes to “Allowed,” that means your robots txt is valid.
Once you test the contents of the file you can edit the file and upload the new version to the root directory of your website.