SEO Guide: Creating robot txt to crawl your site

October 4, 2020

What is a robot txt file?

This is a small text file of your website public folder. We call this file as robot txt also known as the robots exclusion protocol or standard. Basically it has instructions to tell search engine spiders to not to crawl some pages or some sections of your website.

Now, we will see some details of the robot txt file.

Where do I put the robot txt file?

You can place robot txt file inside the root for your web folder. Search engines can access it www.techiecontent.com/robot.txt 

A search engine can read the contents of this file so that it can crawl your site according to the instructions of the robot txt file.

You can edit the robot txt file to give instructions to search engine how to crawl your website.

This is super easy to edit the file but lots of web site owners do not know about this.

Please read the full article and you will understand what we are talking about.

How do add robot.txt file to your site?

There are two ways to add robot.txt file to your web server

use a FTP tool like FileZilla and connect to your server

You can use the File Manager from your cPanel

How to edit robot txt for SEO

Now you have the question: what should be in robot txt file?

This is the basic syntax of the standard robot txt file and this is the ideal robots txt for your site too. So you can use this as your sample robot txt file.

Syntax of the robot.txt

Robot txt user agent

You can specify the user agent or Search engine bot. Normally we allow all the search engine to crawl the web site.

User-agent: *

The * means that any search engine can crawl the entire website

Robot txt allow and disallow path

You can specify the path here to block being accessed by the search engine.

Disallow: /wp-admin

Disallow syntax tells Bot not to visit the /wp-admin folder.

If you are running your website on WordPress you do not want Bot to crawl your entire pages of the website.

You can use the robot txt file to block the wp-admin folder where you have the pages related to the admin side of the website.

In other words, search engines should not crawl the pages which you are not going to show to the public.

Add sitemap to robot txt

You can add your site map to the robot txt file using the following code.

sitemap: https://www.techiecontent.com/sitemap.xml

Although you do not add your sitemap to robots txt file search engine boot will look it in your web server.

How to test the robot file?

You can find the Robots Testing Tool in your google webmasters tool. Once you open the link you can select the domain name to run your test.

Test robot.txt file

You can click the test button at the bottom of the screen 

If the “Test” text changes to “Allowed,” that means your robots txt is valid.

Once you test the contents of the file you can edit the file and upload the new version to the root directory of your website.