Robots.txt explained for small business owners

Robots.txt is a small text file at the root of your website, at yourbusiness.com/robots.txt, that tells search engine crawlers which parts of the site they may visit and which to skip. Most small-business sites need nothing special in it, and website builders write a sensible one for you. The thing to check is that it does not accidentally block Google from your whole site, which takes only one line and happens more often than you would think.

What robots.txt is for

Search engines send automated visitors, called crawlers or bots, to read websites. Before a well-behaved crawler reads anything on your site, it checks robots.txt for instructions. The file can say things like "do not visit the admin area" or "do not visit the shopping cart pages".

It exists mostly for large sites, to stop crawlers wasting time on endless filter pages, search results or login screens. For a ten-page site, it has very little to do.

Two things it is not:

• It is not a lock. Robots.txt is a polite request. Google and other reputable search engines follow it; bad bots ignore it. Never rely on it to hide anything private.
• It is not a way to remove a page from Google. That surprises people, and it is the source of the most common mistake, explained below.

How to read yours

Type your domain followed by /robots.txt into any browser. You will see a few lines of plain text. Here is what they mean:

• User-agent: says which crawler the rules below apply to. An asterisk (*) means all crawlers. "Googlebot" means Google's.
• Disallow: followed by a path means "do not visit addresses starting with this". "Disallow: /admin" blocks anything under /admin.
• Allow: makes an exception inside a disallowed area.
• Sitemap: gives the address of your sitemap, so crawlers can find it.

A typical, healthy robots.txt for a small business has three lines, something like:

• User-agent: *
• Disallow: /admin
• Sitemap: https://www.yourbusiness.com/sitemap.xml

An empty "Disallow:" line, with nothing after it, means "nothing is blocked". That is fine too.

If there is no robots.txt at all and you see a "not found" page, that is also fine. Crawlers treat a missing file as permission to visit everything.

The line that blocks your whole site

This is the pair of lines to look for:

• User-agent: *
• Disallow: /

A single slash after Disallow means "do not visit anything on this site". It is often added on purpose while a site is being built, so search engines do not show an unfinished site, and then forgotten when the site goes live. The site then sits on the internet, working perfectly for anyone who types the address, and invisible to Google.

Signs this has happened:

• A site: search for your domain on Google shows nothing, or shows your address with a note that no information is available for the page.
• Google Search Console reports pages as "Blocked by robots.txt".

If you find it, remove that line, or change it to "Disallow:" with nothing after it. On a builder, the file is usually controlled by a "hide site from search engines" setting rather than edited by hand; turning that setting off fixes the file. Then request indexing of your home page in Search Console. Why your website isn't showing up on Google covers the other causes if this is not it.

Robots.txt vs noindex: the mistake that keeps pages in Google

Owners sometimes want a page out of Google: an old price list, a staff-only page, a page for a service they stopped offering. The natural instinct is to block it in robots.txt. That does not do what they expect.

Robots.txt stops Google from visiting a page. It does not stop Google from listing it. If other pages link to it, Google can still show the address in results, just without a description, because it was told not to read the page.

A noindex instruction stops Google from listing a page. It is a tag inside the page itself, usually switched on with a "hide this page from search engines" option in your builder's page settings. Google has to visit the page to see that instruction.

So the two get in each other's way: if you block a page in robots.txt and mark it noindex, Google never visits, never sees the noindex and may keep listing the page.

To take a page out of Google:

1. Set it to noindex in its page settings (or delete the page if you no longer need it).
2. Make sure robots.txt does not block it, so Google can see the instruction.
3. Wait for Google to revisit, or request it with URL Inspection in Search Console.
4. For urgent cases, Search Console also has a Removals tool that hides a page from results temporarily while the noindex takes effect.

Can you edit it?

That depends on your platform:

• Wix, Squarespace, Shopify and GoDaddy's builder create robots.txt automatically. Some let you edit it in an advanced SEO setting; most owners never need to.
• WordPress creates a basic one automatically. SEO plugins and some hosts let you edit it.
• Hand-built sites have it as a real file in the site's main folder, edited by whoever maintains the site.

Unless you have a specific reason, leave it as the builder wrote it, and only check that it does not contain the blocking line.

What about AI crawlers?

Some owners now see lines in robots.txt naming AI companies' crawlers, such as GPTBot or Google-Extended. These ask those companies not to use your pages for training their AI models. Whether to include them is your choice; blocking them does not affect whether your site appears in regular Google search. How AI answers in Google relate to your site is covered in AI answers in Google search and small businesses.

When a new site launches

Launch day is when the blocking line is most likely to slip through. Add these to your launch checklist:

• Open yourdomain.com/robots.txt and look for "Disallow: /".
• Check the builder's "hide from search engines" setting is off.
• Check the Sitemap line, if present, points at your real domain, not a temporary or staging address.

Is your website hidden from Google by mistake? has a fuller list of the settings that cause this.

What to do this week

• Open your site's robots.txt in a browser.
• Look for "Disallow: /" under "User-agent: *". If it is there and your site is live, remove it or turn off the hiding setting.
• Note whether it lists your sitemap.
• If you have pages you want out of Google, use the page's noindex setting instead of robots.txt.

If your site was set up years ago and nobody is sure what its settings say, you can see your site rebuilt first with search settings checked and open to Google from launch day.