Not every page on a business website belongs in Google. Thank you pages, staging copies and internal search results rarely help anyone who finds them in search. Two tools are often used to manage this. They are the robots.txt file and the noindex rule.
The two are easy to mix up, and using the wrong one can backfire. One controls what search engines crawl. The other controls what appears in results. Knowing the difference saves a lot of confusion later.
What robots.txt does
A robots.txt file sits at the root of your domain and gives instructions to crawlers. Google’s introduction to robots.txt says its main purpose is to manage crawler traffic. It tells crawlers which URLs they may access. This helps stop your server being overloaded by requests.
The same guide is clear that robots.txt is not a way to keep a page out of Google. A blocked URL can still show in search results if other pages link to it. In that case it usually appears without a description. That surprises many site owners who thought blocking meant hiding.
Robots.txt also has limits beyond Google. Following its rules is voluntary, so some crawlers ignore it. Different crawlers can also read the syntax in different ways. It is a polite request, not a lock on the door.
What noindex does
Noindex is a rule that tells search engines not to show a page in their results. It can be added as a robots meta tag in the page head. It can also be sent as an X-Robots-Tag in the HTTP response header. Google’s guide to blocking search indexing with noindex says both have the same effect.
The header option is useful for files that are not web pages. Google notes it works for PDFs, videos and images. Once Googlebot crawls a page and finds the rule, Google drops it from search results. That holds even when other sites link to it.
Google also says it does not support noindex rules inside robots.txt. The rule has to live on the page or in the header. Some other search engines may read noindex differently. A page could still appear in their results.
The mistake that cancels both out
The most common trap is using both tools on the same page. If robots.txt blocks a page, Googlebot never crawls it. That means it never sees the noindex rule. The page can then stay in results when other pages link to it.
So if you want a page out of Google, let it be crawled and add noindex. Leave it out of your robots.txt blocking rules. Give Google time to recrawl the page and drop it. Check the result in Search Console before changing anything else.
Which pages usually suit noindex
Every site is different, so think about who would land on each page from a search. Pages that only make sense after an action are common candidates. So are pages with thin or repeated content. Typical examples on a business site are listed below.
- Thank you pages shown after a form or purchase
- Internal site search results
- Login and account pages
- Tag or archive pages with very little content
Your main service pages, location pages and blog posts should almost always stay indexed. These are the pages people actually search for. If you work on local SEO, check that none of your suburb or service pages carry a stray noindex. It is an easy setting to switch on by accident.
Where to set these on a typical site
Many content management systems handle this through settings rather than code. Google’s robots.txt guide notes that sites on platforms like Wix or Blogger may need to use the platform’s own search settings. Editing files directly is not always possible or wise. Ask whoever built your site where these settings live.
Some platforms also have a single setting that hides the whole site from search engines. It is sometimes switched on while a site is being built. If it stays on after launch, new pages may never show up. Check it first if a new site gets no search traffic.
Checking your work in Search Console
Google needs to recrawl a page before it notices a new noindex rule. Its guide says this can take months for less important pages. You can ask for a recrawl using the URL Inspection tool. The tool also shows the HTML Googlebot received, so you can confirm the rule is there.
The Page Indexing report in Search Console shows which pages Google found a noindex rule on. Review it after any big change to your site. If a page you care about appears there, remove the rule and request a recrawl. Our tips on dealing with web designers cover how to agree on who checks what.
Robots.txt, noindex and your wider SEO
These tools are about housekeeping, not rankings. They help Google spend its time on the pages that matter. They also stop odd pages from showing up when people search your business name. Good housekeeping starts early, from choosing a domain name for SEO through to launch.
Remember that hidden pages can still be shared directly. A noindex page linked from a social post will still open for anyone who clicks. That can suit campaign pages you only want people to reach through social media marketing. Keep in mind that noindex is not privacy. For truly private content, Google’s guide suggests password protection.
Frequently asked questions
Can I use robots.txt to remove a page from Google?
Not reliably. Google says robots.txt is not a mechanism for keeping a web page out of Google. A blocked page can still appear if other sites link to it. Use noindex, password protection or remove the page instead.
How long does noindex take to work?
Google has to recrawl the page first. Google says this can take months for less important pages. You can request a recrawl with the URL Inspection tool. Check the Page Indexing report afterwards.
Does noindex work on PDFs?
Yes, through the X-Robots-Tag response header. Google says the header works for files such as PDFs, videos and images. A meta tag cannot be added to a PDF, so the header is the way to do it. Your host or developer can usually set it up.
Should I block CSS and script files in robots.txt?
Usually not. Google says not to block resources its crawler needs to understand the page. Blocking them can stop Google seeing your pages the way visitors do. Only block files the page works fine without.
