robots.txt and sitemap.xml for Small Business Sites
What robots.txt and sitemap.xml do, why your small business site needs them, and how to set them up correctly (with common mistakes to avoid).
Two small files quietly control whether Google can crawl and index your site properly: robots.txt and sitemap.xml. Get them wrong and your pages may never show up in search. Here's what every small business site needs.
robots.txt — the crawl instructions
This file tells search engines which parts of your site they may crawl. A simple, safe version for most small business sites looks like this:
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlThe #1 mistake: a broken or 500-erroring robots.txt. If Googlebot can't load it, it may refuse to crawl your entire site. Make sure it returns a clean 200.
sitemap.xml — the map of your pages
A sitemap lists the URLs you want indexed, helping search engines discover every page. Keep it to your real, canonical URLs (no redirects, no www/non-www duplicates), and submit it in Google Search Console so Google re-crawls when you update it.
Common mistakes to avoid
- Listing URLs that redirect (use the final, canonical URL instead).
- Blocking your whole site with
Disallow: /left over from a staging build. - Forgetting to submit the sitemap to Search Console.
- A robots.txt that errors instead of returning 200.
Check yours in seconds
Run your homepage through the free Audit SEO tool — it checks whether your robots.txt and sitemap are present and valid, along with 18 other factors. For large sites, Semrush or SE Ranking will crawl every URL and flag indexing problems automatically.
A faster site = a better score
Page speed is a genuine ranking factor, and slow hosting quietly drags down both your SEO score and your visitors' patience. If your site feels sluggish, moving to fast hosting is one of the highest-impact fixes you can make:
- Hostinger — cheap, fast hosting that's ideal for small business and starter sites. Get fast hosting →
- Cloudways — managed cloud hosting (DigitalOcean/Vultr under the hood) for sites that need more power and scalability. Try Cloudways →