robots.txt for Gatsby Sites

robots.txt template for Gatsby static sites. Optimized for the static site generation build process.

robots.txt

User-agent: *
Disallow: /404
Disallow: /404.html
Disallow: /dev-404-page/
Allow: /

Sitemap: https://example.com/sitemap-index.xml

Line-by-Line Explanation

User-agent: * — applies to all crawlers

Disallow: /404 and /404.html — prevents the 404 page from being indexed

Disallow: /dev-404-page/ — blocks the Gatsby development 404 page

Allow: / — all other pages are freely crawlable (they are static HTML)

Sitemap — points to the Gatsby-generated sitemap index

Best Practices for Gatsby

✓ Use gatsby-plugin-robots-txt for automatic robots.txt generation during builds.
✓ Gatsby generates static HTML, so crawling is inherently efficient.
✓ Combine with gatsby-plugin-sitemap for automatic sitemap generation.
✓ Set different robots.txt rules for staging vs production using environment variables.

Build a custom robots.txt for your Gatsby site

Open robots.txt Generator

Frequently Asked Questions

How do I add robots.txt to a Gatsby site?▾

Use gatsby-plugin-robots-txt (recommended) or place a robots.txt file in the /static directory. The plugin approach allows environment-specific configuration.

Does Gatsby need a complex robots.txt?▾

No. Since Gatsby generates static HTML, pages are easily crawlable. A minimal robots.txt blocking only error pages and including a sitemap is usually sufficient.

Should I block /page-data/ in Gatsby?▾

No. Gatsby uses /page-data/ for client-side navigation. Blocking it does not affect SEO since Google crawls the HTML pages directly, and blocking it could break preloading.

Related Templates

Next.js Production-ready robots.txt template for Next.js applications. Handles API routes, internal pages, and build artifacts properly. Astro robots.txt template for Astro framework sites. Minimal configuration needed thanks to static-first architecture. React (Single Page Application) robots.txt template for client-side React applications. Handles build artifacts and ensures proper crawling of SPA routes.