robots.txt Templates
Production-ready robots.txt templates for popular frameworks and platforms. Copy the template, customize it for your site, and upload to your server root.
WordPress
Optimized robots.txt template for WordPress sites. Blocks admin, login, and plugin directories while allowing important content to be crawled.
Next.js
Production-ready robots.txt template for Next.js applications. Handles API routes, internal pages, and build artifacts properly.
Shopify
Recommended robots.txt configuration for Shopify stores. Handles collection filtering, checkout, and admin pages.
Laravel
Secure robots.txt template for Laravel applications. Blocks framework internals, API endpoints, and admin routes.
Django
Python/Django robots.txt template. Blocks admin, static files directory, and internal URLs while allowing public content.
React (Single Page Application)
robots.txt template for client-side React applications. Handles build artifacts and ensures proper crawling of SPA routes.
Angular
robots.txt template for Angular apps. Handles build output, source maps, and ensures proper SPA crawling.
Gatsby
robots.txt template for Gatsby static sites. Optimized for the static site generation build process.
Astro
robots.txt template for Astro framework sites. Minimal configuration needed thanks to static-first architecture.
Nuxt
robots.txt template for Nuxt.js (Vue.js) applications. Handles both SSR and static generation modes.
Drupal
Drupal-optimized robots.txt template. Blocks admin, system files, and internal paths while allowing content pages.
Magento / Adobe Commerce
E-commerce robots.txt template for Magento stores. Handles layered navigation, checkout, and admin pages.
Wix
Recommended robots.txt configuration for Wix websites. Understand what Wix generates and how to customize it.
Squarespace
robots.txt guide for Squarespace sites. Understand the auto-generated rules and available customization options.
GitHub Pages
robots.txt template for GitHub Pages sites and documentation. Simple configuration for static hosting.
Need a custom robots.txt?
Use our free robots.txt Generator to build custom rules with a visual editor.