Free tools · Technical
Robots.txt generator
Build a robots.txt with a visual rule editor, presets and rules for AI crawlers.
How to use it
Three steps.
Good to know. Robots.txt controls crawling, not indexing. To keep a page out of search results, use a noindex tag. Blocking AI crawlers is a business decision, so weigh visibility against control.
Step 1
Start from a preset.
Step 2
Add or edit rules for each crawler.
Step 3
Copy the file to the root of your domain.
More tools
Try these next.
Technical
XML sitemap generator
Turn a list of URLs into a valid sitemap.xml with change frequency, priority and last-modified date.
Use the tool →TechnicalHreflang tag generator
Create hreflang alternate tags or sitemap entries for multilingual and multi-region sites, with x-default.
Use the tool →TechnicalOpen Graph preview
Preview how a link will look when shared on Facebook, X and LinkedIn, using your own image.
Use the tool →Next step
Want this done properly across your whole site?
Tools give you the numbers. I turn them into a plan, fix what matters and report what changed.
Book a free discovery call →