Robots.txt generator

Github

With recent advances in AI, online web crawlers are constantly on the hunt for new training data. This may lead to your web content (files, images, text etc.) being used to train large language models (LLMs) without your consent.

Robots.txt are a way of letting these crawlers know which files are allowed to be crawled and which are not. However, they are difficult to generate manually.

BotBlock automatically generates these Robots.txt files based on your preferences, and even has options to ensure that your site is SEO-compliant and will show up in search engines.