Cloudflare Is Blocking AI Crawlers by Default
Last year, internet infrastructure firm Cloudflare launched tools enabling its customers to block AI scrapers. Today the company has taken its fight against permissionless scraping several steps further. It has switched to blocking AI crawlers by default for its customers and is moving forward with a Pay Per Crawl program that lets customers charge AI companies to scrape their websites.
I paid a visit to my Cloudflare dashboard and I saw the option to turn on the blocking of AI bots. I’m just not sure that I want to silence myself from LLM training. What if everyone does the same?