An infrastructure that artificial intelligence cannot penetrate has been developed

The company will directly block AI crawlers in an effort to prevent AI firms from collecting data from the internet.

Large language models like ChatGPT require vast amounts of training data to improve their performance. Many of these companies use crawlers that access and store the content of different web pages to train the systems. Cloudflare says these AI crawlers will be blocked outright, meaning these automated systems will not be able to access pages without “permission or fee.”

NEW SOLUTIONS FOR THE ADVANCEMENT OF ARTIFICIAL INTELLIGENCE

According to Independent Turkish, publishers of these websites will be able to grant AI crawlers access to their sites. Cloudflare says it will add a "pay-per-crawl" fee option, where AI companies can choose whether to pay.

The use of online text to train AI systems has become one of the most divisive issues of the ongoing AI era, with many publishers taking legal action against AI companies, claiming their content is being used without permission to train the systems.

As the conflict escalates in 2023, Cloudflare initially said it would allow websites to ask AI sites not to access them by adding a special tag, but that proved unworkable. Last year, it began developing technology that detects such AI bots, allowing websites to automatically block them.

The latter tool will be turned on by default. Cloudflare CEO Matthew Prince told the New York Times that “Cloudflare is completely changing the rules of the internet.” Prince said the move was motivated by concerns that allowing AI companies to scrape the web freely would discourage people from publishing new articles or other types of content.