Should lemmy.ml block chatgpt scraping in robots.txt?
Should lemmy.ml block chatgpt scraping in robots.txt?
Some context about this here: https://arstechnica.com/information-technology/2023/08/openai-details-how-to-keep-chatgpt-from-gobbling-up-website-data/
the robots.txt would be updated with this entry
User-agent: GPTBot
Disallow: /
Obviously this is meaningless against non-openai scrapers or anyone who just doesn't give a shit.
You're viewing a single thread.
12 comments
If they'll pay us when they scrape our content, sure.
... Is that like a non-argument? How do you suppose they would pay sites, let alone site users to scrape their content?
Yes that's the point
12 comments