Google to explore alternatives to robots.txt in wake of generative AI and other emerging technologies
Search Engine Land » SEO » Google to explore alternatives to robots.txt in wake of generative AI and other emerging technologies
Google said it will engage with the web and AI communities with public discussions during this process.
Google is exploring alternatives or supplemental ways of controlling crawling and indexing beyond the 30 year standard of the robots.txt protocol. “We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases,” Google wrote.
Engaging with the community. Google said it is inviting members from the web and AI communities to discuss in a new protocol. Google said it is “kicking off a public discussion,” with a “broad range of voices from across web publishers, civil society, academia and more fields from around the world.