Constellation Network and Common Crawl Foundation are revolutionizing web data accessibility and AI development through blockchain technology.
SAN FRANCISCO, Oct. 24, 2024 /PRNewswire/: The Common Crawl Foundation, a non-profit organization founded in 2007, dedicated to providing a copy of the internet to the public, and Constellation Network, a Web3 blockchain ecosystem notable for providing solutions to the US Department of Defense, today announced a strategic partnership aimed at democratizing and enhancing the accessibility and utility of web-crawled data on blockchain technology for artificial intelligence (AI) and data applications.
This collaboration will explore potential opportunities for improving large language models used by AI, starting with Common Crawl’s vast dataset that is used by 80% of large language models, has crawled over 250 billion web pages to date (19 billion in 2024 alone), and consists of an archive of nearly 9 petabytes of archived crawled data. By leveraging Constellation’s decentralized network, Hypergraph, to add immutability, provenance, and auditability around the data, the partnership aligns to provide joint solutions around responsible and transparent AI.
With AI projected to be a $3T industry by 2030, there are growing demands for secure solutions to sharing common data sets being used for the training of large language models, improving storage of queried and cleaned data, monetization opportunities for data, and enhanced transparency with the source of data. With Constellation’s unique approach to providing tools to converge existing infrastructure with distributed and decentralized networks and Common Crawl’s history of data and growth of data utility, this partnership aligns to further democratize data.
“This partnership represents a significant step forward in securing trusted distribution of Common Crawl,” said Rich Skrenta, Executive Director of the Common Crawl Foundation. “By combining our comprehensive web archive with Constellation’s proven implementation of blockchain technology, researchers and developers from around the world can trust what they’re getting from Common Crawl and have a model for authenticating large open data sets, such as those used for AI training.”.
Ben Jorgensen, CEO of Constellation Network, states, “The partnership between Constellation Network and Common Crawl highlights mainstream adoption of Web3 solutions outside the echo chambers of crypto. This alignment continues Constellation’s mission of our zero trust network being used as a public good for a data-focused future.” Jorgensen continues, “Our aim is to further attract new developers by showcasing capabilities, such as integrating immutability throughout digital workflows, and thus further differentiate ourselves from earlier generations of blockchain technology.”
The two organizations will begin a phased approach to implementing this initiative, starting with a customizable subnet, called a metagraph, which will integrate a subset of Common Crawl’s data. This subnet is currently live on their test network and will be soon deployed to Constellation’s public network, Hypergraph. Further details of the live metagraph will be featured in the coming weeks, along with information on how organizations and developers can participate.
For more information, please visit:
- Common Crawl Foundation: https://commoncrawl.org
- Constellation Network: https://constellationnetwork.io
- X.com @Conste11ation
Also read: Viksit Workforce for a Viksit Bharat
Do Follow: CIO News LinkedIn Account | CIO News Facebook | CIO News Youtube | CIO News Twitter
About us:
CIO News is the premier platform dedicated to delivering the latest news, updates, and insights from the CIO industry. As a trusted source in the technology and IT sector, we provide a comprehensive resource for executives and professionals seeking to stay informed and ahead of the curve. With a focus on cutting-edge developments and trends, CIO News serves as your go-to destination for staying abreast of the rapidly evolving landscape of technology and IT. Founded in June 2020, CIO News has rapidly evolved with ambitious growth plans to expand globally, targeting markets in the Middle East & Africa, ASEAN, USA, and the UK.
CIO News is a proprietary of Mercadeo Multiventures Pvt Ltd.
