COMMON CRAWLBIG PICTURETHE DATAABOUTBLOGCONNECTDONATE Access to data is a good thing, right? Please donate today, so we can continue to provide you and others like you with this priceless resource.. DONATE NOW. Don't forget, Common Crawl is a registered 501(c)(3) non-profit so your donation is tax deductible! SO YOU’RE READY TO GET STARTED. SEARCH RESULTS FOR “WEB GRAPHS” We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018

Webgraphs).

WANT TO USE OUR DATA? Want to use our data? The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling. The corpus contains raw web page data, metadata extracts and text extracts. Common Crawl data is stored on Amazon Web Services’ Public Data Sets and on multiple academic cloud platforms across the world. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. JANUARY 2020 CRAWL ARCHIVE NOW AVAILABLE January 2020 crawl archive now available. February 3, 2020 Sebastian Nagel. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. COMMON CRAWL INDEX SERVER Search Page Crawl API endpoint Index File List on s3://commoncrawl/ /CC-MAIN-2021-21: May 2021 /CC-MAIN-2021-21-index: CC-MAIN-2021-21/cc-index.paths.gz /CC-MAIN-2021-17 HYPERLINK GRAPH FROM WEB DATA COMMONS Hyperlink Graph from Web Data Commons. The talented team at Web Data Commons recently extracted and analyzed the hyperlink graph within the Common Crawl 2012 corpus. Altogether, they found 128 billion hyperlinks connecting 3.5 billion pages. They have published resulting graph today together with some results from the analysis of the graph. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. COMMON CRAWLBIG PICTURETHE DATAABOUTBLOGCONNECTDONATE Access to data is a good thing, right? Please donate today, so we can continue to provide you and others like you with this priceless resource.. DONATE NOW. Don't forget, Common Crawl is a registered 501(c)(3) non-profit so your donation is tax deductible! SO YOU’RE READY TO GET STARTED. SEARCH RESULTS FOR “WEB GRAPHS” We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018

Webgraphs).

WANT TO USE OUR DATA? Want to use our data? The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling. The corpus contains raw web page data, metadata extracts and text extracts. Common Crawl data is stored on Amazon Web Services’ Public Data Sets and on multiple academic cloud platforms across the world. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. JANUARY 2020 CRAWL ARCHIVE NOW AVAILABLE January 2020 crawl archive now available. February 3, 2020 Sebastian Nagel. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. COMMON CRAWL INDEX SERVER Search Page Crawl API endpoint Index File List on s3://commoncrawl/ /CC-MAIN-2021-21: May 2021 /CC-MAIN-2021-21-index: CC-MAIN-2021-21/cc-index.paths.gz /CC-MAIN-2021-17 HYPERLINK GRAPH FROM WEB DATA COMMONS Hyperlink Graph from Web Data Commons. The talented team at Web Data Commons recently extracted and analyzed the hyperlink graph within the Common Crawl 2012 corpus. Altogether, they found 128 billion hyperlinks connecting 3.5 billion pages. They have published resulting graph today together with some results from the analysis of the graph. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. EXAMPLES USING COMMON CRAWL DATA Common Crawl WARC/WET/WAT examples and processing code for Java + Hadoop by Stephen Merity. Java and Clojure examples for processing Common Crawl WARC files by Mark Watson. Common web archive utility code by the IIPC. A distributed system for mining Common Crawl using SQS, AWS-EC2 and S3 by Akshay Bhat. TUTORIALS AND PRESENTATIONS ON USING COMMON CRAWL DATA Description of using the Common Crawl data to perform wide scale analysis over billions of web pages to investigate the impact of Google Analytics and what this means for privacy on the web at large. Discussion of how open, public datasets can be harnessed using the AWS cloud. Covers large data collections (such as the 1000 Genomes Project

and

SEARCH RESULTS FOR “WEB GRAPHS” We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018

Webgraphs).

ENSURE COMMON CRAWL CAN CONTINUE TO MAKE WEB DATA Ensure Common Crawl can continue to make web data available freely. Common Crawl is a California 501 (c) (3) registered non-profit organization. We are dedicated to contributing to the thriving commons of open data that will drive innovation, research, and education in the 21st century. Please join Common Crawl’s growing community of

supporters.

APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. FEBRUARY/MARCH 2021 CRAWL ARCHIVE NOW AVAILABLE February/March 2021 crawl archive now available. The crawl archive for February/March 2021 is now available! The data was crawled between February 24th and March 9th and contains 2.7 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls. WEB DATA COMMONS EXTRACTION FRAMEWORK FOR Web Data Commons Extraction Framework for the Distributed Processing of CC Data. August 29, 2014 Robert Meusel. This is a guest blog post by Robert Meusel. Robert Meusel is a researcher at the University of Mannheim in the Data and Web Science Research Group and a key member of the Web Data Commons project. LEXALYTICS TEXT ANALYSIS WORK WITH COMMON CRAWL DATA Lexalytics Text Analysis Work with Common Crawl Data. February 4, 2014 Lisa Green. Oskar Singer. This is a guest blog post by Oskar Singer Oskar Singer is a Software Developer and Computer Science student at University of Massachusetts Amherst. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. COMMON CRAWLBIG PICTURETHE DATAABOUTBLOGCONNECTDONATE Access to data is a good thing, right? Please donate today, so we can continue to provide you and others like you with this priceless resource.. DONATE NOW. Don't forget, Common Crawl is a registered 501(c)(3) non-profit so your donation is tax deductible! SO YOU’RE READY TO GET STARTED. SEARCH RESULTS FOR “WEB GRAPHS” We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018

Webgraphs).

WANT TO USE OUR DATA? Want to use our data? The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling. The corpus contains raw web page data, metadata extracts and text extracts. Common Crawl data is stored on Amazon Web Services’ Public Data Sets and on multiple academic cloud platforms across the world. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. JANUARY 2020 CRAWL ARCHIVE NOW AVAILABLE January 2020 crawl archive now available. February 3, 2020 Sebastian Nagel. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. COMMON CRAWL INDEX SERVER Search Page Crawl API endpoint Index File List on s3://commoncrawl/ /CC-MAIN-2021-21: May 2021 /CC-MAIN-2021-21-index: CC-MAIN-2021-21/cc-index.paths.gz /CC-MAIN-2021-17 HYPERLINK GRAPH FROM WEB DATA COMMONS Hyperlink Graph from Web Data Commons. The talented team at Web Data Commons recently extracted and analyzed the hyperlink graph within the Common Crawl 2012 corpus. Altogether, they found 128 billion hyperlinks connecting 3.5 billion pages. They have published resulting graph today together with some results from the analysis of the graph. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. COMMON CRAWLBIG PICTURETHE DATAABOUTBLOGCONNECTDONATE Access to data is a good thing, right? Please donate today, so we can continue to provide you and others like you with this priceless resource.. DONATE NOW. Don't forget, Common Crawl is a registered 501(c)(3) non-profit so your donation is tax deductible! SO YOU’RE READY TO GET STARTED. SEARCH RESULTS FOR “WEB GRAPHS” We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018

Webgraphs).

WANT TO USE OUR DATA? Want to use our data? The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling. The corpus contains raw web page data, metadata extracts and text extracts. Common Crawl data is stored on Amazon Web Services’ Public Data Sets and on multiple academic cloud platforms across the world. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. JANUARY 2020 CRAWL ARCHIVE NOW AVAILABLE January 2020 crawl archive now available. February 3, 2020 Sebastian Nagel. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. COMMON CRAWL INDEX SERVER Search Page Crawl API endpoint Index File List on s3://commoncrawl/ /CC-MAIN-2021-21: May 2021 /CC-MAIN-2021-21-index: CC-MAIN-2021-21/cc-index.paths.gz /CC-MAIN-2021-17 HYPERLINK GRAPH FROM WEB DATA COMMONS Hyperlink Graph from Web Data Commons. The talented team at Web Data Commons recently extracted and analyzed the hyperlink graph within the Common Crawl 2012 corpus. Altogether, they found 128 billion hyperlinks connecting 3.5 billion pages. They have published resulting graph today together with some results from the analysis of the graph. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. EXAMPLES USING COMMON CRAWL DATA Common Crawl WARC/WET/WAT examples and processing code for Java + Hadoop by Stephen Merity. Java and Clojure examples for processing Common Crawl WARC files by Mark Watson. Common web archive utility code by the IIPC. A distributed system for mining Common Crawl using SQS, AWS-EC2 and S3 by Akshay Bhat. TUTORIALS AND PRESENTATIONS ON USING COMMON CRAWL DATA Description of using the Common Crawl data to perform wide scale analysis over billions of web pages to investigate the impact of Google Analytics and what this means for privacy on the web at large. Discussion of how open, public datasets can be harnessed using the AWS cloud. Covers large data collections (such as the 1000 Genomes Project

and

SEARCH RESULTS FOR “WEB GRAPHS” We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of July, August and September 2020. Additional information about the data formats, the processing pipeline, our objectives, and credits can be found in the announcements of prior webgraph releases (e.g., Nov/Dec/Jan 2017-2018

Webgraphs).

ENSURE COMMON CRAWL CAN CONTINUE TO MAKE WEB DATA Ensure Common Crawl can continue to make web data available freely. Common Crawl is a California 501 (c) (3) registered non-profit organization. We are dedicated to contributing to the thriving commons of open data that will drive innovation, research, and education in the 21st century. Please join Common Crawl’s growing community of

supporters.

APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. FEBRUARY/MARCH 2021 CRAWL ARCHIVE NOW AVAILABLE February/March 2021 crawl archive now available. The crawl archive for February/March 2021 is now available! The data was crawled between February 24th and March 9th and contains 2.7 billion web pages or 280 TiB of uncompressed content. It includes page captures of 1.2 billion new URLs, not visited in any of our prior crawls. WEB DATA COMMONS EXTRACTION FRAMEWORK FOR Web Data Commons Extraction Framework for the Distributed Processing of CC Data. August 29, 2014 Robert Meusel. This is a guest blog post by Robert Meusel. Robert Meusel is a researcher at the University of Mannheim in the Data and Web Science Research Group and a key member of the Web Data Commons project. LEXALYTICS TEXT ANALYSIS WORK WITH COMMON CRAWL DATA Lexalytics Text Analysis Work with Common Crawl Data. February 4, 2014 Lisa Green. Oskar Singer. This is a guest blog post by Oskar Singer Oskar Singer is a Software Developer and Computer Science student at University of Massachusetts Amherst. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. COMMON CRAWLBIG PICTURETHE DATAABOUTBLOGCONNECTDONATE Access to data is a good thing, right? Please donate today, so we can continue to provide you and others like you with this priceless resource.. DONATE NOW. Don't forget, Common Crawl is a registered 501(c)(3) non-profit so your donation is tax deductible! SO YOU’RE READY TO GET STARTED. WANT TO USE OUR DATA? Want to use our data? The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling. The corpus contains raw web page data, metadata extracts and text extracts. Common Crawl data is stored on Amazon Web Services’ Public Data Sets and on multiple academic cloud platforms across the world. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. EXAMPLES USING COMMON CRAWL DATA Common Crawl WARC/WET/WAT examples and processing code for Java + Hadoop by Stephen Merity. Java and Clojure examples for processing Common Crawl WARC files by Mark Watson. Common web archive utility code by the IIPC. A distributed system for mining Common Crawl using SQS, AWS-EC2 and S3 by Akshay Bhat. APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. JANUARY 2020 CRAWL ARCHIVE NOW AVAILABLE January 2020 crawl archive now available. February 3, 2020 Sebastian Nagel. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. COMMON CRAWL INDEX SERVER Search Page Crawl API endpoint Index File List on s3://commoncrawl/ /CC-MAIN-2021-21: May 2021 /CC-MAIN-2021-21-index: CC-MAIN-2021-21/cc-index.paths.gz /CC-MAIN-2021-17 HYPERLINK GRAPH FROM WEB DATA COMMONS Hyperlink Graph from Web Data Commons. The talented team at Web Data Commons recently extracted and analyzed the hyperlink graph within the Common Crawl 2012 corpus. Altogether, they found 128 billion hyperlinks connecting 3.5 billion pages. They have published resulting graph today together with some results from the analysis of the graph. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. COMMON CRAWLBIG PICTURETHE DATAABOUTBLOGCONNECTDONATE Access to data is a good thing, right? Please donate today, so we can continue to provide you and others like you with this priceless resource.. DONATE NOW. Don't forget, Common Crawl is a registered 501(c)(3) non-profit so your donation is tax deductible! SO YOU’RE READY TO GET STARTED. WANT TO USE OUR DATA? Want to use our data? The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling. The corpus contains raw web page data, metadata extracts and text extracts. Common Crawl data is stored on Amazon Web Services’ Public Data Sets and on multiple academic cloud platforms across the world. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. EXAMPLES USING COMMON CRAWL DATA Common Crawl WARC/WET/WAT examples and processing code for Java + Hadoop by Stephen Merity. Java and Clojure examples for processing Common Crawl WARC files by Mark Watson. Common web archive utility code by the IIPC. A distributed system for mining Common Crawl using SQS, AWS-EC2 and S3 by Akshay Bhat. APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. JANUARY 2020 CRAWL ARCHIVE NOW AVAILABLE January 2020 crawl archive now available. February 3, 2020 Sebastian Nagel. The crawl archive for January 2020 is now available! It contains 3.1 billion web pages or 300 TiB of uncompressed content, crawled between January 17th and 29th. COMMON CRAWL INDEX SERVER Search Page Crawl API endpoint Index File List on s3://commoncrawl/ /CC-MAIN-2021-21: May 2021 /CC-MAIN-2021-21-index: CC-MAIN-2021-21/cc-index.paths.gz /CC-MAIN-2021-17 HYPERLINK GRAPH FROM WEB DATA COMMONS Hyperlink Graph from Web Data Commons. The talented team at Web Data Commons recently extracted and analyzed the hyperlink graph within the Common Crawl 2012 corpus. Altogether, they found 128 billion hyperlinks connecting 3.5 billion pages. They have published resulting graph today together with some results from the analysis of the graph. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. MAY 2021 CRAWL ARCHIVE NOW AVAILABLE May 2021 crawl archive now available. May 23, 2021 Sebastian Nagel. The crawl archive for May 2021 is now available! The data was crawled May 5 – 19 and contains 2.6 billion web pages or 280 TiB of uncompressed content. TUTORIALS AND PRESENTATIONS ON USING COMMON CRAWL DATA Description of using the Common Crawl data to perform wide scale analysis over billions of web pages to investigate the impact of Google Analytics and what this means for privacy on the web at large. Discussion of how open, public datasets can be harnessed using the AWS cloud. Covers large data collections (such as the 1000 Genomes Project

and

EXAMPLES USING COMMON CRAWL DATA Common Crawl WARC/WET/WAT examples and processing code for Java + Hadoop by Stephen Merity. Java and Clojure examples for processing Common Crawl WARC files by Mark Watson. Common web archive utility code by the IIPC. A distributed system for mining Common Crawl using SQS, AWS-EC2 and S3 by Akshay Bhat. APRIL 2021 CRAWL ARCHIVE NOW AVAILABLE April 2021 crawl archive now available. April 27, 2021 Sebastian Nagel. The crawl archive for April 2021 is now available! The data was crawled April 10 – 23 and contains 3.1 billion web pages or 320 TiB of uncompressed content. ENSURE COMMON CRAWL CAN CONTINUE TO MAKE WEB DATA Ensure Common Crawl can continue to make web data available freely. Common Crawl is a California 501 (c) (3) registered non-profit organization. We are dedicated to contributing to the thriving commons of open data that will drive innovation, research, and education in the 21st century. Please join Common Crawl’s growing community of

supporters.

NEW CRAWL DATA AVAILABLE! New crawl data is located in the commoncrawl bucket at /crawl-data/ path. Under this base path, crawl data is organized hierarchically as follows: CRAWL-NAME-YYYY-MM – The name of the crawl and year + week# initiated on. segments. SEGMENTNAME – A segment directory, typically a unix timestamp. warc – contains the WARC files with the HTTP LEXALYTICS TEXT ANALYSIS WORK WITH COMMON CRAWL DATA Lexalytics Text Analysis Work with Common Crawl Data. February 4, 2014 Lisa Green. Oskar Singer. This is a guest blog post by Oskar Singer Oskar Singer is a Software Developer and Computer Science student at University of Massachusetts Amherst. WEB DATA COMMONS EXTRACTION FRAMEWORK FOR Web Data Commons Extraction Framework for the Distributed Processing of CC Data. August 29, 2014 Robert Meusel. This is a guest blog post by Robert Meusel. Robert Meusel is a researcher at the University of Mannheim in the Data and Web Science Research Group and a key member of the Web Data Commons project. THE OPEN CLOUD CONSORTIUM’S OPEN SCIENCE DATA CLOUD The Open Cloud Consortium’s Open Science Data Cloud. July 3, 2012 Dave Lester. Common Crawl has started talking with the Open Cloud Consortium (OCC) about working together. If you haven’t already heard of the OCC, it is an awesome nonprofit organization managing and operating cloud computing infrastructure that supports scientific, environmental, medical and health care research. DATA SETS CONTAINING ROBOTS.TXT FILES AND NON-200 Data Sets Containing Robots.txt Files and Non-200 Responses. September 16, 2016 Sebastian Nagel. Together with the crawl archive for August 2016 we release two data sets containing. robots.txt files (or what servers return in response to a GET request /robots.txt); server responses with HTTP status code other than 200 (404s, redirects, etc.)

Skip to content

Toggle navigation

COMMON CRAWL

* Big Picture

* What We Do

* What You Can Do

* FAQs

* The Data

* Get Started

* Example Projects

* Tutorials

* Developer’s List

* About

* Our Team

* Job Opportunities

* Media

* Blog

* Connect

* Donate

* Contact Us

* Terms of Use

* Donate

Us

We build and maintain an open repository of WEB CRAWL DATA that can be ACCESSED AND ANALYZED BY ANYONE.

You

Need YEARS OF FREE web page data to help CHANGE THE WORLD. ACCESS TO DATA IS A GOOD THING, RIGHT? Please donate today, so we can continue to provide you and others like you with this PRICELESS RESOURCE.

DONATE NOW

Don't forget, Common Crawl is a registered 501(c)(3) non-profit so your donation is tax deductible!

* Big Picture

* What We Do

* What You Can Do

* FAQs

* The Data

* Get Started

* Example Projects

* Tutorials

* Developer’s List

* About Us

* Our Team

* Media

* Jobs

* Connect

* Donate

* Blog

* Contact Us

* Terms Of Use

Common Crawl on Twitter