https://github.com/zakird/crux-top-lists Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} zakird / crux-top-lists Public * Notifications * Fork 4 * Star 205 Downloadable snapshots of the Google Chrome (CrUX) Top Million Websites pulled from public CrUX data in Google BigQuery. 205 stars 4 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights zakird/crux-top-lists This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/z] Use Git or checkout with SVN using the web URL. [gh repo clone zakird] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @zakird zakird Link to supporting research ... 20bb5e2 Dec 31, 2022 Link to supporting research 20bb5e2 Git stats * 29 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time cruxdownloader code to maintain current Dec 30, 2022 data/global code to maintain current Dec 30, 2022 .gitignore some initial data Dec 30, 2022 README.md Link to supporting research Dec 31, 2022 pyproject.toml some initial data Dec 30, 2022 View code Cached Chrome Top Million Websites Data Structure Why 1 Million Sites? Country-Specific Websites Supporting Research README.md Cached Chrome Top Million Websites Recent research showed that the top million most popular websites published by Google Chrome via their UX Report (CrUX) is significantly more accurate than other top lists like the Alexa Top Million and Tranco Top Million. This repository caches a CSV version of the Chrome top sites, queried from the CrUX data in Google BigQuery. You can browse all of the cached lists here. The most up-to-date top million global websites can be downloaded directly at: https://raw.githubusercontent.com/ zakird/crux-top-lists/main/data/global/current.csv.gz. Data Structure The CrUX dataset has several important differences from other top lists: 1. Websites are bucketed by rank magnitude order, not by specific rank. Rank will be 1000, 10K, 100K, or 1M in the provided files. The data is ordered by rank magnitude. Within each order of magnitude, websites are listed randomly. 2. Websites are identified by origin (e.g., https://www.google.com) not by domain or FQDN. 3. Data is released monthly, typically on the second Tuesday of the month. This is an example of what the data looks like: origin,rank https://www.ptwxz.com,1000 https://ameblo.jp,1000 https://danbooru.donmai.us,1000 https://game8.jp,1000 https://www.google.com.au,1000 https://www.repubblica.it,1000 https://www.w3schools.com,1000 https://animekimi.com,1000 Websites are ranked by completed pageloads (measured by First Contentful Paint) and aggregated by web origin. The dataset adheres as closely as possible to user-initiated pageloads (e.g., it excludes traffic from iframes). More information about CrUX and its data collection methodology can be found on its official website: https:// developer.chrome.com/docs/crux/about/. Why 1 Million Sites? This repository does not contain all of the website ranking data published by Chrome. Their global list of popular websites contains approximately 15M websites. The top million websites captures over 95% of user traffic in Chrome by both Page Loads and Time on Page ( Ruth et al.) and is a reasonable approximation: CDF of User Traffic If you want to use more or fewer websites, this is the approximate breakdown of coverage: Websites Page Loads 1000 50% 10K 70% 100K 87% 1M 95% 5M 99% The following SQL can be used to generate a similar list of all globally popular websites: SELECT distinct origin, experimental.popularity.rank FROM `chrome-ux-report.experimental.global` WHERE yyyymm = ? -- e.g., integer 202210 GROUP BY origin, experimental.popularity.rank ORDER BY experimental.popularity.rank; Country-Specific Websites Ruth et al. also showed that browsing behavior is localized and a global top list skews towards global sites (e.g., technology and gaming) and away from local sites (e.g., education, government, and finance). As such, researchers may also want to investigate whether trends hold across individual countries. Skew in Websites Chrome publishes country-specific top lists in BigQuery and the following SQL can be used to dump out country-specific top websites: SELECT distinct country_code, origin, experimental.popularity.rank FROM `chrome-ux-report.experimental.country` WHERE yyyymm = ? -- e.g., integer 202210 AND experimental.popularity.rank <= 1000000 GROUP BY country_code, origin, experimental.popularity.rank ORDER BY country_code, experimental.popularity.rank; The CrUX dataset is based on data collected from Google Chrome and is thus biased away from countries with limited Chrome usage (e.g., China). If you're specifically interested in looking at domain popularity in China, consider Building an Open, Robust, and Stable Voting-Based Domain Top List, which is based on data collected from 114DNS, a large DNS provider in China. Supporting Research The data in this repo is all publicly posted by Google to their CrUX dataset in Google BigQuery. This is simply a cache of that public data. Many of the arguments in this README are based on two recent research papers. The first describes how we evaluated the accuracy of lists of top websites. The second is a study on web browsing more broadly. Toppling Top Lists: Evaluating the Accuracy of Popular Website Lists Kimberly Ruth, Deepak Kumar, Brandon Wang, Luke Valenta, and Zakir Durumeric ACM Internet Measurement Conference (IMC), October 2022 A World Wide View of Browsing the World Wide Web Kimberly Ruth, Aurore Fass, Jonathan Azose, Mark Pearson, Emma Thomas, Caitlin Sadowski, and Zakir Durumeric ACM Internet Measurement Conference (IMC), October 2022 About Downloadable snapshots of the Google Chrome (CrUX) Top Million Websites pulled from public CrUX data in Google BigQuery. Resources Readme Stars 205 stars Watchers 1 watching Forks 4 forks Languages * Python 100.0% Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.