https://annas-archive.org/blog/all-isbns.html

Anna's Blog
Updates about Anna's Archive, the largest truly open library in human
history.

Visualizing All ISBNs -- $10k by 2025-01-31

annas-archive.li/blog, 2024-12-15

This picture is 1000x800 pixels. Each pixel represents 2,500 ISBNs.
If we have a file for an ISBN, we make that pixel more green. If we
know an ISBN has been issued, but we don't have a matching file, we
make it more red.

[all_isbns_]

In less than 300kb, this picture succinctly represents the largest
fully open "list of books" ever assembled in the history of humanity
(a few hundred GB compressed in full).

It also shows: there is a lot of work left in backing up books (we
only have 16%).

Background

How can Anna's Archive achieve its mission of backing up all of
humanity's knowledge, without knowing which books are still out
there? We need a TODO list. One way to map this out is through ISBN
numbers, which since the 1970s have been assigned to every book
published (in most countries).

There is no central authority that knows all ISBN assignments.
Instead, it's a distributed system, where countries get ranges of
numbers, who then assign smaller ranges to major publishers, who
might further sub-divide ranges to minor publishers. Finally
individual numbers are assigned to books.

We started mapping ISBNs two years ago with our scrape of ISBNdb.
Since then, we have scraped many more metadata sources, such as
Worldcat, Google Books, Goodreads, Libby, and more. A full list can
be found on the "Datasets" and "Torrents" pages for Anna's Archive.
We now have by far the largest fully open, easily downloadable
collection of book metadata (and thus ISBNs) in the world.

We've written extensively about why we care about preservation, and
why we're currently in a critical window. We must now identify rare,
underfocused, and uniquely at-risk books and preserve them. Having
good metadata on all books in the world helps with that.

Visualizing

Besides the overview image, we can also look at individual datasets
we've acquired. Use the dropdown and buttons to switch between them.

[all_isbns_] [md5_isbns_] [cadal_ssno] [cerlalc_is] [duxiu_ssid]
[edsebk_isb] [gbooks_isb] [goodreads_] [ia_isbns_s] [isbndb_isb]
[isbngrp_is] [libby_isbn] [nexusstc_i] [oclc_isbns] [ol_isbns_s]
[rgb_isbns_] [trantor_is]

[All ISBNs [all_isbns]                       ]    [?] [?] 

[all_isbns_]

There are lots of interesting patterns to see in these pictures. Why
is there some regularity of lines and blocks, that seems to happen at
different scales? What are the empty areas? Why are certain datasets
so clustered? We'll leave these questions as an exercise for the
reader.

$10,000 bounty

There is much to explore here, so we're announcing a bounty for
improving the visualization above. Unlike most of our bounties, this
one is time-bound. You have to submit your open source code by
2025-01-31 (23:59 UTC).

The best submission will get $6,000, second place is $3,000, and
third place is $1,000. All bounties will be awarded using Monero
(XMR).

Below are the minimal criteria. If no submission meets the criteria,
we might still award some bounties, but that will be at our
discretion.

  * Fork this repo, and edit this blog post HTML (no other backends
    besides our Flask backend are allowed).
  * Make the picture above smoothly zoomable, so you can zoom all the
    way to individual ISBNs. Clicking ISBNs should take you to a
    metadata page or search on Anna's Archive.
  * You must still be able to switch between all different datasets.
  * Country ranges and publisher ranges should be highlighted on
    hover. You can use e.g. data4info.py in isbnlib for country info,
    and our "isbngrp" scrape for publishers (dataset, torrent).
  * It must work well on desktop and mobile.

For bonus points (these are just ideas -- let your creativity run
wild):

  * Strong consideration will be given to usability and how good it
    looks.
  * Show actual metadata for individual ISBNs when zooming in, such
    as title and author.
  * Better space-filling curve. E.g. a zig-zag, going from 0 to 4 on
    the first row and then back (in reverse) from 5 to 9 on the
    second row -- recursively applied.
  * Different or customizable color schemes.
  * Special views for comparing datasets.
  * Ways to debug issues, such as other metadata that don't agree
    well (e.g. vastly different titles).
  * Annotating images with comments on ISBNs or ranges.
  * Any heuristics for identifying rare or at-risk books.
  * Whatever creative ideas you can come up with!

You MAY completely veer off from the minimal criteria, and do a
completely different visualization. If it's really spectacular, then
that qualifies for the bounty, but at our discretion.

Make submissions by posting a comment to this issue with a link to
your forked repo, merge request, or diff.

Code

The code to generate these images, as well as other examples, can be
found in this directory.

We came up with a compact data format, with which all the required
ISBN information is about 75MB (compressed). The description of the
data format and code to generate it can be found here. For the bounty
you're not required to use this, but it is probably the most
convenient format to get started with. You can transform our metadata
however you want (though all your code has to be open source).

We can't wait to see what you come up with. Good luck!

- Anna and the team (Reddit, Telegram)