[HN Gopher] Inside The Internet Archive's Infrastructure
___________________________________________________________________
Inside The Internet Archive's Infrastructure
https://github.com/internetarchive/heritrix3
Author : dvrp
Score : 184 points
Date : 2026-01-14 07:26 UTC (1 days ago)
(HTM) web link (hackernoon.com)
(TXT) w3m dump (hackernoon.com)
| BryantD wrote:
| They have come a very long way since the late 1990s when I was
| working there as a sysadmin and the data center was a couple of
| racks plus a tape robot in a back room of the Presidio office
| with an alarmingly slanted floor. The tape robot vendor had to
| come out and recalibrate the tape drives more often than I might
| have wanted.
| textfiles wrote:
| There is a fundamental resistance to tape technology that
| exists to this day as a result of all those troubles.
| hinkley wrote:
| We had a little server room where the AC was mounted directly
| over the rack. I don't think we ever put an umbrella in there
| but it sure made everyone nervous the drain pipe would clog.
|
| Much more recently, I worked at a medium-large SaaS company but
| if you listened to my coworkers you'd think we were Google
| (there is a point where optimism starts being delusion, and a
| couple of my coworkers were past it.)
|
| Then one day I found the telemetry pages for Wikipedia. I am
| hoping some of those charts were per hour not per second,
| otherwise they are dealing with mind numbing amounts of
| traffic.
| brcmthrowaway wrote:
| Does IA do deduplication?
| textfiles wrote:
| Not in the way I think you're talking about. The archive has
| always tried to maintain a situation where the racks could be
| pushed out of the door or picked up after being somewhere and
| the individual drives will contain complete versions of the
| items. We have definitely reached out to people who seem to be
| doing redundant work and ask them to stop or for permission to
| remove the redundant item. But that's a pretty curatorial
| process.
| hedora wrote:
| It's frustrating that there's no way for people to (selectively)
| mirror the Internet Archive. $25-30M per year is a lot for a non-
| profit, but it's nothing for government agencies, or private
| corporations building Gen AI models.
|
| I suspect having a few different teams competing (for funding) to
| provide mirrors would rapidly reduce the hardware cost too.
|
| The density + power dissipation numbers quoted are extremely poor
| compared to enterprise storage. Hardware costs for the enterprise
| systems are also well below AWS (even assuming a short 5 year
| depreciation cycle on the enterprise boxes). Neither this article
| nor the vendors publish enough pricing information to do a
| thorough total cost of ownership analysis, but I can imagine
| someone the size of IA would not be paying normal margins to
| their vendors.
| toomuchtodo wrote:
| Pick the items you want to mirror and seed them via their
| torrent file.
|
| https://help.archive.org/help/archive-bittorrents/
|
| https://github.com/jjjake/internetarchive
|
| https://archive.org/services/docs/api/internetarchive/cli.ht...
|
| u/stavros wrote a design doc for a system (codename "Elephant")
| that would scale this up:
| https://news.ycombinator.com/item?id=45559219
|
| (no affiliation, I am just a rando; if you are a library,
| museum, or similar institution, ask IA to drop some racks at
| your colo for replication, and as always, don't forget to
| donate to IA when able to and be kind to their infrastructure)
| billyhoffman wrote:
| There are real problems with the Torrent files for
| collections. They are automatically created when a collection
| is first created and uploaded, and so they only include the
| files of the initial upload. For very large collections (100+
| GB) it is common for a creator to add/upload files into a
| collection in batches, but the torrent file is never
| regenerated, so download with the torrent results in just a
| small subset of the entire collection.
|
| https://www.reddit.com/r/torrents/comments/vc0v08/question_a.
| ..
|
| The solution is to use one of the several IA downloader
| script on GitHub, which download content via the collection's
| file list. I don't like directly downloading since I know
| that is most cost to IA, but torrents really are an option
| for some collections.
|
| Turns out, there are a lot of 500BG-2TB collections for
| ROMs/ISOs for video game consoles through the 7th and 8th
| generation, available on the IA...
| toomuchtodo wrote:
| I agree it is not perfect current state, and improvements
| are going to require coordination with the Internet
| Archive. With that said, I was confident that torrent files
| were regenerated when items changed (via a derive
| operation), so I'll have to do more research and test to
| speak authoritatively on this.
| Wowfunhappy wrote:
| Is this something the Internet Archive could fix? I would
| have expected the torrent to get replaced when an upload is
| changed, maybe with some kind of 24 hour debounce.
| rincebrain wrote:
| "They're working on it." [1]
|
| It sounds like they put this mechanism into place that
| stops regenerating large torrents incrementally when it
| caused massive slowdowns for them, and haven't finished
| building something to automatically fix it, but will go
| fix individual ones on demand for now.
|
| [1] - https://www.reddit.com/r/theinternetarchive/comment
| s/1ij8go9...
| philipkglass wrote:
| I would like to be able to pull content out of the Wayback
| Machine with a proper API [1]. I'd even be willing to pay a
| combination of per-request and per-gigabyte fees to do it. But
| then I think about the Archive's special status as a non-profit
| library, and I'm not sure that offering paid API access (even
| just to cover costs) is compatible with the organization as it
| exists.
|
| [1] It looks like this might exist at some level, e.g.
| https://github.com/hartator/wayback-machine-downloader, but
| I've been trying to use this for a couple of weeks and every
| day I try I get a HTTP 5xx error or "connection refused."
| toomuchtodo wrote:
| https://github.com/internetarchive/wayback/tree/master/wayba.
| ..
|
| https://akamhy.github.io/waybackpy/
|
| https://wiki.archiveteam.org/index.php/Restoring
| philipkglass wrote:
| Yes, there are documents and third party projects
| indicating that it has a free public API, but I haven't
| been able to get it to work. I presume that a paid API
| would have better availability and the possibility of
| support.
|
| I just tried waybackpy and I'm getting errors with it too
| when I try to reproduce their basic demo operation:
| >>> from waybackpy import WaybackMachineSaveAPI >>>
| url = "https://nuclearweaponarchive.org" >>>
| user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0)
| Gecko/20100101 Firefox/40.0" >>> save_api =
| WaybackMachineSaveAPI(url, user_agent) >>>
| save_api.save() Traceback (most recent call last):
| File "<python-input-4>", line 1, in <module>
| save_api.save() ~~~~~~~~~~~~~^^ File
| "/Users/xxx/nuclearweapons-
| archive/venv/lib/python3.13/site-
| packages/waybackpy/save_api.py", line 210, in save
| self.get_save_request_headers()
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^ File
| "/Users/xxx/nuclearweapons-
| archive/venv/lib/python3.13/site-
| packages/waybackpy/save_api.py", line 99, in
| get_save_request_headers raise
| TooManyRequestsError( ...<4 lines>... )
| waybackpy.exceptions.TooManyRequestsError: Can not save
| 'https://nuclearweaponarchive.org'. Save request refused by
| the server. Save Page Now limits saving 15 URLs per
| minutes. Try waiting for 5 minutes and then try again.
| toomuchtodo wrote:
| Reach out to patron services, support @ archive dot org.
| Also, your API limits will be higher if you specify your
| API key from your IA user versus anonymous requests when
| making requests.
| nodja wrote:
| It's insane to me that in 2008 a bunch of pervs decentralized
| storage and made hentai@home to host hentai comics. Yet here we
| are almost 20 years later and we haven't generalized this
| solution. Yes I'm aware of the privacy issues h@h has (as a
| hoster you're exposing your real IP and people reading comics
| are exposing their IP to you) but those can be solved with
| tunnels, the real value is the redundant storage.
| qingcharles wrote:
| The fact AI companies are stripping mining IA for content and
| not helping to be part of the solution is egregious.
| astrange wrote:
| Has any evidence been provided for this fact?
| hinkley wrote:
| I'd like a Public Broadcasting Service for the Internet but I'm
| afraid that money would just be pulled from actual PBS at this
| point to support it.
| quux wrote:
| Is running an IPFS node and pinning the internet archive's
| collections a good way to do this?
| cowhax wrote:
| >And the rising popularity of generative AI adds yet another
| unpredictable dimension to the future survival of the public
| domain archive.
|
| I'd say the nonprofit has found itself a profitable reason for
| its existence
| schmuckonwheels wrote:
| Disappointed with the lack of pictures.
| parttimelarry wrote:
| Probably because this looks more like a Deep Research agent
| "delving" into the infrastructure -- with a giant list of
| sources at the end. The Archive is not just a library; it is a
| service provider.
| schmuckonwheels wrote:
| I wasn't expecting to _read a podcast_ when clicking.
| mcpar-land wrote:
| Is this some kind of copypasted AI output? There are unformatted
| footnote numbers at the end of many sentences.
| NetOpWibby wrote:
| I was thinking the same thing. No proofreading is a sure sign
| to me. I also feel like I've read parts of this before.
| sltkr wrote:
| Some of the images are AI generated (see the Gemini watermark
| in the bottom right), and the final paragraph also reads
| extremely AI-generated.
| lysace wrote:
| The IA needs perhaps not just more money, but also more talented
| people, IMO. I worry that it has stagnated, from a tech pov.
| rarisma wrote:
| I think this was writen wholly by deep research.
|
| It just reads like a clunky low quality article
| astrange wrote:
| It's clearly AI writing ("hum", "delve") but oddly I don't
| think deep research models use those words.
| bpiche wrote:
| IA is hosting a couple more of Rick Prelinger's shows this month.
| Looking forward to visiting
___________________________________________________________________
(page generated 2026-01-15 23:00 UTC)