[HN Gopher] ArchiveBox: Open-source self-hosted web archiving
___________________________________________________________________
ArchiveBox: Open-source self-hosted web archiving
Author : pcr910303
Score : 204 points
Date : 2021-04-19 02:34 UTC (20 hours ago)
(HTM) web link (archivebox.io)
(TXT) w3m dump (archivebox.io)
| chrisweekly wrote:
| Wow, great! Self-hosted, open-source, solid UI, tie-ins to the
| broader ecosystem... seems to check all the right boxes. Looking
| fwd to trying it and if all goes well, maybe see about
| integrating it into AthensResearch. Thanks for sharing!
| sleavey wrote:
| Even works on the Raspberry Pi, apparently. This would be nice in
| combination with a Pi-Hole.
| dehrmann wrote:
| Am I the only one who spins up a new VM in VMWare ESXi for
| things like this?
| thies226j wrote:
| Yeah, why don't you just use a container?
| Karunamon wrote:
| VIC containers for me - a full VM is a bit overkill for
| something this light :)
| gzer0 wrote:
| What's your setup like? I use VMWare workstation pro- I
| have Windows Enterprise 2019 LTSC N installed and
| snapshotted to a base VM.
|
| Any time I need to do anything, I will full clone the base;
| with a decent SSD it takes maybe 10 seconds for the full
| clone and I have a full OS.
| Karunamon wrote:
| Actual ESXI on a decently powerful (33 cores, 512 gigs)
| machine. VMWare's been really good to me, minus some
| points for occasional stupidity on upgrades.
| ramraj07 wrote:
| Can I export my history from my iPhone web browsing?
| nojito wrote:
| Safari reading list already does this
|
| https://support.apple.com/guide/safari/keep-a-reading-list-s...
| dewey wrote:
| I'm not sure how the iCloud Sync works in this case but if you
| use Desktop Safari where it has your iOS history too you might
| be able to get it out of the Safari sqlite DB on your computer:
|
| https://stackoverflow.com/questions/28628385/sqlite-safari-h...
| rakoo wrote:
| If you can somehow extract your browsing history, archivebox
| can ingest a list of links
| nikisweeting wrote:
| ArchiveBox comes with a script that exports Safari history to
| a text file (which can then be imported into AB):
| ./bin/export_browser_history.sh --safari
|
| https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/export.
| ..
| thedanbob wrote:
| I've been using ArchiveBox since the last time it popped up on HN
| and I like it a lot. It recently got a significant UI upgrade.
| pacifika wrote:
| No mention of search?
| nikisweeting wrote:
| It has full-text search using ripgrep and sonic.
| dugite-code wrote:
| I literally just spun up a copy but it looks like it has Sonic
| full-text integration, however I'm not 100% sure if it's
| working via the UI as there isn't much feed back letting you
| know why a site has displayed in the results.
| weekay wrote:
| It saves pages to archive.org as well. You might want to be
| careful while using this to archive personal content.
| nikisweeting wrote:
| Yes, I can write a long article about why it's the default
| someday. I've agonized over this decision for many many months,
| and it's flipped flopped a few times as well.
|
| The short version is that defaults in software are really
| important (90% of users wont change them), and I don't trust
| myself to code ArchiveBox 100% correctly so as to never lose
| data, or the majority of people to store their archives
| correctly so as to never lose data on their own. Archive.org is
| the redundant failsafe. Another good reason is that Archive.org
| is not the only way that your archive content can be leaked,
| the security model means that archived pages can read each
| other's content, so I want to make it abundantly clear to users
| that by default it's designed to only archive content thats
| already public (in which case it's already fair game for
| Archive.org).
|
| I've settled on leaving it on as the default, but I do mention
| 3 times in the README how to disable it, most notably in the
| CAVEATS section which explains both the security model
| drawbacks and how to prevent your content from being leaked to
| Archive.org or other 3rd party APIs.
| jka wrote:
| Although I tend privacy-by-default for most deployed
| technologies, the context of archiving does change the
| criteria quite a lot; you've selected a sensible and
| reasonable default, I reckon. Hopefully integrity is a
| consideration too? Glad to read that article, one day :)
| kenniskrag wrote:
| Can be disabled but on in default mode.
|
| https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#...
| Proven wrote:
| "Democratizing Tragedy of Commons at scale"
| asaddhamani wrote:
| archive.org will only archive publicly visible content and it
| respects robots.txt
| tyingq wrote:
| _" and it respects robots.txt"_
|
| Not since 2017. https://blog.archive.org/2017/04/17/robots-
| txt-meant-for-sea...
|
| They now have a clunky manual process to exclude your site.
| https://help.archive.org/hc/en-
| us/articles/360004651732-Usin... ("How can I exclude...")
|
| They don't spoof user agents, but blocking them actively
| doesn't remove their history.
| Karunamon wrote:
| When did this change? It used to be that adding robots.txt
| would retroactively remove archives for a domain.
| tyingq wrote:
| 2017.
| pseudalopex wrote:
| Hide. Not remove.
| gravypod wrote:
| I can't wait for the API to be completed. I want to build
| something to archive HN (article + comments) and turn it into
| epub go read offline. Hard to do currently.
| jcoscolla wrote:
| I was trying to do something similar with walls bag, but no
| epub file to send :(
| antman wrote:
| I assume its auto corrected, do you mean Wallabag?
| https://github.com/wallabag/wallabag
| rcMgD2BwE72F wrote:
| How does it differ from https://wallabag.org?
| unicornporn wrote:
| https://www.linkace.org/
|
| Just a heads-up. Found that a while ago and much prefer it over
| wallabag.
| Kovah wrote:
| Thanks for sharing LinkAce! Maintainer here. If you or others
| have any questions, feel free to ask.
| pseudalopex wrote:
| What makes it better?
| Kovah wrote:
| I think it's primarily personal preference of features and
| how things are stored and presented. While Wallabag is more
| a Pocket/Read It Later alternative, LinkAce does not save
| the website itself, but a reference to it including the
| taxonomy you assign to it. It is intended to be a long-term
| bookmark archive, but without handling all the website
| archiving on its own.
| coffeeri wrote:
| It is quiet convenient to combine the two. You are able to
| export your wallabag list via RSS and import it on a schedule
| in archivebox.
| hlasdjlfhalwjk wrote:
| Wallabag only stores links. ArchiveBox archives a snapshot of
| the actual content of a page at a specific point in time.
| ernesth wrote:
| Wallabag extracts the content and stores it. Not just the
| links.
| kwhitefoot wrote:
| How do tools like this cope with pages that are rendered by
| Javascript. What do the tools actually save? For instance if I
| save a Quora page using Firefox I can open it but if Quora is not
| accessible it doesn't work.
| gogopuppygogo wrote:
| Would be great to allow for it to save to worm storage.
| asaddhamani wrote:
| Worm storage?
| kinard wrote:
| write once read many
| throwaway823882 wrote:
| Last time I tried to do this same thing, I didn't know about
| these, and ended up spending a couple days on wget and httrack.
| Do all these alternatives work from the command line, or are they
| their own little proprietary ecosystem?
| elric wrote:
| If you're interested in this sort of thing, you might also be
| interested in Archivy [1], which is somewhat similar but it
| (thankfully) doesn't upload your stuff to archive.org
|
| [1] https://archivy.github.io/
| dugite-code wrote:
| > doesn't upload your stuff to archive.org
|
| FIY You just have to set the environment variable
| SUBMIT_ARCHIVE_DOT_ORG=False
| pseudalopex wrote:
| It's an accident waiting to happen.
| nikisweeting wrote:
| See my answer here for why it's the default:
| https://news.ycombinator.com/item?id=26866689
| faitswulff wrote:
| What's the danger in uploading to archive.org?
| elric wrote:
| Aside from the can of worms that is copyright infringement?
| There was a recent HN discussion about how much of a pain it
| is to get something removed from archive.org.
|
| Not uploading people's stuff to permanent, public archives
| seems like a good rule of thumb.
| notRobot wrote:
| > There was a recent HN discussion about how much of a pain
| it is to get something removed from archive.org.
|
| I really wanna read this but can't find the thread, do you
| happen to have a link?
| tashbarg wrote:
| Isn't this just passing the URL to archive.org which then
| does the actual archiving?
|
| If it isn't already public (i.e., reachable by
| archive.org), it won't be afterwards?
| elric wrote:
| Something can be public today and not tomorrow. Something
| can be made public by accident. Something can be
| publically reachable (i.e. a private URL but one without
| a login) without the intention of being searchable.
| dugite-code wrote:
| Last time I saw this you could only view the archive via the UI.
| It's really come a long way.
|
| Could be very useful now.
___________________________________________________________________
(page generated 2021-04-19 23:02 UTC)