[HN Gopher] Reading the web offline and distraction-free
___________________________________________________________________
Reading the web offline and distraction-free
Author : owulveryck
Score : 52 points
Date : 2021-10-08 15:42 UTC (2 days ago)
(HTM) web link (blog.owulveryck.info)
(TXT) w3m dump (blog.owulveryck.info)
| Syonyk wrote:
| How is this different from the Wallabag project, which, as I
| understand it (it's on my list of "Things to mess with at some
| point") does exactly the same thing - website to epub for offline
| reading?
| phkx wrote:
| I've been using pandoc to extract texts next to my notes (both in
| Markdown) in order to add links between them. I haven't extracted
| too many pages yet, but the results were reasonable so far,
| although sometimes lots of html tags remain. Also, none of them
| contained any math so far.
| ivan_ah wrote:
| Very cool.
|
| The take-any-webpage-offline need is also common in the education
| space (teachers want to save a webpage and send it to their
| students as part of a lesson and don't want to worry about
| availability or ads etc).
|
| I used to work on tools for this
| https://github.com/learningequality/ricecooker/blob/develop/...
| and
| https://github.com/learningequality/BasicCrawler/blob/master...
| which worked quite well for most sites, but still very far from a
| general-purpose solution.
|
| There is also more powerful/general-purpose scraper that
| generates a ZIM file here: https://github.com/openzim/zimit
|
| It would be really nice to a "common" scraper code base that
| takes care of scraping (possibly with a real headless browser)
| and outputs all assets as files + info as JSON. This common code
| base could then be used by all kinds of programs to package the
| content as standalone HTML zip files, ePub, ZIM, or even PDF for
| crazy people like me who like to print things ;)
| tasogare wrote:
| That's a solved problem: click right, save the page as MHTML.
| It can be opened later and online with a browser again.
|
| If I recall correctly the only gotcha is that the option to
| save in this format needs to be enable using the flags
| settings.
| mro_name wrote:
| occasionally I use https://github.com/gildas-lormeau/SingleFile
| haroldtreen wrote:
| I built a Chrome Extension that does this exact thing :). There's
| also a WebAPI.
|
| https://epub.press/
| marban wrote:
| Needless to say that extractability hasn't gotten easier in
| recent years but I'm even more concerned about archive.org's
| quality/capabilities -- They really need to step up their game to
| remain useful in this area.
| alexmcc81 wrote:
| On a related note, does anyone know of any open source project
| that could parse the website text and generate a list of tags?
| marban wrote:
| Textblob/NLTK. I use Google (paid) though these days.
| captn3m0 wrote:
| I do a lot of this work[3] (web to documents) and it's
| interesting to see other approaches. The medium image problem is
| something I've faced as well, but never got around to fixing. I'm
| planning to get a Remarkable soon, so will definitely be trying
| this out.
|
| My personal solution has been https://github.com/captn3m0/url-to-
| epub/ (Node/readability), which I've tested against the entirety
| of Tor's original fiction collection[0] where it performs well
| enough (I'm biased). Another tool that does this beautifully well
| is percollate[1], but it doesn't give enough control of the
| metadata to the user - something I really care about.
|
| I've also started to use rdrview[2], which is a C-port of the
| current Firefox implementation of "reader view". It is very
| unix-y, so it is easy to pipe content to it (I usually run it
| through tidy first). Quite helpful in building web-archiving or
| web-to-pdf or web-to-kindle pipelines easily.
|
| [0]: https://www.tor.com/category/all-fiction/original-fiction/
|
| [1]: https://github.com/danburzo/percollate
|
| [2]: https://github.com/eafer/rdrview
|
| [3]: https://captnemo.in/ebooks/
| atsaloli wrote:
| I run "lynx --dump $URL | vim -" to read the text in Vim when the
| web page gets too cluttered (I use Vim as a pager because I know
| "Vim" better than "less").
| bredren wrote:
| Newspaper3k is a Python package I'm using to extract content from
| articles across the web.
|
| But it has not been maintained, since the author joined Facebook.
|
| It works alright, but it has many issues.
|
| If I understand correctly, a full on replacement for newspaper is
| in the wings, seeking to offer a sustainable content extraction
| tool in Python.
|
| But it isn't ready yet. And some of the problems in this area
| mirror those faced by web scrapers.
| owulveryck wrote:
| This is a post about a tool I am building to generate an epub
| from a website
| spikej wrote:
| Good luck! Those HTML issues you're coming across are tough and
| so varied across the web!
|
| I was working with Mercury Parser (pluggable parsing for
| different sites) in the past.
|
| https://github.com/postlight/mercury-parser
___________________________________________________________________
(page generated 2021-10-10 23:01 UTC)