webdump.md - www.codemadness.org - www.codemadness.org saait content files
(HTM) git clone git://git.codemadness.org/www.codemadness.org
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
webdump.md (4463B)
---
1 webdump is (yet another) HTML to plain-text converter tool.
2
3 It reads HTML in UTF-8 from stdin and writes plain-text to stdout.
4
5
6 ## Goals and scope
7
8 The main goal of this tool for me is to use it for converting HTML mails to
9 plain-text and to convert HTML content in RSS feeds to plain-text.
10
11 The tool will only convert HTML to stdout, similarly to links -dump or lynx
12 -dump but simpler and more secure.
13
14 * HTML and XHTML will be supported.
15 * There will be some workarounds and quirks for broken and legacy HTML code.
16 * It will be usable and secure for reading HTML from mails and RSS/Atom feeds.
17 * No remote resources which are part of the HTML will be downloaded:
18 images, video, audio, etc. But these may be visible as a link reference.
19 * Data will be written to stdout. Intended for plain-text or a text terminal.
20 * No support for Javascript, CSS, frame rendering or form processing.
21 * No HTTP or network protocol handling: HTML data is read from stdin.
22 * Listings for references and some options to extract them in a list that is
23 usable for scripting. Some references are: link anchors, images, audio, video,
24 HTML (i)frames, etc.
25 * Security: on OpenBSD it uses pledge("stdio", NULL).
26 * Keep the code relatively small, simple and hackable.
27
28
29 ## Features
30
31 * Support for word-wrapping.
32 * A mode to enable basic markup: bold, underline, italic and blink ;)
33 * Indentation of headers, paragraphs, pre and list items.
34 * Basic support to query elements or hide them.
35 * Show link references.
36 * Show link references and resources such as img, video, audio, subtitles.
37 * Export link references and resources to a TAB-separated format.
38
39
40 ## Usage examples
41
42 url='https://codemadness.org/sfeed.html'
43
44 curl -s "$url" | webdump -r -b "$url" | less
45
46 curl -s "$url" | webdump -8 -a -i -l -r -b "$url" | less -R
47
48 curl -s "$url" | webdump -s 'main' -8 -a -i -l -r -b "$url" | less -R
49
50 Yes, all these option flags look ugly, a shellscript wrapper could be used :)
51
52
53 ## Practical examples
54
55 To use webdump as a HTML to text filter for example in the mutt mail client,
56 change in ~/.mailcap:
57
58 text/html; webdump -i -l -r < %s; needsterminal; copiousoutput
59
60 In mutt you should then add:
61
62 auto_view text/html
63
64
65 Using webdump as a HTML to text filter for sfeed_curses (otherwise the default is lynx):
66
67 SFEED_HTMLCONV="webdump -d -8 -r -i -l -a" sfeed_curses ~/.sfeed/feeds/*
68
69
70 # Query/selector examples
71
72 The query syntax using the -s option is a bit inspired by CSS (but much more limited).
73
74 To get the title from a HTML page:
75
76 url='https://codemadness.org/sfeed.html'
77
78 title=$(curl -s "$url" | webdump -s 'title')
79 printf '%s\n' "$title"
80
81 List audio and video-related content from a HTML page, redirect fd 3 to fd 1 (stdout):
82
83 url="https://media.ccc.de/v/051_Recent_features_to_OpenBSD-ntpd_and_bgpd"
84 curl -s "$url" | webdump -x -s 'audio,video' -b "$url" 3>&1 >/dev/null | cut -f 2
85
86
87 ## Clone
88
89 git clone git://git.codemadness.org/webdump
90
91
92 ## Browse
93
94 You can browse the source-code at:
95
96 * <https://git.codemadness.org/webdump/>
97 * <gopher://codemadness.org/1/git/webdump>
98
99
100 ## Download releases
101
102 Releases are available at:
103
104 * <https://codemadness.org/releases/webdump/>
105 * <gopher://codemadness.org/1/releases/webdump>
106
107
108 ## Build and install
109
110 $ make
111 # make install
112
113
114 ## Dependencies
115
116 * C compiler.
117 * libc + some BSDisms.
118
119
120 ## Trade-offs
121
122 All software has trade-offs.
123
124 webdump processes HTML in a single-pass. It does not buffer the full DOM tree.
125 Although due to the nature of HTML/XML some parts like attributes need to be
126 buffered.
127
128 Rendering tables in webdump is very limited. Twibright Links has really nice
129 table rendering. However implementing a similar feature in the current design of
130 webdump would make the code much more complex. Twibright links
131 processes a full DOM tree and processes the tables in multiple passes (to
132 measure the table cells) etc. Of course tables can be nested also, or HTML tables
133 that are used for creating layouts (these are mostly older webpages).
134
135 These trade-offs and preferences are chosen for now. It may change in the
136 future. Fortunately there are the usual good suspects for HTML to plain-text
137 conversion, each with their own chosen trade-offs of course:
138
139 * twibright links: <http://links.twibright.com/>
140 * lynx: <https://lynx.invisible-island.net/>
141 * w3m: <https://w3m.sourceforge.net/>
142 * xmllint (part of libxml2): <https://gitlab.gnome.org/GNOME/libxml2/-/wikis/home>
143 * xmlstarlet: <https://xmlstar.sourceforge.net/>