sfeedrc.5 - sfeed - RSS and Atom parser
(HTM) git clone git://git.codemadness.org/sfeed
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
sfeedrc.5 (6669B)
---
1 .Dd August 8, 2025
2 .Dt SFEEDRC 5
3 .Os
4 .Sh NAME
5 .Nm sfeedrc
6 .Nd sfeed_update(1) configuration file
7 .Sh DESCRIPTION
8 .Nm
9 is the configuration file for
10 .Xr sfeed_update 1
11 and is evaluated as a shellscript.
12 .Sh VARIABLES
13 .Bl -tag -width Ds
14 .It Va sfeedpath
15 can be set for the directory to store the TAB-separated feed files.
16 The default is
17 .Pa $HOME/.sfeed/feeds .
18 .It Va maxjobs
19 can be used to change the amount of concurrent
20 .Fn feed
21 jobs.
22 The default is 16.
23 .El
24 .Sh FUNCTIONS
25 .Bl -tag -width Ds
26 .It Fn feeds
27 This function is the required "main" entry-point function called from
28 .Xr sfeed_update 1 .
29 .It Fn feed "name" "feedurl" "basesiteurl" "encoding"
30 Inside the
31 .Fn feeds
32 function feeds can be defined by calling the
33 .Fn feed
34 function.
35 Its arguments are:
36 .Bl -tag -width Ds
37 .It Fa name
38 Name of the feed, this is also used as the filename for the TAB-separated
39 feed file.
40 The feed name cannot contain the '/' character because it is a path separator,
41 they will be replaced with '_'.
42 Each
43 .Fa name
44 should be unique.
45 .It Fa feedurl
46 URL to fetch the RSS/Atom data from.
47 This is usually a HTTP or HTTPS URL.
48 .It Op Fa basesiteurl
49 Base URL of the feed links.
50 This argument allows fixing relative item links.
51 .Pp
52 According to the RSS and Atom specification, feeds should always have absolute
53 URLs, but this is not always the case in practice.
54 .It Op Fa encoding
55 Feeds are converted from this
56 .Ar encoding
57 to UTF-8.
58 The
59 .Ar encoding
60 should be a usable
61 character-set name for the
62 .Xr iconv 1
63 tool.
64 .El
65 .El
66 .Sh OVERRIDE FUNCTIONS
67 Because
68 .Xr sfeed_update 1
69 is a shellscript each function can be overridden to change its behaviour.
70 Notable functions are:
71 .Bl -tag -width Ds
72 .It Fn fetch "name" "url" "feedfile"
73 Fetch feed from URL and write the data to stdout.
74 Its arguments are:
75 .Bl -tag -width Ds
76 .It Fa name
77 Feed name.
78 .It Fa url
79 URL to fetch.
80 .It Fa feedfile
81 Used feedfile (useful for comparing modification times).
82 .El
83 .Pp
84 By default the tool
85 .Xr curl 1
86 is used.
87 .It Fn convertencoding "name" "from" "to"
88 Convert data from stdin from one text-encoding to another and write it to
89 stdout.
90 Its arguments are:
91 .Bl -tag -width Ds
92 .It Fa name
93 Feed name.
94 .It Fa from
95 From text-encoding.
96 .It Fa to
97 To text-encoding.
98 .El
99 .Pp
100 By default the tool
101 .Xr iconv 1
102 is used.
103 .It Fn parse "name" "feedurl" "basesiteurl"
104 Read RSS/Atom XML data from stdin, convert and write it as
105 .Xr sfeed 5
106 data to stdout.
107 Its arguments are:
108 .Bl -tag -width Ds
109 .It Fa name
110 Feed name.
111 .It Fa feedurl
112 URL of the feed.
113 .It Fa basesiteurl
114 Base URL of the feed links.
115 This argument allows to fix relative item links.
116 .El
117 .It Fn filter "name" "url"
118 Filter
119 .Xr sfeed 5
120 data from stdin and write it to stdout.
121 Its arguments are:
122 .Bl -tag -width Ds
123 .It Fa name
124 Feed name.
125 .It Fa url
126 URL of the feed.
127 .El
128 .It Fn merge "name" "oldfile" "newfile"
129 Merge
130 .Xr sfeed 5
131 data of oldfile with newfile and write it to stdout.
132 Its arguments are:
133 .Bl -tag -width Ds
134 .It Fa name
135 Feed name.
136 .It Fa oldfile
137 Old file.
138 .It Fa newfile
139 New file.
140 .El
141 .It Fn order "name" "url"
142 Sort
143 .Xr sfeed 5
144 data from stdin and write it to stdout.
145 Its arguments are:
146 .Bl -tag -width Ds
147 .It Fa name
148 Feed name.
149 .It Fa url
150 URL of the feed.
151 .El
152 .El
153 .Sh EXAMPLES
154 An example configuration file is included named sfeedrc.example and also
155 shown below:
156 .Bd -literal -offset 4n
157 #sfeedpath="$HOME/.sfeed/feeds"
158
159 # list of feeds to fetch:
160 feeds() {
161 # feed <name> <feedurl> [basesiteurl] [encoding]
162 feed "codemadness" "https://www.codemadness.org/atom_content.xml"
163 feed "explosm" "http://feeds.feedburner.com/Explosm"
164 feed "golang github releases" "https://github.com/golang/go/releases.atom"
165 feed "linux kernel" "https://www.kernel.org/feeds/kdist.xml" "https://www.kernel.org"
166 feed "reddit openbsd" "https://old.reddit.com/r/openbsd/.rss"
167 feed "slashdot" "http://rss.slashdot.org/Slashdot/slashdot" "http://slashdot.org"
168 feed "tweakers" "http://feeds.feedburner.com/tweakers/mixed" "http://tweakers.net" "iso-8859-1"
169 # get youtube Atom feed: curl -s -L 'https://www.youtube.com/user/gocoding/videos' | sfeed_web | cut -f 1
170 feed "youtube golang" "https://www.youtube.com/feeds/videos.xml?channel_id=UCO3LEtymiLrgvpb59cNsb8A"
171 feed "xkcd" "https://xkcd.com/atom.xml" "https://xkcd.com"
172 }
173 .Ed
174 .Pp
175 To change the default
176 .Xr curl 1
177 options for fetching the data, the
178 .Fn fetch
179 function can be overridden and added at the top of the
180 .Nm
181 file, for example:
182 .Bd -literal -offset 4n
183 # fetch(name, url, feedfile)
184 fetch() {
185 # allow for 1 redirect, set User-Agent, timeout is 15 seconds.
186 curl -L --max-redirs 1 -H "User-Agent: 007" -f -s -m 15 \e
187 "$2" 2>/dev/null
188 }
189 .Ed
190 .Pp
191 Caching, incremental data updates and bandwidth saving
192 .Pp
193 For HTTP servers that support it some bandwidth saving can be done by changing
194 some of the default curl options.
195 These options can come at a cost of some privacy, because it exposes additional
196 metadata from the previous request.
197 .Pp
198 .Bl -bullet -compact
199 .It
200 The curl ETag options (--etag-save and --etag-compare) can be used to store and
201 send the previous ETag header value.
202 curl version 7.73+ is recommended for it to work properly.
203 .It
204 The curl -z option can be used to send the modification date of a local file as
205 a HTTP If-Modified-Since request header.
206 The server can then respond if the data is modified or not or respond with only
207 the incremental data.
208 .It
209 The curl --compressed option can be used to indicate the client supports
210 decompression.
211 Because RSS/Atom feeds are textual XML data this generally compresses very
212 well.
213 .It
214 The example below also sets the User-Agent to sfeed, because some CDNs block
215 HTTP clients based on the User-Agent request header.
216 .El
217 .Pp
218 Example:
219 .Bd -literal -offset 4n
220 mkdir -p "$HOME/.sfeed/etags" "$HOME/.sfeed/lastmod"
221
222 # fetch(name, url, feedfile)
223 fetch() {
224 basename="$(basename "$3")"
225 etag="$HOME/.sfeed/etags/${basename}"
226 lastmod="$HOME/.sfeed/lastmod/${basename}"
227 output="${sfeedtmpdir}/feeds/${basename}.xml"
228
229 curl \e
230 -f -s -m 15 \e
231 -L --max-redirs 0 \e
232 -H "User-Agent: sfeed" \e
233 --compressed \e
234 --etag-save "${etag}" --etag-compare "${etag}" \e
235 -R -o "${output}" \e
236 -z "${lastmod}" \e
237 "$2" 2>/dev/null || return 1
238
239 # succesful, but no file written: assume it is OK and Not Modified.
240 [ -e "${output}" ] || return 0
241
242 # use server timestamp from curl -R to set Last-Modified.
243 touch -r "${output}" "${lastmod}" 2>/dev/null
244 cat "${output}" 2>/dev/null
245 # use write output status, other errors are ignored here.
246 fetchstatus="$?"
247 rm -f "${output}" 2>/dev/null
248 return "${fetchstatus}"
249 }
250 .Ed
251 .Pp
252 The README file has more examples.
253 .Sh SEE ALSO
254 .Xr curl 1 ,
255 .Xr iconv 1 ,
256 .Xr sfeed_update 1 ,
257 .Xr sh 1
258 .Sh AUTHORS
259 .An Hiltjo Posthuma Aq Mt hiltjo@codemadness.org