hurl.html - www.codemadness.org - www.codemadness.org saait content files
(HTM) git clone git://git.codemadness.org/www.codemadness.org
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
hurl.html (4834B)
---
1 <!DOCTYPE html>
2 <html dir="ltr" lang="en">
3 <head>
4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
5 <meta http-equiv="Content-Language" content="en" />
6 <meta name="viewport" content="width=device-width" />
7 <meta name="keywords" content="Gopher, HTTP, HTTPS" />
8 <meta name="description" content="hurl: HTTP, HTTPS and Gopher file grabber" />
9 <meta name="author" content="Hiltjo" />
10 <meta name="generator" content="Static content generated using saait: https://codemadness.org/saait.html" />
11 <title>hurl: HTTP, HTTPS and Gopher file grabber - Codemadness</title>
12 <link rel="stylesheet" href="style.css" type="text/css" media="screen" />
13 <link rel="stylesheet" href="print.css" type="text/css" media="print" />
14 <link rel="alternate" href="atom.xml" type="application/atom+xml" title="Codemadness Atom Feed" />
15 <link rel="alternate" href="atom_content.xml" type="application/atom+xml" title="Codemadness Atom Feed with content" />
16 <link rel="icon" href="/favicon.png" type="image/png" />
17 </head>
18 <body>
19 <nav id="menuwrap">
20 <table id="menu" width="100%" border="0">
21 <tr>
22 <td id="links" align="left">
23 <a href="index.html">Blog</a> |
24 <a href="/git/" title="Git repository with some of my projects">Git</a> |
25 <a href="/releases/">Releases</a> |
26 <a href="gopher://codemadness.org">Gopherhole</a>
27 </td>
28 <td id="links-contact" align="right">
29 <span class="hidden"> | </span>
30 <a href="feeds.html">Feeds</a> |
31 <a href="pgp.asc">PGP</a> |
32 <a href="mailto:hiltjo@AT@codemadness.DOT.org">Mail</a>
33 </td>
34 </tr>
35 </table>
36 </nav>
37 <hr class="hidden" />
38 <main id="mainwrap">
39 <div id="main">
40 <article>
41 <header>
42 <h1>hurl: HTTP, HTTPS and Gopher file grabber</h1>
43 <p>
44 <strong>Last modification on </strong> <time>2020-07-20</time>
45 </p>
46 </header>
47
48 <p>hurl is a relatively simple HTTP, HTTPS and Gopher client/file grabber.</p>
49 <h2>Why?</h2>
50 <p>Sometimes (or most of the time?) you just want to fetch a file via the HTTP,
51 HTTPS or Gopher protocol.</p>
52 <p>The focus of this tool is only this.</p>
53 <h2>Features</h2>
54 <ul>
55 <li>Uses OpenBSD pledge(2) and unveil(2). Allow no filesystem access (writes to
56 stdout).</li>
57 <li>Impose time-out and maximum size limits.</li>
58 <li>Use well-defined exitcodes for reliable scripting (curl sucks at this).</li>
59 <li>Send as little information as possible (no User-Agent etc by default).</li>
60 </ul>
61 <h2>Anti-features</h2>
62 <ul>
63 <li>No HTTP byte range support.</li>
64 <li>No HTTP User-Agent.</li>
65 <li>No HTTP If-Modified-Since/If-* support.</li>
66 <li>No HTTP auth support.</li>
67 <li>No HTTP/2+ support.</li>
68 <li>No HTTP keep-alive.</li>
69 <li>No HTTP chunked-encoding support.</li>
70 <li>No HTTP redirect support.</li>
71 <li>No (GZIP) compression support.</li>
72 <li>No cookie-jar or cookie parsing support.</li>
73 <li>No Gopher text handling (".\r\n").</li>
74 <li>... etc...</li>
75 </ul>
76 <h2>Dependencies</h2>
77 <ul>
78 <li>C compiler (C99).</li>
79 <li>libc + some BSD functions like err() and strlcat().</li>
80 <li>LibreSSL(-portable)</li>
81 <li>libtls (part of LibreSSL).</li>
82 </ul>
83 <h2>Optional dependencies</h2>
84 <ul>
85 <li>POSIX make(1) (for Makefile).</li>
86 <li>mandoc for documentation: <a href="https://mdocml.bsd.lv/">https://mdocml.bsd.lv/</a></li>
87 </ul>
88 <h2>Clone</h2>
89 <pre><code>git clone git://git.codemadness.org/hurl
90 </code></pre>
91 <h2>Browse</h2>
92 <p>You can browse the source-code at:</p>
93 <ul>
94 <li><a href="https://git.codemadness.org/hurl/">https://git.codemadness.org/hurl/</a></li>
95 <li><a href="gopher://codemadness.org/1/git/hurl">gopher://codemadness.org/1/git/hurl</a></li>
96 </ul>
97 <h2>Download releases</h2>
98 <p>Releases are available at:</p>
99 <ul>
100 <li><a href="https://codemadness.org/releases/hurl/">https://codemadness.org/releases/hurl/</a></li>
101 <li><a href="gopher://codemadness.org/1/releases/hurl">gopher://codemadness.org/1/releases/hurl</a></li>
102 </ul>
103 <h2>Build and install</h2>
104 <pre><code>$ make
105 # make install
106 </code></pre>
107 <h2>Examples</h2>
108 <p>Fetch the Atom feed from this site using a maximum filesize limit of 1MB and
109 a time-out limit of 15 seconds:</p>
110 <pre><code>hurl -m 1048576 -t 15 "https://codemadness.org/atom.xml"
111 </code></pre>
112 <p>There is an -H option to add custom headers. This way some of the anti-features
113 listed above are supported. For example some CDNs like Cloudflare are known to
114 block empty or certain User-Agents.</p>
115 <p>User-Agent:</p>
116 <pre><code>hurl -H 'User-Agent: some browser' 'https://codemadness.org/atom.xml'
117 </code></pre>
118 <p>HTTP Basic Auth (base64-encoded username:password):</p>
119 <pre><code>hurl -H 'Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=' \
120 'https://codemadness.org/atom.xml'
121 </code></pre>
122 <p>GZIP (this assumes the served response Content-Type is gzip):</p>
123 <pre><code>hurl -H 'Accept-Encoding: gzip' 'https://somesite/' | gzip -d
124 </code></pre>
125
126 </article>
127 </div>
128 </main>
129 </body>
130 </html>