json2tsv.html - www.codemadness.org - www.codemadness.org saait content files
(HTM) git clone git://git.codemadness.org/www.codemadness.org
(DIR) Log
(DIR) Files
(DIR) Refs
(DIR) README
(DIR) LICENSE
---
json2tsv.html (8660B)
---
1 <!DOCTYPE html>
2 <html dir="ltr" lang="en">
3 <head>
4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
5 <meta http-equiv="Content-Language" content="en" />
6 <meta name="viewport" content="width=device-width" />
7 <meta name="keywords" content="json2tsv, JSON, tsv, TAB-Separated Value Format" />
8 <meta name="description" content="json2tsv: a JSON to TAB-Separated Value converter" />
9 <meta name="author" content="Hiltjo" />
10 <meta name="generator" content="Static content generated using saait: https://codemadness.org/saait.html" />
11 <title>json2tsv: a JSON to TSV converter - Codemadness</title>
12 <link rel="stylesheet" href="style.css" type="text/css" media="screen" />
13 <link rel="stylesheet" href="print.css" type="text/css" media="print" />
14 <link rel="alternate" href="atom.xml" type="application/atom+xml" title="Codemadness Atom Feed" />
15 <link rel="alternate" href="atom_content.xml" type="application/atom+xml" title="Codemadness Atom Feed with content" />
16 <link rel="icon" href="/favicon.png" type="image/png" />
17 </head>
18 <body>
19 <nav id="menuwrap">
20 <table id="menu" width="100%" border="0">
21 <tr>
22 <td id="links" align="left">
23 <a href="index.html">Blog</a> |
24 <a href="/git/" title="Git repository with some of my projects">Git</a> |
25 <a href="/releases/">Releases</a> |
26 <a href="gopher://codemadness.org">Gopherhole</a>
27 </td>
28 <td id="links-contact" align="right">
29 <span class="hidden"> | </span>
30 <a href="feeds.html">Feeds</a> |
31 <a href="pgp.asc">PGP</a> |
32 <a href="mailto:hiltjo@AT@codemadness.DOT.org">Mail</a>
33 </td>
34 </tr>
35 </table>
36 </nav>
37 <hr class="hidden" />
38 <main id="mainwrap">
39 <div id="main">
40 <article>
41 <header>
42 <h1>json2tsv: a JSON to TSV converter</h1>
43 <p>
44 <strong>Last modification on </strong> <time>2021-09-25</time>
45 </p>
46 </header>
47
48 <p>Convert JSON to TSV or separated output.</p>
49 <p>json2tsv reads JSON data from stdin. It outputs each JSON type to a TAB-
50 Separated Value format per line by default.</p>
51 <h2>TAB-Separated Value format</h2>
52 <p>The output format per line is:</p>
53 <pre><code>nodename<TAB>type<TAB>value<LF>
54 </code></pre>
55 <p>Control-characters such as a newline, TAB and backslash (\n, \t and \) are
56 escaped in the nodename and value fields. Other control-characters are
57 removed.</p>
58 <p>The type field is a single byte and can be:</p>
59 <ul>
60 <li>a for array</li>
61 <li>b for bool</li>
62 <li>n for number</li>
63 <li>o for object</li>
64 <li>s for string</li>
65 <li>? for null</li>
66 </ul>
67 <p>Filtering on the first field "nodename" is easy using awk for example.</p>
68 <h2>Features</h2>
69 <ul>
70 <li>Accepts all <strong>valid</strong> JSON.</li>
71 <li>Designed to work well with existing UNIX programs like awk and grep.</li>
72 <li>Straightforward and not much lines of code: about 475 lines of C.</li>
73 <li>Few dependencies: C compiler (C99), libc.</li>
74 <li>No need to learn a new (meta-)language for processing data.</li>
75 <li>The parser supports code point decoding and UTF-16 surrogates to UTF-8.</li>
76 <li>It does not output control-characters to the terminal for security reasons by
77 default (but it has a -r option if needed).</li>
78 <li>On OpenBSD it supports <a href="https://man.openbsd.org/pledge">pledge(2)</a> for syscall restriction:
79 pledge("stdio", NULL).</li>
80 <li>Supports setting a different field separator and record separator with the -F
81 and -R option.</li>
82 </ul>
83 <h2>Cons</h2>
84 <ul>
85 <li>For the tool there is additional overhead by processing and filtering data
86 from stdin after parsing.</li>
87 <li>The parser does not do complete validation on numbers.</li>
88 <li>The parser accepts some bad input such as invalid UTF-8
89 (see <a href="https://tools.ietf.org/html/rfc8259#section-8.1">RFC8259 - 8.1. Character Encoding</a>).
90 json2tsv reads from stdin and does not do assumptions about a "closed
91 ecosystem" as described in the RFC.</li>
92 <li>The parser accepts some bad JSON input and "extensions"
93 (see <a href="https://tools.ietf.org/html/rfc8259#section-9">RFC8259 - 9. Parsers</a>).</li>
94 <li>Encoded NUL bytes (\u0000) in strings are ignored.
95 (see <a href="https://tools.ietf.org/html/rfc8259#section-9">RFC8259 - 9. Parsers</a>).
96 "An implementation may set limits on the length and character contents of
97 strings."</li>
98 <li>The parser is not the fastest possible JSON parser (but also not the
99 slowest). For example: for ease of use, at the cost of performance all
100 strings are decoded, even though they may be unused.</li>
101 </ul>
102 <h2>Why Yet Another JSON parser?</h2>
103 <p>I wanted a tool that makes parsing JSON easier and work well from the shell,
104 similar to <a href="https://stedolan.github.io/jq/">jq</a>.</p>
105 <p>sed and grep often work well enough for matching some value using some regex
106 pattern, but it is not good enough to parse JSON correctly or to extract all
107 information: just like parsing HTML/XML using some regex is not good (enough)
108 or a good idea :P.</p>
109 <p>I didn't want to learn a new specific <a href="https://stedolan.github.io/jq/manual/#Builtinoperatorsandfunctions">meta-language</a> which jq has and wanted
110 something simpler.</p>
111 <p>While it is more efficient to embed this query language for data aggregation,
112 it is also less simple. In my opinion it is simpler to separate this and use
113 pattern-processing by awk or an other filtering/aggregating program.</p>
114 <p>For the parser, there are many JSON parsers out there, like the efficient
115 <a href="https://github.com/zserge/jsmn">jsmn parser</a>, however a few parser behaviours I want to have are:</p>
116 <ul>
117 <li>jsmn buffers data as tokens, which is very efficient, but also a bit
118 annoying as an API as it requires another layer of code to interpret the
119 tokens.</li>
120 <li>jsmn does not handle decoding strings by default. Which is very efficient
121 if you don't need parts of the data though.</li>
122 <li>jsmn does not keep context of nested structures by default, so may require
123 writing custom utility functions for nested data.</li>
124 </ul>
125 <p>This is why I went for a parser design that uses a single callback per "node"
126 type and keeps track of the current nested structure in a single array and
127 emits that.</p>
128 <h2>Clone</h2>
129 <pre><code>git clone git://git.codemadness.org/json2tsv
130 </code></pre>
131 <h2>Browse</h2>
132 <p>You can browse the source-code at:</p>
133 <ul>
134 <li><a href="https://git.codemadness.org/json2tsv/">https://git.codemadness.org/json2tsv/</a></li>
135 <li><a href="gopher://codemadness.org/1/git/json2tsv">gopher://codemadness.org/1/git/json2tsv</a></li>
136 </ul>
137 <h2>Download releases</h2>
138 <p>Releases are available at:</p>
139 <ul>
140 <li><a href="https://codemadness.org/releases/json2tsv/">https://codemadness.org/releases/json2tsv/</a></li>
141 <li><a href="gopher://codemadness.org/1/releases/json2tsv">gopher://codemadness.org/1/releases/json2tsv</a></li>
142 </ul>
143 <h2>Build and install</h2>
144 <pre><code>$ make
145 # make install
146 </code></pre>
147 <h2>Examples</h2>
148 <p>An usage example to parse posts of the JSON API of <a href="https://www.reddit.com/">reddit.com</a> and format them
149 to a plain-text list using awk:</p>
150 <pre><code>#!/bin/sh
151 curl -s -H 'User-Agent:' 'https://old.reddit.com/.json?raw_json=1&limit=100' | \
152 json2tsv | \
153 awk -F '\t' '
154 function show() {
155 if (length(o["title"]) == 0)
156 return;
157 print n ". " o["title"] " by " o["author"] " in r/" o["subreddit"];
158 print o["url"];
159 print "";
160 }
161 $1 == ".data.children[].data" {
162 show();
163 n++;
164 delete o;
165 }
166 $1 ~ /^\.data\.children\[\]\.data\.[a-zA-Z0-9_]*$/ {
167 o[substr($1, 23)] = $3;
168 }
169 END {
170 show();
171 }'
172 </code></pre>
173 <h2>References</h2>
174 <ul>
175 <li>Sites:
176 <ul>
177 <li><a href="http://seriot.ch/parsing_json.php">seriot.ch - Parsing JSON is a Minefield</a></li>
178 <li><a href="https://github.com/nst/JSONTestSuite">A comprehensive test suite for RFC 8259 compliant JSON parsers</a></li>
179 <li><a href="https://json.org/">json.org</a></li>
180 </ul>
181 </li>
182 <li>Current standard:
183 <ul>
184 <li><a href="https://tools.ietf.org/html/rfc8259">RFC8259 - The JavaScript Object Notation (JSON) Data Interchange Format</a></li>
185 <li><a href="https://www.ecma-international.org/publications/standards/Ecma-404.htm">Standard ECMA-404 - The JSON Data Interchange Syntax (2nd edition (December 2017)</a></li>
186 </ul>
187 </li>
188 <li>Historic standards:
189 <ul>
190 <li><a href="https://tools.ietf.org/html/rfc7159">RFC7159 - The JavaScript Object Notation (JSON) Data Interchange Format (obsolete)</a></li>
191 <li><a href="https://tools.ietf.org/html/rfc7158">RFC7158 - The JavaScript Object Notation (JSON) Data Interchange Format (obsolete)</a></li>
192 <li><a href="https://tools.ietf.org/html/rfc4627">RFC4627 - The JavaScript Object Notation (JSON) Data Interchange Format (obsolete, original)</a></li>
193 </ul>
194 </li>
195 </ul>
196
197 </article>
198 </div>
199 </main>
200 </body>
201 </html>