https://ishan.page/blog/2023-11-06-jq-by-example/

Avatar 

Ishan Das Sharma

Self-taught developer, writing about the things I have learned

 1.  
 2.  
 3.  
 4.  

 1. Home
 2. About
 3. All Posts
 4. Search
 5. Dark Mode

Table of contents

 1. Introduction
 2. Basic Operations
     1. Selecting values
     2. Filtering Arrays
     3. Mapping Arrays
     4. Combining Filters
     5. Splitting Strings
     6. Conditional Logic
     7. Handling Null Values
     8. Formatting Output
     9. Multiple Outputs
 3. Dealing with Nested Items
     1. Recursive Descent
     2. Filtering Nested Arrays
     3. Flattening Nested JSON Objects
     4. Recursive Object Manipulation
     5. Complex Object Transformation
     6. Walk through object and apply a transformation conditionally
 4. Statistical Operations
     1. Sorting Arrays
     2. Extracting Unique Values from an Array
     3. Calculating Averages
     4. Grouping and Aggregating
     5. Filtering after Aggregation
     6. Custom Aggregation with reduce
     7. Calculating Histogram Bins
 5. Other Common Operations
     1. Extracting Values Based on Multiple Conditions
     2. Formatting Unix Timestamps
     3. Enumerating by Top Level Key and Value
 6. Closing Thoughts
     1. How Does This Article Work?
     2. Get In Touch
     3. Extra Resources

Featured image of post The Ultimate Interactive JQ Guide
Featured Articles

The Ultimate Interactive JQ Guide

Learn how to search, query, and modify JSON data with 25 interactive
jq examples and explainations

Nov 06, 2023
9 minute read


    Cover Photo by Pixabay

This article requires JavaScript. Sorry...

Has this ever happened to you?

You've just received a massive JSON file that looks like it was
designed to confuse you. Or maybe you entered a command, and you got
so much JSON that it looks incomprehensible.

The data you need is buried inside, and you're dreading the hours
it'll take to extract and clean it up.

I've been there. I've grepped my way through JSON and written ad-hoc
Python scripts to process it for me.

But things don't have to be like this.

Introduction

jq is one of the best-kept secrets in the data processing world.

Here are some scenarios where jq could swoop in to save your day (and
saves mine regularly):

 1. Integrating with APIs in shell scripts often means handling JSON
    responses, requiring data extraction and manipulation.

 2. Data from different sources may need to be converted to or from
    JSON format for compatibility.

 3. Managing software configuration files in JSON format can be a
    regular task.

 4. Extracting data from websites often results in dealing with JSON
    data that requires parsing and filtering.

 5. Server logs and monitoring data often use JSON, necessitating
    parsing and analysis.

 6. Infra as Code tools like Ansible and Terraform use JSON-like
    configurations, requiring management. JSON is a subset of YAML,
    so every valid JSON file is also a valid YAML file.

All examples are fully interactive, so I encourage you to play
around!
In fact, I'll be downright heartbroken if you don't, because I put a
lot of effort into it. You can edit both the input JSON data, and the
jq program as well.

Let's dive in! We'll start off easy, and get slowly deeper into the
weeds.

Basic Operations

Selecting values

Everything in jq is a filter. The dot . is used to select the current
object or element, and we can put the property name after it to
access a key from an object:

```bash echo '{"name": "Alice", "age": 30}' | \ jq '.name' # Output:
"Alice" ```

Filtering Arrays

The .[] notation is used to iterate over the elements of an array in
a JSON document. It allows you to access each element of an array and
perform operations on them.

The select() function is used to filter JSON data based on a
specified condition or criteria. It is a powerful tool for extracting
specific elements from a JSON document that meet certain conditions.

Similiar to shell scripting, jq works on a pipes-and-filters manner.
We use the | to send the data from one filter to the next.

```bash echo '[{"name": "Alice", "age": 30}, {"name": "Bob", "age":
25}]' | \ jq '.[] | select(.age > 28)' # Output: {"name": "Alice",
"age": 30} ```

Mapping Arrays

We can use the map function to run any operation on every element of
the array and return a new array containing the outputs of that
operation:

```bash echo '[1, 2, 3, 4, 5]' | jq 'map(. * 2)' # Output:
[2,4,6,8,10] ```

Combining Filters

The pipe operator | can be used to chain as many filters or functions
as we want:

```bash echo '[{"name": "Alice", "age": 30}, {"name": "Bob", "age":
25}]' | jq '.[] | select(.age > 28) | .name' # Output: "Alice" ```

Splitting Strings

We can use the split() function to a split a string on a particular
separator character.
Note also the usage of .[0] to select the first index from the split
array.

```bash echo '{"name": "Alice Smith"}' | jq '.name | split(" ") | .
[0]' # Output: "Alice" ```

Conditional Logic

We can use if to create expressions

```bash echo '{"name": "Alice", "age": 30}' | jq 'if .age > 18 then
"Adult" else "Child" end' # Output: "Adult" ```

Handling Null Values

Null values can often mess up logic in our scripts, so we can filter
them all out using map and select

```bash echo '[1, null, 3, null, 5]' | jq 'map(select(. != null))' #
Output: [1,3,5] ```

Formatting Output

Sometimes we don't want JSON output. We want it in a particular
string format.
Note the use of the -r flag, it makes the output raw. Without it, it
would be displayed with quote marks around it.

```bash echo '{"name": "Alice", "age": 30}' | jq -r '"Name: \(.name),
Age: \(.age)"' # Output: Name: Alice, Age: 30 ```

Multiple Outputs

Curly braces create a new object, which we can use for multiple
outputs:

```bash echo '{"name": "Alice", "age": 30}' | jq '{name: .name, age:
(.age + 5)}' # Output: {"name": "Alice", "age": 35} ```

Dealing with Nested Items

[russian_dolls]

    Photo by cottonbro studio

JSON is very commonly used to store nested objects, and we often need
to traverse or manipulate such structures. jq gives us all the tools
we need to make it easy:

Recursive Descent

We can use .. to recursively descend through a tree of an object.

```bash echo '{"data": {"value": 42, "nested": {"value": 24}}}' | jq
'.. | .value?' # Output: 42, 24 ```

Filtering Nested Arrays

```bash echo '{"data": [{"values": [1, 2, 3]}, {"values": [4, 5,
6]}]}' | jq '.data[].values[] | select(. > 3)' # Output: 4, 5, 6 ```

Flattening Nested JSON Objects

Often, we just want all the key-values, and flattening the object may
be the most convenient way to go:

```bash echo '{"person": {"name": {"first": "Alice", "last":
"Smith"}, "age": 30}}' | jq 'paths as $p | select(getpath($p) | type
!= "object") | ($p | join(".")) + " = " + (getpath($p) | tostring)' #
Output: "person.name.first = Alice", "person.name.last = Smith",
"person.age = 30" ```

Recursive Object Manipulation

We can use the recurse as well, to traverse a tree.

```bash echo '{"data": {"value": 42, "nested": {"value": 24}}}' | jq
'recurse | .value? | select(. != null) | { value: (. * 5) } | add' #
Output: 210, 120 ```

Complex Object Transformation

```bash echo '{"items": [{"name": "Apple", "price": 1}, {"name":
"Banana", "price": 0.5}]}' | jq '.items | map({(.name): (.price *
2)}) | add' # Output: {"Apple": 2, "Banana": 1.0} ```

Walk through object and apply a transformation conditionally

The walk() function provides a convenient way to traverse a nested
object and apply some transformation to it.

```bash echo '{"data": {"values": [1, 2, 3], "nested": {"values": [4,
5, 6]}}}' | jq 'recurse(.values) |= map(. * 2)' # Output: {"data":
{"values":[2,4,6],"nested":{"values":[8,10,12]}}} ```

Statistical Operations

[stats]

    Photo by Leeloo Thefirst

jq is incredibly handy for doing quick and dirty statistical analysis
in the field. Here's most of the common operations related to that

Sorting Arrays

Sorting an array is a basic operation that is useful for many things
in statistics.

```bash echo '[3, 1, 4, 2, 5]' | jq 'sort' # Output: [1,2,3,4,5] ```

Extracting Unique Values from an Array

Extracting unique values from an array is another fairly basic
operation that we need for many things.

```bash echo '[1, 2, 2, 3, 4, 4, 5]' | jq 'unique' # Output:
[1,2,3,4,5] ```

Calculating Averages

Calculating the mean or average of a dataset is a common statistical
operation we may often need to do

```bash echo '[{"score": 90}, {"score": 85}, {"score": 95}]' | jq
'map(.score) | add / length' # Output: 90 ```

Grouping and Aggregating

We can group an array of objects by a particular key and get an
aggregated value of the other keys fairly easily:

```bash echo '[{"category": "A", "value": 10}, {"category": "B",
"value": 20}, {"category": "A", "value": 5}]' | jq 'group_by
(.category) | map({category: .[0].category, sum: map(.value) | add})'
# Output: [{"category": "A", "sum": 15}, {"category": "B", "sum":
20}] ```

Filtering after Aggregation

```bash echo '[{"category": "A", "value": 10}, {"category": "B",
"value": 20}, {"category": "A", "value": 5}]' | jq 'group_by
(.category) | map({category: .[0].category, sum: (map(.value) |
add)}) | .[] | select(.sum > 17)' # Output: {"category":"B","sum":20}
```

Custom Aggregation with reduce

We can also use reduce to perform a single-output aggregation from an
array

```bash echo '[{"value": 10}, {"value": 20}, {"value": 30}]' | jq
'reduce .[] as $item (0; . + $item.value)' # Output: 60 ```

Calculating Histogram Bins

We may want to calculate a histogram from an array of data.

```bash echo '[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]' |
jq 'group_by(. / 5 | floor * 5) | map({ bin: .[0], count: length })'
# Output: [ { "bin": 1, "count": 4 }, { "bin": 5, "count": 5 }, {
"bin": 10, "count": 5 }, { "bin": 15, "count": 1 } ] ```

Other Common Operations

These are some other common operations I frequently find myself doing
every day, but I couldn't think of a better way to categorize them.

Extracting Values Based on Multiple Conditions

We can combine multiple conditions in a select call. The test()
function is used to check if the passed string contains one of the
substrings or not.

```bash echo '[{"name": "Alice", "age": 30}, {"name": "Bob", "age":
25}, {"name": "John", "age": 35}]' | jq 'map(select(.age > 28 and
(.name | test("a", "i"))))' # Output: [{"name": "Alice", "age": 30}]
```

Formatting Unix Timestamps

Various tools emit Unix Timestamps, and we can use the handy strftime
function to format it so it's easier to understand at a glace.

```bash echo '{"timestamp": 1630768200}' | jq '.timestamp | strftime
("%Y-%m-%d %H:%M:%S")' # Output: "2021-09-04 12:30:00" ```

Enumerating by Top Level Key and Value

```bash echo '{"a": 1, "b": 2, "c": 3}' | jq 'to_entries[] | "\(.key)
is \(.value)"' # Output: "a is 1", "b is 2", "c is 3" ```

Closing Thoughts

Whew! That's been a long article  If you're still here, then I
appreciate you staying till the very end.

I hope you've learned something new, and that you'll be able to
quickly identify use cases for jq in your current workflow and apply
your learnings there.

How Does This Article Work?

  * I have used web components to create a custom component
    <jq-view>.
  * There is some Javascript in this page which renders all the
    <jq-view>s when the page is loaded.
  * The WebAssembly build of jq, as well as all the code for calling
    out to it is provided by BioWasm.
  * I have used AlpineJs to make the examples interactive. When the
    button is clicked, it sends an event to a listener, which makes
    it run jq and then update the output.
  * Since I am not good at front-end, this was a substantial learning
    experience for me.

Get In Touch

If you have any suggestions on how this may be improved, errors that
I might have made, or you just want to discuss any other topic,
please feel free to email me. I always love to hear from you.

Extra Resources

JQ Manual

jq featured Licensed under CC BY-NC-SA 4.0

Related content

 

Programming "with the grain"

 

The secret life of .well-known

(c) 2013 - 2023 Ishan Das Sharma Built with Hugo
Theme Stack designed by Jimmy