http://www.weaselhat.com/2021/06/29/processing-semi-structured-data-in-the-unix-shell/

weaselhat Logo

Processing Semi-Structured Data in the Unix Shell

29 June 20212021-06-29POSIX shell, Software2 Comments

The Unix shell is incredibly powerful. I use it routinely for simple
tasks (moving files around), routine work (grading scripts), and in
my development process (building, deploying, etc.). When I'm working
with text, the shell and its ecosystem is excellent: patching
together cat, find, grep, sed, tr, and cut with shell pipelines and
redirections is a convenient, expressive, and fast way to inspect and
edit files.

But my shell toolchain is much less helpful when working with
semi-structured data, like JSON and YAML. Folks have made wonderful
contributions to the shell ecosystem to help--tools like jq and gron.
These two tools provide new languages for manipulating JSON. It may
be embarrassing to admit for a programming languages researcher, but...
I'm kind of maxed out on new languages.

So I built a new tool that lets you use your usual shell tools to
work with modern file formats: ffs, the file filesystem.

A GIF showing the following shell interaction, editing JSON in
place.~/ffs/demo $ echo '{}' >demo.json~/ffs/demo $ ffs -i demo.json
&[1] 56827~/ffs/demo $ cd demo~/ffs/demo/demo $ echo 47 >
favorite_number~/ffs/demo/demo $ mkdir likes~/ffs/demo/demo $ echo
true >likes/dogs~/ffs/demo/demo $ echo false >likes/cats~/ffs/demo/
demo $ touch mistakes~/ffs/demo/demo $ echo Michael Greenberg >name~/
ffs/demo/demo $ echo https://mgree.github.io >website~/ffs/demo/demo
$ cd ..~/ffs/demo $ umount demo~/ffs/demo $ [1]+ Done ffs -i
demo.json~/ffs/demo $ cat demo.json {"favorite_number":47,"likes":
{"cats":false,"dogs":true},"mistakes":null,"name":"Michael
Greenberg","website":"https://mgree.github.io"}~/ffs/demo $ ~/ffs/
demo $Editing JSON in place using ffs.

ffs lets you mount semi-structured data as a filesystem: objects and
lists correspond to directories, while other types correspond to
regular files. You can mount a file in one format, edit the
filesystem, and write it back in another.

All you need to run ffs is FUSE, a kernel module that supports
userspace filesystem. You'll want libfuse on Linux, or macFUSE on
macOS. Download a binary and play around!

Post navigation

SIGPLAN Blog: Making PL Ideas Accessible

2 Comments

 1. [646b6] Jay McCarthy says:
    2021-07-04 at 9:37am

    You might consider using 9p instead of FUSE, or in addition to.
    This is very cool.

    Reply
 2. [7d2b9] Michael Greenberg says:
    2021-07-04 at 6:28pm

    Thanks! I had never heard of 9p, and it seems cool! I had some
    trouble finding details, but my impression is that FUSE is faster
    than 9p, and FUSE itself is already substantially slower than
    in-kernel filesystems. Bento seems like a cool option (and it's
    Rust based!). Please feel free to chime in on my tracking issue <
    https://github.com/mgree/ffs/issues/9> if you have thoughts on
    performance!

    I'm hopeful that fuser will just get Windows support (https://
    github.com/cberner/fuser/issues/129) and I can use that.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked 
*

        [                                             ]
        [                                             ]
        [                                             ]
        [                                             ]
        [                                             ]
        [                                             ]
        [                                             ]
Comment [                                             ]

Name * [                              ]

Email * [                              ]

Website [                              ]

[Post Comment] 

[                                             ]
[                                             ]
[                                             ]
[                                             ]
[                                             ]
[                                             ]
[                                             ]
[                                             ]
Pages

  * boolopt
  * Bread
  * MA158
  * PHPEnkoder
  * About/Contact

Archives

  * June 2021
  * April 2021
  * January 2021
  * September 2020
  * August 2020
  * May 2020
  * December 2019
  * May 2019
  * April 2019
  * November 2018
  * March 2018
  * October 2017
  * December 2016
  * August 2016
  * April 2016
  * February 2016
  * March 2015
  * February 2015
  * December 2014
  * October 2014
  * July 2014
  * March 2014
  * January 2014
  * December 2013
  * October 2013
  * July 2013
  * November 2012
  * August 2012
  * July 2012
  * February 2012
  * October 2011
  * March 2011
  * February 2011
  * January 2011
  * October 2010
  * May 2010
  * March 2010
  * February 2010
  * January 2010
  * November 2009
  * September 2009
  * August 2009
  * July 2009
  * May 2009
  * April 2009
  * March 2009
  * February 2009
  * January 2009
  * October 2008
  * July 2008
  * June 2008
  * February 2008
  * January 2008
  * December 2007
  * November 2007
  * August 2007
  * April 2007
  * March 2007
  * February 2007
  * January 2007
  * December 2006
  * November 2006
  * October 2006
  * August 2006
  * July 2006
  * June 2006

CS

  * dave
  * lambda the ultimate
  * phil wadler
  * pl club
  * the little calculist

Misc

  * language log

People

  * benjamin
  * family
  * leo
  * me @ github
  * me @ google scholar
  * me @ last.fm
  * me @ pomona cs
  * me @ smugatarian
  * me @ twitter
  * shriram

Projects

  * flapjax

Copyright 2006-2021 (c) Michael Greenberg - Theme by Puro