jq (programming language)

jq

The official jq logo
Paradigms	Purely functional programming, JSON-oriented processing, tacit programming
Designed by	Stephen Dolan
First appeared	August 21, 2012 (2012-08-21)
Stable release	1.7.1  / December 13, 2023 (December 13, 2023)
Implementation language	jq: C; gojq: Go
Platform	Cross-platform
OS	Cross-platform
License	MIT
Website	stedolan.github.io/jq

jq is a very high-level lexically-scoped functional programming language in which every JSON value is a constant. jq supports backtracking and managing indefinitely long streams of JSON data. It is related to the Icon and Haskell programming languages. The language supports a namespace-based module system, and has some support for closures. In particular, functions and functional expressions can be used as parameters of other functions. The original implementation of jq was in C; gojq is a "Pure Go" implementation.

History

jq was created by Stephen Dolan, and released in October 2012. It was famously described as being "like sed for JSON data". Support for regular expressions was added in jq version 1.5.

A "wrapper" program for jq named yq adds support for YAML, XML and TOML. It was first released in 2017.

The Go implementation, gojq, was initially released in 2019. gojq notably extends jq to include support for YAML.

Usage

Command-line usage

jq is typically used at the command line, and can be used with other command-line utilities, such as curl. Here is an example showing how the output of a curl command can be piped to a jq filter to determine the category names associated with this Wikipedia page:

$ curl 'https://en.wikipedia.org/w/api.php?action=parse&page=jq_(programming_language)&format=json' | jq '.parse.categories[]["*"]'

The output produced by this pipeline consists of a stream of JSON strings, the first few of which are:

"Articles_with_short_description"
"Short_description_matches_Wikidata"
"Dynamically_typed_programming_languages"
"Functional_languages"
"Programming_languages"

The `curl` command above uses the MediaWiki API for this page to produce a JSON response. The pipe `|` allows the output of curl to be accessed by jq, a standard Unix shell mechanism.

The jq filter shown is an abbreviation for the jq pipeline:

.["parse"] | .["categories"] | .[] | .["*"]

This corresponds to the nested JSON structure produced by the call to curl. Notice that the jq pipeline is constructed in the same manner using the '|' character as the Unix-style pipeline.

Embedded usage

Both the C and the Go implementations provide libraries so that jq functionality can be embedded in other applications and programming environments.

For example, gojq has been integrated with SQLite so that a `jq` function is available in SQL statements. This function is marked as "deterministic" and can therefore be used in "CREATE INDEX" commands.

Syntax and semantics

Types

Every JSON value is itself a value in jq, which accordingly has the types shown in the table below. The gojq implementation distinguishes between integers and non-integer numbers, and supports unbounded-precision integer arithmetic.

Summary of jq's supported types

Type	Examples
"number"	3 3.2 1e6 nan infinite
"string"	"Hello" "😐"
"boolean"	true false
"array"	[1, "2", {"mixed": "type"}, [3,4]]
"object"	{"one": 1, "two": "2", "three": [3]}
"null"	null

`null` is a value, just like any other JSON scalar; it is not a pointer or a "null-pointer". `nan` (corresponding to NaN) and `infinite` (see IEEE 754) are the only two jq scalars that are not also JSON values.

Forms

There are special syntactic forms for function creation, conditionals, stream reduction, and the module system.

Filters

Here is an example which shows how to define a named, parameterized filter for formatting an integer in any base from 2 to 36 inclusive. The implementation illustrates tacit (or point-free) programming:

# Use gojq for infinite precision integer arithmetic
def tobase($b):
    def digit: "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"[.:.+1];
    def mod: . % $b;
    def div: ((. - mod) / $b);
    def digits: recurse( select(. >= $b) | div) | mod ;

    select(2 <= $b and $b <= 36)
    [digits | digit] | reverse | add;

The next example demonstrates the use of generators in the classic "SEND MORE MONEY" verbal arithmetic game:

def send_more_money:
    def choose(m;n;used): ([range(m;n+1)] - used)[];
    def num(a;b;c;d): 1000*a + 100*b + 10*c + d;
    def num(a;b;c;d;e): 10*num(a;b;c;d) + e;
    first(
      1 as $m
      | 0 as $o
      | choose(8;9;[]) as $s
      | choose(2;9;[$s]) as $e
      | choose(2;9;[$s,$e]) as $n
      | choose(2;9;[$s,$e,$n]) as $d
      | choose(2;9;[$s,$e,$n,$d]) as $r
      | choose(2;9;[$s,$e,$n,$d,$r]) as $y
      | select(num($s;$e;$n;$d) + num($m;$o;$r;$e) == num($m;$o;$n;$e;$y))
      | [$s,$e,$n,$d,$m,$o,$r,$e,$m,$o,$n,$e,$y] );

Parsing Expression Grammars

There is a very close relationship between jq and the Parsing Expression Grammar (PEG) formalism. The relationship stems from the equivalence of the seven basic PEG operations and the jq constructs shown in the following table.

Correspondence between PEG operations and jq equivalents

PEG operation name	PEG notation	jq operation or def
Sequence	e1 e2	e1 | e2
Ordered choice	e1 / e2	e1 // e2
Zero-or-more	e*	def star(E): ((E | star(E)) // .) ;
One-or-more	e+	def plus(E): E | (plus(E) // . );
Optional	e?	def optional(E): (E // .);
And-predicate	&e	def amp(E): . as $in | E | $in;
Not-predicate	!e	def neg(E): select( [E] == [] );

References

Bibliography

• Janssens, Jeroen (2021). Data Science at the Command Line. O'Reilly Media. ISBN 9781492087885.
• Janssens, Jeroen (2014). Data Science at the Command Line: Facing the Future with Time-Tested Tools. O'Reilly Media. ISBN 9781491947807.
• Marrs, Tom (2017). JSON at Work: Practical Data Integration for the Web. O'Reilly Media. ISBN 9781491982419.

External links

• jq homepage
• gojq - the Pure Go implementation
• jq FAQ
• Awesome jq - curated listing of jq-related resources
• The jq Programming Language page on the Rosetta Code comparative programming tasks project site