| JData | |
| Mime: | application/jdata-text, application/jdata-binary |
| Type Code: | TEXT and BINARY |
| Extension: | .jdt, .jdb |
| Developer: | Qianqian Fang |
| Released: | [1] |
| Latest Release Version: | 1.0 Draft 2 |
| Latest Release Date: | [2] |
| Genre: | Data interchange |
| Extended From: | JSON |
| Open: | Yes |
JData is a light-weight data annotation and exchange open-standard designed to represent general-purpose and scientific data structures using human-readable (text-based) JSON and (binary) UBJSON formats. JData specification specifically aims at simplifying exchange of hierarchical and complex data between programming languages, such as MATLAB, Python, JavaScript etc. It defines a comprehensive list of JSON-compatible "name":value constructs to store a wide range of data structures, including scalars, N-dimensional arrays, sparse/complex-valued arrays, maps, tables, hashes, linked lists, trees and graphs, and support optional data grouping and metadata for each data element. The generated data files are compatible with JSON/UBJSON specifications and can be readily processed by most existing parsers. JData-defined annotation keywords also permit storage of strongly-typed binary data streams in JSON, data compression, linking and referencing.
The initial development of the JData annotation scheme started in 2011 as part of the development of the JSONLab Toolbox - a widely used open-source MATLAB/GNU Octave JSON reader/writer. The majority of the annotated N-D array constructs, such as _ArrayType_, _ArraySize_, and _ArrayData_, had been implemented in the early releases of JSONLab. In 2015, the first draft of the JData Specification was developed in the Iso2Mesh Wiki; since 2019, the subsequent development of the specification has been migrated to Github.
The v0.5 version of the JData specification is the first complete draft and public request-for-comment (RFC) of the specification, made available on May 15, 2019. This preview version of the specification supports a majority of the data structures related to scientific data and research, including N-D arrays, sparse and complex-valued arrays, binary data interface, data-record-level compression, hashes, tables, trees, linked lists and graphs. It also describes the general approach for data linking and referencing. The reference implementation of this specification version is released as JSONLab v1.8.
The Draft 1 of the JData specification Version 1 was released on June 4, 2019. The major changes in this release include 1) the serialization order of N-D array elements changes from column-major to row-major, 2) _ArrayData_ construct for complex N-D array changes from a 1-D vector to a two-row matrix, 3) support non-string valued keys in the hash data JSON representation, and 4) add a new _ByteStream_ object to serialize generic binary data or binary large object (BLOB). The reference implementation of this specification version is released as JSONLab v1.9.
The Draft 2 of the JData specification Version 1 was released on July 25, 2019. The major changes in this release include 1) support storage of special matrices via the _ArrayShape_ tag, 2) renamed all _ArrayCompression*_ tags to _ArrayZip*_, 3) add dedicated table data keywords: _TableCols_, _TableRows_, and _TableRecords_. The reference implementation of this specification version is released as JSONLab v2.0.
Numerical values are directly supported by either JSON or UBJSON specifications. A numerical value is typically unchanged when converting to the JData annotation. When storing as files, they are directly stored in the JSON/UBJSON numerical value forms. For example
|}There are a few special constants, namely "NaN", "Infinity" and "-Infinity", they are encoded as special string keywords when stored in the JSON/text-JData formats, but stay unchanged when stored in the binary JData format
|-| || ➡️ || ||}|-| || ➡️ || ||}|-| || ➡️ || || }|-| || ➡️ || || }|-| || ➡️ || || }|-| || ➡️ || ||}|}Hierarchical structures are often needed when representing metadata or simple lists with named-members. Because "structure" data-type can be directly mapped to the "object" construct in JSON and UBJSON, therefore, they do not need to be converted when using the JData annotation.
Simple 1-dimensional vectors are supported in both JSON and UBJSON using the "array" construct. For example
| Native data | text-JData/JSON form | binary-JData(BJData/UBJSON) | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
a=[ 1,2,3 4,5,6 ] | ➡️ | {
"a":[
[1,2,3],
[4,5,6]
]
} |
[{]
[U][1][a]
[[]
[[] [U][1][U][2][U][3] []]
[[] [U][4][U][5][U][6] []]
[]]
[}] | |||||||||||||||||||||||
| similar to the 1-D row vector example above, we can use the type [$] and count [#] markers to simplify this array in the binary form |
[{]
[U][1][a]
[[]
[[] [$][U] [#][U][3] [1][2][3]
[[] [$][U] [#][U][3] [4][5][6]
[]]
[}]|-| colspan="3"| to simplify this further, in the JData Specification, we further extended UBJSON array count marker [#] to accept a 1-D array count-type, representing the dimension vector of an N-D array, in this case [2,3] for a 2x3 matrix |
[{]
[U][1][a]
[[] [$][U] [#][[] [$][U][#][2] [2][3]
[1][2][3][4][5][6]
[}]|}2-D arrays in the annotated formatIn JData specification, we introduced a light-weight data annotation approach to allow one to specify additional information, such as data type, data size and compression, in the stored data record. This is achieved using a "structure-like" data container (a structure is supported in almost all programming language) with JData-specified human-readable subfield keywords. This construct is also easily serialized using many of the existing JSON/UBJSON libraries. For example, the above 2-D array can be alternatively stored using the annotated format to allow fine-grained data storage
| |||||||||||||||||||||||||