The Dataset Project provides tools for working with collections of JSON documents stored on the local file system in a pairtree or in a SQL database supporting JSON columns. Two tools are provided by the project – a command line interface (dataset) and a RESTful web service (datasetd).
dataset is a command line tool for working with collections of JSON documents. Collections can be stored on the file system in a pairtree directory structure or stored in a SQL database that supports JSON columns (currently SQLite3 or MySQL 8 are supported). Collections using the file system store the JSON documents in a pairtree. The JSON documents are plain UTF-8 source. This means the objects can be accessed with common Unix text processing tools as well as most programming languages.
The dataset command line tool supports common data management operations such as initialization of collections; document creation, reading, updating and deleting; listing keys of JSON objects in the collection; and associating non-JSON documents (attachments) with specific JSON documents in the collection.
See Getting started with dataset for a tour and tutorial.
datasetd is a RESTful web service implementation of the dataset command line program. It features a sub-set of capability found in the command line tool. This allows dataset collections to be integrated safely into web applications or used concurrently by multiple processes. It achieves this by storing the dataset collection in a SQL database using JSON columns.
dataset and datasetd are intended to be simple tools for managing collections JSON object documents in a predictable structured way.
dataset is guided by the idea that you should be able to
work with JSON documents as easily as you can any plain text document on
the Unix command line. dataset is intended to be simple to use
with minimal setup (e.g. dataset init mycollection.ds
creates a new collection called ‘mycollection.ds’).
.ds extension for easy identificationdatatset collection storage options - pairtree is the default disk organization of a dataset collection - the pairtree path is always lowercase - non-JSON attachments can be associated with a JSON document and found in a directories organized by semver (semantic version number) - versioned JSON documents are created along side the current JSON document but are named using both their key and semver - SQL store stores JSON documents in a JSON column - SQLite3 and MySQL 8 are the current SQL databases support - A “DSN URI” is used to identify and gain access to the SQL database - The DSN URI maybe passed through the environment
datasetd is a web service - is intended as a back end web service run on localhost - by default it runs on localhost port 8485 - supports collections that use the SQL storage engine - should never be used as a public facing web service - there are no user level access mechanisms - anyone with access to the web service end point has access to the dataset collection content
The choice of plain UTF-8 is intended to help future proof reading dataset collections. Care has been taken to keep dataset simple enough and light weight enough that it will run on a machine as small as a Raspberry Pi Zero while being equally comfortable on a more resource rich server or desktop environment. dataset can be re-implement in any programming language supporting file input and output, common string operations and along with JSON encoding and decoding functions. The current implementation is in the Go language.
dataset supports - Initialize a new dataset collection - Define metadata about the collection using a codemeta.json file - Define a keys file holding a list of allocated keys in the collection - Creates a pairtree for object storage - Codemeta file support for describing the collection contents - Simple JSON object versioning - Listing Keys in a collection - Object level actions - create - read - update - delete - keys - has-key - sample - clone - clone-sample - Documents as attachments - attachments (list) - attach (create/update) - retrieve (read) - prune (delete) - The ability to create data frames from while collections or based on keys lists - frames are defined using a list of keys and a lost dot paths describing what is to be pulled out of a stored JSON objects and into the frame - frame level actions - frames, list the frame names in the collection - frame, define a frame, does not overwrite an existing frame with the same name - frame-def, show the frame definition (in case we need it for some reason) - frame-keys, return a list of keys in the frame - frame-objects, return a list of objects in the frame - refresh, using the current frame definition reload all the objects in the frame given a key list - reframe, replace the frame definition then reload the objects in the frame using the existing key list - has-frame, check to see if a frame exists - delete-frame remove the frame
datasetd supports
Both dataset and datasetd maybe useful for general data science applications needing JSON object management or in implementing repository systems in research libraries and archives.
dataset has many limitations, some are listed below
datasetd is a simple web service intended to run on “localhost:8485”.
Compiled versions are provided for Linux (x86), Mac OS X (x86 and M1), Windows 11 (x86) and Raspberry Pi OS (ARM7).
github.com/caltechlibrary/dataset/releases
You can use dataset from Python via the py_dataset package.