Caltech Library logo

datatools

datatools provides a variety of command line programs for working with data in different formats as well as to ease Posix shell scripting (e.g. writing scripts that run under Bash). The tools are group as data, strings and scripting.

For data

Command line utilities for simplifying work with CSV, JSON, Excel Workbooks and plain text files or content.

Compiled versions are provided for Linux (amd64), Mac OS X (amd64), Windows 10 (amd64) and Raspbian (ARM7). See https://github.com/caltechlibrary/datatools/releases.

Use “-help” option for a full list of options for each utility (e.g. csv2json -help).

For strings

datatools provides the string command for working with text strings (limited to memory available). This is commonly needed when cleanup data for analysis. The string command was created for when the old Unix stand bys- grep, awk, sed, tr are unwieldly or inconvient. string provides operations are common in most language like, trimming, spliting, and transforming letter case. The string command also makes it easy to join JSON string arrays into single a string using a delimiter or split a string into a JSON array based on a delimiter. The form of the command is string [OPTIONS] [ACTION] [ARCTION_PARAMETERS...]

    string toupper "one two three"

Would yield “ONE TWO THREE”.

Some of the features included

See string for full details

For scripting

Various utilities for simplifying work on the command line.

Compiled versions are provided for Linux (amd64), Mac OS X (amd64), Windows 10 (amd64) and Raspbian (ARM7). See https://github.com/caltechlibrary/datatools/releases.

Use the utilities try “-help” option for a full list of options.

Installation

See INSTALL.md for details for installing pre-compiled versions of the programs.

datatools are go get-able. If you have go v1.8 (or newer) you can install with the command below.

    go get github.com/caltechlibrary/datatools/...