Caltech Library logo

eprinttools

eprinttools is a collection of command line tools written in Go, a Go package and set of command line utilities for working with EPrints 3.x EPrint XML and REST API written in Python 3.8. Eventually it is planned to have this project become a pure Python project.

This project also hosts demonstration code to replicate a public facing version of an EPrints repository outside of EPrints. Think of it as the public views and landing pages.

Go base code

The command line programs

The first two utilities can be configured from the environment or command line options. The environment settings are overridden by command line options. For details running either command envoke the tool name with the ‘-help’ option.

Python base code

Python Modules

eprints3x

This python module wraps the eputil Go comand in Python. It makes it trivial to implement harvesting an EPrints repository using the EPrints REST API.

eprintviews

This python module uses py_dataset and the harvested content to generate a htdocs directory similar to the URL layout of EPrints. It features classes for working with Views, Users (needed to attribute names to userid fields EPrint XML harvested from the REST API), Subjects (a way to load the subjects text file from an EPrints archive and generate the path to label mapping used when rendering views into an htdocs directory) and Aggregator (this does the heavy lifting of processing a dataset collection of harvested EPrint XML and generating the views as JSON documents in the htdocs directory).

command line tools

harvester_full.py, harvester_recent.py

These two Python programs use eprints3x module to implement harvesters of EPrint XML and any related digitl objects (e.g. PDFs, images) into a dataset collection

genviews.py

This Python program processes a dataset collection and renders an htdocs tree populating it with JSON documents and key lists. This skeleton of metadata and directory structure can then be processed into a rendered website mirroring the content from an EPrints repository. This module relies on eprintviews.

indexer.py

This Python program indexes the contents of our replicated EPrints site by creating scheme.json files along side the index.json files that represent the landing pages for the replicated repository. These can then be easily ingested into Lunr.js or Elasticsearch. Currently the proof of concept targets Lunr.js. This module relies on eprintviews.

mk_website.py

This Python program creates the HTML pages from Markdown documents in the static folder (e.g. home page and major landing pages) as well as the individual views and abstracts from the JSON documents created by genviews.py. The final result is a static website ready to serve out to the public. This module relies on eprintviews.

publisher.py

This Python program copys (syncs) the content with an AWS S3 bucket via the AWS command line tools.

py_dataset

This Python module provides access to dataset collections which we use as intermediate storage for JSON documents and related attachments.

AMES

The eprintools command line programs have been made available to Python via the AMES project. This include support for both read and write to EPrints repository systems.