Newt, a small system experiment

R. S. Doiel, rsdoiel@caltech.edu

Caltech Library, Digital Library Development

July 14, 2023

The experiment

How do we make building web applications for Libraries, Archives and Museums simpler?

Focus on three abstractions

A JSON source for managing data => Postgres + PostgREST
A template engine => Pandoc
A data router and form validator => Newt

Three implementations a bird sighting website

static site derived from CSV data file
- (CSV file holds our sighting list)
dynamic, SQL and browser JavaScript
- (bird sighting list held in SQL database)
dynamic, SQL and no browser JavaScript
- (bird sighting list held in SQL database)

What are trade-offs?

birds version 1, static site, Pandoc

CSV file, Pandoc, 2 directories, 5 files, 75 total line count, static site

Lines	Files
26	README.md
4	birds.csv <– this is used in each of the demos
6	build.sh
7	page.tmpl
32	htdocs/index.html

birds version 2, dynamic site, browser JavaScript

SQL (Postgres + PostgREST), Browser JavaScript, 2 directories, 8 files, 232 total line count, dynamic site

Lines	Files
29	README.md
4	birds.csv <– from birds1
34	setup.sql
60	models.sql <– implements our data models
15	models_test.sql
3	postgrest.conf
24	htdocs/index.html <– hand coded
63	htdocs/sightings.js <– hand coded

birds version 3, dynamic site, no browser JavaScript

SQL (Postgres + PostgREST), Pandoc, Newt, 1 directory, 9 files, 225 total line count, dynamic site

Lines	Files
43	README.md
4	birds.csv <– from birds1
34	setup.sql <– from birds2
60	models.sql <– from birds2
15	models_test.sql <– from birds2
3	postgrest.conf <– from birds2
23	birds.yaml
36	page.tmpl
7	post_result.tmpl

Insights from experiment

A few “Off the shelf” microservices can make application construction easier
Orchestrating the data pipeline in YAML is reasonable
SQL turns some people off
- models could be bootstrapped from Newt’s YAML
Pandoc templates are simple to learn and well documented at pandoc.org
Newt stack plays well with HTML5 and front-end best practices
I encountered an unexpected result …

An unexpected result

Newt like PostgREST and Pandoc do not require shared synchronous state
Postgres can be deployed in a HA cluster

The Newt stack can scale really big

Newt has weaknesses

Newt is an experimental prototype (June/July 2023, six weeks old)
Newt doesn’t support file uploads

Newt stack has strengths

A very mature foundation

20th Century tech
- SQL (1974), HTTP (1991), HTML (1993), Postgres (1996)
21st Century tech
- JSON (2001), YAML (2001), Pandoc (2006), PostgREST (2014)

Next steps for Newt?

Test with Solr/Elasticsearch as alternate JSON sources
Build staff facing applications this Summer (2023)
Explore generating PostgREST configuration/SQL Models from Newt’s YAML
(hopefully) move beyond my proof of concept in Fall/Winter (2023)

Newt’s someday, maybe …

Have Newt delegate file uploads to an S3 like service
- One approach would be Minio using file streams
Explore integrating SQLite3 support as a JSON data source
Consider implementing Newt in Haskell for richer Pandoc integration
A Newt community to share YAML, SQL and Pandoc templates

Thank you!

This Presentation https://caltechlibrary.github.io/newt/presentation/
Project: https://github.com/caltechlibrary/newt
Email: rsdoiel@caltech.edu