A recipe for applications: Dataset & Web
Components
R. S. Doiel, rsdoiel@caltech.edu
Caltech Library, Digital Library Development
TBD
Welcome to “A recipe for applications”
Welcome everyone. This is a talk and hands on workshop.
An approach to building web applications using Dataset and Web
Components
Getting started
You
probably already have these, if not install them
- A computer running macOS, Lunix, Raspberry Pi OS, Windows using
LSW
- Terminal application
- Text Editor
- Web Browser
(I’m assuming Firefox for this tutorial)
Install dataset
We can start our first iteration of our application once you have
these available.
Part 1: What are we building?
GOAL: A simple web application that lets us curate a list of
recipes.
We’re going start from the backend and spend most of our time on the
front end.
Part 1.1: What are the parts of our application?
- A web service for managing the recipe collection
- A page to browse recipes by name
- A page to display a recipe
- A page and web form to add or edit a recipe
Part 1.1: What is a recipe?
- A “key”, the unique identifier of a recipe (url friendly)
- A name (human friendly and readable)
- A list of ingredients and measures (CSV data)
- A procedure describing the preparation process (text)
Part 1.1: Basic Strategy
- Setting up our web service
- Sketch your app using HTML
- Wire up and test
Part 1.2: Setting up our web service
- create our
recipes.ds
collection
- load sample data into our
recipes.ds
collection
- Configure and run our collection as a web service
Part 1.2: creating our collection
We use the dataset
command line program to initialize a
dataset collection.
dataset init recipes.ds
Part 1.2: Verify we loaded our data OK
dataset keys recipes.ds
dataset read recipes.ds frybread
Part 1.2: Example terminal interaction
$ dataset keys recipes.ds
blondies
brownies
cafe-con-leche
frybread
poi
waffles
Part 1.2: Example terminal interaction
$ dataset read recipes.ds frybread
{
"ingredients": "backing powder,2 tsp\r\nflour,2 cups\r\nhot water,2/3 cups\r\nsalt,1 tsp salt\r\nshortening,2 Tbsp\r\n",
"key": "frybread",
"name": "Fry Bread",
"procedure": "1. Combine flour, baking powder, and salt in a bowl.\r\n2. Use a pastry blender (or two butter knives) to cut the shortening into the flour.\r\n3. Add the hot water, and mix until the water is incorporated and you get a dough.\r\n4. Turn out the dough on a lightly floured board. Knead the dough until it is soft and smooth.\r\n5. Wrap the dough in plastic, and let the dough rest for 30 minutes.\r\n6. Divide the dough into 6 pieces, roll each into a ball, and roll each into a flat disk with a rolling pin.\r\n7. Brush one side of each disk with melted margarine and place on a barbecue over a 3 Mississippi fire.\r\n8. Brush the opposing side of the bread with margarine and flip the bread on the barbecue.\r\n9. Cook until both sides are golden brown. Serve hot.\r\n\r\n- You can substitute either 1/4 cup plain yogurt, or a 1/4 cup soured milk as a leavening agent\r\ninstead of baking powder. If you do, add 2 hours to the rest time for the dough, and leave\r\nthe dough somewhere warm. You can optionally include the baking powder as well to get a\r\nvery puffy version of frybread. If using yogurt or soured milk for leavening, use 1\2 cup\r\ninstead of 1/3 cup water.\r\n- You can use mayonnaise in place of shortening for a crispy, crunchier texture.\r\n- You can also fry the dough in hot oil over a stovetop. The dough cooks rapidly and will brown\r\nin about 12 seconds and must be turned over to allow the opposite side to brown, then be removed\r\nfrom the oil and placed sideways into a colander or large bowl lined with paper towels to allow\r\nthe oil to drain off the finished product\r\n\r\nurl: https://en.wikibooks.org/wiki/Cookbook:Fry_Bread_I\r\n"
}
Part 1.2: create the static content directories
mkdir htdocs
mkdir htdocs/modules
mkdir htdocs/css
Part 1.2: recipes_api.yaml
#!/usr/bin/env -S datasetd -debug
host: localhost:8001
htdocs: htdocs
collections:
- dataset: recipes.ds
keys: true
create: true
read: true
update: true
delete: true
query:
list_recipes: |
select src
from recipes
order by src->>'name'
Part 1.2: Starting and stopping the web service
Starting the web service.
datasetd -debug recipes_api.yaml
- Go do http://localhost:8001/api/version
- Look at the terminal window, do you see the log message for the
request?
- You can shutdown the service by press control and C (Ctrl-C) in the
terminal session
Part 1.2: Starting and stopping the web service
- Tired to typing
datasetd -debug recipes_api.yaml
?
- Make the YAML file executable!
chmod 775 recipes_api.yaml
Now you can shorten start up to
./recipes_api.yaml
Part 1.3: What about our static content?
The web service is running but haven’t populated the htdocs
directory.
What do you see when you go to http://localhost:8001/?
Part 1.3: What are our web pages?
- htdocs/index.html
-
Display a list of our recipes
- htdocs/display_recipe.html
-
A page that shows the recipe
- htdocs/edit_recipe.html
-
A page used to add and edit recipes we’ve collected
Part 1.3: Populating our pages using JavaScript
We’ll create four modules, one specific to each HTML page and one
utility module
- htdocs/modules/list_recipes.js
-
Display a list of our recipes
- htdocs/modules/display_recipe.js
-
A page that shows the recipe
- htdocs/modules/edit_recipe.js
-
A page used to add and edit recipes we’ve collected
- htdocs/modules/utils.js
-
This module handles retrieving data from the JSON API and finding the
object’s key
Part 1.3: Fire up our web service
In a terminal run our startup command
datasetd -debug recipes_api.yaml
Part 1.3: Test using your web browser
- Go to http://localhost:8001
- In your browser turn on your developer tools
- Reload the page and explore using your developer tools
- Click through the site
Part 1.3: Debugging and improving
- Are there issues to debug?
- What happens when you add a recipe?
- What happens when you hen you update a receipt?
- Can any of this be improved?
Intermission
Let’s take a short break then we’ll comeback and iterate.
I’m available for questions.
Part 2: Recipes version 2
What we are doing next
- Creating a new dataset collection called,
recipes2.ds
- Creating a new,
recipes_api2.yaml
- Creating a new directory structure for our static content called,
htdocs2
Part 2.1: Bootstrap from version 1
dataset init recipes2.ds
cp recipes_api.yaml recipes_api2.yaml
cp -vR htdocs htdocs2
(NOTE: The first line should look familiar, the others are just time
savers)
Part 2.1: Updating our YAML configuration
- edit our recipes_api2.yaml
- update the
htdocs
reference
- update the port number in hosts to 8002
- update the dataset to
recipes2.ds
- What additional files need to change?
- did we hard code the collection name in HTML?
- did we hard code the collection name in JavaScript?
Part 2.1: Testing a new instance
- Test our new instance
- What is broken?
- Did we catch all the places with hard coded collection names?
- What about behaviors?
- Shutdown down and restart datasetd to debug YAML changes
dataset recipes_api2.yaml
Part 2.2: Desirable changes
- Handle form submission sends us to a useless URL, how do we fix
that?
- Typing in comma seperated values is cumbersum, can me improve
that?
Part 2.2: Update the HTML for edit_recipe.html
Add the following at the bottom of the page before the
</body>
.
<script type="module">
import { saveRecipe } from './modules/utils.js';
document.addEventListener('DOMContentLoaded', () => {
const form = document.addEventListener('submit', saveRecipe);
});
</script>
Part 2.2: Restart datasetd and test
datasetd -debug recipes_api2.yaml
Test using your web browser.
Part 2.3: Improving the UI with Web Components
- Extend the HTML elements available
- Implement components as JavaScript Modules
Part 2.3: Copy the web components to the modules directory
- Unzip just the JavaScript files
- Move the JavaScript files in the zip file to
htdocs2/modules/
.
unzip $HOME/Downloads/cl-web-components-0.0.6.zip *.js
mv -v *.js htdocs2/models/
Part 2.2: Adding CSVTextarea to edit_recipe.html
- edit
htdocs2/edit_recipe.html
- Include the CSVTextarea JavaScript module in the document head
- Wrapping the “ingredients” textarea with
<csv-textarea>
See: https://github.com/caltechlibrary/t2t3_dataset_web_apps/blob/main/htdocs2/edit_recipe.html
Part 2.2: What are the attributes needed in a
<csv-textarea>
?
- copy the attributes form the “ingredients” textarea to the
<csv-textarea>
- Add an these attributes to
<csv-textarea>
column-headings="Ingredients,Units"
debug="true"
Part 2.2: Restart recipes_api2.yaml and test
Start up our web service
dataset recipes_api2.yaml
- Point your browser at http://localhost:8002/edit_recipe.html
- Turn on your developer tools
- Test the web component, what’s the problem you see?
Part 2.2: Getting the table populated, update
utils.js
CSVTextarea has the ability to be updated from CSV text. Let’s do
that.
In edit_recipe.js
you need to find this.
if (data["ingredients"] !== undefined) {
ingredientsTextarea.innerHTML = data["ingredients"];
}
And replace it with something like this.
if (data["ingredients"] !== undefined) {
ingredientsTextarea.fromCSV(data['ingredients']);
}
Part 2.2: Test and debug
- Do you find other problems?
Part 3: Exploring further
- The server side can be turn key using a JavaScript web page
- When is it a good idea?
- When is be an bad idea?
- Moving from a single layer stack to a two or three layer stack
- Dataset behind a front end web server
- Dataset behind middle ware
- Is this approach sustainable?
Part 3: Exploring Human Interfaces
- Why bother with Web Components?
- What are the assumptions in this approach?
Part 3: Exploring Human Interfaces
- The traditional division of responsibilities in the browser is
- HTML for structured data markup
- CSS for visual design and layout
- JavaScript to orchestrate behaviors
- Does Web components contradict that the division of
responsibilities?
- Is progressive enhancement is still relevant in 2025?
- Is it OK to require JavaScript in a web page?
Part 3: My Recomendations
- Build with the grain of the web
- Building blocks are HTML, CSS, JavaScript and HTTP protocol
- Take advantage of localhost
- Production, build in layers
- access control with front end web service (Apache+Shibboleth,
NginX+Shibboleth)
- data validation with middle ware (localhost: Go, TypeScript or
Python)
- object storage with Dataset (localhost)
Part 3: What I am still mulling over?
- Dataset can shrink the stack but does not remove the need for
middleware (yet)
- Web Components offer the possibility of consistent interfaces across
sites
- They can help with accessibility
- I think Web Components ultimately simplify things
- Trade off: individual compontents can be complex
- REST services force us to middleware or Browser JavaScript
- Is it reasonable to require JavaScript (or WASM)?
- Is there a simpler abstraction?
Reference: CL-web-components
- CSVTextarea
-
Wraps a textarea element and presents a editable table of cells
- AToZUL
-
Wraps a UL list and creates an A to Z list
- SortableTable
-
Wraps an HTML table making it sort-able and filterable on a column
Reference: Programming Languages