riki static site generator

2023-06-04 12:02

Introduction

riki is a small subset of ikiwiki rewritten in Rust, for speed. This document describes the requirements and acceptance criteria for the software, and how to verify that riki meets them in an automated way. This is done using the Subplot software.

Software architecture

riki converts files in a source tree into a files that form a static website in an output, or target, tree. The files in the source tree are either "pages" or "blobs". The files in the target tree are HTML files or blobs.

Source pages contain "wiki text", which adds on top of Markdown syntax for wiki links and directives:

plain wiki link: [[pagename]]
- this corresponds to Markdown: [pagename](pagename)
- or HTML: <a href="pagename">pagename</a>
wiki link with link text: ``[[link text|pagename]]`
- this corresponds to Markdown: [link text](pagename)
- or HTML: <a href="pagename">link text</a>
directive: [[!foo arg other="value for other" more="""value for more"""]]
- directive arguments may contain values
- values may be single or triple quoted: triple quoted may span multiple lines

Directives cause some processing to be done. That processing may take as its input only values of arguments, or the page, where the directive is used, or all other files in the site.

Wiki text is converted into plain Markdown by replacing wiki links and directives with Markdown text, before the whole page is parsed into HTML, which gets written to the target directory.

Blobs are copied to the target directory as-is, without any processing.

Processing pipeline

When the site is processed from source to target, the processing pipeline is roughly like this:

read in all files in the source tree
parse each page of wiki text into snippets
- a snippet is plain markdown, a wiki link, or a directive
prepare directives in each page
- this collects or produces data needed for processing the directive, but doesn't produce output
process directives in each page
- this produces markdown output
process wiki links in each page
combine all processed snippets into one markdown string for each page and parse that into HTML
write HTML files to target tree
copy all blobs to target tree

Note that when preparing or processing directives, directives on all pages are prepared first, before any directives are processed. This allows things like defining shortcuts on any page of the site: the shortcut definitions are recognized and obeyed during the preparation stage.

Verification scenarios

The approach used for verifying acceptance criteria is to run riki against known inputs, and check that the output is as expected. Specifically this is done by comparing the Pandoc abstract syntax trees of the input and output. Pandoc is a well-known, well-respected tool that we rely on as an "oracle".

Markdown features

Empty Markdown file

Requirement: Given an empty input Markdown file, the output must be an empty HTML file.

given an installed riki
given file site/empty.mdwn from empty
when I run riki build --plain-body site output
then AST of site/empty.mdwn matches that of output/empty/index.html

Plain text

Requirement: Given a Markdown file with plain text, the output must be an HTML file with the same text, without extra elements.

given an installed riki
given file site/page.mdwn from para
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

Hello, world.

There are two paragraphs.

Quoted block

_Requirement: Given a Markdown file with an quoted block of text, the output must have a blockquote element.

given an installed riki
given file site/page.mdwn from blockquote
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

> This is a quoted block.

Indented code block

_Requirement: Given a Markdown file with an indented code block, the output must have a pre element.

given an installed riki
given file site/page.mdwn from indented-code
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

    This is indented by four spaces.

Fenced code block

_Requirement: Given a Markdown file with a fenced code block, the output must have a pre element.

given an installed riki
given file site/page.mdwn from fenced-code
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

```
This is a fenced code block.
```

Image link

_Requirement: Given a Markdown file linking to an image, the output must have an img element.

given an installed riki
given file site/page.mdwn from image-link
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

![my kitten](cat.jpg "image-title")

Emphasized text

Requirement: Inline markup for emphasis must result in an em element in HTML output.

given an installed riki
given file site/page.mdwn from emph
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

There is *emphasized*, and so is _this_.

Strongly emphasised text

Requirement: Inline markup for strong emphasis must result in a strong element in HTML output.

given an installed riki
given file site/page.mdwn from strong
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

There is **emphasized**, and so is __this__.

Strike through in text

Requirement: Inline markup for strike through must result in a del element in HTML output.

given an installed riki
given file site/page.mdwn from strike
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

There is ~~struck through~~.

Headings

Requirement: Given a Markdown file with headings of various levels, the output must be an HTML file with corresponding h1, h2, etc, elements, without extra elements. Up to six levels of headings must be supported.

given an installed riki
given file site/page.mdwn from headings
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

# Heading one
## Heading two
### Heading three
#### Heading four
##### Heading five
###### Heading six

Inline code

Requirement: Inline code markup with backticks must result in a code element in HTML output.

given an installed riki
given file site/page.mdwn from backticks
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

There is `code` lurking here.

Table

Requirement: Markup of a table result in a table element in HTML output.

Note: This is disabled. Pandoc doesn't seem to handle the HTML table OK.*

given an installed riki
given file site/page.mdwn from table
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

Horizontal rule

Requirement: Markup of a horizontal rule must result in hr element in HTML output.

given an installed riki
given file site/page.mdwn from rule
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

foo

---------------------------------------------------------------------------------------

bar

Unordered list

Requirement: Markup of an unordered list must result in a ul element in HTML output.

given an installed riki
given file site/page.mdwn from ul
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

* first
* second

Ordered list

Requirement: Markup of an ordered list must result in an ol element in HTML output.

Note: This is disabled. Pandoc doesn't seem to parse the HTML list the same as the Markdown.*

given an installed riki
given file site/page.mdwn from ol
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

Task list

Requirement: Markup of a task list must result in a ul element in HTML output.

given an installed riki
given file site/page.mdwn from tasklist
when I run riki build --plain-body site output
then AST of site/page.mdwn matches that of output/page/index.html

* [ ] not done
* [x] done

Definition list

Requirement: Markup indicating use of a definition list should be flagged as an error.

Justification: Neither the CommonMark specification, nor GitHub Flavored Markdown, supports definition lists, even though some Markdown variants do. The Markdown parser Riki uses doesn't support it.

given an installed riki

given file site/page.mdwn from dl-1
when I try to run riki build --plain-body site output
then command fails
then stderr contains "definition list"

given file site/page.mdwn from dl-2
when I try to run riki build --plain-body site output
then command fails
then stderr contains "definition list"

given file site/page.mdwn from dl-3
when I run riki build --plain-body site output
then file output/page/index.html contains ": bar"

foo
: bar

foo

: bar

foo

<!-- no colon at beginning of line here -->: bar

Wiki links to other pages on the site

Requirement: Pages can link to other pages on the site, the same way ikiwiki does, including subpages.

given an installed riki
given file site/dir/foo.mdwn from foo
given file site/absolute.mdwn from empty
given file site/dir/sibling.mdwn from empty
given file site/dir/foo/child.mdwn from empty
given file site/dir/foo/child/grandchild.mdwn from empty
when I run riki build --plain-body site output
then file output/dir/foo/index.html contains "href="../absolute""
then file output/dir/foo/index.html contains "href="sibling""
then file output/dir/foo/index.html contains "href="foo/child""
then file output/dir/foo/index.html contains "href="foo/child/grandchild""

Note the uppercase link to the child page in the test page below.

[[/absolute]]
[[sibling]]
[[child]]
[[child/grandchild]]
[[CHILD]]

Wiki links to pages that don't exist

Requirement: Linking to a page that doesn't exist is an error.

given an installed riki
given file site/dir/foo.mdwn from badlink
when I try to run riki build --plain-body site output
then command fails

[[missing]]

Directives

`img`

The [ikiwiki img directive][] allow including images in the site source tree.

ikiwiki img directive

Simple image inclusion

Requirement: the img directive embeds an image in the generated HTML page.

given an installed riki
given file site/index.mdwn from img
given file site/img.jpg from jpeg
when I run riki build site output
then file output/index.html contains "<img src="img.jpg""

[[!img img.jpg]]

This is a dummy JPEG image.

Image size

Requirement: the img directive can set size.

From the [ikiwiki img directive][] documentation:

size---The size parameter is optional, defaulting to full size. You can specify only the width or the height, and the other value will be calculated based on it: "200x", "x200".

given an installed riki
given file site/index.mdwn from img-size
given file site/a.jpg from jpeg
given file site/b.jpg from jpeg
given file site/c.jpg from jpeg
when I run riki build site output
then file output/index.html contains "<img src="a.jpg" width="100" height="200">"
then file output/index.html contains "<img src="b.jpg" width="100">"
then file output/index.html contains "<img src="c.jpg" height="200">"

[[!img a.jpg size="100x200"]]
[[!img b.jpg size="100x"]]
[[!img c.jpg size="x200"]]

Image attributes

Requirement: the img directive allows useful attributes to be set.

The ikiwiki img directive allows arguments:

alt, title, class, align, id, hspace, and vspace---These are passed through unchanged to the html img tag.

given an installed riki
given file site/index.mdwn from img-attr
given file site/img.jpg from jpeg
when I run riki build site output
when I run cat output/index.html
then file output/index.html contains "<img src="img.jpg""
then file output/index.html contains "alt="halt malt""
then file output/index.html contains "title="tightle""
then file output/index.html contains "class="klass""
then file output/index.html contains "align="malign""
then file output/index.html contains "id="kid""
then file output/index.html contains "hspace="hspc""
then file output/index.html contains "vspace="vspc""

[[!img img.jpg alt="halt malt" title="tightle" class="klass" align="malign"
   id="kid" hspace="hspc" vspace="vspc"]]

Image link generation

Requirement: the img directive can make an image be a link.

The ikiwiki img directive allows arguments:

The link parameter is used to control whether the scaled image links to the full size version. By default it does; set "link=somepage" to link to another page instead, or "link=no" to disable the link, or "link=http://url" to link to a given url.

given an installed riki
given file site/index.mdwn from img-link
given file site/a.jpg from jpeg
given file site/b.jpg from jpeg
when I run riki build site output
when I run cat output/index.html
then file output/index.html contains "<a href="a.jpg"><img src="a.jpg""
then file output/index.html doesn't contain "<a href="b.jpg"><img src="b.jpg""

[[!img a.jpg]]
[[!img b.jpg link=no]]

meta title

Requirement: the meta title directive sets page title.

given an installed riki
given file site/index.mdwn from meta
when I run riki build site output
then file output/index.html contains "<TITLE>Yo</TITLE>"

[[!meta title=Yo]]]

shortcut

Requirement: the shortcut directive created a shortcut that looks like a directive.

given an installed riki
given file site/a.mdwn from use_shortcut
given file site/b.mdwn from define_shortcut
when I run riki build site output
when I run cat output/a/index.html
then file output/a/index.html contains "<A href="https://example.com/foo/123">foo!123</A>"

[[!foo 123]]

[[!shortcut name="foo" url="https://example.com/foo/%s" desc="foo!%s"]]

table

Requirement: the table directive creates a simple table.

given an installed riki
given file site/index.mdwn from table
when I run riki build site output
when I run cat output/index.html
then file output/index.html contains "<TABLE>"
then file output/index.html contains "<TH><TD>Greeting</TD>"
then file output/index.html contains "<TD>Greetee</TD>"
then file output/index.html contains "<TR><TD>hello</TD>"
then file output/index.html contains "<TD>world</TD>"
then file output/index.html contains "<TR><TD>goodbye</TD>"
then file output/index.html contains "<TD>cruel world</TD>"

[[!table data="""
Greeting | Greetee
hello | world
goodbye | cruel world
"""]]

Source file tree

Listing source files

Requirement: source files can be listed.

given an installed riki
given file site/index.mdwn from empty
given file site/img.jpg from empty
when I run riki list site
then stdout contains "img.jpg"
then stdout contains "index.mdwn"

Exclude unusual files

Requirement: files and directories that aren't meant to be part of the site content should be excluded.

given an installed riki
given file site/index.mdwn from empty
given file site/img.jpg from empty
given file site/.git from empty
given file site/index.mdwn~ from empty
given file site/#index.mdwn# from empty
when I run riki list site
then stdout contains "img.jpg"
then stdout contains "index.mdwn"
then stdout doesn't contain ".git"
then stdout doesn't contain "index.mdwn~"
then stdout doesn't contain "#index.mdwn#"

Input files other than Markdown

Requirement: Input files that aren't Markdown files must be copied into the destination directory as-is.

given an installed riki
given file site/image.jpg from image
when I run riki build --plain-body site output
then files site/image.jpg and output/image.jpg match

# Dummy
Pretend this is an image.

Input files in sub-directories

Requirement: If an source page or file is in a sub-directory, it should be put in the corresponding sub-directory in the target directory.

given an installed riki
given file site/foo/page.mdwn from image
given file site/bar/image.jpg from para
when I run riki build --plain-body site output
then AST of site/foo/page.mdwn matches that of output/foo/page/index.html
then files site/bar/image.jpg and output/bar/image.jpg match

Output directory tree

No markdown files in output tree

Requirement: Markdown files are not copied to the output tree.

given an installed riki
given file site/index.mdwn from empty
when I run riki build site output
then file output/index.html exists
then file output/index.mdwn does not exist

Output files have source file modification times

Requirement: Files in the output directory have the same time stamp as the corresponding files in the source directory.

Note that due to limitations in the Subplot lib/files library, our check for modification times is imprecise.

given an installed riki
given file site/index.mdwn from empty
given file site/index.mdwn has modification time 1970-01-01 00:00:00
given file site/index.jpg from empty
given file site/index.jpg has modification time 1970-01-01 00:00:00
when I run riki build site output
then file output/index.html has a very old modification time
then file output/index.jpg has a very old modification time

Output files have source `meta date` modification times

Requirement: Files in the output directory have the time stamp specified in a meta date directive.