Citeproc Test Runner for styles

CSL Development
1

I finally succumbed to modernity and set up continuous integration in the citeproc-js repo, rewriting the old Python test runner in node. In the process, I spun the test runner off as a separate project, and included the support files (locales, juris modules, CSL and CSL-M schemata) as dependencies, so that the runner can be used as a standalone tool outside of the citeproc-js source repository. If you have node, mocha, and java installed on your system, you can install with this:

    npm install -g citeproc-test-runner

After installing, you can invoke the runner with cslrun. It will spit out instructions on how to set the path to a directory where style tests will be saved and read. After configuration, cslrun -h will display a list of options to help you get started.

The tool can be used both for building and for running tests. If you export a set of items in CSL JSON, you can build tests from them with:

    cslrun -S <style-nickname> -C <csl-json-file>

That will create a subdirectory with the style nickname under your styles root, and populate it with boilerplate test fixtures built from the data in the export file. You can then run:

    cslrun -S <style-nickname> -w <csl-style-file> -a -k

That will run each of the tests in turn, failing miserably but with an option to adopt the processor’s return as the expected result for the test. If you hit Y for each fixture, you have a set of passing tests, which you can tweak if there are anomalies in the style that generated the output. Be sure to rename the fixtures from their boilerplate names, since another run with the -C option will overwrite them otherwise.

To use the tests as a validation and output monitor when editing a style, remove the -k option:

    cslrun -S <style-nickname> -w <csl-style-file> -a

The runner will watch the style file, and rerun validation and tests when it changes on disk, for pretty-good dynamic validation with any editor that saves files. Long-suffering users of nano and edlin rejoice!

As a nice extra in the CSL-M zone, the -w option can be repeated, to watch one or more jurisdiction submodules in addition to the parent style, for unified integration testing of the two working together.

The script seems to work fine, but parts of it needed to be coded with async functions, which I don’t understand at all well. If you spot something awful in there and would like to file a PR, the script is in the Citeproc Test Runner repo on GitHub.

Enjoy!

FB

1 Like
2

I have made some changes to Citeproc Test Runner to improve its utility for style tests. Getting-started details on usage are in the README under the link. Here are the main changes:

  1. The -C option (to read test data from a CSL JSON export file) has been removed.
  2. Item data for machine-composed tests is now drawn directly from a Zotero public group using a new -U option that calls the Zotero API. By default the tool draws items from the Jurism Style Test Items group, looking for a collection with the same name as the “slug name” of the style (i.e. chicago-author-date).
  3. The key of each item added to the test set with the -U option is recorded in the test, and the -U option is idempotent—items will be added only once.
  4. Each test result covers the forms that the style appears to support, so only one test should be needed per item.
  5. I have started a GitHub project jm-style-tests to hold finished tests in subdirectories under the slug name of each style.

This takes a bit more work to set up than the initial version, but once configured, the workflow should be less burdensome. Feel free to fork, adapt, or file pull requests. This was built for some upcoming legal project work in Jurism, but it should be equally useful to CSL style maintenance.