test suite redux; need comments

CSL Development
1

So moving these comments from Frank over here …

People from Zotero, Mendeley, etc. listen up! We need feedback on the
excellent test suite work that Frank has been doing; otherwise, you’re
stuck with our decisions.

On how to express quotes in the result. I use unicode characters, and Frank …

Re proper quotes, the tests are HTML, quotes are expressed as entities.

I have a general aversion to entities (what’s the point; we have
unicode?), but for the moment won’t object. I may come back to this
though.

The tests should be kept in single files. It would be annoying to have to scratch around in parallel filesystem hierarchies when writing, editing or interpreting tests (I’ve written about eighty of them so far, but we’ll need five or six times that number eventually, so ease of editing is important).

OK, here’s the big question. I agree that whatever solution needs to
be easy to edit and interpret (by both humans and machines). But it
also needs to be easy to distribute and use.

To review; current test suite works like so if you’d like to add a test:

  1. write the test case in Frank’s syntax, add to the std/human dir.

  2. run grind.py to get the humans file converted to json (in the
    std/machines dir).

So there are two directories full of test cases: one for human
editing, and the other for machine reading.

The only difference between them is that one is valid JSON, and the
other is not.

My point is this: it’s awkward to have two directories of tests, when
the only reason this is necessary is because of forcing the CSL
fragments into JSON.

So rather than have a single …

/sort_test.json

… file, I’m wondering about:

/sort_test.json
/sort_test.csl

In other words, only the CSL fragment would get moved into a separate
file. Everything else would remain as json.

Advantage?

  1. no need for separate humans and machines dirs.

  2. csl fragments can be validated

I don’t see this as any particular burden.

I also don’t have a super strong opinion on this; am just looking
towards the horizon and wondering which approach suits our needs best.
It might be that we defer this decision until later; when the test
suite is “done.”

Bruce

3

…

I don’t see this as any particular burden.

Tests have to be written by somebody, and it’s a pain to scratch
around in the filesystems to prepare the matched-up files. You would
have to do that for authoring, and you would need to track down both
files to interpret the result of the test when things go haywire.

Don’t you already have to do this? One can’t (humanly) read the
machine file CSL, so you have to now switch to the human version of it
in order to understand a test (or, of course, only read the human
versions).

That doubles the amount of keyboard work for two high-frequency
operations (writing and interpreting test content).

I’m not understanding this. But, moving on …

By moving the editable files to pure json and pure xml, you would get
colorful syntax highlighting,

Just talking about the XML, the point is more about real-time
validation and completion. In the current approach, you already should
be writing the CSL fragment as a separate file (quicker and more
reliable), and then copy-and-pasting it into the test file.

but you’d need too work out separate
commenting schemes for each, with separate extraction methods, if you
want to build docs off of the test suite.

You’re referring to the text that may precede each section? OK, that’s
a good point.

Applications would also
either need to implement a consolidation mechanism (i.e. replicate
what grind.py does), or implement two separate APIs for reading data
(remember that JS has no native I/O, so this has to be built
separately for every supported JS interpreter that goes on the
test-bed).

You mean that JS requires you to write an unreasonable amount of extra
code just to read the XML from disk?

Sounds like a lot of extra work.

But as I wrote over on the other list, the machines dir could be
dropped from the distribution, if the new grinder is known to work
reliably on the common platforms (Linux, Mac, Windows), and builds
tests that pass. Then after downloading the archive, you would need
to run the grinder to build the tests (or run python ./setup.py
install or whatever). The build could also spin off a copy of the CSL
files for validation. (The old grind.sh script used to do that, I
just dropped the dir to save traffic. If only the source is archived,
there’s no reason not to add it back.)

Bruce

4

…

I don’t see this as any particular burden.

Tests have to be written by somebody, and it’s a pain to scratch
around in the filesystems to prepare the matched-up files. You would
have to do that for authoring, and you would need to track down both
files to interpret the result of the test when things go haywire.

Don’t you already have to do this? One can’t (humanly) read the
machine file CSL, so you have to now switch to the human version of it
in order to understand a test (or, of course, only read the human
versions).

I go straight to the human version.

That doubles the amount of keyboard work for two high-frequency
operations (writing and interpreting test content).

I’m not understanding this. But, moving on …

Maybe a concrete example will help then. Suppose you get the
following error trace:

expected: (2005)
but got: (Robert Jones 2005)
in test: discretionary_SuppressAuthorSolo

To start working out what the system attempted to do, you need to
identify the input data, the citation request, and the CSL that was
applied. If that information is spread across two files, you have to
open both of them. Hence double the keyboard work.

With a major refactoring of the engine, you can have a hundred failing
tests or more. Saving keystrokes matters.

By moving the editable files to pure json and pure xml, you would get
colorful syntax highlighting,

Just talking about the XML, the point is more about real-time
validation and completion. In the current approach, you already should
be writing the CSL fragment as a separate file (quicker and more
reliable), and then copy-and-pasting it into the test file.

If you’re offering to provide a bespoke emacs mode or an IDE that ties
the various files together and will be pleasant to work with, I’ll
look forward to using it.

but you’d need too work out separate
commenting schemes for each, with separate extraction methods, if you
want to build docs off of the test suite.

You’re referring to the text that may precede each section? OK, that’s
a good point.

Applications would also
either need to implement a consolidation mechanism (i.e. replicate
what grind.py does), or implement two separate APIs for reading data
(remember that JS has no native I/O, so this has to be built
separately for every supported JS interpreter that goes on the
test-bed).

You mean that JS requires you to write an unreasonable amount of extra
code just to read the XML from disk?

Actually this isn’t such a big deal. In fact, this entire discussion
isn’t a big issue; it’s all just test code, and I can reshuffle it
locally to make it play nice with the existing code base. Go ahead
with reorganization if you like, I’ll happily work with whatever you
produce.

5

…

Actually this isn’t such a big deal. In fact, this entire discussion
isn’t a big issue; it’s all just test code, and I can reshuffle it
locally to make it play nice with the existing code base. Go ahead
with reorganization if you like, I’ll happily work with whatever you
produce.

It’s not that big a deal for me either, and it’s not hard to change
things later depending on how are needs evolve, or based on the
comments of implementers.

So given that, unless there are other comments, we can hold off and
reassess as we get closer to 1.0 and the associated stuff.

Bruce

6

Hi Frank, Bruce,

Thank-you for your work on the test suite. The current approach is
fine for us I think. Regarding having separate JSON/CSL files - I
think the current format is probably preferable with all the data that
the test harness needs to read being in one file and all the data for
human preparation in another - although it isn’t a big issue.

Regards,
Robert.2009/6/17 Bruce D’Arcus <@Bruce_D_Arcus1>: