Tests: curating the test suite

CSL Development
1

Brecht writes:> * A large number of tests test functionality that is not in the CSL spec,

but is provided by citeproc-js (raw dates, static ordering, literal names,
…). I think these should be indicated as such, or perhaps moved to a
separate directory. This would make it easier to check the other CSL
processor’s compatibility.

Rintze responds:

Sylvester Keil proposed using a Cucumber format for unit tests, which
would allow tests to be tagged:

If somebody else helps with the technical infrastructure, I’d be happy
to help reclassifying the existing unit tests.

I’ve moved quite a few citeproc-js-specific tests out of the archive,
but I’ve obviously not caught them all: I’ll fix up the tests with
numeric page and page-first. If there are others that don’t belong
there, please call attention to them, and I’ll pull them out.

The suite does need attention. It would be really good to recast the
tests in a more manageable form, and Cucumber seems to be the best
candidate going. We may be able to automate the conversion, but the
tests should really be individually reviewed, commented, tagged and
stripped down to their essentials. It will be a lot of work.

Streamlined editing processes would be very helpful to the cause. I’m
out of my league there, but I would be happy to test things. I’m not
sure if it’s relevant, but this might be interesting for
infrastructure:

https://github.com/tauberer/jot

(Note that Josh hasn’t yet decided on a licence, but I suspect he
would be sympathetic if someone were to turn the code to CSL
purposes.)

Frank

2

Hello,

Brecht writes:

  • A large number of tests test functionality that is not in the CSL
    spec,
    but is provided by citeproc-js (raw dates, static ordering, literal
    names,
    …). I think these should be indicated as such, or perhaps moved to a
    separate directory. This would make it easier to check the other CSL
    processor’s compatibility.

Rintze responds:

Sylvester Keil proposed using a Cucumber format for unit tests, which
would allow tests to be tagged:
citeproc-ruby/features/condition/is_numeric.feature at 1c420de0f7a86b7c35782dee86ce62cbebb47ab9 · inukshuk/citeproc-ruby · GitHub

If somebody else helps with the technical infrastructure, I’d be happy
to help reclassifying the existing unit tests.

I’ve moved quite a few citeproc-js-specific tests out of the archive,
but I’ve obviously not caught them all: I’ll fix up the tests with
numeric page and page-first. If there are others that don’t belong
there, please call attention to them, and I’ll pull them out.

Here’s a short list I have ready. The following tests use raw/literal
dates:

  • date_Accessed (raw)
  • date_InPress (literal)
  • date_LoneJapaneseMonth (raw)
  • date_LopsidedDataYearSuffixCollapse (raw)
  • date_OtherWithDate (season)
  • date_RawParseSimpleDate (raw)
  • date_RawSeasonRange1 (raw)
  • date_RawSeasonRange2 (raw)
  • date_RawSeasonRange3 (raw)
  • date_String (raw)

It might be valuable to convert the raw dates to the structured format, so
the tests can be used by all CSL processors. But I assume you will want to
keep testing the citeproc-js raw date parser? I would suggest separating
these tests from the rendering tests, testing only the conversion from raw
to structured dates.

The suite does need attention. It would be really good to recast the
tests in a more manageable form, and Cucumber seems to be the best
candidate going. We may be able to automate the conversion, but the
tests should really be individually reviewed, commented, tagged and
stripped down to their essentials. It will be a lot of work.

What is the problem with the current test format? It can be easily
extended to support adding tags.

I’m not sure I see the benefit of moving to Cubumber. There are only a
small number of different scenarios (citation, note, bibliography). It’s
basically only the input/style data that differs between tests. Also note
that there’s no real flow (steps) in the citeproc tests, which seems to be
central in Cucumber tests.

Cucumber would introduce a dependency on Ruby, making it more difficult to
support by non-Ruby citeprocs. That said, it would be good to get rid of
the Python dependency of the current test suite. However, the ‘human’ test
files should be rather easy to parse, so the conversion step might not be
necessary.

If you insist on dropping the current test format, may I suggest YAML as
an alternative? YAML is easy to write and read and has parsers for most
platforms. It would not require the extra conversion step (human →
machine).

Here’s date_InPress in YAML format as an example:

comment

mode: citation
result: |
(John Doe (in press))
input:
id: ITEM-1
type: book
author:
- family: Doe
given: John
issued:
date-parts:
literal: (in press)
csl: |




2009-08-10T04:49:00+09:00











Note that you don’t need to quote strings and can include the XML as is
(no escaping). This translates into the current JSON format (try
http://yaml-online-parser.appspot.com/).

Conversion from the current format files can be largely automated.

Cheers,
Brecht

3

Hello,

Brecht writes:

  • A large number of tests test functionality that is not in the CSL
    spec,
    but is provided by citeproc-js (raw dates, static ordering, literal
    names,
    …). I think these should be indicated as such, or perhaps moved to a
    separate directory. This would make it easier to check the other CSL
    processor’s compatibility.

Rintze responds:

Sylvester Keil proposed using a Cucumber format for unit tests, which
would allow tests to be tagged:

If somebody else helps with the technical infrastructure, I’d be happy
to help reclassifying the existing unit tests.

I’ve moved quite a few citeproc-js-specific tests out of the archive,
but I’ve obviously not caught them all: I’ll fix up the tests with
numeric page and page-first. If there are others that don’t belong
there, please call attention to them, and I’ll pull them out.

Here’s a short list I have ready. The following tests use raw/literal
dates:

  • date_Accessed (raw)
  • date_InPress (literal)
  • date_LoneJapaneseMonth (raw)
  • date_LopsidedDataYearSuffixCollapse (raw)
  • date_OtherWithDate (season)
  • date_RawParseSimpleDate (raw)
  • date_RawSeasonRange1 (raw)
  • date_RawSeasonRange2 (raw)
  • date_RawSeasonRange3 (raw)
  • date_String (raw)

It might be valuable to convert the raw dates to the structured format,
so

the tests can be used by all CSL processors. But I assume you will want
to

keep testing the citeproc-js raw date parser? I would suggest separating

these tests from the rendering tests, testing only the conversion from
raw

to structured dates.

The suite does need attention. It would be really good to recast the
tests in a more manageable form, and Cucumber seems to be the best
candidate going. We may be able to automate the conversion, but the
tests should really be individually reviewed, commented, tagged and
stripped down to their essentials. It will be a lot of work.

What is the problem with the current test format? It can be easily
extended to support adding tags.

I’m not sure I see the benefit of moving to Cubumber. There are only a
small number of different scenarios (citation, note, bibliography). It’s

basically only the input/style data that differs between tests. Also
note
that there’s no real flow (steps) in the citeproc tests, which seems to
be

central in Cucumber tests.

Cucumber would introduce a dependency on Ruby, making it more difficult
to

support by non-Ruby citeprocs. That said, it would be good to get rid of

the Python dependency of the current test suite. However, the ‘human’
test

files should be rather easy to parse, so the conversion step might not
be
necessary.

All along the idea was to use the Cucumber format solely as a replacement
for the ‘human’ test cases; we would still convert them automatically to
the JSON/‘machine’ format for processor implementers to use for their own
testing. So this would not create any additional dependencies at all. The
reason I suggested the Cucumber format were:

  • They’re easy to read and write for non-programmers
  • The format isn’t very compact and natively allows for a lot of meta
    information
  • It’s a proven format so we don’t need to invent anything new (plus there
    are parsers available and we don’t need to write our own)
  • The tags are very useful in marking tests as experimental,
    processor-specific etc.

Personally, I don’t have anything against YAML either, but in my
experience the tabs/spaces issue of YAML files is an annoying hurdle,
especially for non-programmers.

Sylvester