# Infrastructure for style-level testing

**URL:** <https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997>\
**Category:** CSL Development\
**Created:** [April 7, 2011, 7:40am UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997 "2011-04-07T07:40:52Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 7, 2011, 7:40am UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/1 "2011-04-07T07:40:52Z")

</div>

Hi, all,

The experience of two major deployments of citeproc-js has driven home  
the value of the test suite fixtures. But after writing upward of 640  
of the things, I don’t relish the idea of continuing to do this by  
hand. 🙂

Thanks to generous feedback from many quarters, the processor itself  
now seems to be pretty well under control. The problem of style  
testing remains with us, however, and the bottleneck is obvious; no  
one in their right mind is going to sit down and write the hundreds of  
tests that would be required to prove that any individual style is  
trouble-free. Yet that level of testing will be needed, it seems to  
me, if we want CSL to establish solid credibility with and find favor  
among major publishers.

The “syntax” of the test fixtures is admittedly a dog’s breakfast  
currently, set up in a rush before speeding forward with the coding.  
JSON has been used to describe the input data mostly because it does  
not require conversion (saving me the time required to write a  
converter), and because typing hundreds of tests in JSON by hand is at  
least a little less painful than writing hundreds of tests in XML.  
However, now that we have the processor running in at least two  
interactive applications, recasting the fixtures in a single uniform  
syntax seems a good idea, and the obvious choice would be XML.

What would make XML attractive at this point is that we now have  
citeproc-js running in two (2) interactive applications, with many  
thousands of users between them. Users typically recognize style  
errors when they are entering citations in the word processor. At that  
point, the document state is instantiated in the processor registry,  
and can be readily accessed via JS. It would be very simple to build a  
DOM construct containing the items cited, the citations set in the  
document, the context (bibliography/citation), and the expected result  
string (provided by the user), and the style code or a pointer to the  
style in use (if a standard style), and to dump that to disk. Such a  
reporting facility would save time in the fielding of user error  
reports, and it would make it possible to contemplate building  
comprehensive test suites for individual styles.

So … it would be a great help to the cause if a persevering soul or  
hungry intern could be persuaded to produce a plugin or extension for  
use with one or both of our citeproc-js-consuming projects, offering  
a “Report style error” button in the respective word-processor plugin  
menu. I would be very (very) happy to adapt the citeproc-js test  
runner to process such output, and (although I’m only guessing) I  
reckon that the same goes for Andrea Rossato and other developers.

One could even argue that a robust set of style-level tests, coupled  
with a simple CSL IDE built with, say, xulrunner, would have a greater  
impact than a style editor, since you could then (safely) rely on  
contributions by amateur programmers and relative newcomers to the CSL  
scene as collaborative style maintainers.

So that’s the pitch. I should add that I’m unlikely to undertake such  
a project off my own bat, as the processor is looking pretty stable,  
and need to get on with other tasks. But I do think this would be a  
good thing for CSL, so I thought I would float the suggestion, in case  
anyone out there is in a Christmas mood vis-a-vis our expanding  
community.

Frank

(cross-posted to citeproc-js list, with apologies for any double-deliveries)

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [April 7, 2011, 12:05pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/2 "2011-04-07T12:05:47Z")

</div>

…

> One could even argue that a robust set of style-level tests, coupled  
> with a simple CSL IDE built with, say, xulrunner, would have a greater  
> impact than a style editor, since you could then (safely) rely on  
> contributions by amateur programmers and relative newcomers to the CSL  
> scene as collaborative style maintainers.

Just want to note that I think this highlights that a lot of us are  
thinking about the importance of what comes down to smart previewing  
as a possible way to facilitate a number of things, including style  
creation and editing. E.g.:

[https://github.com/citation-style-language/styles/issues#issue/4](https://github.com/citation-style-language/styles/issues#issue/4)

Some of us, for example, were again talking about Dan Stillman’s  
original thought about a direct-edit preview style creation interface.  
With good preview data, and some UI magic to deal with hairy things  
like names and dates, this might be a really good idea.

Bruce

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [April 7, 2011, 12:28pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/3 "2011-04-07T12:28:40Z")

</div>



---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 7, 2011, 12:39pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/4 "2011-04-07T12:39:49Z")

</div>



---

<div class="post-metadata">

**Author:** ![andrea\_rossato2](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@andrea\_rossato2](https://discourse.citationstyles.org/u/andrea_rossato2)\
**Post date:** [April 7, 2011, 4:20pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/5 "2011-04-07T16:20:49Z")

</div>

Frank Bennett \<@Frank\_Bennett\> writes:

> So … it would be a great help to the cause if a persevering soul or  
> hungry intern could be persuaded to produce a plugin or extension for  
> use with one or both of our citeproc-js-consuming projects, offering  
> a “Report style error” button in the respective word-processor plugin  
> menu. I would be very (very) happy to adapt the citeproc-js test  
> runner to process such output, and (although I’m only guessing) I  
> reckon that the same goes for Andrea Rossato and other developers.
> 
> One could even argue that a robust set of style-level tests, coupled  
> with a simple CSL IDE built with, say, xulrunner, would have a greater  
> impact than a style editor, since you could then (safely) rely on  
> contributions by amateur programmers and relative newcomers to the CSL  
> scene as collaborative style maintainers.

Frank’s guess is correct: I’d be delighted to support such a facility  
for style development and I agree that this could be more important than  
a style editor for the long term robustness of CSL.–  
andrea rossato

---

<div class="post-metadata">

**Author:** ![Sylvester\_Keil](https://avatars.discourse-cdn.com/v4/letter/s/85e7bf/32.png) [@Sylvester\_Keil](https://discourse.citationstyles.org/u/Sylvester_Keil)\
**Post date:** [April 7, 2011, 8:03pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/6 "2011-04-07T20:03:43Z")

</div>

Absolutely. The citeproc-test suite is an invaluable resource for processor development, so to make the tests more accessible to style authors or to an even larger user base is an effort I would definitely support.

By the way, (and because I still owe Bruce and Rintze an example) I have been playing around with converting the citeproc-test JSON data into cucumber features; you can take a look at an example at:

[https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is\_numeric.feature](https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is_numeric.feature)

The advantage of cucumber features is that they are extremely intuitive and easy to write. Although, in this case the main complexity in formulating a test case is in defining the style and input data. If I understand it correctly, this is also Frank’s position and I would agree that it is a good approach to aim at generating test cases from within an application that already maintains the relevant input data.

Sylvester

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [April 7, 2011, 8:18pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/7 "2011-04-07T20:18:25Z")

</div>

Yes. Only downside is it’s then Ruby-specific.

Seems you have some other news of sorts: that one can now do “gem  
install citeproc-ruby”.

Cool!

Bruce

---

<div class="post-metadata">

**Author:** ![Sylvester\_Keil](https://avatars.discourse-cdn.com/v4/letter/s/85e7bf/32.png) [@Sylvester\_Keil](https://discourse.citationstyles.org/u/Sylvester_Keil)\
**Post date:** [April 7, 2011, 8:55pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/8 "2011-04-07T20:55:45Z")

</div>

> > By the way, (and because I still owe Bruce and Rintze an example) I have been playing around with converting the citeproc-test JSON data into cucumber features; you can take a look at an example at:
> > 
> > [https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is\_numeric.feature](https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is_numeric.feature)
> > 
> > The advantage of cucumber features is that they are extremely intuitive and easy to write. Although, in this case the main complexity in formulating a test case is in defining the style and input data.
> 
> Yes. Only downside is it’s then Ruby-specific.

Yes (although I believe there is a Java implementation around); however, the step definitions in this case are extremely simple (e.g., see [https://github.com/inukshuk/citeproc-ruby/blob/master/features/step\_definitions/citeproc\_steps.rb](https://github.com/inukshuk/citeproc-ruby/blob/master/features/step_definitions/citeproc_steps.rb)), therefore, you could easily use Ruby sorto of like a shell script that fires up rhino, hugs, or ghc and feeds the test data to citeproc-js or citeproc-hs.

Not to stray too far off topic, though, you are of course right that an implementation agnostic test suite isthe best option going forward.

> Seems you have some other news of sorts: that one can now do “gem  
> install citeproc-ruby”.
> 
> Cool!

It is still very experimental and incomplete but I guess ‘release early’ won’t hurt at this point. (It only works in Ruby 1.9. at the moment, though, because of differences in unicode handling.) I have also reconsidered your advice and changed the license to a two-clause BSD license.

Sylvester

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [April 7, 2011, 11:17pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/9 "2011-04-07T23:17:30Z")

</div>

> -----BEGIN PGP SIGNED MESSAGE-----  
> Hash: SHA1
> 
> > > By the way, (and because I still owe Bruce and Rintze an example) I have been playing around with converting the citeproc-test JSON data into cucumber features; you can take a look at an example at:
> > > 
> > > [https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is\_numeric.feature](https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is_numeric.feature)
> > > 
> > > The advantage of cucumber features is that they are extremely intuitive and easy to write. Although, in this case the main complexity in formulating a test case is in defining the style and input data.
> > 
> > Yes. Only downside is it’s then Ruby-specific.
> 
> Yes (although I believe there is a Java implementation around); however, the step definitions in this case are extremely simple (e.g., see [https://github.com/inukshuk/citeproc-ruby/blob/master/features/step\_definitions/citeproc\_steps.rb](https://github.com/inukshuk/citeproc-ruby/blob/master/features/step_definitions/citeproc_steps.rb)), therefore, you could easily use Ruby sorto of like a shell script that fires up rhino, hugs, or ghc and feeds the test data to citeproc-js or citeproc-hs.
> 
> Not to stray too far off topic, though, you are of course right that an implementation agnostic test suite isthe best option going forward.
> 
> > Seems you have some other news of sorts: that one can now do “gem  
> > install citeproc-ruby”.
> > 
> > Cool!
> 
> It is still very experimental and incomplete but I guess ‘release early’ won’t hurt at this point. (It only works in Ruby 1.9. at the moment, though, because of differences in unicode handling.)

I remembered 1.8 didn’t deal with unicode very well. But even with  
1.9.2, I get this:

> x = [‘ö’,‘o’,‘a’,‘x’]  
> =\> [“ö”, “o”, “a”, “x”]  
> x.sort  
> =\> [“a”, “o”, “x”, “ö”]

I was used this just working in saxon. Is there any way to get it to  
sort right in 1.9.2?

> I have also reconsidered your advice and changed the license to a two-clause BSD license.

Still compatible with GPL?

Bruce

---

<div class="post-metadata">

**Author:** ![Sylvester\_Keil](https://avatars.discourse-cdn.com/v4/letter/s/85e7bf/32.png) [@Sylvester\_Keil](https://discourse.citationstyles.org/u/Sylvester_Keil)\
**Post date:** [April 8, 2011, 9:23am UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/10 "2011-04-08T09:23:48Z")

</div>

> > -----BEGIN PGP SIGNED MESSAGE-----  
> > Hash: SHA1
> > 
> > > > By the way, (and because I still owe Bruce and Rintze an example) I have been playing around with converting the citeproc-test JSON data into cucumber features; you can take a look at an example at:
> > > > 
> > > > [https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is\_numeric.feature](https://github.com/inukshuk/citeproc-ruby/blob/master/features/condition/is_numeric.feature)
> > > > 
> > > > The advantage of cucumber features is that they are extremely intuitive and easy to write. Although, in this case the main complexity in formulating a test case is in defining the style and input data.
> > > 
> > > Yes. Only downside is it’s then Ruby-specific.
> > 
> > Yes (although I believe there is a Java implementation around); however, the step definitions in this case are extremely simple (e.g., see [https://github.com/inukshuk/citeproc-ruby/blob/master/features/step\_definitions/citeproc\_steps.rb](https://github.com/inukshuk/citeproc-ruby/blob/master/features/step_definitions/citeproc_steps.rb)), therefore, you could easily use Ruby sorto of like a shell script that fires up rhino, hugs, or ghc and feeds the test data to citeproc-js or citeproc-hs.
> > 
> > Not to stray too far off topic, though, you are of course right that an implementation agnostic test suite isthe best option going forward.
> > 
> > > Seems you have some other news of sorts: that one can now do “gem  
> > > install citeproc-ruby”.
> > > 
> > > Cool!
> > 
> > It is still very experimental and incomplete but I guess ‘release early’ won’t hurt at this point. (It only works in Ruby 1.9. at the moment, though, because of differences in unicode handling.)
> 
> I remembered 1.8 didn’t deal with unicode very well. But even with  
> 1.9.2, I get this:
> 
> > x = [‘ö’,‘o’,‘a’,‘x’]  
> > =\> [“ö”, “o”, “a”, “x”]  
> > x.sort  
> > =\> [“a”, “o”, “x”, “ö”]
> 
> I was used this just working in saxon. Is there any way to get it to  
> sort right in 1.9.2?

Unicode-Strings are a big mess right now; encoding works like a charm but manipulation is painful; for example this does not work either:

x.map(&:upcase)  
=\> [“ö”, “O”, “A”, “X”]

For this reason I’m currently using the unicode-utils gem (which doesn’t support sorting yet I believe), but it is something that I need to assess separately. I remember that Unicode strings worked fairly well under 1.8 using active-support.

> > I have also reconsidered your advice and changed the license to a two-clause BSD license.
> 
> Still compatible with GPL?

Yes. It’s the ‘FreeBSD’ License and it is listed as GPL compatible:

[http://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses](http://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses)

Sylvester

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [April 8, 2011, 12:50pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/11 "2011-04-08T12:50:51Z")

</div>

We could start with:

In any case, what do we need?

That work?

Anything else?

[https://gist.github.com/909764](https://gist.github.com/909764)

Might be cool if the XML test could get fed to cucumber.

Bruce

---

<div class="post-metadata">

**Author:** ![Sylvester\_Keil](https://avatars.discourse-cdn.com/v4/letter/s/85e7bf/32.png) [@Sylvester\_Keil](https://discourse.citationstyles.org/u/Sylvester_Keil)\
**Post date:** [April 9, 2011, 7:12am UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/12 "2011-04-09T07:12:19Z")

</div>

Off the top of my head:

An optional akin to the ‘citation items’ or ‘bibsection’ in the current test suite (if no input is specified, all the elements in are to be processed).

The tag should support inline styles as well as URIs to load a certain predefined style (because styles can change that is not ideal for processor testing, but it makes writing the tests easier and is useful for style testing — that way we can use the same test format for testing processors and styles.

An optional tag outside of the definition. Again, this should support inline definition or definition by reference.

An optional tag.

An optional tag (or similar); this is something I find quite useful in cucumber: you can add any number of tags to a scenario description. For example, the tags ‘experimental’, ‘v1.0’, ‘edtf’, or ‘html’ could denote that the test is experimental, expected to work in CSL 1.0, uses EDTF input data, or formats the expected result in HTML). Tags are easily extensible, optional as they don’t alter the test itself, but useful for filtering (e.g., I want to run tests that are tagged ‘v1.0’ but not ‘v1.1’ etc.).

Sylvester

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [April 9, 2011, 3:19pm UTC](https://discourse.citationstyles.org/t/infrastructure-for-style-level-testing/997/13 "2011-04-09T15:19:01Z")

</div>

I’ve added the start of the schema:

[https://github.com/citation-style-language/schema/commit/b98aecf31577eb8c188e57c56d381332a2f58881](https://github.com/citation-style-language/schema/commit/b98aecf31577eb8c188e57c56d381332a2f58881)

I mostly focused on the getting the basic RNC structure right, with  
details still to be worked to be worked out. For example, apropos the  
last point, do we want to constrain the tag values?

I have a few comments in the source to indicate other questions.

Feel free to comment on the commit, or fork and hack.

Bruce