# test result formats

**URL:** <https://discourse.citationstyles.org/t/test-result-formats/684>\
**Category:** CSL Development\
**Created:** [July 10, 2009, 5:23pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684 "2009-07-10T17:23:49Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [July 10, 2009, 5:23pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684/1 "2009-07-10T17:23:49Z")

</div>

So we’ve got a little issue I earlier worried about with the test  
suite results format: the mixed-content HTML. In short, there are many  
ways to achieve particular visual output results in HTML, but the  
current format would require everyone to agree on one.

I don’t really know how to solve this problem, but perhaps we need to  
consider the possibility that we simply can’t reasonably test the rich  
formatting options, and should just test plain text output?

Any opinions or ideas on this?

Bruce

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [July 10, 2009, 10:09pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684/2 "2009-07-10T22:09:56Z")

</div>

> So we’ve got a little issue I earlier worried about with the test  
> suite results format: the mixed-content HTML. In short, there are many  
> ways to achieve particular visual output results in HTML, but the  
> current format would require everyone to agree on one.
> 
> I don’t really know how to solve this problem, but perhaps we need to  
> consider the possibility that we simply can’t reasonably test the rich  
> formatting options, and should just test plain text output?
> 
> Any opinions or ideas on this?

Formatting needs to be tested, unfortunately. One of the hardert  
things to implement has been second-field-align, which I think is  
going to require some pretty arcane twisting and turning in any  
implementation. … The punctuation order issue has to have formatting  
in there so you can see sequencing. … Layout has to capture its  
affixes with formatting … there are probably others.

The idea we had at the start was to allow the fixture to define  
multiple results, and pass if one of them matches. That’s easy enough  
to set up in the fixture. It would take a little work to set up in  
each test suite, but it’s possible to do.

An alternative would be to have alternative “result sets” defined  
separately, and allow an implementation to select one with a  
parameter. It could work a bit like locale fallback, with the result  
written into the fixture serving as default. That way when an  
implementation encounters a failing test on valid output, the  
implementer could register the valid result in the suite and move on.

I think I like the alternative solution, immediately above.

?

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [July 10, 2009, 10:32pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684/3 "2009-07-10T22:32:19Z")

</div>

First, I’d like again to suggest that other implementers spend some  
time integrating the test suite and offering feedback on these sorts  
of issues. I’m a little frustrated that Frank and I are really the  
only people working on this …

> > So we’ve got a little issue I earlier worried about with the test  
> > suite results format: the mixed-content HTML. In short, there are many  
> > ways to achieve particular visual output results in HTML, but the  
> > current format would require everyone to agree on one.
> > 
> > I don’t really know how to solve this problem, but perhaps we need to  
> > consider the possibility that we simply can’t reasonably test the rich  
> > formatting options, and should just test plain text output?
> > 
> > Any opinions or ideas on this?
> 
> Formatting needs to be tested, unfortunately. One of the hardert  
> things to implement has been second-field-align, which I think is  
> going to require some pretty arcane twisting and turning in any  
> implementation. …

“Second-field align”?

> The punctuation order issue has to have formatting  
> in there so you can see sequencing. …

This I don’t follow. For sake of argument, what if you had the affixes  
include obviously delimiters (“|”) to signal the transitions?

> Layout has to capture its affixes with formatting … there are probably others.

I don’t deny this might be important; I just don’t see ATM how to do  
it. In my implementation, for example, I have no control over things  
like whitespace, or attribute order (which is not significant in xml).  
I just let the tostring function in the xml library takes care of all  
these details (along with namespaces and prefixes).

But as I said, it’d be helpful if others would look into this.

> The idea we had at the start was to allow the fixture to define  
> multiple results, and pass if one of them matches. That’s easy enough  
> to set up in the fixture. It would take a little work to set up in  
> each test suite, but it’s possible to do.
> 
> An alternative would be to have alternative “result sets” defined  
> separately, and allow an implementation to select one with a  
> parameter. It could work a bit like locale fallback, with the result  
> written into the fixture serving as default. That way when an  
> implementation encounters a failing test on valid output, the  
> implementer could register the valid result in the suite and move on.
> 
> I think I like the alternative solution, immediately above.

Bruce

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [July 10, 2009, 11:02pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684/4 "2009-07-10T23:02:25Z")

</div>

That’s the one! Here’s the comment I was working from:

## second-field-align is useful for formatting a numbered list. It aligns

## any subsequent lines of a bibliography entry with the beginning of the

## second field. For example, if the first field is

## On Sat, Jul 11, 2009 at 7:32 AM, Bruce D’Arcus\<@Bruce\_D\_Arcus1\> wrote:

> First, I’d like again to suggest that other implementers spend some  
> time integrating the test suite and offering feedback on these sorts  
> of issues. I’m a little frustrated that Frank and I are really the  
> only people working on this …
> 
> On Fri, Jul 10, 2009 at 6:09 PM, Frank Bennett\<@Frank\_Bennett\> wrote:
> 
> > On Sat, Jul 11, 2009 at 2:23 AM, Bruce D’Arcus\<@Bruce\_D\_Arcus1\> wrote:
> > 
> > > So we’ve got a little issue I earlier worried about with the test  
> > > suite results format: the mixed-content HTML. In short, there are many  
> > > ways to achieve particular visual output results in HTML, but the  
> > > current format would require everyone to agree on one.
> > > 
> > > I don’t really know how to solve this problem, but perhaps we need to  
> > > consider the possibility that we simply can’t reasonably test the rich  
> > > formatting options, and should just test plain text output?
> > > 
> > > Any opinions or ideas on this?
> > 
> > Formatting needs to be tested, unfortunately. One of the hardert  
> > things to implement has been second-field-align, which I think is  
> > going to require some pretty arcane twisting and turning in any  
> > implementation. …
> 
> “Second-field align”?

## 

## 1. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do

## eiusmod

## 

## If set to “margin,” then the first field is put in the margin and all

## subsequent lines of text are aligned with the margin. This is useful for

## IEEE style.

second-field-align-opt = attribute name { “second-field-align” },  
attribute value { “true” | “margin” }

To implement that, you need to put a wrapper around (take a breath)  
the remaining top-level rendering objects after the first one. It’s  
not going to be obvious how to code that in any implementation, so  
there should be a test to check that it’s working.

> > The punctuation order issue has to have formatting  
> > in there so you can see sequencing. …
> 
> This I don’t follow. For sake of argument, what if you had the affixes  
> include obviously delimiters (“|”) to signal the transitions?

Sorry, I should have unpacked that a bit. It’s important to check  
the nesting sequence (punctuation should not get boldfaced). It’s  
easy to get that part right, but it’s harder to get that right when  
moving punctuation inside quotes, so testing it is a good idea. Not a  
bit issue, it’s just one example you can think of.

> > Layout has to capture its affixes with formatting … there are probably others.
> 
> I don’t deny this might be important; I just don’t see ATM how to do  
> it. In my implementation, for example, I have no control over things  
> like whitespace, or attribute order (which is not significant in xml).  
> I just let the tostring function in the xml library takes care of all  
> these details (along with namespaces and prefixes).

Are there libraries around for doing XML diffs?

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [July 10, 2009, 11:12pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684/5 "2009-07-10T23:12:21Z")

</div>

…

> > “Second-field align”?
> 
> That’s the one! Here’s the comment I was working from:
> 
> ## second-field-align is useful for formatting a numbered list. It aligns
> 
> ## any subsequent lines of a bibliography entry with the beginning of the
> 
> ## second field. For example, if the first field is
> 
> ## 
> 
> ## 
> 
> ## 1. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
> 
> ## eiusmod
> 
> ## 
> 
> ## If set to “margin,” then the first field is put in the margin and all
> 
> ## subsequent lines of text are aligned with the margin. This is useful for
> 
> ## IEEE style.
> 
> second-field-align-opt = attribute name { “second-field-align” },  
> attribute value { “true” | “margin” }

I’m not sure this is as complicated as you think, particularly when  
dealing with HTML and CSS. From the CSS perspective, I _think_ we’re  
saying whether the list item has a negative margin or not. It’s for  
numbered styles where the number goes in the margin.

> To implement that, you need to put a wrapper around (take a breath)  
> the remaining top-level rendering objects after the first one. It’s  
> not going to be obvious how to code that in any implementation, so  
> there should be a test to check that it’s working.

But this is really about the rendering engine; not sure we really can.

> > > The punctuation order issue has to have formatting  
> > > in there so you can see sequencing. …
> > 
> > This I don’t follow. For sake of argument, what if you had the affixes  
> > include obviously delimiters (“|”) to signal the transitions?
> 
> Sorry, I should have unpacked that a bit. It’s important to check  
> the nesting sequence (punctuation should not get boldfaced). It’s  
> easy to get that part right, but it’s harder to get that right when  
> moving punctuation inside quotes, so testing it is a good idea. Not a  
> bit issue, it’s just one example you can think of.
> 
> > > Layout has to capture its affixes with formatting … there are probably others.
> > 
> > I don’t deny this might be important; I just don’t see ATM how to do  
> > it. In my implementation, for example, I have no control over things  
> > like whitespace, or attribute order (which is not significant in xml).  
> > I just let the tostring function in the xml library takes care of all  
> > these details (along with namespaces and prefixes).
> 
> Are there libraries around for doing XML diffs?

Good question; I’ll check.

Bruce

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [July 10, 2009, 11:53pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684/6 "2009-07-10T23:53:02Z")

</div>

> …
> 
> > > “Second-field align”?
> > 
> > That’s the one! Here’s the comment I was working from:
> > 
> > ## second-field-align is useful for formatting a numbered list. It aligns
> > 
> > ## any subsequent lines of a bibliography entry with the beginning of the
> > 
> > ## second field. For example, if the first field is
> > 
> > ## 
> > 
> > ## 
> > 
> > ## 1. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
> > 
> > ## eiusmod
> > 
> > ## 
> > 
> > ## If set to “margin,” then the first field is put in the margin and all
> > 
> > ## subsequent lines of text are aligned with the margin. This is useful for
> > 
> > ## IEEE style.
> > 
> > second-field-align-opt = attribute name { “second-field-align” },  
> > attribute value { “true” | “margin” }
> 
> I’m not sure this is as complicated as you think, particularly when  
> dealing with HTML and CSS. From the CSS perspective, I _think_ we’re  
> saying whether the list item has a negative margin or not. It’s for  
> numbered styles where the number goes in the margin.

All you need to test is whether the right tag wrappers are going in  
the right place – the specifics of the formatting attributes can all  
be kicked out to CSS, in HTML mode at least. But the tag wrappers do  
need to be tested, to be sure the engine is capable of handling the  
option.

It’s true that it’s not traumatic to handle or anything, but there is  
some coding involved, and untested is broken code and all that.

> > To implement that, you need to put a wrapper around (take a breath)  
> > the remaining top-level rendering objects after the first one. It’s  
> > not going to be obvious how to code that in any implementation, so  
> > there should be a test to check that it’s working.
> 
> But this is really about the rendering engine; not sure we really can.
> 
> > > > The punctuation order issue has to have formatting  
> > > > in there so you can see sequencing. …
> > > 
> > > This I don’t follow. For sake of argument, what if you had the affixes  
> > > include obviously delimiters (“|”) to signal the transitions?
> > 
> > Sorry, I should have unpacked that a bit. It’s important to check  
> > the nesting sequence (punctuation should not get boldfaced). It’s  
> > easy to get that part right, but it’s harder to get that right when  
> > moving punctuation inside quotes, so testing it is a good idea. Not a  
> > bit issue, it’s just one example you can think of.
> > 
> > > > Layout has to capture its affixes with formatting … there are probably others.
> > > 
> > > I don’t deny this might be important; I just don’t see ATM how to do  
> > > it. In my implementation, for example, I have no control over things  
> > > like whitespace, or attribute order (which is not significant in xml).  
> > > I just let the tostring function in the xml library takes care of all  
> > > these details (along with namespaces and prefixes).
> > 
> > Are there libraries around for doing XML diffs?
> 
> Good question; I’ll check.

If there is something for Python, that could be a way forward, using a  
testing script written in Python that runs arbitrary processors via a  
standard API. Citerpoc-js will have trouble with that in the short  
term, because I can’t find a way of setting the locale across the  
python-spidermonkey bridge to anything other than “C”. But someone  
will eventually sort that out, I’m sure, or another bridge module will  
come along.

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [July 11, 2009, 12:16am UTC](https://discourse.citationstyles.org/t/test-result-formats/684/7 "2009-07-11T00:16:47Z")

</div>

But this sort of highlights my point, I don’t intend to to do anything  
special with this feature in terms of output markup, and my solution  
may be as simple as:

...

… and leave it to CSS to render. There’s just no way to test this adequately.

Bruce

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [July 11, 2009, 1:16am UTC](https://discourse.citationstyles.org/t/test-result-formats/684/8 "2009-07-11T01:16:23Z")

</div>

> > > …
> > > 
> > > > > “Second-field align”?
> > > > 
> > > > That’s the one! Here’s the comment I was working from:
> > > > 
> > > > ## second-field-align is useful for formatting a numbered list. It aligns
> > > > 
> > > > ## any subsequent lines of a bibliography entry with the beginning of the
> > > > 
> > > > ## second field. For example, if the first field is
> > > > 
> > > > ## 
> > > > 
> > > > ## 
> > > > 
> > > > ## 1. Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
> > > > 
> > > > ## eiusmod
> > > > 
> > > > ## 
> > > > 
> > > > ## If set to “margin,” then the first field is put in the margin and all
> > > > 
> > > > ## subsequent lines of text are aligned with the margin. This is useful for
> > > > 
> > > > ## IEEE style.
> > > > 
> > > > second-field-align-opt = attribute name { “second-field-align” },  
> > > > attribute value { “true” | “margin” }
> > > 
> > > I’m not sure this is as complicated as you think, particularly when  
> > > dealing with HTML and CSS. From the CSS perspective, I _think_ we’re  
> > > saying whether the list item has a negative margin or not. It’s for  
> > > numbered styles where the number goes in the margin.
> > 
> > All you need to test is whether the right tag wrappers are going in  
> > the right place – the specifics of the formatting attributes can all  
> > be kicked out to CSS, in HTML mode at least. But the tag wrappers do  
> > need to be tested, to be sure the engine is capable of handling the  
> > option.
> > 
> > It’s true that it’s not traumatic to handle or anything, but there is  
> > some coding involved, and untested is broken code and all that.
> 
> But this sort of highlights my point, I don’t intend to to do anything  
> special with this feature in terms of output markup, and my solution  
> may be as simple as:
> 
> ...
> 
> … and leave it to CSS to render. There’s just no way to test this adequately.

To quote Monty Python, “This isn’t argument, it’s contradiction!” We  
don’t have to throw up our hands over this. It’s doable.

For the example using the ol tag and CSS, you won’t want to implement  
a render bibiliography in that way. Think of how far removed it is  
from the original CSL structures. Also, remember that it must also  
render in RTF, and in formats that don’t have CSS behind them.

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [July 11, 2009, 1:38am UTC](https://discourse.citationstyles.org/t/test-result-formats/684/9 "2009-07-11T01:38:45Z")

</div>

…

> For the example using the ol tag and CSS, you won’t want to implement  
> a render bibiliography in that way.

Well, I haven’t made a decision yet, but this is certainly a perfectly  
reasonable approach: use the classes, and provide a little CSS file  
for developers to include in their documents.

> Think of how far removed it is from the original CSL structures. Also,  
> remember that it must also render in RTF …

RTF support is not a requirement of CSL at all.

Most formats that are important these days (HTML, ODF, OOXML, etc.;  
maybe even RTF) support lists, and some way to style them like this  
using classes, and so without having to get low-level.

So I don’t think we can design the test suite around that assumption.

Bruce

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [July 11, 2009, 2:02am UTC](https://discourse.citationstyles.org/t/test-result-formats/684/10 "2009-07-11T02:02:25Z")

</div>

> …
> 
> > For the example using the ol tag and CSS, you won’t want to implement  
> > a render bibiliography in that way.
> 
> Well, I haven’t made a decision yet, but this is certainly a perfectly  
> reasonable approach: use the classes, and provide a little CSS file  
> for developers to include in their documents.

It’s not the use of classes for styling – that’s all good – it’s  
that using ol as the top-level wrapper would have CSS providing the  
citation-number value automagically, on the assumption that it will  
correspond to the values used in CSL to generate citations. That  
might work, at least in current use cases, although it does seem  
risky. A tougher question is how you would map the formatting  
attributes on the first-to-render   
element through to the CSS in a robust way. CSL might change the  
formatting of the element depending on the reference item content  
(boldface for books, say – never mind whether it seems likely or a  
good idea, the language allows it, and the language needs to behave  
correctly).

You’ll want to keep the output markup as close as possible to the CSL  
layout structure. It’s simpler all around that way.

Frank

---

<div class="post-metadata">

**Author:** ![Bruce\_D\_Arcus1](https://avatars.discourse-cdn.com/v4/letter/b/4bbf92/32.png) [@Bruce\_D\_Arcus1](https://discourse.citationstyles.org/u/Bruce_D_Arcus1)\
**Post date:** [July 11, 2009, 1:54pm UTC](https://discourse.citationstyles.org/t/test-result-formats/684/11 "2009-07-11T13:54:09Z")

</div>

True; I’ll need to look out for this before making any final  
decisions. The key test is whether I can more-or-less easily modify  
the wrappers around a list item using CSS or other other styling  
language. I know I can modify other characteristics of the list mark.

Bruce