how much bugged a style may be?

CSL Development
41

As I anticipated I coded a few tests for getting the numbers. I didn’t
have much time, so this is just some preliminary data. If you think it
is worth I can do some more testing. Take into account this tests
could expose citeproc-hs bugs. They would be probably more meaningful
if conducted with citeproc-js.

First a description of the test procedure.

I’m not very much familiar with Zotero, and I don’t know the style
repository structure. I first tried to update, with the csl-utils, the
"dependent" directory (with ~1150 styles) but I failed with some error
I had no time to investigate. So I tried with the root directory (with
about 275 styles) and I succeeded at the first try.

I had to remove all.csl, blank.csl, irish-historical-studies.csl,
journalistica.csl, sp-legal.csl, test.csl, unisa-harvard.csl,
unisa-harvard3.csl, vienna-legal.csl. This for various reasons -
citeproc-hs bugs I suppose: the “sp” locale is not recognized
(shouldn’t it be “es”?); I do not have “en-EI” and “en-AU” locale
files; citeproc-hs fails if the element is not present.

I tested the styles using only on type, article-journal, because it
instantiates a large number of variables. So I stressed the styles
with missing data: no container-title, no volume, etc. One entry
(ITEM-3) is just a date with a page range.

There are two tests.

The first one tests the presence of two (or more) consecutive spaces.
The results seems to depend quite a lot on the quality of the input
data.

With ITEM-3 106 styles out of 271 will produce consecutive spaces.
Removing ITEM-3 70 styles will fail. I had no time to read the styles.

The citeproc-js approach will entirely remove the problem, here.

The second test checks if the style is producing a space followed by a
punctuation. This is another common case where the possible absence of
some data has not be taken into account.

With ITEM-3 111 styles out of 271 will fail. Without ITEM-3 54.

So I finally checked if the two problems overlaps, but I had no time
for doing the math (I spent the afternoon/night with my two year old
daughter). Still I think maybe 50-60 suffer of both problems (with
ITEM-3).

Now, I don’t know how meaningful this data may be. Is it meaningful to
test styles with low quality input data? I think it is, if we want to
appraise the quality of styles and their use of CSL expressiveness.

The test logs can be found here:
http://gorgias.mine.nu/citeproc/zoteroStyle/

If you want the code (which is not optimized - it takes five minutes
to run a test on my eeepc) I can post it.

Andrea

42

Andrea,

I’ll happily adjust to do whatever the specification calls for. The
CSL behind the tests you raise in your mail all could be written to
avoid the need for duplicate suppression. So no dispute there.

I’ve laid out the reasoning behind space suppression in my previous
mail. The trade-off for eliminating this behavior would be breakage in
a number of existing styles. To protect against that, both at this
point and in future development, we would need a test framework, with
a good foundation of test cases for all extant styles. I don’t think
anyone is proposing to build that infrastructure, so the breakage
would mostly emerge from user feedback. That would mean a lot of
back-and-forth correspondence for the debugging styles, often under
severe time pressure at both ends.

Can we isolate the types of conditions which are likely to lead to
these problems? Do we know of some examples, beyond the ones in the
test suite that Andrea identified?

We should have specific tests for the various combinations. They are
currently embedded in full styles, which is not terribly transparent.
I’m pretty familiar with the possibilities from that recent work with
Carles, and I’ve been thinking about simplifying the test cases. I’ll
move that up in the todo list.

Tests of the node combinations that can result in duplicate spaces are
now available in the citeproc-js local tests area:

http://bitbucket.org/fbennett/citeproc-js/src/4a50c093cb22/tests/fixtures/local/

The relevant fixtures are listed below. The test result strings
assume the suppression of duplicate spaces; this should be changed to
whatever value CSL requires, if similar tests are incorporated into
the standard test suite. Hope they are of use.

spaces_ExplicitDoubleSpaceDelimiter.txt
spaces_ExplicitDoubleSpacePrefix.txt
spaces_ExplicitDoubleSpaceSuffix.txt
spaces_ParentalDelimiterPrefixPrefix.txt
spaces_ParentalDelimiterPrefix.txt
spaces_ParentalPrefixPrefix.txt
spaces_ParentalSuffixDelimiter.txt
spaces_ParentalSuffixPrefixDownhill.txt
spaces_ParentalSuffixPrefixOverTheHill.txt
spaces_ParentalSuffixPrefixUphill.txt
spaces_ParentalSuffixSuffixDelimiter.txt
spaces_ParentalSuffixSuffix.txt
spaces_SiblingDelimiterPrefix.txt
spaces_SiblingSuffixDelimiter.txt
spaces_SiblingSuffixPrefix.txt

Frank

43

Andrea,

I’ll happily adjust to do whatever the specification calls for. The
CSL behind the tests you raise in your mail all could be written to
avoid the need for duplicate suppression. So no dispute there.

I’ve laid out the reasoning behind space suppression in my previous
mail. The trade-off for eliminating this behavior would be breakage in
a number of existing styles. To protect against that, both at this
point and in future development, we would need a test framework, with
a good foundation of test cases for all extant styles. I don’t think
anyone is proposing to build that infrastructure, so the breakage
would mostly emerge from user feedback. That would mean a lot of
back-and-forth correspondence for the debugging styles, often under
severe time pressure at both ends.

Can we isolate the types of conditions which are likely to lead to
these problems? Do we know of some examples, beyond the ones in the
test suite that Andrea identified?

We should have specific tests for the various combinations. They are
currently embedded in full styles, which is not terribly transparent.
I’m pretty familiar with the possibilities from that recent work with
Carles, and I’ve been thinking about simplifying the test cases. I’ll
move that up in the todo list.

Tests of the node combinations that can result in duplicate spaces are
now available in the citeproc-js local tests area:

http://bitbucket.org/fbennett/citeproc-js/src/4a50c093cb22/tests/fixtures/local/

The relevant fixtures are listed below. The test result strings
assume the suppression of duplicate spaces; this should be changed to
whatever value CSL requires, if similar tests are incorporated into
the standard test suite. Hope they are of use.

spaces_ExplicitDoubleSpaceDelimiter.txt
spaces_ExplicitDoubleSpacePrefix.txt
spaces_ExplicitDoubleSpaceSuffix.txt
spaces_ParentalDelimiterPrefixPrefix.txt
spaces_ParentalDelimiterPrefix.txt
spaces_ParentalPrefixPrefix.txt
spaces_ParentalSuffixDelimiter.txt
spaces_ParentalSuffixPrefixDownhill.txt
spaces_ParentalSuffixPrefixOverTheHill.txt
spaces_ParentalSuffixPrefixUphill.txt
spaces_ParentalSuffixSuffixDelimiter.txt
spaces_ParentalSuffixSuffix.txt
spaces_SiblingDelimiterPrefix.txt
spaces_SiblingSuffixDelimiter.txt
spaces_SiblingSuffixPrefix.txt

As an afterthought to assembling these tests, it occurred to me that
strict correspondence between the space characters written into the
CSL file and space characters written on output could be enforced
rather easily in the schema by banning leading and trailing space
characters from the prefix and suffix attributes, and thereby
requiring that such spacing reside on a delimiter.

Such a rule would have the desired effect, and could be enforced
through style validation. It would require the refactoring of every
style in the respository, but at least it would be clear when the job
was done.

Frank

44

Sure they are but I’m not able to find them in the repository (neither
at the link cited above). Am I missing anything?

Andrea

45

Please forget. Now I was able to pull these changes.

andrea

46

Sure they are but I’m not able to find them in the repository (neither
at the link cited above). Am I missing anything?

Please forget. Now I was able to pull these changes.

I forgot to push. Sorry for the time that cost you.

47

No costs at all.

I think there are a few minor issues:

spaces_SiblingDelimiterPrefix.txt: shouldn’t be
(citeproc-hs’s parsers fails otherwise: is this a bug?)

spaces_ParentalSuffixPrefixDownhill.txt: the input ends with a comma.

You also pushed spaces_SiblingSuffixDelimiter.txt~

I’m attaching a patch.

citeproc-hs passes all these tests.

Andrea

space_tests.diff (1013 Bytes)

48

This seems a very interesting proposal. I could help with the style
refactoring if the adoption of this proposal would be deemed worth the
effort it would require for being implemented.

Andrea

49

Am not entirely following this discussion over the last day or so, but
on Frank’s punchline that a primary problem appears to be “leading and
trailing space characters” in prefix and suffix attributes, that would
be easy enough to achieve with a regular expression pattern on those
attributes. Do we really want to do that though? Or do we simply want
to use this information to fix styles?

Alternately, we could create a schematron rule that would simply warn
people running it about the problem?

Bruce

50

Hello,

I didn’t say anything in this thread because I didn’t have a strong
opinion about that (well, more that I have had two contradictory
opinions)

One thing is that it’s important for us to not break already working
styles. Apart from that: to take any decision we would need to know
how difficult is to change the current behaviour (possibly in terms of
time for the style authors, for already existing styles or new
styles). My understanding is that when we have this information we can
keep discussing.

I’ll keep an open eye.

51

Further to this issue, I’ve added an option in citeproc-js to turn off the
purging of duplicate spaces and punctuation, so that citeproc-js can also be
used to test styles for these unintended formatting glitches.

While I got a little worked up in the discussion (for which I apologize,
particularly to Andrea), I do agree that would be cleaner to control this in
the styles, rather than relying on processor magic.

Thinking that a strict version of the CSL schema might be a handy for
editing, and for checking the scope of the problem in the repositories, I
had a go at restricting the leading prefix and trailing suffix spaces
outside of cs:date and cs:names – and ran up against what I think is a
limitation in RelaxNG. What I would like to do is restrict leading spaces on
the cs:label prefix when it appears before cs:name, and trailing spaces on
the suffix when it appears after. That requires two instances of the element
(cs:label), and in Emacs nXML mode, at least, one of them clobbers the
other, so you get false positives for one error or the other, depending on
the order in which the RelaxNG elements are referenced during validation.

If anyone knows of at way around this (schematron?), it would be great to
get it going. It will be easier to get people building styles that nest
cleanly if real time validation that covers this can be set up in an editor.–
View this message in context: http://xbiblio-devel.2463403.n2.nabble.com/how-much-bugged-a-style-may-be-tp5784767p6818349.html
Sent from the xbiblio-devel mailing list archive at Nabble.com.

52

Ha. I take back the question about how to get separate validation of cs:label
before and after cs:name to work.

It’s working fine, I was just being dense; the error markup in nXML
sometimes turns up on the “wrong” affix is all, but fixing the syntax causes
errors to clear. I’ve committed the changes to the mlz-schema fork:

To examine existing styles in the main repo, you can either use the MLZ
extended schema (which should treat CSL 1.0 styles as valid, apart from
affix spacing issues), or set up a “strict” branch to the official schema.
The changes probably won’t go in cleanly to CSL 1.0 official, but the patch
shows what would need to be changed where.–
View this message in context: http://xbiblio-devel.2463403.n2.nabble.com/how-much-bugged-a-style-may-be-tp5784767p6818525.html
Sent from the xbiblio-devel mailing list archive at Nabble.com.

53

Ha. I take back the question about how to get separate validation of cs:label
before and after cs:name to work.

It’s working fine, I was just being dense; the error markup in nXML
sometimes turns up on the “wrong” affix is all, but fixing the syntax causes
errors to clear. I’ve committed the changes to the mlz-schema fork:

To examine existing styles in the main repo, you can either use the MLZ
extended schema (which should treat CSL 1.0 styles as valid, apart from
affix spacing issues), or set up a “strict” branch to the official schema.
The changes probably won’t go in cleanly to CSL 1.0 official, but the patch
shows what would need to be changed where.–
View this message in context: http://xbiblio-devel.2463403.n2.nabble.com/how-much-bugged-a-style-may-be-tp5784767p6818531.html
Sent from the xbiblio-devel mailing list archive at Nabble.com.