test cases

OK, David and I are working on tackling some bugs. David seems to have
wrapped up Ticket 3, and 1 is next up.

http://citeproc.x2x2x.org/cgi-bin/trac.cgi/report/1

James; any interest in working with David on a test suite? I just added
to the existing docbook-test.xml and mods.xml files an example that
exposes the problem in #1, but certainly the more extensive the tests,
the better.

I do use my book manuscript as a test bed, but that won’t go in the
repository of course.

Bruce

OK, David and I are working on tackling some bugs. David seems to
have wrapped up Ticket 3, and 1 is next up.

http://citeproc.x2x2x.org/cgi-bin/trac.cgi/report/1

Trac looks cool. One thing I’ve always wanted is the link between a
ticket and the changes to the code made to fix it, or test it. So for
those one uses
http://citeproc.x2x2x.org/cgi-bin/trac.cgi/wiki/TracLinks right? Nice.

James; any interest in working with David on a test suite?

I general yes. Not sure how we’ll pull this off yet. Basically I’m
thinking about a tests system that looks like this

A test:
testinput, configuration, expected output

And would like to have those in a folder hierarchy so that adding a new
one is as simple as dropping the required pieces into folder. Then we
can have a ticket for each test?

Then we need something to actually run the tests. Is there an obvious
xlst way of building a test suite? Is there something in oXygen and/or
eclipse that would be most appropriate, or should we look to the
testing frameworks in perl/python or maybe Ant?

I just added to the existing docbook-test.xml and mods.xml files an
example that exposes the problem in #1, but certainly the more
extensive the tests, the better.

Ok, I’d like to run these tests. I pulled the source tree with:

svn co http://citeproc-source.x2x2x.org/svn/citeproc

You are saying that something in change set [17]
http://citeproc.x2x2x.org/cgi-bin/trac.cgi/changeset/17
is a test for the author year stuff in #1
http://citeproc.x2x2x.org/cgi-bin/trac.cgi/ticket/1

Ok. can you let me know how you run it? I do notice that there are
some hard coded paths in there:

eg.

http://citeproc.x2x2x.org/cgi-bin/trac.cgi/file/trunk/styles/author-
year.csl?rev=17

line 2:

<?oxygen RNGSchema="file:/Users/darcusb/Desktop/test/citeproc/trunk/schemas/ citationstyle.rnc" type="compact"?>

But could I request a ‘step-by-step’ on how you run these? Command
line would be best (for me to get a picture of the whole tool-chain
required for testing, but I can follow along in oXygen too.

Are you using the exist methods or the flat file?

I do use my book manuscript as a test bed, but that won’t go in the
repository of course.

I think ideally you’d break elements out and create tests as you go.
Of course the developers actually using the software is the best test
of all!

–James
+1 315 395 4056
Details: <http://freelancepropaganda.com/jameshowison.vcf

But could I request a ‘step-by-step’ on how you run these? Command
line would be best (for me to get a picture of the whole tool-chain
required for testing, but I can follow along in oXygen too.

I think David whipped together some batch script for running this.
Let’s see if we can get him to clarify. I gotta go to a meeting …

Are you using the exist methods or the flat file?

I use eXist, but the flatfile is probably easier. Just add
bibdb=flatfile to the command line.

Bruce

The, ah, command line of what? I thought. Then I tracked it through the
code and added bibdb as a parameter in the oXyen transform and it
worked a treat.

–James
+1 315 395 4056
Details: <http://freelancepropaganda.com/jameshowison.vcf

I’m arriving to this thread a little later than I had thought. I have
been on a “hidden” alias all day in which only Bruce had access to
me… he forwarded this to me and not even thinking I responded from
this alias which is not registered in the system for this list and as
such a… very… lonnnnnnnnnnggggg… time … later… I was
reminded of this by the SF.net servers.

Anyway, James, in short I created an XSLT test-suite that uses an XML
configuration file that contains a combination of settings to both set
the test files (or portions of a file) as well as create the batch
files that will auto process these files based on a sequence that is
dynamically built, a final batch file created as a “controller” that
will invoke each batch file sequentially, storing the output in
individual files that can then be referenced for verification of
proper output. Its actually probably easier to understand if you look
at the actual code base so heres a link to a zip of the most recent
updates I made to it a few days ago to compare Saxon and Saxon.NET
output.

link > http://66.93.224.14/~mdavid/test-suite/xslt-test-suite.zip

A slightly outdated explanation of the test suite can be found on my blog

link > http://www.xsltblog.com/archives/2005/01/idea_for_usage.html

In reality the explanation was more of a side effect of a sudden
realization I had for how some points Dimitre was making on XSL-List
could be used in a real world situation, but this gives the general
idea of how the test suite works. Keep in mind that I wrote this in
about a 1 hour block of time + some extended wait time while I
searched for a way to get the commanline message output of Saxon into
a stream and saved to a file. None-the-less I knew I was using
shortcuts that would need to be updated at a later date so this code
is in no way optimized nor does it contain even a small portion of the
functionality it could contain. This project will be a nice way to
use a real world test need to expand the code base appropriately.

Also, the current state of the suite is designed to compare two
different stylesheets and the output they produce, in the time it took
to produce it. It was original thrown together to compare a challenge
to the Muenchian Method of grouping so that it could be quickly
showcased that in fact the challenge was only plausible at a very
superficial level and when used against extensive amounts of data that
needed to be grouped the MM was clearly the winner as we all new it
would be. Still, the resulting suite, slightly modified and enhanced
would allow an extensive library of test files to be run against a
series of output definitions and then used to further verify the
correct output is recieved.

Something just occured to me that would make this web service sample I
just built (still working on a few things and as soon as they are
finished I will shoot you a link) something more than just a sample.
I can set it up such that anyone can submit a test file via a web
form, defining a test sequence to be run against it, returning the
results in either the Response stream or zipped up in a package and
the link returned to the browser via a Response.Redirect such that it
will automatically invoke a download of these files… The other
possibility (and probably even better now that I think about it) would
be to create a custom report page with links to each individual file
for viewing online, with a link to download the files in a zip file if
desired.

Anyway, see if the downloaded files make sense as to whats taking
place. I will take a few minutes now to update this to enable
processing of a library of test files to be output using a series of
defined outputs. We can then turn that into a web service and you all
can coach me on how to make it something that will be truly useful to
the community to begin running tests against and as such contributing
another test file to the system such that it will help us further
understand the usage patterns of the user base which will aid in
further optimizations as well as the future creation of features,
etc…

Back soon with the update…

Any of you still around?

I’ve got maybe an hour of steam left in me at the moment at which
point I will be heading off to lala land to get some much needed rest.
I will check again in a little bit, otherwise will catch up with you
both first thing tomorrow. I have several questions, none of which
are tremendously important, but if you’re around I will shoot them
your way, otherwise see where things are at before I sign off and pose
them, if necessary, at that point.

The one BIG question is the simple answer as to whether you feel a
web-based interface that captures the input and any potential test
patterns (if they dont already exist in the system) and returns the
results as described before. The simple piece of this (a web form to
copy the data to process and push it to the server in an
XmlHttpRequest(), returning the result in the Response stream) already
exists where as taking things to the next level will take a bit more
time. I personally feel the extra dev time is well worth it but I
will leave it to you decide whether or not you feel the same way.

Let me know if either of you are still around and, if so, able to
discuss any of this tonight. If I dont hear back Ill assume your down
for the night and will pick things up again with you in the morning.

Cheers :slight_smile:

<M:D/

Any of you still around?

Yes, though not by choice (insomnia!).

The one BIG question is the simple answer as to whether you feel a
web-based interface that captures the input and any potential test
patterns (if they dont already exist in the system) and returns the
results as described before. The simple piece of this (a web form to
copy the data to process and push it to the server in an
XmlHttpRequest(), returning the result in the Response stream) already
exists where as taking things to the next level will take a bit more
time. I personally feel the extra dev time is well worth it but I
will leave it to you decide whether or not you feel the same way.

I’m not really clear on the choices above. What is the “next level” in
this context?

Let me know if either of you are still around and, if so, able to
discuss any of this tonight. If I dont hear back Ill assume your down
for the night and will pick things up again with you in the morning.

I’m pretty much down. I’ll be able to check email mid-morning
probably. I’ll be entertaining a guest whose flying in for a lecture
though.

Bruce

Oh, ok… well, I will work through this and create demos for you to
evaluate as this will better enable you to make a decision for
extended plans…

FYI… I went back through Ticket #3 and determined that I need some
info from you to finalize the fix, although the current fix is checked
in there still needs to be some attention to it to determine just what
it is that is logically trying to be determined… see comments in the
same line # section as mentioned in the ticket >
http://citeproc.x2x2x.org/cgi-bin/trac.cgi/file/trunk/xsl/citeproc.xsl?rev=21

BTW… Is there a way with TracLink to also specify the line number
to scroll to?

Ive been burning things from both ends of the candle tonight and am
now just heading off to bed… wont be back online for another 10
hours or so but will be right back at it when I arise…

Cheers :slight_smile:

<M:D/

FYI… I went back through Ticket #3 and determined that I need some
info from you to finalize the fix, although the current fix is checked
in there still needs to be some attention to it to determine just what
it is that is logically trying to be determined…

You’re referring to the below comment?

Considering this and the added fact that this variable is then never
used within its scope leads me to determine that what you are
attempting to determine is if the current group contains more than one
element and if so then add add the bib:key element and associated
values to the output tree.

Right. IIRC, it’s also used to get proper punctuation and such.

An easier way to do this is to simply test if the position() is 1 and
if yes then add the addition elements and if no then don’t. While this
does seem to fix the inclusion of the a, b etc… being appended to
the end of the date as well as excluding a second reference with the
same name and year value it also seems to exclude the a,b appended to
the end of the date as it seems is the correct format based on the
corrected output you provided. Obviously this is something we need to
discuss to determine what it is that needs to be output to the result
true such that it can be processed correctly by the output driver of
choice…

Expected output should be (Tilly, 1992, 1993, 1995a, b)

… where the leaving-off the year on the last one should probably be
determined with a parameter from the CSL.

So if there’s only one in an author-year group for a document, then
there is no suffix, while if there’s more than one, there’s always a
suffix.

Make sense?

I would like to know why that code doesn’t work now, but did before the
switch to the global variable approach.

Bruce

Don’t know, but it’d certainly be helpful!

Bruce

Believe it or not Im still awake… I wanted to tie this Web Service
together such that you could play with it as you have time… I had
set it aside to dig deep into this other issue as it was starting to
really bug me that I wasnt getting the output expected…

Make sense?

Complete sense. I couldnt see any other possibility but just wanted to be sure.

I would like to know why that code doesn’t work now, but did before the
switch to the global variable approach.

Is the global variable your refering to the output format variable?
Hey, maybe with a few minutes now we can tackle this… all I need is
a few more tidbits of info and we can smack it down and mark it as
fixed…

M. David Peterson wrote:

Believe it or not Im still awake…

Wow!

I would like to know why that code doesn’t work now, but did before the
switch to the global variable approach.

Is the global variable your refering to the output format variable?

And “enhanced-biblist” and so forth.

Bruce

I just tested the latest code. For some strange reason I’m getting some
funky character output:

pp. 154–76

It’s always in the locator element. Did you change something there David?

Bruce

No… I wonder if the encoding is set incorrectly? I noticed
yesterday that for some strange reason Eclipse was reverting to a 1252
encoding instead of UTF-8 and as such causing me hell for a good hour
while I tried to trace what ultimately proved to be a 1252 base
encoding when it was declared as UTF-8. I changed the default in
Eclipse and everything seemed to be fine… Im guess though that the
source file hasnt been updated properly and is still set at 1252…
let me see if I can fix it locally and if so I will resubmit the
change…

M. David Peterson wrote:

No… I wonder if the encoding is set incorrectly?

That’s probably it, as the offending character is an em-dash.

Bruce

Bruce,

I tried changing the encoding but neither of the docs are reporting
anything but UTF-8, both delcaration and in the encoding property for
each page. And without making a change to the file Eclipse wont allow
me to post my local copy as it is seen as the same copy. I guess I
could make a simple change and as such force it but neither of these
files are files I have actually checked-out and back in so my
assumption is that the problem may be stemming from your end. Have
you checked your local files encoding?

Yes, I think a web interface for submitting test input, configs and
outputs is a very good idea. It allows us to leverage what people are
doing. I wonder if we could rig it so that a Trac ticket is
automatically created (in a special category) so that we can track the
code that is directly relevant to the test case?

I’ll take a look at the testing suite in the next few days but my
familiarity with XLST is only now beginning so this could take some
time.

–J

(

FYI, I am out of commission for about the next 12 hours but will
reply to the above parts related to me when I return…