Performance testing?

CSL Development
1

@bwiernik and I received a report that the new version of the APA style is running much more slowly than the old one (using citeproc-java, which is citeproc-js with a java wrapper). I’d like to run a performance test on that to see if we can replicate that with citeproc-js – there’s nothing in the style that should cause this, but if it does slow down that much (a factor 6 was mentioned) it’d be good to know why.

@Frank_Bennett – do you have any ideas or maybe even something set up that would allow us to quickly run this? I figured just creating a large bibliography a couple of times running on node.js should do?

2

Sure thing. You can build tests from a CSL JSON export of library items in a few minutes with Citeproc Test Runner. After accepting each output as the accepted result, you’ll have a bundle of passing tests as large as you need. Then run the tests with the option -r spec to get a listing. If you want better feedback on timings, you can run with -r spectrum, which should give you some guidance on installing the spectrum reporter.

3

In addition, though, if the reports are against a checkout of Michel Kraemer’s repo, it looks like it’s set to use the 1.1.206 processor version, which is over a year old. Some of the changes to the processor since then should affect both memory consumption and speed in a positive way.

1 Like
4

I recently removed DOM support from the releases, because string parsing is much faster in modern JS engines; but if citeproc-java still requires it, I can set up to restore it in a separate copy of the processor for their use.

5

No, I don’t think we’d want to change anything at this time. For now I just want to know what, if anything, is going on. If performance is fine with the current citeproc-js, that’s useful information.

That said, I set up the tests and then ran

cslrun -S apa-current -w ~/Zotero/styles/apa.csl -a -k

But got

Using processor from package

Error: Invalid string length

Any thoughts?

6

Ugh, that doesn’t look very useful, does it.

I had a few tweaks in my local copy that I’ve now published to npm. I don’t think they are relevant (my first time to see this error), but you can do npm update citeproc-test-runner and see if it helps. If that doesn’t work, next things to check will be your node version and your OS (so I can test on something similar), a copy of the source of one of the fixtures, and a copy of ~/.cslTestFixtures/fixtures.js, if it has generated that file.

7

This still doesn’t quite work for me (the feedback when running -k doesn’t work for some reason) but I was able to quickly run a test on a 250 item library with both the new and the old APA style, and there’s a very significant difference, with the new style taking about 40 secs, the old one about 10 secs. I also tested with chicago-fullnote-bibliography in case this was due simply to the size of the file, but that one also completed in about 10 secs.

Is there any way you can try to figure out (or even just eyball) what part of the APA code makes it run so slowly? I don’t see anything obvious. The new APA.csl is the one on master, https://github.com/citation-style-language/styles/blob/master/apa.csl – you can see a diff here: https://github.com/citation-style-language/styles/commit/0b94edb2db77c381487149daf846f9536f5fed54 although so much changed that it’s only moderately useful.

8

I have a Jurism release queued up for a research project, once that’s done I’ll take a look. Meanwhile, what is your OS? I’d also like to get the -k bug fixed for you.

9

Also, can you point me at the items you’ve used to generate the tests? The spectrum reporter might offer some hints as to which inputs are most draggy.

10

This post began as a wordy series of observations jotted down as I explored the performance issue. Some of the “facts” that I declared along the way were quite wrong, so I am replacing the original with this summary, which I hope will be more readable, and know to be less misleading.

A pair of traces dumped from the processor while rendering a single citation showed the CSL nodes hit by the old and new versions of the APA style (originally linked to hastebin, but since removed by their expiry rules, oops). The traces show the greater complexity of the new version. It hits 302 nodes, as compared with 175 nodes in the older version. It also has more deeply nested groups, reaching a maximum depth of 10 levels, as opposed to 6 levels in the previous version.

So the new APA version is more complex. But does that have a severe impact on performance? I initially thought that we were in deep water here, but it turns out that the answer is, “No, not really.”

The new APA version does take substantially longer to instantiate when it is installed to the processor—roughly three times longer, at 257ms against 87ms. In normal operation, though, instantiation is infrequent, since a built processor instance can and should be reused. A rise in build-time latency isn’t particularly worrying.

At the rendering level, the picture is much brighter. On my laptop, for a run of 1,000 citations + bibliography, I’m getting timings like 18803ms for the old APA version, against 19248ms for the new one, for a difference of 0.000445 seconds per citation.

I did trial runs with older and newer processor versions, and there isn’t much of a difference. In the course of exploring this issue, I made a few small tweaks that might speed things up very slightly for all styles, but the difference will be so small as to be lost in noise on anything short of a massive data set.

So congratulations to Brenton on a job well done!

PS: @Sebastian_Karcher Is that responsive?

PPS: As a followup note on this issue, I note an enviable report of 7-8ms rendering time in @Sylvester_Keil’s citeproc-ruby, so citeproc-js is lagging a bit there. On the other hand, the test run discussed here was of incremental addition of citations to a document followed by output of the bibliography, so (apart from hardware differences) there would be some additional overhead for disambiguation operations, update reformatting, and DOM latency. All things considered, and to my surprise, citeproc-js maybe isn’t doing too badly on the speed thing.

11

Thanks – that’s indeed great news, thanks for continuing to explore. For test fixtures, does the style get re-instantiated for every fixture? Because that would explain my observation on the fixtures as opposed to your observation on the large bibliography.

12

Yes, exactly. That’s what initially suggested a big performance hit.