# Performance testing?

**URL:** <https://discourse.citationstyles.org/t/performance-testing/1527>\
**Category:** CSL Development\
**Created:** [April 11, 2019, 2:27am UTC](https://discourse.citationstyles.org/t/performance-testing/1527 "2019-04-11T02:27:25Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sebastian\_Karcher](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/sebastian_karcher/32/176_2.png) [@Sebastian\_Karcher](https://discourse.citationstyles.org/u/Sebastian_Karcher)\
**Post date:** [April 11, 2019, 2:27am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/1 "2019-04-11T02:27:25Z")

</div>

@bwiernik and I received a report that the new version of the APA style is running much more slowly than the old one (using citeproc-java, which is citeproc-js with a java wrapper). I’d like to run a performance test on that to see if we can replicate that with citeproc-js – there’s nothing in the style that should cause this, but if it does slow down that much (a factor 6 was mentioned) it’d be good to know why.

@Frank_Bennett – do you have any ideas or maybe even something set up that would allow us to quickly run this? I figured just creating a large bibliography a couple of times running on node.js should do?

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 11, 2019, 2:32am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/2 "2019-04-11T02:32:36Z")

</div>

Sure thing. You can build tests from a CSL JSON export of library items in a few minutes with [Citeproc Test Runner](https://github.com/juris-m/citeproc-test-runner#citeproc-test-runner). After accepting each output as the accepted result, you’ll have a bundle of passing tests as large as you need. Then run the tests with the option `-r spec` to get a listing. If you want better feedback on timings, you can run with `-r spectrum`, which should give you some guidance on installing the spectrum reporter.

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 11, 2019, 2:42am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/3 "2019-04-11T02:42:21Z")

</div>

In addition, though, if the reports are against a checkout of Michel Kraemer’s repo, it looks like it’s set to use the 1.1.206 processor version, which is over a year old. Some of the changes to the processor since then should affect both memory consumption and speed in a positive way.

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 11, 2019, 2:46am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/4 "2019-04-11T02:46:27Z")

</div>

I recently removed DOM support from the releases, because string parsing is much faster in modern JS engines; but if citeproc-java still requires it, I can set up to restore it in a separate copy of the processor for their use.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Karcher](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/sebastian_karcher/32/176_2.png) [@Sebastian\_Karcher](https://discourse.citationstyles.org/u/Sebastian_Karcher)\
**Post date:** [April 11, 2019, 12:11pm UTC](https://discourse.citationstyles.org/t/performance-testing/1527/5 "2019-04-11T12:11:48Z")

</div>

No, I don’t think we’d want to change anything at this time. For now I just want to know what, if anything, is going on. If performance is fine with the current citeproc-js, that’s useful information.

That said, I set up the tests and then ran

```auto
cslrun -S apa-current -w ~/Zotero/styles/apa.csl -a -k

```

But got

```auto
Using processor from package

Error: Invalid string length

```

Any thoughts?

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 11, 2019, 2:44pm UTC](https://discourse.citationstyles.org/t/performance-testing/1527/6 "2019-04-11T14:44:31Z")

</div>

Ugh, that doesn’t look very useful, does it.

I had a few tweaks in my local copy that I’ve now published to `npm`. I don’t think they are relevant (my first time to see this error), but you can do `npm update citeproc-test-runner` and see if it helps. If that doesn’t work, next things to check will be your `node` version and your OS (so I can test on something similar), a copy of the source of one of the fixtures, and a copy of `~/.cslTestFixtures/fixtures.js`, if it has generated that file.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Karcher](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/sebastian_karcher/32/176_2.png) [@Sebastian\_Karcher](https://discourse.citationstyles.org/u/Sebastian_Karcher)\
**Post date:** [April 12, 2019, 12:01am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/7 "2019-04-12T00:01:19Z")

</div>

This still doesn’t quite work for me (the feedback when running -k doesn’t work for some reason) but I was able to quickly run a test on a 250 item library with both the new and the old APA style, and there’s a very significant difference, with the new style taking about 40 secs, the old one about 10 secs. I also tested with chicago-fullnote-bibliography in case this was due simply to the size of the file, but that one also completed in about 10 secs.

Is there any way you can try to figure out (or even just eyball) what part of the APA code makes it run so slowly? I don’t see anything obvious. The new APA.csl is the one on master, [https://github.com/citation-style-language/styles/blob/master/apa.csl](https://github.com/citation-style-language/styles/blob/master/apa.csl) – you can see a diff here: [https://github.com/citation-style-language/styles/commit/0b94edb2db77c381487149daf846f9536f5fed54](https://github.com/citation-style-language/styles/commit/0b94edb2db77c381487149daf846f9536f5fed54) although so much changed that it’s only moderately useful.

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 12, 2019, 1:30am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/8 "2019-04-12T01:30:49Z")

</div>

I have a Jurism release queued up for a research project, once that’s done I’ll take a look. Meanwhile, what is your OS? I’d also like to get the `-k` bug fixed for you.

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 12, 2019, 1:34am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/9 "2019-04-12T01:34:11Z")

</div>

Also, can you point me at the items you’ve used to generate the tests? The spectrum reporter might offer some hints as to which inputs are most draggy.

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 12, 2019, 2:11pm UTC](https://discourse.citationstyles.org/t/performance-testing/1527/10 "2019-04-12T14:11:31Z")

</div>

This post began as a wordy series of observations jotted down as I explored the performance issue. Some of the “facts” that I declared along the way were quite wrong, so I am replacing the original with this summary, which I hope will be more readable, and know to be less misleading.

A pair of traces dumped from the processor while rendering a single citation showed the CSL nodes hit by the old and new versions of the APA style (originally linked to `hastebin`, but since removed by their expiry rules, oops). The traces show the greater complexity of the new version. It hits 302 nodes, as compared with 175 nodes in the older version. It also has more deeply nested groups, reaching a maximum depth of 10 levels, as opposed to 6 levels in the previous version.

So the new APA version is more complex. But does that have a severe impact on performance? I initially thought that we were in deep water here, but it turns out that the answer is, “No, not really.”

The new APA version does take substantially longer to instantiate when it is installed to the processor—roughly three times longer, at `257ms` against `87ms`. In normal operation, though, instantiation is infrequent, since a built processor instance can and should be reused. A rise in build-time latency isn’t particularly worrying.

At the rendering level, the picture is much brighter. On my laptop, for a run of 1,000 citations + bibliography, I’m getting timings like `18803ms` for the old APA version, against `19248ms` for the new one, for a difference of `0.000445` seconds per citation.

I did trial runs with older and newer processor versions, and there isn’t much of a difference. In the course of exploring this issue, I made a few small tweaks that might speed things up very slightly for all styles, but the difference will be so small as to be lost in noise on anything short of a massive data set.

So congratulations to Brenton on a job well done!

PS: @Sebastian_Karcher Is that responsive?

PPS: As a followup note on this issue, I note an enviable [report](https://github.com/inukshuk/citeproc-ruby/issues/53#issuecomment-369368047) of 7-8ms rendering time in @Sylvester_Keil’s `citeproc-ruby`, so `citeproc-js` is lagging a bit there. On the other hand, the test run discussed here was of incremental addition of citations to a document followed by output of the bibliography, so (apart from hardware differences) there would be some additional overhead for disambiguation operations, update reformatting, and DOM latency. All things considered, and to my surprise, `citeproc-js` maybe isn’t doing too badly on the speed thing.

---

<div class="post-metadata">

**Author:** ![Sebastian\_Karcher](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/sebastian_karcher/32/176_2.png) [@Sebastian\_Karcher](https://discourse.citationstyles.org/u/Sebastian_Karcher)\
**Post date:** [April 16, 2019, 1:02am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/11 "2019-04-16T01:02:52Z")

</div>

Thanks – that’s indeed great news, thanks for continuing to explore. For test fixtures, does the style get re-instantiated for every fixture? Because that would explain my observation on the fixtures as opposed to your observation on the large bibliography.

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [April 16, 2019, 1:04am UTC](https://discourse.citationstyles.org/t/performance-testing/1527/12 "2019-04-16T01:04:09Z")

</div>

Yes, exactly. That’s what initially suggested a big performance hit.