A recent thread requested a new CSL attribute, date-range-format, that has existed (as year-range-format) as a CSL-M extension in citeproc-js for the last 14 years.
Had this not already existed, we would’ve been happy to add it in Zotero’s version of citeproc-js and submit upstream, but it’s of limited utility if the official Chicago styles can’t use it.
I can’t remember exactly what the issues were several years ago when we talked about new CSL additions, but given the ease of updating processors these days, I do think we need a mechanism for fast, granular additions like this — particularly ones with no backward-compatibility issues — as minor CSL version bumps, treating them more like software updates. I don’t think there’s much reason that something like year-range-format couldn’t just be 1.0.3, maybe with one or two other new features to reduce churn.
I’m not really familiar with standard processes for specification updates (or how appropriate those processes are in the AI era), but for something like this, I could imagine a process like the following:
- Someone makes a request for an addition. The proposal should state what existing processors will do when they encounter the new markup. (The answer should generally be “ignore it and produce the current behavior”, but worth noting that while citeproc-js ignores unknown node-level elements, it currently fails on unrecognized style-level attributes and unknown elements, so we should change that as part of this, and other processors should do the same.)
- Someone — the same person or another person — provides a reference implementation in one of the main CSL processors, together with test fixtures for the CSL test suite and a proposed diff to the spec and schema.
- There’s a short, fixed comment period (e.g., a few weeks).
- Absent objections, it’s merged as or as part of a minor CSL version bump.
- The styles repository starts accepting styles that use the new feature — either immediately for backwards-compatible changes or, for breaking changes, after a reasonable period that allows major processors to add support and major consumers to test and roll out new versions with the updated processors.
It would be good to hear from the specification maintainers and from other projects using citeproc implementations whether something like this sounds reasonable.
@Dan_Stillman I think that process makes sense, and I think experimenting with anything is a good idea.
But it might also be worth considering integrating citeproc-js (or a rewritten or refactored typescript alternative as mentioned here) into the process?
When a number of us were working on enhancements during the COVID shutdown, the main issue that held us back in my view was what had previously been an advantage: the super loose coupling of schema and code. We never really had a sense of if or when anyone would implement any changes we were making, so we never felt comfortable cutting a release.
And, I think, we were also held back by the lack of compatibility rules, per one of your other points; what must CSL processors do with unknown attributes and elements. I agree you need that.
It’s one of the reason I decided to take the complete opposite approach in Citum; where everything is defined in and generated from (Rust) code: schemas, docs, etc. Styles are developed alongside that code in the same repo, and the most important ones embedded in the engine, and styles can inherit.
Well, this is the point of my proposed Step 2: providing a reference implementation, ideally in a processor in wide usage, that shows support for the new feature. We should probably amend that to a processor in wide usage that already passes the CSL test suite. In practice, maybe that’s always citeproc-js, but if someone wants to demonstrate a feature with another processor that meets those criteria, that seems OK to me. It’s mostly just a quick confirmation step to demonstrate that a processor can be updated without any obvious problems and a guarantee that at least one major processor will support it from Day 1.
For Zotero’s part, we can commit to supporting spec updates in Zotero and submitting citeproc-js patches upstream.
1 Like
Sounds good. How are you thinking that might work, logistically?
Dev opens a PR for the schema, linked to an implementation PR, and a test-suite PR? Something like that?
There are PRs on citeproc-js going back as far as 4 years where no-one ha commented on whether this is something that is desired or not [1]. So my impression is that citeproc-js isn’t really maintained currently - at least to the level where committed patches are being considered for inclusion.
Also, you recently said in a PR where I tried to streamline citeproc-js and turn it into typescript that:
I don’t think humans are working on citeproc-js ever again, so no, I don’t think a massive change like this is necessary or a good idea. [2]
I think you could be right about programming being an activity mainly done by AIs that are being directed by humans through chats and similar in the near future. But if that is the case, and no humans will ever touch citation-related libraries again and for that reason we shouldn’t make any big changes to the libraries – how does fit with the idea og making CSL specification updates and patches to citeproc-js?
[1] Avoid for...in to iterate through array by paulnotley · Pull Request #212 · Juris-M/citeproc-js · GitHub
[2] Typescript by johanneswilm · Pull Request #283 · Juris-M/citeproc-js · GitHub
Some citeproc-js PRs have been merged recently, but maintenance of the official citeproc-js repo isn’t really relevant to this discussion, which is about CSL specification updates. I’m saying that Zotero can commit to adopting spec updates like this for its millions of users. Other projects are welcome to use Zotero’s version of citeproc-js, make their own changes, or switch to another processor if the official citeproc-js doesn’t keep up.
OK, so I take it that you are going to continue development of the citeproc-js library. Some of the PRs currently waiting for approval are really quite urgent, yet have been just let waiting for years. Some of them really should not take more than a few minutes of consideration, such as this one [1].
It’s great there are so many users. I am one of them. I don’t think I would be happy about breaking changes though. I want to be able to rerender the book manuscript I created ten years ago without having to worry whether it still renders or not. Non-breaking changes that add new things would be very much appreciated though.
[1] fix incorrect license field in package.json by bramhaag · Pull Request #245 · Juris-M/citeproc-js · GitHub
+1 to this. I work on a publishing platform that uses citeproc-js, and we’ve hit several cases where Chicago 18 rules aren’t supported by the current spec (date range abbreviation being one of them). Right now our fastest option would be to maintain our own modified CSL files, which means we drift from upstream.
A faster process for small, non-breaking spec additions would be really valuable. We’d much rather pull in official updates than patch things in our own modified files. The process by @Dan_Stillman described sounds good on our end.
I’d be happy to contribute PRs for spec additions and reference implementations if this is the path forward.
On the compatibility issue, I think the easiest way to handle that is to say:
- the XML template element model is frozen (this is the most complex part of the schema, where small changes will break code)
- new features are added via attributes
- CSL processors must not reject unknown attributes
There is one technical wrinkle though, I think.
In CSL right now, all elements are namespaced, but all attributes are not.
With namespaced attributes, it’s easy to express “allow all attributes in X namespace” (@foo:*), and to preserve validation, IDE support, etc for known attributes.
You could even consider two namespaces; an experimental, and a stable one?
I don’t have a specific proposal for how to handle this, but just wanted to flag it.
Also, on 1, you could also consider one breaking change where you do modify the template model. But that could be costly.
I’ve opened a PR for citeproc-js to ignore unknown style-level attributes. We’ll ship this in Zotero 10, which will be out in a few weeks. Node-level attributes were already being ignored in citeproc-js with a console warning.
I think it’s fine to avoid element-level changes, at least until there’s a compelling case for one, or hold those for a major version bump in the future. The failure mode for an unknown attribute is to continue showing a value in the wrong format, perhaps only slightly so. The failure mode for an unknown element is essentially undefined, and might involve omitting entire portions of citations.
It would be good to hear how other major processors handle unknown attributes, so we can understand what the impacts would be on existing users. If they don’t currently ignore them, it would be great if they could be updated to do so. But I don’t think existing processor behavior should be much of a blocker regardless — beyond a reasonable allowance for getting new app versions out — because if projects are blindly pulling in styles from the official repo, they should also be monitoring this list. In any case, at least for the entire deployed citeproc-js user base, we can add node-level attributes (and year-range-format on style) today without breaking anything for users.
I don’t think we should do anything regarding namespacing attributes. The whole point here is that we’re creating new CSL schema versions with new attributes that will validate in the repo. Clients that validate styles at install time and that haven’t updated to a new CSL version might show a warning, but that’s appropriate.
copying a few processor maintainers @John_MacFarlane @zepinglee @asimonyi @Jason-Abbott
See Dan’s question above on unknown attributes. Do you already ignore, but accept, them? If not, could you easily change your code to do that?
Thanks for the @ , @Bruce_D_Arcus1 . For me, an unknown attribute would just be ignored and supporting it would be pretty trivial.
I welcome the initiative and would be happy to help where I can.
What do folks think about extension namespaces similar to how expath handles things?
Actually, no need for expath. But extension namespaces might still be a good idea.
I mentioned that above. Dan was dismissing it, but I think it will matter for validation, IDE support, etc.
Since if you change the schema to allow any attribute, without naming scoping, but definition that will mean typos and such for known attributes will no longer be errors.
But we’re not changing the schema to allow any attribute? That’s processor behavior. Each new CSL version will define the new attributes that we’ve approved for that version. As I said, an older version of an app that validated styles at install time against a previous schema would flag the new style as invalid to the user, as it should, but its processor would continue to produce the same output as before.