Feature proposal: Custom Schemas (custom-schemas)

[cormacrelf]

Just a new thread for an idea I had, didn’t want it to get buried in the weeds.

@cormacrelf

Option 5:

• Let people & style authors do whatever they want with first-class, namespaced escape hatches, BUT provide a way that users can be made aware of breaking changes that styles make for people who depend on those escape hatches.

I got all excited about giving styles version numbers a while back, but for the wrong reasons – I had thought reusability of style code was the goal. It isn’t. (I’m such a programmer.) The correct goal is letting people keep creating documents with the same reference library and data-entry conventions while upgrading to new releases of styles.

Say you allowed extensions in a <schema> block, that the processor can validate.

<style style-version="2.3.0" xmlns:chicago="https://citationstyles.org/schema/chicago">
  <schema namespace="chicago">
    <!-- - bool can be checked in a condition
             <if chicago:var="true" />
         - string is a regular variable
             <text variable="chicago:var" />
         - name is a name variable
             <names variable="chicago:var" />
         - date is a date variable
             <date variable="chicago:var" />
         - number is a number variable
             <number variable="chicago:var" /> -->
    <key name="chapter-is-introduction" type="bool" />
    <!-- - enum could be like a CSL type: <if chicago:some-enum="two" /> -->
    <key name="some-enum" type="enum" values="one two three" />
  </schema>
  <citation>
    <layout>
      <choose>
        <if chicago:chapter-is-introduction="true" />
      </choose>
    </layout>
  </citation>
</style>

A researcher with a library full of introductory chapters is using Chicago. She realises these are being formatted incorrectly, and asks @Sebastian_Karcher to improve the style for her. He releases Chicago version 2.3.0 as above, and she puts in her Zotero settings (e.g.) that her library depends on Chicago version range “2.3”, i.e. ^2.3.0 in Node semver notation. Zotero then creates some field UI to edit this.

[{
  "id": "citekey",
  "type": "chapter",
  "title": "What is Your Dangerous Idea?",
  "schema:https://citationstyles.org/schema/chicago": {
    "chapter-is-introduction": true
  }
}]

Version 2.4 might add another key to the schema (a semver ‘feature’); version 2.5 might adopt some CSL spec addition that covers this, but retain backwards compatibility with people who are using the old schema entry; version 3.0.0 (breaking) might stop supporting the schema and only support the official way. Styles should have a changelog, so that researchers who are using one version know what might break or be improved if they were to upgrade.

By validate, I mean emit warnings or errors if a user enters:

{ ..., "schema:some-unsupported-schema": { ...} }
=> info: reference ID "citekey" included properties for "some-unsupported-schema"
   (may not be an error, because e.g. most styles do not treat introductions differently.
    This is an 'include X hacks for chicago' type of deal.)

{ ..., "schema:...": { "some-unsupported-key": "..." } }
=> error: schema "chicago" does not support this key.

{ ..., "schema:...": { "chapter-is-introduction": "some-improperly-typed-value" } }
=> error: schema "chicago" key "..." expected a boolean value, found string

I included namespacing in this concept so that multiple similar styles can share a namespace and the properties within it. That way one library could mostly work across a larger number of styles, e.g. all of the variants of a particular style. But it also means that you don’t have to worry about name collisions with future official variable names in the CSL-JSON input. Just reserve schema:* and you’re good.

Obviously not all of this works very well (esp the sharing of namespaces between styles), but it’s maybe a start. I’m thinking now it would be better for schemas to have a version, and styles to support a version range? Anway, there are my thoughts.

PS, <key> could also have for="chapter othertypes" , to enforce usage only with a specific set of types.

@Denis_Maier

I guess this is more than a start. The more I think about this the more I like the idea of allowing schema additions. This could possibly solve many problems at once: Keeping CSL core specs simple and maintainable while at the same time giving more options to those who need them. This could be especially useful for certain disciplines with rather special demands, like law, classical philology, biblical studies, film studies, musicology. I don’t think this would necessarily entail convoluted GUIs. It might be reasonable to have users who need this use the extra -Field in Zotero for example. (Of course, the idea that GUI tools could add fields if required by a style sounds intriguing.)

I don’t know how difficult it is to implement this. But if is feasible I am absolutely in favor. @cormacrelf Have you already thought about this?

@cormacrelf

Have I thought about how easy it is to implement? For me? Pretty easy.

It would need some other things to be successful.

• It would need backpressure. Using a schema addition instead of the ‘standard library’ of variables reduces the portability of people’s references between styles. So its use should be generally discouraged where existing stuff would do the trick. I don’t imagine this stance will be difficult to take for the CSL maintainers, given how easily saying no can be justified!
• It would also need documentation. Not of how to use the feature, but when to use a custom-schemas variable and what it will do, whether it’s deprecated. Essentially you need javadoc/rustdoc-style documentation for each key. Ideally this would be compulsory and pull requests to the styles repo would enforce it. With a deprecated = "for X reason" tag. That GUIs will actually warn you about. That’s the third thing:
• This is a feature for end-users, not style authors. Without GUI support and attention to detail over the experience, the validation capabilities are worthless. “STYLE HAD AN ERROR CODE -4188” is much worse than what we have now. That’s where the actual implementation burden is: with Zotero, that means error codes and translations for UI elements. Building a engine->user warning path would be useful in other ways, however. I would like to be able to bubble up things like “invalid raw date string” and “could not disambiguate”, not only do a best-effort rendering.
• Because it connects end-users to style authors more directly, this might have implications for how people seek support. Are people still going to think to post on the Zotero forums and file GitHub issues if they see docs/text written by individuals? I’m a programmer so I have “find where the source code is hosted, read it, maybe fix it yourself and don’t assume the original author actually has time to deal with your problem individually” drilled into me, not sure if everyone else does.

---

Addendum

I would add that it’s easy to determine which styles people care about seeing GUI fields for: there’s already a list of styles you care about in Zotero settings! I wonder if that would end up showing a lot by default. I think if done right, the idea generally strikes an OK balance complicating the spec and flexibility. The actual work for implementors is miniscule; every implementation already tests variables and checks conditions. You just parse 2 extra nodes, schema:-prefixed variable names, and make the possibilities a little more dynamic. So I wouldn’t say there is undue burden or unnecessary complication. The complications saved and flexibility gained for authors and users far outweighs any implementation difficulty. I also think people who are bound by absolute compliance to style guides but don’t want to format all their references themselves deserve a bit of a break.

Some misc other benefits, which I will proceed to exaggerate a little:

• It moves the decision of whether to add something to CSL proper away from the people who would be annoyed if you said no.
• It lets people use free software in a way that works for them, without requiring their opinionated approaches to affect everyone else.
• It enables collecting information about what kind of hacks the community finds necessary, whereas before this is hidden in 2 ways:
  • Hacks by misusing fields (not sure how prevalent this is on the main repo, I suspect it’s policed a little)
  • People using local overriding versions of styles that misuse fields or use notes to set overrides (I know I do this constantly)
• The rampant abuse of the Extra/notes field by every single implementation or downstream user of CSL, without exception, tells us that people really do need the functionality it brings, and that it’s worth formalising. The negatives of that approach shouldn’t colour this solution, which is structured, easier to use, and actually viable for permanent non-spec-addition hacks for various styles. I would have no trouble with forever using the schema for my idiotic style guide to do the annoying bits.
• You can also avoid the the four hundred million different formats there are for information embedded in Extra. I will ultimately have to, but really don’t want to, implement that stuff.
• It can encode many of CSL-M’s extensions just fine (not eg institutions, but the variables yes)

It would be under a custom-schemas feature flag for a good while, and would probably require users to opt in before they can use the feature. Styles should generally still do best-effort rendering if their custom variables are not set, not render nothing and break for people who aren’t using the tweaks they offer.

[Denis_Maier]

I’ve already written that I really like the proposal. Just to add to this: I think this could actually lead to metadata portability since it would render hackish solutions obsolete. Consider the problem with introductions in Chicago I have outlined in the other thread: Using @bwiernik’s idea to use genre or medium without a title here could be nice solution of CMoS, but you won’t be able to cite such items as-is with other styles that treat introductions just like an ordinary chapter. With a special boolean for CMoS your general item metadata would still be usable with other styles.

[Denis_Maier]

@cormacrelf

That’s where the actual implementation burden is: with Zotero, that means error codes and translations for UI elements.

@Dan_Stillman any thoughts about this and about the idea of schema additions in general?

[bwiernik]

With respect to the introduction idea—it’s really not a hack so much writing styles to robustly handle diverse metadata formats. It’s not an uncommon thing for style guides to specify that untitled items should display a description of the item. I’ve accommodated this sort of requirement in APA and other psychology styles I’ve written by writing the title macros to handle an item that does not have a title but instead has just genre or medium. This sort of thing can already be handled with the existing CSL spec—it just requires writing styles that are robust to nuanced selections of metadata. That’s not how most styles are currently written, but I would prefer to take that approach, especially with the big guides like APA, Chicago, MLA, Vancouver, etc., rather than bifurcating schemas in individual styles.

[Denis_Maier]

Yes, I’m totally in favor of not bifurcating schemas without a good reason. In my opinion the problem here is that one style treats this as an untitled item while other styles take “Introduction” as a proper title. So, if we change styles we’d also have to move the title from title to genre or medium.

[Sebastian_Karcher]

I’m afraid I dont’ have the time necessary to fully do justice in comments to this well-thought-out proposal, but I don’t think it’s feasible.
It adds significant complexity upstream, including to end-users e.g.

He releases Chicago version 2.3.0 as above, and she puts in her Zotero settings (e.g.) that her library depends on Chicago version range “2.3”, i.e. ^2.3.0 in Node semver notation. Zotero then creates some field UI to edit this.

will be a disaster in getting users to understand and in providing support for.

It also seems miserable to update all these differentyl versioned styles (e.g. when a style guide updates, do I have to update all existing versions of the style? Which ones are deprecated? And having a unified suit of styles is imo a core strenght of CSL

So while I understand this is attractive from the pov of a style and potentially citeproc authors, I think it’s a bad idea for repository maintainers, end-users, implementers and, because it creates a multiverse of CSL styles, ultimately the CSL project as a whole.

(FWIW, the excess of Extra/Note stuff is much more on the Zotero end because of the – soon to be removed – block on field updates. So while that is indeed a problem, I don’t agree it’s as big of a CSL problem as you suggest)

[Dan_Stillman]

Yeah, I’m afraid I’m in complete agreement with @Sebastian_Karcher on this. This would just add massively too much complexity all around, particularly around support.

(And the Extra issue will indeed be resolved in Zotero soon, but the supported formats are explained in the citeproc-js docs. It’s really just a basic hack that’s pretty trivial to support.)

[Denis_Maier]

@Dan_Stillman

And the Extra issue will indeed be resolved in Zotero soon, but the supported formats are explained in the citeproc-js docs. It’s really just a basic hack that’s pretty trivial to support.

What does that it mean that the Extra issue will be resolved soon? Some details somewhere? Will this mean that entering additional varialbes via Extra/Note will stop working?

[Dan_Stillman]

@Denis_Maier

What does that it mean that the Extra issue will be resolved soon?

We’ll finally be able to start making changes to types and fields in Zotero again. That’s been on hold for many years for technical and user-experience reasons related to syncing, but in the current Zotero beta we finally have a plan for dealing with it. Once that rolls out, and we give it enough time (probably a month or two) that we can do a sync cut-off, we can start updating types and fields.

Known fields in Extra will be automatically migrated to new fields. (E.g., original-date: 1908 will move to an Original Date field.)

@Denis_Maier

Will this mean that entering additional varialbes via Extra/Note will stop working?

That’s just a feature of citeproc-js, so no.