# locale files - Part II

**URL:** <https://discourse.citationstyles.org/t/locale-files-part-ii/1198>\
**Category:** CSL Development\
**Created:** [October 6, 2013, 1:01pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198 "2013-10-06T13:01:03Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paulo\_Ney\_de\_Souza](https://avatars.discourse-cdn.com/v4/letter/p/76d3ee/32.png) [@Paulo\_Ney\_de\_Souza](https://discourse.citationstyles.org/u/Paulo_Ney_de_Souza)\
**Post date:** [October 6, 2013, 1:01pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/1 "2013-10-06T13:01:03Z")

</div>

2- Fallback scheme defined inside the XML files themselves: I understand  
from Rintze that the fall-back scheme from one locale to another is defined  
and execute from withing the code, and that the sequence

```
 de-AT --> de-DE --> en-US

```

is hardcoded in and will vary for other languages. That makes the creation  
of a new locale file very hard - one has to request and a fall-back schema  
to be created and introduced in the code.

The natural place for the fall-back language to be would be inside the XML  
files themselves. So one could create -ar-DZ for Arabic-Algeria and have it  
fall-back to the language -ar and be in business in a minute! That would  
make testing easier and lower the bar on the creation of new contributions.

Care would have to be exercised to prevent an user from creating an  
infinite loop in the fall-back. The best way to handle both organization  
and this technical problem would be to allow a locale file to fall-back to  
a language-file only, and then to the English file. In the previous example  
the fall back would:

```
 -de-AT.xml --> -de.xml --> -en.xml

```

in this new schema, there would be 5 new files that will be language  
specific: -de, -en, -fr, -pt and -zh containing most of the language  
material and a sequence of very small files:

de-AT  
de-CH  
de-DE

en-GB  
en-US

fr-CA  
fr-FR

pt-BR  
pt-PT

zh-CN  
zh-TW

that would contain the real locale-specific settings. None of this files  
will need to be created by hand - explanation coming up on Part III.

Paulo Ney

---

<div class="post-metadata">

**Author:** ![Sebastian\_Karcher](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/sebastian_karcher/32/176_2.png) [@Sebastian\_Karcher](https://discourse.citationstyles.org/u/Sebastian_Karcher)\
**Post date:** [October 6, 2013, 5:14pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/2 "2013-10-06T17:14:32Z")

</div>

to me this sound reasonable. I’ll say that creating a new locale file when  
a similar one exists is not really “very hard”, since you can just copy  
over anything that you want to maintain - e.g. for the fr-CA locale, folks  
simply took fr-FR and changed the filename & one or two terms, took a total  
of 5mins. That said allowing fallbacks to be specified is certainly more  
elegant & cleaner. The other concerns would be implementers. I’m not sure  
how well locales are implemented in the various citeproc versions and we’d  
want to make sure that implementers are happy with this solution - e.g.  
even with the current local set-up isn’t work everywhere and that’s a  
problem for CSL, so if citeproc maintainers or ref managers object to this,  
imho those objections would override concerns for elegance.  
SebastianOn Sun, Oct 6, 2013 at 6:01 AM, Paulo Ney de Souza \<@Paulo\_Ney\_de\_Souza\>wrote:

---

<div class="post-metadata">

**Author:** ![Paulo\_Ney\_de\_Souza](https://avatars.discourse-cdn.com/v4/letter/p/76d3ee/32.png) [@Paulo\_Ney\_de\_Souza](https://discourse.citationstyles.org/u/Paulo_Ney_de_Souza)\
**Post date:** [October 6, 2013, 6:47pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/3 "2013-10-06T18:47:18Z")

</div>

True! Creating a new locale from a nearby one is never hard, but  
maintaining it later becomes hard. Because tokens are no longer  
single-sourced, and people end up modifying one file and leaving the other  
behind - this is clearly what happened to the pt-PT, pt-BR double.

I’ll explain it a bit better on Part III and IV, but I need more than a  
phone to do that message!

Paulo Ney

---

<div class="post-metadata">

**Author:** ![Dan\_Stillman](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@Dan\_Stillman](https://discourse.citationstyles.org/u/Dan_Stillman)\
**Post date:** [October 6, 2013, 9:11pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/4 "2013-10-06T21:11:52Z")

</div>

As far as I know de-AT -\> de-\>DE -\> en-US isn’t hardcoded anywhere, at  
least for Zotero, unless citeproc-js does something I’m not aware of. At  
least in Zotero’s pre-citeproc-js cite code, it just looked for a full  
code matching the current locale (e.g., ‘de-AT’), and if that didn’t  
exist it looked for a matching language code (‘de’, but expanded to  
’de-DE’ for historical reasons), and if that didn’t exist it used  
’en-US’. I would guess that citeproc-js still does the same. I don’t  
think there’s a reason anything needs to be hardcoded, in the code or in  
the XML files, or requested.

Looking for ‘de-DE’ instead of ‘de’, though not technically accurate,  
doesn’t strike me as having much of a downside, but with appropriate  
warning I imagine the various implementers would be OK with changing  
those to the two-character codes. (Some implementations might support  
two-character codes already, given the presence of locales-eu.xml,  
though it’s also possible that one just doesn’t work in some  
implementations.)

If I’m understanding you, this doesn’t address your larger request, to  
fall back on individual missing terms. If Frank and others want to  
implement that, I guess they can, but I’m not convinced it’s worth the  
effort. I’ve actually wanted en-US fallback in Zotero proper (not for  
CSLs) for a long time, to avoid errors when new English strings are  
added but other locale files haven’t been updated, but unlike the Zotero  
strings, which change whenever we add or change features, the CSL  
locales seem largely fixed, with relatively little maintenance involved  
after the initial creation. It doesn’t really seem beyond the scope of  
copy and paste. But, again, it’s really up to the implementers.

---

<div class="post-metadata">

**Author:** ![Charles\_Parnot](https://avatars.discourse-cdn.com/v4/letter/c/57b2e6/32.png) [@Charles\_Parnot](https://discourse.citationstyles.org/u/Charles_Parnot)\
**Post date:** [October 7, 2013, 6:59am UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/5 "2013-10-07T06:59:52Z")

</div>

Here is what Papers does, implementation-wise. If a term is not available for the requested locale, it first falls back on the “parent” language locale. This parent language is harcoded in our implementation so for instance, de-DE is the parent language for all the de-XX locales. In other words, each family of yy-XX locales have a designated parent yy-??. If the term is not found, it falls back on the en-US locale (again, the fact that this locale is the grandmother of all is hard-coded).

The proposal to formalize that process makes sense.

Charles

---

<div class="post-metadata">

**Author:** ![Rintze\_Zelle](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/rintze_zelle/32/175_2.png) [@Rintze\_Zelle](https://discourse.citationstyles.org/u/Rintze_Zelle)\
**Post date:** [October 7, 2013, 8:56pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/6 "2013-10-07T20:56:22Z")

</div>

Currently the spec dictates

de-AT \> de-DE \> en-US

for fallback behavior among the locale files, and the spec provides a list  
of “primary” locales like “de-DE” (see  
[http://citationstyles.org/downloads/specification.html#locale-fallback](http://citationstyles.org/downloads/specification.html#locale-fallback) ).  
It also says “If the chosen output locale is a language (e.g. ‘de’), the  
(primary) dialect is used in step 1 (e.g. ‘de-DE’).” Any CSL processor that  
supports this will have to hardcode the primary locales, right?

To avoid hardcoding, we could rename “de-DE” and all other primary locales  
to their 2-letter language tag (i.e. “de”), so the locale file fallback  
simply becomes:

de-AT \> de \> en-US

I’m not sure if that’s sufficient or in agreement with what Avram or Paulo  
wish for, though. Avram seems to argue against locale file fallback  
altogether in the Appendum thread?

Rintze

---

<div class="post-metadata">

**Author:** ![Sebastian\_Karcher](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/sebastian_karcher/32/176_2.png) [@Sebastian\_Karcher](https://discourse.citationstyles.org/u/Sebastian_Karcher)\
**Post date:** [October 7, 2013, 9:24pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/7 "2013-10-07T21:24:45Z")

</div>

> To avoid hardcoding, we could rename “de-DE” and all other primary locales  
> to their 2-letter language tag (i.e. “de”), so the locale file fallback  
> simply becomes:
> 
> de-AT \> de \> en-US
> 
> I’m not sure if that’s sufficient or in agreement with what Avram or Paulo  
> wish for, though. Avram seems to argue against locale file fallback  
> altogether in the Appendum thread?

this was something I considered, too. So as long as we only have one  
version for a language we just use the two language code. After that we  
just add locales with countries.

To me this has the advantage of being both elegant _and_ to work out of the  
box with current citeproc behavior.

The biggest problem I see with that is different scripts. As both Avram and  
Paulo points out, this would create very confusing (and, per Avram,  
potentially offensive) substitutions.  
I’m less concerned about Avram’s other concern - i.e. “how complete is a  
locale” - since those interested can just check the local files. I’m not  
sure how big the first issue is and if it may have a simple solution. I  
disagree with Avram about not supporting any fallbacks (except to en-US to  
prevent errors), since Paulo is correct that this can possibly make  
supporting multiple locales for one language rather tedious. Not that  
that’s a current problem, but might become one long term.

Sebastian\>

---

<div class="post-metadata">

**Author:** ![Avram\_Lyon](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@Avram\_Lyon](https://discourse.citationstyles.org/u/Avram_Lyon)\
**Post date:** [October 7, 2013, 9:28pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/8 "2013-10-07T21:28:50Z")

</div>

Yes. I think we should focus on being good at matching users to the best  
matching single locale file for their specified locale, but we shouldn’t  
attempt to do it on the level of terms.

---

<div class="post-metadata">

**Author:** ![Dan\_Stillman](https://avatars.discourse-cdn.com/v4/letter/d/ecb155/32.png) [@Dan\_Stillman](https://discourse.citationstyles.org/u/Dan_Stillman)\
**Post date:** [October 7, 2013, 9:38pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/9 "2013-10-07T21:38:35Z")

</div>

> ```
> As far as I know de-AT -> de->DE -> en-US isn't hardcoded anywhere, at
> least for Zotero, unless citeproc-js does something I'm not aware
> of. At
> least in Zotero's pre-citeproc-js cite code, it just looked for a full
> code matching the current locale (e.g., 'de-AT'), and if that didn't
> exist it looked for a matching language code ('de', but expanded to
> 'de-DE' for historical reasons), and if that didn't exist it used
> 'en-US'. I would guess that citeproc-js still does the same. I don't
> think there's a reason anything needs to be hardcoded, in the code
> or in
> the XML files, or requested.
> 
> ```
> 
> Currently the spec dictates
> 
> de-AT \> de-DE \> en-US
> 
> for fallback behavior among the locale files, and the spec provides a  
> list of “primary” locales like “de-DE” (see  
> [Redirecting…](http://citationstyles.org/downloads/specification.html#locale-fallback) ).  
> It also says “If the chosen output locale is a language (e.g. ‘de’),  
> the (primary) dialect is used in step 1 (e.g. ‘de-DE’).” Any CSL  
> processor that supports this will have to hardcode the primary  
> locales, right?

I overlooked ‘zh-CN’, but that’s the only one of that list that needs to  
be hardcoded, along with the general fallback of en-US. ‘de-DE’ (or  
‘de’) and ‘pt-PT’ could just be rule based: if the user’s full locale  
(e.g., ‘de-CH’) doesn’t exist, try the root (‘de’ or ‘de-DE’).

> To avoid hardcoding, we could rename “de-DE” and all other primary  
> locales to their 2-letter language tag (i.e. “de”), so the locale file  
> fallback simply becomes:
> 
> de-AT \> de \> en-US

Sure, but our ‘de-DE’ was really just ‘de’ expanded for simplicity, and  
I think the code — at least our code, but I imagine citeproc-js too —  
was designed accordingly.

And to be clear, I’m talking solely about choosing a single file here,  
as Avram (I think) is arguing for. I think that term-level substitution  
is extreme overengineering for our purposes, with all the problems that  
Avram points out. But if someone ends up with Standard German instead of  
Austrian German, it’s pretty clear what happened.

---

<div class="post-metadata">

**Author:** ![Frank\_Bennett](https://yyz1.discourse-cdn.com/flex029/user_avatar/discourse.citationstyles.org/frank_bennett/32/198_2.png) [@Frank\_Bennett](https://discourse.citationstyles.org/u/Frank_Bennett)\
**Post date:** [October 7, 2013, 11:17pm UTC](https://discourse.citationstyles.org/t/locale-files-part-ii/1198/10 "2013-10-07T23:17:32Z")

</div>

> > As far as I know de-AT → de-\>DE → en-US isn’t hardcoded anywhere, at  
> > least for Zotero, unless citeproc-js does something I’m not aware of. At  
> > least in Zotero’s pre-citeproc-js cite code, it just looked for a full  
> > code matching the current locale (e.g., ‘de-AT’), and if that didn’t  
> > exist it looked for a matching language code (‘de’, but expanded to  
> > ‘de-DE’ for historical reasons), and if that didn’t exist it used  
> > ‘en-US’. I would guess that citeproc-js still does the same. I don’t  
> > think there’s a reason anything needs to be hardcoded, in the code or in  
> > the XML files, or requested.
> 
> Currently the spec dictates
> 
> de-AT \> de-DE \> en-US
> 
> for fallback behavior among the locale files, and the spec provides a list  
> of “primary” locales like “de-DE” (see  
> [Redirecting…](http://citationstyles.org/downloads/specification.html#locale-fallback) ). It  
> also says “If the chosen output locale is a language (e.g. ‘de’), the  
> (primary) dialect is used in step 1 (e.g. ‘de-DE’).” Any CSL processor that  
> supports this will have to hardcode the primary locales, right?
> 
> I overlooked ‘zh-CN’, but that’s the only one of that list that needs to be  
> hardcoded, along with the general fallback of en-US. ‘de-DE’ (or ‘de’) and  
> ‘pt-PT’ could just be rule based: if the user’s full locale (e.g., ‘de-CH’)  
> doesn’t exist, try the root (‘de’ or ‘de-DE’).
> 
> To avoid hardcoding, we could rename “de-DE” and all other primary locales  
> to their 2-letter language tag (i.e. “de”), so the locale file fallback  
> simply becomes:
> 
> de-AT \> de \> en-US
> 
> Sure, but our ‘de-DE’ was really just ‘de’ expanded for simplicity, and I  
> think the code — at least our code, but I imagine citeproc-js too — was  
> designed accordingly.
> 
> And to be clear, I’m talking solely about choosing a single file here, as  
> Avram (I think) is arguing for. I think that term-level substitution is  
> extreme overengineering for our purposes, with all the problems that Avram  
> points out. But if someone ends up with Standard German instead of Austrian  
> German, it’s pretty clear what happened.

The specification requires us to implement term-level substitution in  
the processors for overrides embedded in the style file, with the same  
selection logic as for full standalone locale selection. Continuing to  
require full coverage in individual standalone locales might be the  
best choice for clarity, as Avram and Dan suggest, but just in case,  
going the other way wouldn’t be an exceptional burden for the  
implementations.