Status page for style repository

I was wondering if there would be any interest in helping me setting up a
status page for the CSL style repository at
https://github.com/citation-style-language/styles. I think it would be
rather nice if we could use a post-commit hook to automatically update a
status page displaying:

  • the number of independent and dependent styles
  • the results of CSL schema validation (do all styles validate), including
    validation against the embedded Schematron rules that scan for calls to
    undeclared macros
  • a number of addition validation tests: is there any overlap in CSL file
    names between dependent and independent styles, do the file names match the
    style IDs, do the file names follow our naming guidelines, do all
    independent styles have a “self” link, do all “template” links point to
    existing styles, does the citation-format of dependent styles match that of
    their independent parent

I currently perform these tests manually on my local git repo with Jing and
a few Python scripts (I mostly use csl-qc.py and csl-rights.py from
https://github.com/citation-style-language/utilities, and I know Charles
Parnot has some scripts of his own), but it would be nice if we could
automate QC a little bit while showing people more information about the
status of the repo. My hacking skills aren’t advanced enough though to set
this up myself.

Rintze

Oh, and another important piece of information to list would be the rights
under which the styles are released. I think it would be neat if people can
easily confirm that all styles are released under the exact same CC BY-SA
license (which they are now, and should be). I currently use csl-rights.py
to make sure all the styles use the same string in the cs:rights element.

RintzeOn Thu, May 3, 2012 at 10:36 AM, Rintze Zelle <@Rintze_Zelle>wrote:

An option to consider …

Remember that I used github pages to setup this:

http://citation-style-language.github.com/styles/

Yeah, it was http://simonster.github.com/csl-validator.js/ that made me
wonder we could do this all on github.com. But Simon uses client-side
JavaScript, and I guess it would be easier to do things server-side.

Rintze

Dear all,

Sylvester Keil has been extraordinary helpful in the past couple of weeks
by setting up an automatic testing environment (using the Travis CI
service) for CSL styles hosted on GitHub. After tinkering a little with
Python (and Jing), he has now settled on Ruby. Tests are automatically run
after each commit to the GitHub repository (currently his fork of
https://github.com/citation-style-language/styles ), and build reports can
be viewed at http://travis-ci.org/#!/inukshuk/styles .

The current tests are defined in
https://github.com/inukshuk/styles/blob/testing-ruby/spec/independent_styles_spec.rb
and
https://github.com/inukshuk/styles/blob/testing-ruby/spec/dependent_styles_spec.rb
and are somewhat self-explanatory. They include XML and CSL schema
validation, and cover all the tests I normally run on my own computer.
Similar tests can be set up for the CSL locales (which also should validate
to the CSL schema).

At this point, there are a few questions for contributors and consumers of
CSL styles.

  1. How the test environment should be incorporated into our workflow.

a) for contributors: once everything is in place, I would like to ask of
everybody who directly commits to the style repository to run the tests
locally to make sure only correct styles are added to the repository.
Alternatively, you could just fork the repo and create pull requests
(Travis CI supports screening of pull requests before they are merged,
which should catch any problems
http://about.travis-ci.org/blog/announcing-pull-request-support/ ).

b) for consumers (Zotero, Mendeley): are you all happy with the way the
repository is being run now? Sometimes the repository contains a bunch of
invalid styles, and an error (e.g. deleting all styles) can quickly affect
a lot of users. We could:

i) revoke commit access from all contributors and require them to use pull
requests, which the repository maintainers could manually merge when Travis
CI gives the green light (merging is a chore, so I’m not a fan of this
approach), or
ii) create a separate repository that automatically pulls in commits from
https://github.com/citation-style-language/styles, but only if all tests
pass. I think this should be relatively easy to set up with a commit hook,
and should be very maintenance free. Consumers of CSL styles could then get
their styles from that verified repository.
iii) keep things as they are.

  1. whether there are any additional tests we can add. Charles, I know you
    do some quality assurance as well. Do you see any missing tests?

Comments welcome,

RintzeOn Thu, May 3, 2012 at 10:36 AM, Rintze Zelle <@Rintze_Zelle>wrote:

Hi Rintze,

This is really great, and something that’s very much needed. I like that it’s using Ruby too :wink:

Regarding adding more control on what gets into the clients, I also think it would be good to have at least one level of control, and prevent commits with failures to make it to the clients. There is much simpler alternative to having a separate repository, and that’s to have instead a separate branch. We can very simply keep the current ‘master’ branch, and simply add a ‘tested’ branch. This would be a simple branch that we can fast-forward automatically to the latest commit on which all tests pass.

In pseudo-code, that would be something like this:

if (csl_tests_all_good_on_master)
git update-ref -m “Fast forward tested branch” origin/tested origin/master

So it really amounts to moving upn the ‘tested’ pointer as soon as tests pass, and no need for a separate repository.

Then clients simply need to pull from the ‘tested’ branch instead of 'master.

charles

The current tests are defined in
https://github.com/inukshuk/styles/blob/testing-ruby/spec/independent_styles_spec.rb
and
https://github.com/inukshuk/styles/blob/testing-ruby/spec/dependent_styles_spec.rb
and are somewhat self-explanatory. They include XML and CSL schema validation, and cover all the tests I normally run on my own computer. Similar tests can be set up for the CSL locales (which also should validate to the CSL schema).

Could we merge this now into the main repository, so we can start using it? Maybe with a few instructions? I like the idea of having those tests right there with the repo. And it will then be possible for us to submit more tests or modify existing ones if needed.

Again, this looks like a great addition to the repository, I am eager to get that integrated into my workflow!

charles

The benefit I see in having a separate repository is that it would be
safer: fewer people could have write access to the “verified repository”,
and people wouldn’t be able to accidentally commit to the wrong branch.

RintzeOn Mon, Jul 16, 2012 at 4:22 PM, Charles Parnot <@Charles_Parnot>wrote:

Yes, that was another question that slipped my mind. Does anybody object to
putting the test scripts in the main repo?

Sylvester helped me along running the tests locally on Windows XP (through
http://rubyinstaller.org/ ), which was a quick setup.

RintzeOn Mon, Jul 16, 2012 at 4:27 PM, Charles Parnot <@Charles_Parnot>wrote:

We could prevent anybody from committing to the ‘tested’ branch with a commit-hook (http://www.galengrover.com/git/prevent-committing-to-master-with-git-hooks/). The branch can then only be used as a pointer.

I honestly don’t know which way is easier, so in the end, whatever works is good for me. You may be right: a separate repo could be easier to enforce.

charles

Could we merge this now into the main repository, so we can start using it? Maybe with a few instructions? I like the idea of having those tests right there with the repo. And it will then be possible for us to submit more tests or modify existing ones if needed.
Yes, that was another question that slipped my mind. Does anybody object to putting the test scripts in the main repo?

If possible, it would be nice to move all the related files inside one directory, e.g. /Tests/?

Sylvester helped me along running the tests locally on Windows XP (through http://rubyinstaller.org/ ), which was a quick setup.

Looks like I will just need to install a few gems, then.

charles

Hello,

Thanks Rintze and Sylvester for working on this. It should make it
easier for us to safely provide frequent style updates to users.

ii) create a separate repository that automatically pulls in commits from https://github.com/citation-style-language/styles,
but only if all tests pass. I think this should be relatively easy to set up with a commit hook, and should be very maintenance free.
Consumers of CSL styles could then get their styles from that verified repository.

This seems sensible and matches our own workflow internally - except
that we apply commit restrictions (which can be on the committer, the
content of the commit message or the content blobs) only to certain
branches in the pre-receive hook rather than the whole repository.

Though git makes it easier to push, pull and merge between different
repositories, I suspect that just applying restrictions on pushing to
certain branches (eg. only the CI system can push to master) might be
easier to work with.

Regards,
Rob.

Been out of town, and will look at this more closely later. But my strong
belief is that (in particular the master branch on) the main csl style repo
should be the canonical one. Is that not possible?

We can definitely arrange it any way we want. I can see how it would make sense that the master branch of the main CSL repository would have the full tested styles and would be the one clients should pull from, and users could download from.

An ‘edge’ branch or fork would be the one with the commits still to be tested.

charles

To keep the tested styles in the master branch is certainly possible. In fact, this is similar to our current test setup, as we created separate branches for the Ruby and Python tests respectively. It is then very easy to configure travis-ci to run the tests on a specific branch only.

By the way, I just pulled in the latest styles from the master branch and all tests are passing now.

Rintze, if you give me write access to the official wiki I will add instructions on how to set up the test environment there.

Sylvester

signature.asc (203 Bytes)

Yes, keeping the master branch of
https://github.com/citation-style-language/styles as the canonical one
makes sense.

So the consensus seems to be that a new branch for untested test should
suffice? I’d be very happy if one of you could pick up the gauntlet and set
this up (I prefer to hide behind the excuse that I’m not a professional
programmer). I can give read/write access to the repo to anybody interested.

Rintze

I’ll gladly create the new branch and set up the test-environment there. Any preferences for the branch name? (I would have suggested ‘development’ or ‘dev’ myself)

Sylvester

signature.asc (203 Bytes)

“development” is fine.

RintzeOn Tue, Jul 17, 2012 at 11:13 AM, Sylvester Keil <@Sylvester_Keil>wrote:

One thing to keep in mind about git: branches are super lightweight.
So, for example, it’s possible to consider dev branches for every
committer.

I’m not saying this is a great idea; just suggesting you all not be
bound by CVS/SVN thinking on branches.

@sylvester: thanks! and ‘development’ seems like a good name :slight_smile:

@bruce: thanks for the trip back to memory lane; had not thought of svn in a while… Given the nature of the CSL repo, and the low granularity, and given that there are almost no dependencies between files, it works very well to have just one shared branch, but I agree it’s good to keep in mind.

In fact, it reminds me of the discussion about CSL 1.0.1. With these tests put in place (and after giving it a bit of time to settle), maybe it would be nice to start thinking about that transition also in terms of branches. I know you had proposed to create a separate directory for the different version (how svn-like!!!), but a branch would likely work as well. Merging changes from development back into a 1.0.1 branch would allow to keep the 1.0.1 styles in sync, and eventually, the ‘development’ branch could be split to leave a 1.0.0 branch behind, and switch its content to 1.0.1 styles. At that point, ‘master’ would pick up those 1.0.1 styles as well. The 1.0.0 branch could also be updated by merging back ‘master’ into it (but being careful to then strip the changes that are 1.0.1 specific).