Re: Finally completed badge, feedback on process


 

Apologies on the lack of brevity in advance.

Getting to 100% passing was relatively easy for BRL-CAD with
only one MUST item arguably being unmet beforehand (our website
certificate didn’t match our domain, fixed). The rest was mostly a matter of
documentation and elaboration.
Good to hear. I would expect most well-run projects to "mostly" do well. Fixing domain certificates, and doing a little documentation & elaboration, Is (to me) a *good* thing.
The "documentation and elaboration” I mentioned was actually on the badging form itself (i.e., the various explain/justify prompts).

In BRL-CAD's case, we fortunately already had everything publicly available and documented in expected locations. Of course, always room for improving, consolidating, and simplifying documentation so information is easier to find.

I also note there isn't a requirement against the certificate being self-signed, having a name mismatch, or having no chain. Might be worthy of a SHOULD clause.

Another feedback point that I’m not sure how / if the badging process could capture: Accessibility of information. In filling out the information for BRL-CAD, I was able to find and cite everything. However, it was spread across a half dozen different resources, some on the web, some in the repo, etc.

For criteria like the license, it spells out a standard location (e.g., COPYING in the repo). What about doing something similar for some of the other documentation items?

For example, under the first Documentation section, it says that docs must exist. Is there best practice on where those docs must be? I’d argue that a source code repository MUST 1) have some sort of obvious introductory documentation (e.g., a README in the top-level directory for most projects) 2) have contributor documentation provided in there or in another obvious location (e.g., in a similarly obvious top-level document like CONTRIBUTING.md, HACKING, DEVINFO, etc.).

We certainly *could* add 3rd-party review gating. For example, we don’t advertise this much, but I *do* review each passing badge to look for nonsense. But it's merely a brief look for nonsense, not a rigorous re-check.
Unfortunately, you don’t scale. ;)

So the question is... how could we scale that 3rd-party review? My current position is to focus on improving automation... which would make everyone happy (faster to get a badge, more rigorous checking). Are there good scaleable alternatives?
I could see two ways working, both requiring implementation effort.

The first would be to simply add a MUST peer-review sign-off criteria that N others must mark as Met/Unmet. The system would randomly contact N (e.g., == 3) of the 100% completed projects and ask them to review their peer. If more than half concur, they join the 100% club. Basically, treat it like an academic paper peer review.

An even better approach — but more work to implement — would be to take a page from stack overflow. Let registered readers mark individual criteria responses as satisfactory or unsatisfactory, and leave comments. When a statistically significant % (e.g., majority after >3 votes) disagree, the item gets set to Unmet and the reviewer must do some work and try again or appeal to higher authority for arbitration. With the social approach, the system easily expands to review queues where the workload is shared.

I certainly agree that centralized version control is viable (having used sccs, rcs, cvs, and subversion at one time or other). The argument for this criterion is that distributed systems tend to make it easier to collaborate (because you can easily re-heal changes initiated at different times)... and since it's only SUGGESTED, it does not *mandate* decentralized version control.
It certainly makes it easier for unaffiliated contributors to collaborate, which can be good but is not necessarily a good thing strictly speaking. For core devs, it’s not true as I can take a team of devs, put them on a centralized repo (regardless of SCM) and establish commit norms that will be more efficient for that team’s collaboration (unquestionably at restricted capability with others, but therein derives the efficiency savings).

There are also other merits of a centralized repository in terms of positively controlling source code quality, conducting verification and validation, QA, etc. If a group is going for ISO 9000 or CMMI certification, this is arguably far easier to ensure, track, and reasonably prove with a centralized repository. You can have failure and successes in communication with both central and distributed.

That said, we could certainly drop this criterion. It's only SUGGESTED (which hints at your next point), so it doesn't have much "oomph" - and shortening the criteria a little bit is a good thing. I think we should *expect* that as we get more projects & experience there will improvements and tightening of the criteria. The *much* more important issue is to *have* version control, and I think it's generally agreed that version control should stay as a MUST.
Absolutely a MUST. I suggest half-merging that last one with the first criteria: “The project MUST have a version-controlled source repository using common version control software (e.g., git, svn), accessible via public URL.

Reasonable enough. I think the SUGGESTED items have some value, because psychologically people don't like to admit they don't do something (if they think they should be doing it). But I could be mistaken.
I might agree for most of the SHOULD items, but adding another level even less weak made me (as reader) dismiss their value even when fully met — you’re not willing to commit to them thus I have no reason to care. If I don’t meet one of them (e.g., use common build tools), I’m probably going to feel completely justified (my tools are superior). There’s nothing to “admit”.

That's certainly true in a sense. The point, though, is that the project has to *tell* reporters how to do the private reporting. A lot of projects just don't tell people how to report vulnerabilities (they've never considered the possibility), so the real goal is to get them to write it down ahead-of-time.
I gathered as much, so perhaps a change in wording to clarify: “If private vulnerability reports are supported, the project MUST publicly document the process for sending information privately.”

Well, I've seen some non-working build systems, but you're right, people don't want to *admit* they're not working.
We’ve all seen them, and most of the time the stream dev will simply (often with justification) claim the failure is a problem with the environment, an unsupported configuration, etc.

Would changing "Working" to "Automated" be an improvement?
I think what’s missing is to define either term. Maybe something like this:

"If the software requires building, the project MUST document supported environments, dependencies (libraries and tools) required for installation, and default build instructions that work automatically for those environments.

The problem is that it depends on the overall platform (including language and underlying OS) & the tools available. Making it a MUST would be too harsh for many cases. For example, you can find more problems by enabling more warnings, but those higher levels tend to be much noisier. We don't want to *discourage* people from using more sensitive (though noisier) tools.
Commented on this in the other e-mail reply to Daniel.

Cheers!
Sean

Join CII-badges@lists.coreinfrastructure.org to automatically receive all group messages.