Date   
Re: CII Best Practices co-developers - need to update Ruby & gems

Dan Kohn
 

On a Mac, it's:

brew update && brew upgrade ruby-build
rbenv install 2.4.2
gem install bundler
bundle

--
Dan Kohn <dan@...>
Executive Director, Cloud Native Computing Foundation https://www.cncf.io
+1-415-233-1000 https://www.dankohn.com

On Sun, Oct 22, 2017 at 5:27 PM, Wheeler, David A <dwheeler@...> wrote:

FYI:

 

If you are creating proposed changes to the CII Best Practices Badge code, I’ve recently updated the Ruby implementation (as well as all the gems I can).  So if you’ve had the badge application running, the next time after doing a “git pull” you’ll need to run this (in its directory):

 

./ruby-update   # Update Ruby

bundle install   # Install current gems

 

This is documented in the CONTRIBUTING.md file, but I thought I’d point this out here, because I want to make it easy for people to propose changes.

 

Thanks!

 

--- David A. Wheeler

 


_______________________________________________
CII-badges mailing list
CII-badges@lists.coreinfrastructure.org
https://lists.coreinfrastructure.org/mailman/listinfo/cii-badges


Re: Daniel Stenberg won the Polhem Prize for his work on curl!

Daniel Stenberg
 

On Fri, 20 Oct 2017, Wheeler, David A wrote:

I'd just like to publicly congratulate Daniel Stenberg for winning the Polhem Prize for his work on curl! The Polhem Prize is awarded "for a high-level technological innovation or an ingenious solution to a technical problem."
Thank you!

--

/ daniel.haxx.se

CII Best Practices co-developers - need to update Ruby & gems

David A. Wheeler
 

FYI:

 

If you are creating proposed changes to the CII Best Practices Badge code, I’ve recently updated the Ruby implementation (as well as all the gems I can).  So if you’ve had the badge application running, the next time after doing a “git pull” you’ll need to run this (in its directory):

 

./ruby-update   # Update Ruby

bundle install   # Install current gems

 

This is documented in the CONTRIBUTING.md file, but I thought I’d point this out here, because I want to make it easy for people to propose changes.

 

Thanks!

 

--- David A. Wheeler

 

Daniel Stenberg won the Polhem Prize for his work on curl!

David A. Wheeler
 

All:

 

I’d just like to publicly congratulate Daniel Stenberg for winning the Polhem Prize for his work on curl!  The Polhem Prize is awarded “for a high-level technological innovation or an ingenious solution to a technical problem.”

 

As you know, curl was one of the early projects in the badging project.  Daniel’s commentary & insights have been very helpful to the badging project, and it’s nice to see him honored elsewhere as well.

 

More info here:

https://daniel.haxx.se/blog/2017/10/16/polhemspriset-2017/

https://daniel.haxx.se/blog/2017/10/20/my-night-at-the-museum/

 

--- David A. Wheeler

 

CII Best Practices Badge, 1.5 years later

David A. Wheeler
 

At the Linux Security Summit 2017 I gave a presentation titled “CII Best Practices Badge, 1.5 years later”.  It’s basically a status report about the CII Best Practices badging project.

 

A few highlights:

* We continue to grow (e.g., in the number of participating projects and the number passing badges).

* In general, the number of passing projects seems to be consistently about 10% of the participating projects.  I’m not sure why!

* The most commonly-missed criteria (at first), among projects close to passing, are now vulnerability_report_process (make sure you tell people how to report vulnerabilities) and sites_https_status (HTTPS).

* The criterion tests_are_added (tests are added for new major functionality) is the third most likely problem (it *was* #1).  Hopefully that’s because more projects are working to create & maintain test suites, but trends are always suspicious when there are only 2 data points J.

* Most importantly: We’re making a difference.  OSS projects are reporting that they’re changing things in their projects to meet the criteria, and as a result improving their project.  In many cases, they kept putting it off (e.g., test suites and HTTPS).  In other cases, they just didn’t think of it (e.g., telling people how to report vulnerabilities).  None of this guarantees that there will never be vulnerabilities, of course, but taking care of these things makes it easier to produce software more resistant to attack & more responsive when vulnerabilities are found – and that helps all of us.

 

You can see much more here:

  http://events.linuxfoundation.org/sites/events/files/slides/cii-bp-badge-2017-09_0.pdf

 

Once again, I want to say THANK YOU VERY MUCH for everyone who has participated in this badging project.  No doubt there are improvements that can be made and need to be made.  That’s true for any project J.  But in my mind, the right test for a project is, “is the world a better place because this project exists?”  I believe the badging project is passing that test with flying colors, and that could not have happened without you.  Thank you very much for ALL the effort that ALL of you have put into the badging project.

 

--- David A. Wheeler

 

Dan Kohn on FLOSS Weekly, & CIL providing free computing power

David A. Wheeler
 

FYI:

 

Dan Kohn was instrumental in getting the CII Best Practices Badge project up-and-running.  Dan is now the Executive Director of the Cloud Native Computing Foundation, which sustains and integrates open source technologies like Kubernetes and Prometheus.  He was recently interviewed on “FLOSS Weekly”, and I suspect many of you will be interested in that interview:

  https://twit.tv/shows/floss-weekly/episodes/452

 

In particular, the CNCF Community Infrastructure Lab (CIL) has free access to state-of-the-art computing resources for open source developers working to advance cloud native computing. They offer access to both x86 and ARMv8 bare metal servers.  If you happen to need some real computing power, and it relates to open source software, this might be the free capability you’ve been looking for.  For example, it could be used for software builds, continuous integration, scale testing, or demonstrations.  Dan’s of the opinion that it’d be sad to see this free capability go unused, and I agree, so please let them know if you need something like this.  More info here:

  https://www.cncf.io/community/infrastructure-lab/

 

--- David A. Wheeler

 

Projects that received badges (monthly summary)

badgeapp@...
 

This is an automated monthly status report of the best practices badge application covering the month 2017-09.

Here are some selected statistics for most recent completed month, preceded by the same statistics for the end of the month before that.

Ending dates2017-08-302017-09-29
Total Projects10031049
Projects 25%+371388
Projects 50%+303321
Projects 75%+239251
Projects passing103110

Here are the projects that first achieved a passing badge in 2017-09:

  1. bolt at 2017-09-01 20:15:38 UTC
  2. User mode file system library for windows with FUSE Wrapper at 2017-09-01 20:27:25 UTC
  3. CoreDNS at 2017-09-12 13:33:53 UTC
  4. C++ front/service proxy at 2017-09-19 22:51:05 UTC
  5. chrony at 2017-09-26 15:00:02 UTC
  6. ibmkr at 2017-09-27 04:57:44 UTC

We congratulate them all!

Do you know a project that doesn't have a badge yet? Please suggest to them that they get a badge now!

Badging project continuing to grow!

David A. Wheeler
 

All – there is continuous steady growth in badging project participation.  We have over 1000 participating projects, and over 100 badges.  I’m not sure why 10% of the participating projects at any one time have a badge, but it’s been approximately true for some time.

 

I gave a presentation about the badging project at the 2017 Linux Security Summit; slides here:

  http://events.linuxfoundation.org/events/linux-security-summit

 

Steady growth isn’t something that “jumps out” at you, but I thought you’d like to know!

 

I thank *everyone* here for your help and insights.

 

--- David A. Wheeler

 

Projects that received badges (monthly summary)

badgeapp@...
 

This is an automated monthly status report of the best practices badge application covering the month 2017-08.

Here are some selected statistics for most recent completed month, preceded by the same statistics for the end of the month before that.

Ending dates2017-07-302017-08-30
Total Projects9341003
Projects 25%+348371
Projects 50%+285303
Projects 75%+224239
Projects passing95103

Here are the projects that first achieved a passing badge in 2017-08:

  1. Kubernetes at 2017-08-16 14:52:28 UTC
  2. dash at 2017-08-21 15:29:31 UTC
  3. libpki at 2017-08-24 17:43:06 UTC
  4. LDAP Tool Box Self Service Password at 2017-08-30 12:44:34 UTC

We congratulate them all!

Do you know a project that doesn't have a badge yet? Please suggest to them that they get a badge now!

Projects that received badges (monthly summary)

badgeapp@...
 

This is an automated monthly status report of the best practices badge application covering the month 2017-07.

Here are some selected statistics for most recent completed month, preceded by the same statistics for the end of the month before that.

Ending dates2017-06-292017-07-30
Total Projects877934
Projects 25%+328348
Projects 50%+269285
Projects 75%+213224
Projects passing8895

Here are the projects that first achieved a passing badge in 2017-07:

  1. umoci at 2017-07-02 13:10:18 UTC
  2. lxd at 2017-07-03 16:34:14 UTC
  3. LXC - Linux Containers at 2017-07-03 16:56:04 UTC
  4. A library for exploring persistent homology at 2017-07-05 10:33:27 UTC
  5. Viua Virtual Machine at 2017-07-08 19:09:29 UTC
  6. ruamel.yaml at 2017-07-18 17:01:35 UTC
  7. QtOSG at 2017-07-25 13:35:57 UTC

We congratulate them all!

Do you know a project that doesn't have a badge yet? Please suggest to them that they get a badge now!

Projects that received badges (monthly summary)

badgeapp@...
 

This is an automated monthly status report of the best practices badge application covering the month 2017-06.

Here are some selected statistics for most recent completed month, preceded by the same statistics for the end of the month before that.

Ending dates2017-05-302017-06-29
Total Projects820877
Projects 25%+306328
Projects 50%+250269
Projects 75%+195213
Projects passing8288

Here are the projects that first achieved a passing badge in 2017-06:

  1. Cypht at 2017-06-03 19:01:49 UTC
  2. Iroha - A simple, decentralized ledger at 2017-06-11 02:59:07 UTC
  3. OWASP dependency-check at 2017-06-19 11:00:16 UTC
  4. The Prometheus monitoring system and time series database. at 2017-06-23 08:01:41 UTC
  5. naxsi at 2017-06-23 13:38:29 UTC
  6. LDAPImporter at 2017-06-29 18:38:08 UTC

We congratulate them all!

Do you know a project that doesn't have a badge yet? Please suggest to them that they get a badge now!

Re: I claim "bus factor" is mostly useless

Daniel Stenberg
 

On Fri, 30 Jun 2017, Wheeler, David A wrote:

To be honest, I really had a judgement call in mind here, not a quantitative measure.
Sure OK, but that's also what I object to. That's a totally, completely and all-through subjective measurement that also most likely can be highly debatable in projects. And I'm supposed to give a binary answer to it.

I claim this makes it a criteria with a too weak foundation, thus making it useless.

I did not understand this "estimate it with your gut" nature from the criteria description.

--

/ daniel.haxx.se

Re: I claim "bus factor" is mostly useless

David A. Wheeler
 

On Tue, 20 Jun 2017, Wheeler, David A wrote:
It's valuable for users to know that even if the current lead died,
the project could simply continue. The basic issue here is that we're
all mortal (vulnerable to being "hit by a bus"). Even if the project
is popular, that doesn't mean that it can be easily continued if the lead
disappears.
Daniel Stenberg [mailto:daniel@...] on Wednesday, June 21, 2017 3:47 AM:
Subject: RE: [CII-badges] I claim "bus factor" is mostly useless

My claim is simply that you cannot figure out how easy, how hard or how
likely that is based on this "bus factor" number. I claim that you instead
wrongly fail to appreciate well-run small projects by this.

Okay, have a seat. I'll elaborate and lay out the whole story as I see it: ...
Seat taken! Sorry I haven't responded sooner, but this was a long email & I wanted to increase my likelihood of grokking it.

A very strong driver and motivator for people in open source is the well
known "scratch your own itch", which also boils down to: if *nobody else* is
scratching that itch - and it is annoying enough - I can do it. It is also very well
known in open source (and elsewhere) that if someone else does the job, you
don't have to do it. (No terribly surprising but I feel I need to spell it out
clearly.)...
Fair enough. Basically, people often get involved in OSS projects only if no one else is doing it. I *have* seen cases where people jump in, but I think your basic point is still sound in general - people tend to only do things if others won't do it for them.


... What happens when things go sour...
Suddenly people will move along those "satisfaction levels" to a much higher
degree and people will suddenly start to actually have to roll up their sleeves
to do work. Then what is important for them to successfully do so? A working
process (so that changes can be received and handled). Quality source code
that isn't too hard to modify. Documentation that helps you understand how
things work and what to change. Tests for making sure your changes don't
cause too many regressions. Quite likely also you want a communication
channel. Etc.
I think we're in total agreement on those things. The intent was that the "passing" level capture these key things; hopefully the passing level does so.

All of that should work totally independent of the bus factor.
If those things don't work, they are independent issues for the project and
are failures that are unrelated to the bus factor.

We simply cannot know who would step up to do the work if the fearless
leaders who did the 51% vanished. Since when people didn't have to do it
before, they avoided it.
I think I have a different mental model about "bus factor" than you do, and perhaps that explains things.

The definition of "bus factor" I like is the "number of key developers who would need to be incapacitated, i.e. hit by a bus, to send the project into such disarray that it would not be able to proceed". Source: M. Bowler, "Truck factor," Tech. Rep., May 2005. [Online]. Available: http://www.agileadvice.com/2005/05/15/agilemanagement/truck-factor/
This is also used in "Assessing the Bus Factor of Git Repositories", https://www.researchgate.net/publication/272794507_Assessing_the_Bus_Factor_of_Git_Repositories

To me, *that* is what matters: the ability to proceed. The quantitative measures are merely indicators, not what matters.

Well, this number is really impossible to figure out without tools and I don't
see how tools can take "general knowledge" into account, or "this person
answers a lot of email on the list", or this person has 48k in reputation on
stack overflow already for responding to questions about the project.
I think it should be relatively easy to figure out without tools. Many non-software organizations can tell you who would take over if the leader died, and have a rational confidence that the successor knows enough to take over. If you can't easily identify someone else who knows things, then the tools will probably tell you what you already know.

To be honest, I really had a judgement call in mind here, not a quantitative measure.

Of course, there are various quantitative methods for estimating the bus factor, e.g., the aforementioned "Assessing the Bus Factor of Git Repositories", https://www.researchgate.net/publication/272794507_Assessing_the_Bus_Factor_of_Git_Repositories
That's not the only one - there are several ways to estimate bus factor (that paper cites some of them).

That said, I think the these quantitative measures are simply algorithms to *estimate* the real issue: Could the project proceed?


We can't prove or assume lack of knowledge or interest by the lack of
commits or changes.
Agreed.

I like curl & often use it myself. So let's be specific: If you died
(please don't any time soon), could someone just pick up & continue?
Who?

curl is soon 20 years old and boasts about 22k commits. I'm the author of
about 57% of them, and the second-most committer (who's not involved
anymore) has about 12%. Two committers at 15300 commits. If we for
simplicity calculate bus factor based on commit numbers, we'd need 8580
commits from others and I should stop completely to reach bus factor >2
(when the 2 top committers have less than 50% of the commits), which at
the current commit rate equals in about 5 years. So even when someone new
comes into the project, they have a really hard time to significantly change
the bus factor... Especially since my personal commit rate have not gone
down (much).
I think the measure you're using for bus factor is excessively harsh... which also explains why you're objecting to using it :-).

The *main* text simply emphasizes a "bus factor" of 2 or more, and the rest are details to try to explain it. Perhaps the problem *is* the use of tools. I think the real point is that "there is a successor if the 'main' contributor unexpectedly dies" & "that successor knows enough to be able to continue the project". The tools for analyzing commits are merely a way to help estimate the knowledge, but as you note, they often don't tell the whole story.


... Now of course the "best practices" need to be generic but I really don't think
our project is that unique and special in this regard. I have my own unique
and special insight into this project and that's why I use it as an example
here, but I'm convinced that there are many more in similar situations.

Just because you have a large bus factor doesn't necessarily make the project
a better place to ask questions. We've seen projects in the past where N
persons involved are all from the same company and when that company
removes its support for that project those people all go away. High bus
factor, no people to ask.
Agreed, but we got you covered there with a "gold" criterion, "contributors_unassociated":
"The project MUST have at least two unassociated significant contributors."

I believe that criterion as even *harder* to achieve on some projects, which is why it's only at the gold level.

(Finally, let me just add that I would of course *love* to have many more
committers and contributors, and I think we would be a better project if we
did. But that's a separate issue.)
Sure. Letting non-terrorists board international flights would be a good thing too :-).

The real issue, as I see it, is that the project needs to be able proceed if any 1 person dies or otherwise becomes unavailable. If we focused more on being able to identify a second person, and ask for some evidence that the second person knows enough for the project to be able to proceed (without overly-picky quantitative measures), would that be an improvement?

--- David A. Wheeler

Automatic locale selection (will change URLs by adding /en/ to English URLs)

David A. Wheeler
 

We now have complete localizations for French, (Simplified) Chinese, and Japanese. German and Russian are past the halfway mark as well. My sincere thanks to all the translators!!

It now makes sense to *automatically* redirect people to their preferred locales. Unfortunately, there's currently no way to distinguish between "users who want English" and "users who want their preferred locale".

As a result, we need to make *some* change to our URLs so that automatic locale selection can work. The current plan is that if you provide a URL *without* a specific locale identified, you'll be redirected to a URL with a locale (it'll be whatever your browser most prefers that we support, and English if none are supported). As a result, if you visit:
https://bestpractices.coreinfrastructure.org/projects/1
and the best fit is English (en), you'll be redirected to:
https://bestpractices.coreinfrastructure.org/en/projects/1

Similarly, a visit to the home page:
https://bestpractices.coreinfrastructure.org/
Will redirect you. If the best fit is English (en), you'll be redirected to:
https://bestpractices.coreinfrastructure.org/en
but you prefer simplified Chinese, you'll instead be redirected to:
https://bestpractices.coreinfrastructure.org/zh-CN

This should work out well. People can, at any time, select their preferred locale on the website (as they do now) and it will continue to work. There seems to be a general consensus that the actual locale should be part of the URL, and *not* be stored in a session or cookie, so that a particular URL will lead to the same material. This also avoids locale selection by geolocation, which is widely considered a bad idea (not everyone in a region shares the same locale preference - at the very least, when you're on travel *you* should be able to choose). More discussion about this approach is here:
https://github.com/coreinfrastructure/best-practices-badge/pull/805

One exception: the plan is that the badge images (e.g., /projects/1/badge) will *not* have a locale prefix. They're constant regardless of locale, and making them locale-less means that they'll cache better (through better sharing).

--- David A. Wheeler

Re: documentation_roadmap

David A. Wheeler
 

On Tue, 20 Jun 2017, Wheeler, David A wrote:
Some projects don't plan to make any changes; some plan to make modest
changes; some plan to rewrite everything & the API is going out the window.
It's helpful for users to know where the project intends to go. Not
everything goes as planned, but at least the user has an inkling.
Does that help?

Daniel Stenberg:
Yes, thanks, that helps. It does however make me question the value of the
criteria.
*Please* feel free to question! It's why this mailing list exists :-).


It asks for a document to be present that lists things that a project
might or might not do at some point.
- project A lists things in the roadmap for several years that haven't
been implemented
- project B doesn't have a roadmap document but develop things
Project A is still considered having the (much) better practice? Why?
I think this quote sums up things:
"Plans are useless, but planning is indispensable." - Dwight D. Eisenhower

As expanded here from DrupalCon:
https://events.drupal.org/baltimore2017/sessions/plans-are-useless-planning-indispensable
"The value of a plan lies in the act and effort of planning: in doing so, you gain understanding ... Planning encourages situational awareness through learning and discussing strengths, weaknesses, opportunities, threats and a flexible strategy that can tie them all together. A good plan establishes a vision, with goals and supporting objectives, and provides context and directionality so the team can move forward and be supported without being locked in."

Clearly execution is *way* more important than just planning. But planning, and sharing your thoughts with collaborators, can increase the likelihood that everyone will work in the same (general) direction. Even just saying "we're in maintenance mode, and focusing on small incremental improvements" can help people understand things.

Also: In principle this criterion isn't difficult. Just figure out your current plan & write it down. The *real* challenge is the planning (to figure out your plan). The hope is that this criterion will encourage people to think that through; it encourages people to step back & think about the larger goals for the project.

--- David A. Wheeler

Re: documentation_roadmap

Daniel Stenberg
 

On Tue, 20 Jun 2017, Wheeler, David A wrote:

Some projects don't plan to make any changes; some plan to make modest changes; some plan to rewrite everything & the API is going out the window. It's helpful for users to know where the project intends to go. Not everything goes as planned, but at least the user has an inkling.

Does that help?
Yes, thanks, that helps. It does however make me question the value of the criteria. It asks for a document to be present that lists things that a project might or might not do at some point.

- project A lists things in the roadmap for several years that haven't
been implemented

- project B doesn't have a roadmap document but develop things

Project A is still considered having the (much) better practice? Why?

--

/ daniel.haxx.se

Re: I claim "bus factor" is mostly useless

Daniel Stenberg
 

On Tue, 20 Jun 2017, Wheeler, David A wrote:

It's valuable for users to know that even if the current lead died, the project could simply continue. The basic issue here is that we're all mortal (vulnerable to being "hit by a bus"). Even if the project is popular, that doesn't mean that it can be easily continued if the lead disappears.
My claim is simply that you cannot figure out how easy, how hard or how likely that is based on this "bus factor" number. I claim that you instead wrongly fail to appreciate well-run small projects by this.

Okay, have a seat. I'll elaborate and lay out the whole story as I see it:

A very strong driver and motivator for people in open source is the well known "scratch your own itch", which also boils down to: if *nobody else* is scratching that itch - and it is annoying enough - I can do it. It is also very well known in open source (and elsewhere) that if someone else does the job, you don't have to do it. (No terribly surprising but I feel I need to spell it out clearly.)

How does this affect the reality of an open source project? It makes people follow this general pattern:

1 - if the project works fine and the products work fine, you're happy

2 - you find a problem, you submit a bug report or complain and hope that it
gets fixed soon. If it does, or you find a work-around, you're happy

3 - if the above steps weren't good enough, you consider actually debugging and working on a fix yourself. If the language/environment the project is using makes you feel comfortable enough. If it doesn't, you're back to mostly waiting and asking the project what else you can do to help.

4 - you debug and ideally end up submitting a patch to the project

5 - if the patch and your responses ot the patch were fine, the patch gets merged

From the project's point of view as well as the user's point of view, users should stay on level 1 or at worst level 2 in this table. The better the project works, the more users stay there. The fewer users that have to go down to 3 or more, the better for everyone involved.

It doesn't matter if that maintenance is done by a single person or an army of thousands. To the end user the satisfaction is independent of that number.

But okay, that's right now when the project runs fine and everyone is merry. What happens when things go sour. If a project has 51% of all commits or 51% of all lines of codes written by one or two persons (or however you want to calculate the bus factor).

Suddenly people will move along those "satisfaction levels" to a much higher degree and people will suddenly start to actually have to roll up their sleeves to do work. Then what is important for them to successfully do so? A working process (so that changes can be received and handled). Quality source code that isn't too hard to modify. Documentation that helps you understand how things work and what to change. Tests for making sure your changes don't cause too many regressions. Quite likely also you want a communication channel. Etc. All of that should work totally independent of the bus factor. If those things don't work, they are independent issues for the project and are failures that are unrelated to the bus factor.

We simply cannot know who would step up to do the work if the fearless leaders who did the 51% vanished. Since when people didn't have to do it before, they avoided it.

All the availabe tools that determine this factor only run on code and check for commits or code churn etc.
True, but we don't have to limit ourselves to stuff that can be determined with automated tools. I do think that tools can help determine a likely value in this case.
Well, this number is really impossible to figure out without tools and I don't see how tools can take "general knowledge" into account, or "this person answers a lot of email on the list", or this person has 48k in reputation on stack overflow already for responding to questions about the project.

So bus factor is about code(?) (amount/changes), which may or may not be a good indicator of who knows what about the code. Those who author and commit changes probably have a good idea but a real problem is that you can't reverse that view and say that just because you *didn't* commit or change something, you don't know.

We can't prove or assume lack of knowledge or interest by the lack of commits or changes.

I like curl & often use it myself. So let's be specific: If you died (please don't any time soon), could someone just pick up & continue? Who?
curl is soon 20 years old and boasts about 22k commits. I'm the author of about 57% of them, and the second-most committer (who's not involved anymore) has about 12%. Two committers at 15300 commits. If we for simplicity calculate bus factor based on commit numbers, we'd need 8580 commits from others and I should stop completely to reach bus factor >2 (when the 2 top committers have less than 50% of the commits), which at the current commit rate equals in about 5 years. So even when someone new comes into the project, they have a really hard time to significantly change the bus factor... Especially since my personal commit rate have not gone down (much).

We're approaching 1600 individually named contributors thanked in the project and every release we do (we ship one every 8 weeks) has around 40 contributors, out of which typically around half are newcomers. The long tail is very long and the amount of drive-by just-once contributors is high.

When we ask our users "why don't you contribute (more) to the project?" (which we do annually) what do they answer? They say its because 1) everything works, 2) I don't have time 3) things get fixed fast enough 4) I don't know the programming language 5) I don't have the energy.

First as the 6th answer (at 5% 2017) comes "other" where some people actually say they wouldn't know where to start and so on.

All of this taken together: no signs of us *suffering* from having a low bus factor. Lots of signs that people can do things when they want to if I don't do it. Lots of signs that the code and concepts are understood.

Lots of signs that bus factor is irrelevant here.

Do I *know* who would pick up the project and move on if I die today? No. We're a 100% volunteer-driven project. We're one of the world's most widely used software components (easily more than three billion instances and counting) but we don't know who'll be around tomorrow to work on it. I can't know because that's not how the project works.

Given the extremly wide use of our stuff, given the huge contributor base, given the vast amounts of documentation and tests I think it'll work out.

Now of course the "best practices" need to be generic but I really don't think our project is that unique and special in this regard. I have my own unique and special insight into this project and that's why I use it as an example here, but I'm convinced that there are many more in similar situations.

Just because you have a large bus factor doesn't necessarily make the project a better place to ask questions. We've seen projects in the past where N persons involved are all from the same company and when that company removes its support for that project those people all go away. High bus factor, no people to ask.

(Finally, let me just add that I would of course *love* to have many more committers and contributors, and I think we would be a better project if we did. But that's a separate issue.)

--

/ daniel.haxx.se

Re: documentation_roadmap

David A. Wheeler
 

So for silver level, a project MUST have a "roadmap".

Does the project also have to actually implement or work on any of the items
in the roadmap? If so, to what degree? If not, what's the purpose of this
criteria?
Text:
The project MUST have a documented roadmap that describes what the project intends to do and not do for at least the next year. [documentation_roadmap]

We tried to define the purpose of this criterion in the details:
Details: The project might not achieve the roadmap, and that's fine; the purpose of the roadmap is to help potential users and constributors understand the intended direction of the project. It need not be detailed.

Some projects don't plan to make any changes; some plan to make modest changes; some plan to rewrite everything & the API is going out the window. It's helpful for users to know where the project intends to go. Not everything goes as planned, but at least the user has an inkling.

Does that help?

It might be useful to look at the BadgeApp's roadmap:
https://github.com/coreinfrastructure/best-practices-badge/blob/master/doc/roadmap.md

You'll notice that at this point we do *not* have grand plans for the BadgeApp - we wanted to add internationalization & higher levels, and did that. At this point we're expecting to do more refinements than big plans, and that's fine... just let users know that.

--- David A. Wheeler

Re: I claim "bus factor" is mostly useless

David A. Wheeler
 

Daniel Stenberg:
Since you've now added "bus factor" as a criteria...
First - just to put it one place, let met copy here the criteria text. This is SHOULD at silver, MUST at gold (it is *not* at passing). The text at silver:
The project SHOULD have a "bus factor" of 2 or more. [bus_factor]
Details: A "bus factor" (aka "truck factor") is the minimum number of project members that have to suddenly disappear from a project ("hit by a bus") before the project stalls due to lack of knowledgeable or competent personnel. The truck-factor tool can estimate this for projects on GitHub. For more information, see Assessing the Bus Factor of Git Repositories by Cosentino et al.

can someone please
explain how this is valuable to users of open source projects?
Absolutely!

It's valuable for users to know that even if the current lead died, the project could simply continue. The basic issue here is that we're all mortal (vulnerable to being "hit by a bus"). Even if the project is popular, that doesn't mean that it can be easily continued if the lead disappears.

All the availabe tools that determine this factor only run on code and check
for commits or code churn etc. They can't tell the general "knowledge level"
or how many people "understand" the code or particular areas? How many
people read and deal with the code without sending back patches?
True, but we don't have to limit ourselves to stuff that can be determined with automated tools. I do think that tools can help determine a likely value in this case.

For a documented, proven, old, established open source project with an
active maintainer, it is *easy* to have a bus factor that is less than two when
checked by scripts like you encourage in the criteria, but yet if said person
would be run over by a bus there's very little indications that the project
would stall or die because of that.
I like curl & often use it myself. So let's be specific: If you died (please don't any time soon), could someone just pick up & continue? Who?

The bus factor could perhaps be relevant if there were no docs, if the code
was poorly written, uncommented or barely tested. But all those are *also*
criteria...
I've had to pick up projects I've never seen before. It's absolutely true that docs, well-written code, and good tests are a big help. But having someone to ask, and work with at the beginning, is also a *huge* help.

I have plans to live on for a while longer (good plan, right?). But if something happened to me, I know that Dan Kohn & Jason Dossett are well-versed in the code & have all the necessary commit rights on this project.

So then. I hold up my finger and sense that more at least a few more people
than me know my project. Is that a met criteria?
I hope you'll stick around on this earth a while longer :-).

But if you suddenly died or were incapacitated, could someone else carry on? Who? Who knows the code well enough, and has the necessary rights, to continue? If you can identify at least one other person, then it's met.

--- David A. Wheeler

documentation_roadmap

Daniel Stenberg
 

Hi,

So for silver level, a project MUST have a "roadmap".

Does the project also have to actually implement or work on any of the items in the roadmap? If so, to what degree? If not, what's the purpose of this criteria?

I think this is a bad criteria.

--

/ daniel.haxx.se