Topics

I claim "bus factor" is mostly useless


Daniel Stenberg
 

Hi,

Since you've now added "bus factor" as a criteria, can someone please explain how this is valuable to users of open source projects?

All the availabe tools that determine this factor only run on code and check for commits or code churn etc. They can't tell the general "knowledge level" or how many people "understand" the code or particular areas? How many people read and deal with the code without sending back patches?

For a documented, proven, old, established open source project with an active maintainer, it is *easy* to have a bus factor that is less than two when checked by scripts like you encourage in the criteria, but yet if said person would be run over by a bus there's very little indications that the project would stall or die because of that.

The bus factor could perhaps be relevant if there were no docs, if the code was poorly written, uncommented or barely tested. But all those are *also* criteria...

So then. I hold up my finger and sense that more at least a few more people than me know my project. Is that a met criteria?

--

/ daniel.haxx.se


David A. Wheeler
 

Daniel Stenberg:
Since you've now added "bus factor" as a criteria...
First - just to put it one place, let met copy here the criteria text. This is SHOULD at silver, MUST at gold (it is *not* at passing). The text at silver:
The project SHOULD have a "bus factor" of 2 or more. [bus_factor]
Details: A "bus factor" (aka "truck factor") is the minimum number of project members that have to suddenly disappear from a project ("hit by a bus") before the project stalls due to lack of knowledgeable or competent personnel. The truck-factor tool can estimate this for projects on GitHub. For more information, see Assessing the Bus Factor of Git Repositories by Cosentino et al.

can someone please
explain how this is valuable to users of open source projects?
Absolutely!

It's valuable for users to know that even if the current lead died, the project could simply continue. The basic issue here is that we're all mortal (vulnerable to being "hit by a bus"). Even if the project is popular, that doesn't mean that it can be easily continued if the lead disappears.

All the availabe tools that determine this factor only run on code and check
for commits or code churn etc. They can't tell the general "knowledge level"
or how many people "understand" the code or particular areas? How many
people read and deal with the code without sending back patches?
True, but we don't have to limit ourselves to stuff that can be determined with automated tools. I do think that tools can help determine a likely value in this case.

For a documented, proven, old, established open source project with an
active maintainer, it is *easy* to have a bus factor that is less than two when
checked by scripts like you encourage in the criteria, but yet if said person
would be run over by a bus there's very little indications that the project
would stall or die because of that.
I like curl & often use it myself. So let's be specific: If you died (please don't any time soon), could someone just pick up & continue? Who?

The bus factor could perhaps be relevant if there were no docs, if the code
was poorly written, uncommented or barely tested. But all those are *also*
criteria...
I've had to pick up projects I've never seen before. It's absolutely true that docs, well-written code, and good tests are a big help. But having someone to ask, and work with at the beginning, is also a *huge* help.

I have plans to live on for a while longer (good plan, right?). But if something happened to me, I know that Dan Kohn & Jason Dossett are well-versed in the code & have all the necessary commit rights on this project.

So then. I hold up my finger and sense that more at least a few more people
than me know my project. Is that a met criteria?
I hope you'll stick around on this earth a while longer :-).

But if you suddenly died or were incapacitated, could someone else carry on? Who? Who knows the code well enough, and has the necessary rights, to continue? If you can identify at least one other person, then it's met.

--- David A. Wheeler


Daniel Stenberg
 

On Tue, 20 Jun 2017, Wheeler, David A wrote:

It's valuable for users to know that even if the current lead died, the project could simply continue. The basic issue here is that we're all mortal (vulnerable to being "hit by a bus"). Even if the project is popular, that doesn't mean that it can be easily continued if the lead disappears.
My claim is simply that you cannot figure out how easy, how hard or how likely that is based on this "bus factor" number. I claim that you instead wrongly fail to appreciate well-run small projects by this.

Okay, have a seat. I'll elaborate and lay out the whole story as I see it:

A very strong driver and motivator for people in open source is the well known "scratch your own itch", which also boils down to: if *nobody else* is scratching that itch - and it is annoying enough - I can do it. It is also very well known in open source (and elsewhere) that if someone else does the job, you don't have to do it. (No terribly surprising but I feel I need to spell it out clearly.)

How does this affect the reality of an open source project? It makes people follow this general pattern:

1 - if the project works fine and the products work fine, you're happy

2 - you find a problem, you submit a bug report or complain and hope that it
gets fixed soon. If it does, or you find a work-around, you're happy

3 - if the above steps weren't good enough, you consider actually debugging and working on a fix yourself. If the language/environment the project is using makes you feel comfortable enough. If it doesn't, you're back to mostly waiting and asking the project what else you can do to help.

4 - you debug and ideally end up submitting a patch to the project

5 - if the patch and your responses ot the patch were fine, the patch gets merged

From the project's point of view as well as the user's point of view, users should stay on level 1 or at worst level 2 in this table. The better the project works, the more users stay there. The fewer users that have to go down to 3 or more, the better for everyone involved.

It doesn't matter if that maintenance is done by a single person or an army of thousands. To the end user the satisfaction is independent of that number.

But okay, that's right now when the project runs fine and everyone is merry. What happens when things go sour. If a project has 51% of all commits or 51% of all lines of codes written by one or two persons (or however you want to calculate the bus factor).

Suddenly people will move along those "satisfaction levels" to a much higher degree and people will suddenly start to actually have to roll up their sleeves to do work. Then what is important for them to successfully do so? A working process (so that changes can be received and handled). Quality source code that isn't too hard to modify. Documentation that helps you understand how things work and what to change. Tests for making sure your changes don't cause too many regressions. Quite likely also you want a communication channel. Etc. All of that should work totally independent of the bus factor. If those things don't work, they are independent issues for the project and are failures that are unrelated to the bus factor.

We simply cannot know who would step up to do the work if the fearless leaders who did the 51% vanished. Since when people didn't have to do it before, they avoided it.

All the availabe tools that determine this factor only run on code and check for commits or code churn etc.
True, but we don't have to limit ourselves to stuff that can be determined with automated tools. I do think that tools can help determine a likely value in this case.
Well, this number is really impossible to figure out without tools and I don't see how tools can take "general knowledge" into account, or "this person answers a lot of email on the list", or this person has 48k in reputation on stack overflow already for responding to questions about the project.

So bus factor is about code(?) (amount/changes), which may or may not be a good indicator of who knows what about the code. Those who author and commit changes probably have a good idea but a real problem is that you can't reverse that view and say that just because you *didn't* commit or change something, you don't know.

We can't prove or assume lack of knowledge or interest by the lack of commits or changes.

I like curl & often use it myself. So let's be specific: If you died (please don't any time soon), could someone just pick up & continue? Who?
curl is soon 20 years old and boasts about 22k commits. I'm the author of about 57% of them, and the second-most committer (who's not involved anymore) has about 12%. Two committers at 15300 commits. If we for simplicity calculate bus factor based on commit numbers, we'd need 8580 commits from others and I should stop completely to reach bus factor >2 (when the 2 top committers have less than 50% of the commits), which at the current commit rate equals in about 5 years. So even when someone new comes into the project, they have a really hard time to significantly change the bus factor... Especially since my personal commit rate have not gone down (much).

We're approaching 1600 individually named contributors thanked in the project and every release we do (we ship one every 8 weeks) has around 40 contributors, out of which typically around half are newcomers. The long tail is very long and the amount of drive-by just-once contributors is high.

When we ask our users "why don't you contribute (more) to the project?" (which we do annually) what do they answer? They say its because 1) everything works, 2) I don't have time 3) things get fixed fast enough 4) I don't know the programming language 5) I don't have the energy.

First as the 6th answer (at 5% 2017) comes "other" where some people actually say they wouldn't know where to start and so on.

All of this taken together: no signs of us *suffering* from having a low bus factor. Lots of signs that people can do things when they want to if I don't do it. Lots of signs that the code and concepts are understood.

Lots of signs that bus factor is irrelevant here.

Do I *know* who would pick up the project and move on if I die today? No. We're a 100% volunteer-driven project. We're one of the world's most widely used software components (easily more than three billion instances and counting) but we don't know who'll be around tomorrow to work on it. I can't know because that's not how the project works.

Given the extremly wide use of our stuff, given the huge contributor base, given the vast amounts of documentation and tests I think it'll work out.

Now of course the "best practices" need to be generic but I really don't think our project is that unique and special in this regard. I have my own unique and special insight into this project and that's why I use it as an example here, but I'm convinced that there are many more in similar situations.

Just because you have a large bus factor doesn't necessarily make the project a better place to ask questions. We've seen projects in the past where N persons involved are all from the same company and when that company removes its support for that project those people all go away. High bus factor, no people to ask.

(Finally, let me just add that I would of course *love* to have many more committers and contributors, and I think we would be a better project if we did. But that's a separate issue.)

--

/ daniel.haxx.se


David A. Wheeler
 

On Tue, 20 Jun 2017, Wheeler, David A wrote:
It's valuable for users to know that even if the current lead died,
the project could simply continue. The basic issue here is that we're
all mortal (vulnerable to being "hit by a bus"). Even if the project
is popular, that doesn't mean that it can be easily continued if the lead
disappears.
Daniel Stenberg [mailto:daniel@haxx.se] on Wednesday, June 21, 2017 3:47 AM:
Subject: RE: [CII-badges] I claim "bus factor" is mostly useless

My claim is simply that you cannot figure out how easy, how hard or how
likely that is based on this "bus factor" number. I claim that you instead
wrongly fail to appreciate well-run small projects by this.

Okay, have a seat. I'll elaborate and lay out the whole story as I see it: ...
Seat taken! Sorry I haven't responded sooner, but this was a long email & I wanted to increase my likelihood of grokking it.

A very strong driver and motivator for people in open source is the well
known "scratch your own itch", which also boils down to: if *nobody else* is
scratching that itch - and it is annoying enough - I can do it. It is also very well
known in open source (and elsewhere) that if someone else does the job, you
don't have to do it. (No terribly surprising but I feel I need to spell it out
clearly.)...
Fair enough. Basically, people often get involved in OSS projects only if no one else is doing it. I *have* seen cases where people jump in, but I think your basic point is still sound in general - people tend to only do things if others won't do it for them.


... What happens when things go sour...
Suddenly people will move along those "satisfaction levels" to a much higher
degree and people will suddenly start to actually have to roll up their sleeves
to do work. Then what is important for them to successfully do so? A working
process (so that changes can be received and handled). Quality source code
that isn't too hard to modify. Documentation that helps you understand how
things work and what to change. Tests for making sure your changes don't
cause too many regressions. Quite likely also you want a communication
channel. Etc.
I think we're in total agreement on those things. The intent was that the "passing" level capture these key things; hopefully the passing level does so.

All of that should work totally independent of the bus factor.
If those things don't work, they are independent issues for the project and
are failures that are unrelated to the bus factor.

We simply cannot know who would step up to do the work if the fearless
leaders who did the 51% vanished. Since when people didn't have to do it
before, they avoided it.
I think I have a different mental model about "bus factor" than you do, and perhaps that explains things.

The definition of "bus factor" I like is the "number of key developers who would need to be incapacitated, i.e. hit by a bus, to send the project into such disarray that it would not be able to proceed". Source: M. Bowler, "Truck factor," Tech. Rep., May 2005. [Online]. Available: http://www.agileadvice.com/2005/05/15/agilemanagement/truck-factor/
This is also used in "Assessing the Bus Factor of Git Repositories", https://www.researchgate.net/publication/272794507_Assessing_the_Bus_Factor_of_Git_Repositories

To me, *that* is what matters: the ability to proceed. The quantitative measures are merely indicators, not what matters.

Well, this number is really impossible to figure out without tools and I don't
see how tools can take "general knowledge" into account, or "this person
answers a lot of email on the list", or this person has 48k in reputation on
stack overflow already for responding to questions about the project.
I think it should be relatively easy to figure out without tools. Many non-software organizations can tell you who would take over if the leader died, and have a rational confidence that the successor knows enough to take over. If you can't easily identify someone else who knows things, then the tools will probably tell you what you already know.

To be honest, I really had a judgement call in mind here, not a quantitative measure.

Of course, there are various quantitative methods for estimating the bus factor, e.g., the aforementioned "Assessing the Bus Factor of Git Repositories", https://www.researchgate.net/publication/272794507_Assessing_the_Bus_Factor_of_Git_Repositories
That's not the only one - there are several ways to estimate bus factor (that paper cites some of them).

That said, I think the these quantitative measures are simply algorithms to *estimate* the real issue: Could the project proceed?


We can't prove or assume lack of knowledge or interest by the lack of
commits or changes.
Agreed.

I like curl & often use it myself. So let's be specific: If you died
(please don't any time soon), could someone just pick up & continue?
Who?

curl is soon 20 years old and boasts about 22k commits. I'm the author of
about 57% of them, and the second-most committer (who's not involved
anymore) has about 12%. Two committers at 15300 commits. If we for
simplicity calculate bus factor based on commit numbers, we'd need 8580
commits from others and I should stop completely to reach bus factor >2
(when the 2 top committers have less than 50% of the commits), which at
the current commit rate equals in about 5 years. So even when someone new
comes into the project, they have a really hard time to significantly change
the bus factor... Especially since my personal commit rate have not gone
down (much).
I think the measure you're using for bus factor is excessively harsh... which also explains why you're objecting to using it :-).

The *main* text simply emphasizes a "bus factor" of 2 or more, and the rest are details to try to explain it. Perhaps the problem *is* the use of tools. I think the real point is that "there is a successor if the 'main' contributor unexpectedly dies" & "that successor knows enough to be able to continue the project". The tools for analyzing commits are merely a way to help estimate the knowledge, but as you note, they often don't tell the whole story.


... Now of course the "best practices" need to be generic but I really don't think
our project is that unique and special in this regard. I have my own unique
and special insight into this project and that's why I use it as an example
here, but I'm convinced that there are many more in similar situations.

Just because you have a large bus factor doesn't necessarily make the project
a better place to ask questions. We've seen projects in the past where N
persons involved are all from the same company and when that company
removes its support for that project those people all go away. High bus
factor, no people to ask.
Agreed, but we got you covered there with a "gold" criterion, "contributors_unassociated":
"The project MUST have at least two unassociated significant contributors."

I believe that criterion as even *harder* to achieve on some projects, which is why it's only at the gold level.

(Finally, let me just add that I would of course *love* to have many more
committers and contributors, and I think we would be a better project if we
did. But that's a separate issue.)
Sure. Letting non-terrorists board international flights would be a good thing too :-).

The real issue, as I see it, is that the project needs to be able proceed if any 1 person dies or otherwise becomes unavailable. If we focused more on being able to identify a second person, and ask for some evidence that the second person knows enough for the project to be able to proceed (without overly-picky quantitative measures), would that be an improvement?

--- David A. Wheeler


Daniel Stenberg
 

On Fri, 30 Jun 2017, Wheeler, David A wrote:

To be honest, I really had a judgement call in mind here, not a quantitative measure.
Sure OK, but that's also what I object to. That's a totally, completely and all-through subjective measurement that also most likely can be highly debatable in projects. And I'm supposed to give a binary answer to it.

I claim this makes it a criteria with a too weak foundation, thus making it useless.

I did not understand this "estimate it with your gut" nature from the criteria description.

--

/ daniel.haxx.se