Date   
NTP

john s
 

as an aside, from Kit:

The NTP problem is really bugging me.  
Not even recognized as a top-10 ‘risk’ as defined by the CII Census (https://www.coreinfrastructure.org/programs/census-project) with a really low popularity rating and 0 committers - yikes.  If it is unreal to believe that not one of the main Linux/UNIX distros wouldn’t pay somebody to be a project lead for that thing.  Then there’s this:  http://netpatterns.blogspot.com/2016/01/the-rising-sophistication-of-network.html

-------------------------------------------
John Scott
 240.401.6574
< jms3rd@... >
http://powdermonkey.blogs.com
@johnmscott

On January 15, 2016 at 9:48:50 AM, Wheeler, David A (dwheeler@...) wrote:

Sebastian Benthall:
> One thing I'm trying to get a sense of (and I still need to read the paper very thoroughly to find out) is what exactly the "risk" you a measuring is risk of. That would make it easier to identify ground truth or proxies for it in existing data.

The title of the supporting paper gives that away: "Open Source Software Projects Needing Security Investments". The CII project was started, in part, as a response to the Heartbleed vulnerability of OpenSSL. We're trying to determine what projects are more likely to have serious vulnerabilities and investment is needed.


> Is there a record of the anomalies and the adjustments?

A high-level discussion is in the paper. See the git log for a record of many of the actual adjustments (the commit text should give you at least a brief reason as to *why* they were adjusted). I don’t think all adjustments we tried are recorded in the git log, since we weren't particularly trying to do that (sorry). But I think you'll find lots of useful information.


> Is there any sort of formal procedure for further expert review?
> I would be interested in designing such a procedure if there isn't one.

No, there's no formal procedure. You can propose one.

That said, we're happy to take good ideas from anyone, even if they're not perceived as experts.

--- David A. Wheeler

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census

NTP

john s
 

as an aside, from Kit:

The NTP problem is really bugging me.  
Not even recognized as a top-10 ‘risk’ as defined by the CII Census (https://www.coreinfrastructure.org/programs/census-project) with a really low popularity rating and 0 committers - yikes.  If it is unreal to believe that not one of the main Linux/UNIX distros wouldn’t pay somebody to be a project lead for that thing.  Then there’s this:  http://netpatterns.blogspot.com/2016/01/the-rising-sophistication-of-network.html

-------------------------------------------
John Scott
 240.401.6574
< jms3rd@... >
http://powdermonkey.blogs.com
@johnmscott

On January 15, 2016 at 9:48:50 AM, Wheeler, David A (dwheeler@...) wrote:

Sebastian Benthall:
> One thing I'm trying to get a sense of (and I still need to read the paper very thoroughly to find out) is what exactly the "risk" you a measuring is risk of. That would make it easier to identify ground truth or proxies for it in existing data.

The title of the supporting paper gives that away: "Open Source Software Projects Needing Security Investments". The CII project was started, in part, as a response to the Heartbleed vulnerability of OpenSSL. We're trying to determine what projects are more likely to have serious vulnerabilities and investment is needed.


> Is there a record of the anomalies and the adjustments?

A high-level discussion is in the paper. See the git log for a record of many of the actual adjustments (the commit text should give you at least a brief reason as to *why* they were adjusted). I don’t think all adjustments we tried are recorded in the git log, since we weren't particularly trying to do that (sorry). But I think you'll find lots of useful information.


> Is there any sort of formal procedure for further expert review?
> I would be interested in designing such a procedure if there isn't one.

No, there's no formal procedure. You can propose one.

That said, we're happy to take good ideas from anyone, even if they're not perceived as experts.

--- David A. Wheeler

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census

Re: NTP

Emily Ratliff
 

This is likely a quirk in the data. CII does fund the NTP project for ongoing maintenance work (part-time - the project certainly could use additional funding). They don't use the github repository as their main development repo, so that may be throwing the numbers off. There are more than 0 committers. 

On the blog article, please also see this thread where the issue is discussed on oss-security:

On Thu, Jan 28, 2016 at 7:30 AM, John Scott <jms3rd@...> wrote:
as an aside, from Kit:

The NTP problem is really bugging me.  
Not even recognized as a top-10 ‘risk’ as defined by the CII Census (https://www.coreinfrastructure.org/programs/census-project) with a really low popularity rating and 0 committers - yikes.  If it is unreal to believe that not one of the main Linux/UNIX distros wouldn’t pay somebody to be a project lead for that thing.  Then there’s this:  http://netpatterns.blogspot.com/2016/01/the-rising-sophistication-of-network.html

-------------------------------------------
John Scott
@johnmscott

On January 15, 2016 at 9:48:50 AM, Wheeler, David A (dwheeler@...) wrote:

Sebastian Benthall:
> One thing I'm trying to get a sense of (and I still need to read the paper very thoroughly to find out) is what exactly the "risk" you a measuring is risk of. That would make it easier to identify ground truth or proxies for it in existing data.

The title of the supporting paper gives that away: "Open Source Software Projects Needing Security Investments". The CII project was started, in part, as a response to the Heartbleed vulnerability of OpenSSL. We're trying to determine what projects are more likely to have serious vulnerabilities and investment is needed.


> Is there a record of the anomalies and the adjustments?

A high-level discussion is in the paper. See the git log for a record of many of the actual adjustments (the commit text should give you at least a brief reason as to *why* they were adjusted). I don’t think all adjustments we tried are recorded in the git log, since we weren't particularly trying to do that (sorry). But I think you'll find lots of useful information.


> Is there any sort of formal procedure for further expert review?
> I would be interested in designing such a procedure if there isn't one.

No, there's no formal procedure. You can propose one.

That said, we're happy to take good ideas from anyone, even if they're not perceived as experts.

--- David A. Wheeler

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census


Re: NTP

Kit Plummer
 

Thanks for the update Emily.  

Yeah, some of these ‘OG’ projects are tough to track for sure.  :)  Perhaps that is why they are less popular?

Kit

On Jan 28, 2016, at 8:51 AM, Emily Ratliff <eratliff@...> wrote:

This is likely a quirk in the data. CII does fund the NTP project for ongoing maintenance work (part-time - the project certainly could use additional funding). They don't use the github repository as their main development repo, so that may be throwing the numbers off. There are more than 0 committers. 

On the blog article, please also see this thread where the issue is discussed on oss-security:

On Thu, Jan 28, 2016 at 7:30 AM, John Scott <jms3rd@...> wrote:
as an aside, from Kit:

The NTP problem is really bugging me.  
Not even recognized as a top-10 ‘risk’ as defined by the CII Census (https://www.coreinfrastructure.org/programs/census-project) with a really low popularity rating and 0 committers - yikes.  If it is unreal to believe that not one of the main Linux/UNIX distros wouldn’t pay somebody to be a project lead for that thing.  Then there’s this:  http://netpatterns.blogspot.com/2016/01/the-rising-sophistication-of-network.html

-------------------------------------------
John Scott
@johnmscott

On January 15, 2016 at 9:48:50 AM, Wheeler, David A (dwheeler@...) wrote:

Sebastian Benthall:
> One thing I'm trying to get a sense of (and I still need to read the paper very thoroughly to find out) is what exactly the "risk" you a measuring is risk of. That would make it easier to identify ground truth or proxies for it in existing data.

The title of the supporting paper gives that away: "Open Source Software Projects Needing Security Investments". The CII project was started, in part, as a response to the Heartbleed vulnerability of OpenSSL. We're trying to determine what projects are more likely to have serious vulnerabilities and investment is needed.


> Is there a record of the anomalies and the adjustments?

A high-level discussion is in the paper. See the git log for a record of many of the actual adjustments (the commit text should give you at least a brief reason as to *why* they were adjusted). I don’t think all adjustments we tried are recorded in the git log, since we weren't particularly trying to do that (sorry). But I think you'll find lots of useful information.


> Is there any sort of formal procedure for further expert review?
> I would be interested in designing such a procedure if there isn't one.

No, there's no formal procedure. You can propose one.

That said, we're happy to take good ideas from anyone, even if they're not perceived as experts.

--- David A. Wheeler

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census



Re: NTP

Emily Ratliff
 

I had to look up 'OG'. :-) Always good to learn new lingo.

Absolutely agree with your sentiment. As projects mature and grow larger, the barriers to entry also grow larger in that the amount of knowledge that you need to have before making a meaningful contribution also grows, especially for highly technical and complex projects like crypto and time.

But, that is why it is a best practice to offer advice for incoming contributors and even tag bugs for newcomers to help them learn the code base like the projects on OpenHatch's Easy Bugs have done:
Producing lists like these also take time so this concept is sadly out of reach for critically short-handed projects.


On Thu, Jan 28, 2016 at 8:09 AM, Kit Plummer <kit.plummer@...> wrote:
Thanks for the update Emily.  

Yeah, some of these ‘OG’ projects are tough to track for sure.  :)  Perhaps that is why they are less popular?

Kit

On Jan 28, 2016, at 8:51 AM, Emily Ratliff <eratliff@...> wrote:

This is likely a quirk in the data. CII does fund the NTP project for ongoing maintenance work (part-time - the project certainly could use additional funding). They don't use the github repository as their main development repo, so that may be throwing the numbers off. There are more than 0 committers. 

On the blog article, please also see this thread where the issue is discussed on oss-security:

On Thu, Jan 28, 2016 at 7:30 AM, John Scott <jms3rd@...> wrote:
as an aside, from Kit:

The NTP problem is really bugging me.  
Not even recognized as a top-10 ‘risk’ as defined by the CII Census (https://www.coreinfrastructure.org/programs/census-project) with a really low popularity rating and 0 committers - yikes.  If it is unreal to believe that not one of the main Linux/UNIX distros wouldn’t pay somebody to be a project lead for that thing.  Then there’s this:  http://netpatterns.blogspot.com/2016/01/the-rising-sophistication-of-network.html

-------------------------------------------
John Scott
@johnmscott

On January 15, 2016 at 9:48:50 AM, Wheeler, David A (dwheeler@...) wrote:

Sebastian Benthall:
> One thing I'm trying to get a sense of (and I still need to read the paper very thoroughly to find out) is what exactly the "risk" you a measuring is risk of. That would make it easier to identify ground truth or proxies for it in existing data.

The title of the supporting paper gives that away: "Open Source Software Projects Needing Security Investments". The CII project was started, in part, as a response to the Heartbleed vulnerability of OpenSSL. We're trying to determine what projects are more likely to have serious vulnerabilities and investment is needed.


> Is there a record of the anomalies and the adjustments?

A high-level discussion is in the paper. See the git log for a record of many of the actual adjustments (the commit text should give you at least a brief reason as to *why* they were adjusted). I don’t think all adjustments we tried are recorded in the git log, since we weren't particularly trying to do that (sorry). But I think you'll find lots of useful information.


> Is there any sort of formal procedure for further expert review?
> I would be interested in designing such a procedure if there isn't one.

No, there's no formal procedure. You can propose one.

That said, we're happy to take good ideas from anyone, even if they're not perceived as experts.

--- David A. Wheeler

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census

_______________________________________________
cii-census mailing list
cii-census@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-census




Re: NTP

David A. Wheeler
 

Kit:
The NTP problem is really bugging me
On Jan 28, 2016, at 8:51 AM, Emily Ratliff <eratliff@...> wrote:
This is likely a quirk in the data. CII does fund the NTP project for ongoing maintenance work (part-time - the project certainly could use additional funding). They don't use the github repository as their main development repo, so that may be throwing the numbers off. There are more than 0 committers.
Actually, ntp was identified as a risky program. The more detailed paper D-5459 has more info that you (Kit) may have missed. See https://github.com/linuxfoundation/cii-census/blob/master/OSS-2015-06-19.pdf (click on “Raw” to use your local PDF reader). If you look at the list “Riskiest OSS Programs (human-identified subset informed by risk measures)”, on page 6-5 we *specifically* identify ntp as one of the riskiest programs. Once you look at the list when combined with human expertise, ntp jumps out as important. As Emily noted, the Linux Foundation is specifically funding the NTP project.

An *ideal* for the census project would be to have no need for human judgement. Ideally we could create quantitative measures, combine them in a clear and simple way, and demonstrably have a perfect list of exactly what’s riskiest (and in what order). We don’t currently have that ideal… but that doesn’t make the work useless. What we have instead are quantitative measures that can *help* humans make a determination of risk. In my experience there are many tough problems where the computer can't really make the decision... it can only be an *aid* to a human who makes the decision. Since the goal was to help humans make investment decisions, we met the goal.

If people have ideas about how to improve the census, we're all ears. We posted not just how we created the census numbers, but the alternatives we looked at & the code to calculate it. We *want* people to suggest improvements. Metrics is a notoriously hard problem in security.

One *big* problem is the lack of known truth - there are a lot of great learning algorithms, but they require truth data we don't have. Vulnerability counts (for example) are terrible proxies; a low number may mean the software is secure, or it may simply mean that no one has seriously reviewed it AND publicly reported the results. Not all vulnerabilities are equal, either.

--- David A. Wheeler

CII Census proposal: zlib

Timofonic
 

Hello.

Here's a proposal to the census: zlib

Website (90s design, does it count as site?): http://zlib.net
Contributors: Only Madler (the famous Mark Adler from JPI's NASA)
since 2013. Before that, only occasional contributors appear.
Popularity: I'm not sure how to interpret that and there are many
similar packages, but the statistics show it's extremely popular.
Check it yourself https://qa.debian.org/popcon.php?package=zlib
Main Language: C.
Network Exposure: I think it does, as some network-based projects use
it. I'm sure more skilled people can look at that.
Repo: https://github.com/madler/zlib/
Dependencies: The list is absurdly enormous, even Linux kernel uses it.
Application Data Only: No idea how to look for that at Debian
Popularity Contest.
Patches: I've seen patches out there. There's a lot of forks and
unmerged pull requests too.
ABRT crash statistics: I have no idea of what is it.

Kind regards.

CII Census proposal: Dropbear

Timofonic
 

Hello.

Here's a proposal to the census: Dropbear

Website (90s design, but it seems effective despite not having an own
domain): https://matt.ucc.asn.au/dropbear/dropbear.html
Contributors: Mostly Matt Johnston with periodical contributions from
others such as Francois Perrad, Ben Gardner. Occasional contributions
from others such as Henrik Nordström, Chocobo1, Jeremy Kerr. I'm not
sure how to analyze it, as the Mercurial web log seems weird to me.
Popularity: I'm not sure how to interpret that and there are many
similar packages, but the statistics show it's very popular. Anyway, I
think this parameter doesn't do it justice enough as this project is
extremely more popular in embedded Linux distros such as OpenWRT,
Alpine Linux and others. Check it yourself
https://qa.debian.org/popcon.php?package=dropbear
Main Language: C.
Network Exposure: It's a lightweight OpenSSH replacement. Of course,
it has tons of network exposure.
Repo: https://secure.ucc.asn.au/hg/dropbear/
Dependencies: I'm not sure about this, as the project tries to be
standalone because it's embedded-like nature.
Application Data Only: No idea how to look for that at Debian
Popularity Contest.
Patches: I've seen patches out there. There's a lot of forks and
unmerged pull requests too.
ABRT crash statistics: I have no idea of what is it.

Kind regards.

Re: CII Census proposal: zlib

Timofonic
 

I forgot to mention this email idea born from the following GitHub
issue tracker: https://github.com/madler/zlib/issues/299

2017-09-15 16:58 GMT+02:00 timofonic timofonic <timofonic@...>:

Hello.

Here's a proposal to the census: zlib

Website (90s design, does it count as site?): http://zlib.net
Contributors: Only Madler (the famous Mark Adler from JPI's NASA)
since 2013. Before that, only occasional contributors appear.
Popularity: I'm not sure how to interpret that and there are many
similar packages, but the statistics show it's extremely popular.
Check it yourself https://qa.debian.org/popcon.php?package=zlib
Main Language: C.
Network Exposure: I think it does, as some network-based projects use
it. I'm sure more skilled people can look at that.
Repo: https://github.com/madler/zlib/
Dependencies: The list is absurdly enormous, even Linux kernel uses it.
Application Data Only: No idea how to look for that at Debian
Popularity Contest.
Patches: I've seen patches out there. There's a lot of forks and
unmerged pull requests too.
ABRT crash statistics: I have no idea of what is it.

Kind regards.

Re: CII Census proposal: zlib

Timofonic
 

I reactivated it on OpenHub

https://www.openhub.net/p/zlib

2017-09-15 17:31 GMT+02:00 timofonic timofonic <timofonic@...>:

I forgot to mention this email idea born from the following GitHub
issue tracker: https://github.com/madler/zlib/issues/299

2017-09-15 16:58 GMT+02:00 timofonic timofonic <timofonic@...>:
Hello.

Here's a proposal to the census: zlib

Website (90s design, does it count as site?): http://zlib.net
Contributors: Only Madler (the famous Mark Adler from JPI's NASA)
since 2013. Before that, only occasional contributors appear.
Popularity: I'm not sure how to interpret that and there are many
similar packages, but the statistics show it's extremely popular.
Check it yourself https://qa.debian.org/popcon.php?package=zlib
Main Language: C.
Network Exposure: I think it does, as some network-based projects use
it. I'm sure more skilled people can look at that.
Repo: https://github.com/madler/zlib/
Dependencies: The list is absurdly enormous, even Linux kernel uses it.
Application Data Only: No idea how to look for that at Debian
Popularity Contest.
Patches: I've seen patches out there. There's a lot of forks and
unmerged pull requests too.
ABRT crash statistics: I have no idea of what is it.

Kind regards.

Re: CII Census proposal: Dropbear

Timofonic
 

I improved the OpenHub profile by adding the Mercurial repo (it's
finishing to analyze it)...

https://www.openhub.net/p/dropbear

2017-09-15 17:15 GMT+02:00 timofonic timofonic <timofonic@...>:

Hello.

Here's a proposal to the census: Dropbear

Website (90s design, but it seems effective despite not having an own
domain): https://matt.ucc.asn.au/dropbear/dropbear.html
Contributors: Mostly Matt Johnston with periodical contributions from
others such as Francois Perrad, Ben Gardner. Occasional contributions
from others such as Henrik Nordström, Chocobo1, Jeremy Kerr. I'm not
sure how to analyze it, as the Mercurial web log seems weird to me.
Popularity: I'm not sure how to interpret that and there are many
similar packages, but the statistics show it's very popular. Anyway, I
think this parameter doesn't do it justice enough as this project is
extremely more popular in embedded Linux distros such as OpenWRT,
Alpine Linux and others. Check it yourself
https://qa.debian.org/popcon.php?package=dropbear
Main Language: C.
Network Exposure: It's a lightweight OpenSSH replacement. Of course,
it has tons of network exposure.
Repo: https://secure.ucc.asn.au/hg/dropbear/
Dependencies: I'm not sure about this, as the project tries to be
standalone because it's embedded-like nature.
Application Data Only: No idea how to look for that at Debian
Popularity Contest.
Patches: I've seen patches out there. There's a lot of forks and
unmerged pull requests too.
ABRT crash statistics: I have no idea of what is it.

Kind regards.