Alton Blom wrote:

The phrase [...] is used multiple times throughout the criteria.
I've got a few ideas we could use to reduce this repetition.

1. Have a footnote that they can all refer to
2. Have a glossary
3. [...]

I vote for a glossary (table or bullet list), an endnote (rather than a footnote) or else a dedicated introductory chapter. As already mentioned, I'd prefer the criteria to be as concise as possible, or else it could be accompanied by an official checklist.

Also the criterion 'Projects SHOULD fix all critical vulnerabilities
rapidly after they are reported' mentions critical vulnerabilities - do
we have a definition of this, i.e. CVSS = 10?

Also do we have a definition of "rapidly"? How about a numeric graph or table ("severity"/"max time for fix per badge"), e.g.

Rating | CVSSv2 | Bronze | Silver | Gold
---------+--------+----------+----------+---------
Critical | 9-10 | 14 days | 7 days | 3 days
High | 7-8.9 | 30 days | 14 days | 7 days
Medium | 4-6.9 | 60 days | 30 days | 14 days
Low | 1-3.9 | 90 days | 60 days | 60 days

HeartBleed only had a CVSSv2 rating of 5.0, but a more appropriate CVSSv3 rating 7.5. Wouldn't it be better to reference CVSSv3 instead, also for longevity?


Kind Regards
--
Enos

Enos:

I vote for a glossary (table or bullet list), an endnote (rather than a

footnote) or else a dedicated introductory chapter. As already mentioned, I'd prefer the criteria to be as concise as possible, or else it could be accompanied by an official checklist.

Okay! That makes sense. Many people will probably be entering a form... we'll need to figure out how to make that info available to them. We've discussed creating text that can be hidden or revealed that provides more detail, perhaps that would go into the "details".

Also do we have a definition of "rapidly"? How about a numeric graph or table ("severity"/"max time for fix per badge"), e.g.

Rating | CVSSv2 | Bronze | Silver | Gold
---------+--------+----------+----------+---------
Critical | 9-10 | 14 days | 7 days | 3 days
High | 7-8.9 | 30 days | 14 days | 7 days
Medium | 4-6.9 | 60 days | 30 days | 14 days
Low | 1-3.9 | 90 days | 60 days | 60 days

At this point we don't have bronze/silver/gold, though that's certainly what we're thinking about longer-term. We have had separate discussions that maybe all the "SHOULD" are needed for silver, and all the "RECOMMENDED" are needed for gold, but that's merely one of many ideas... not something we've committed to.

HeartBleed only had a CVSSv2 rating of 5.0, but a more appropriate CVSSv3 rating 7.5. Wouldn't it be better to reference CVSSv3 instead, also for longevity?

The plan is to use CVSSv3 eventually; the question is *when*. CVSSv3 was released by FIRST on June 10th, 2015, and many have not updated yet. In particular, many tools are currently based on the NIST National Vulnerability Database (NVD), and while the NVD *does* plan to switch to CVSSv3 sometime this fall, they have NOT done so yet, and that won't provide CVSSv3 information on past vulnerabilities. See: https://nvd.nist.gov/CVSS/CVSS-v3-information.aspx

--- David A. Wheeler

On 09/29/2015 06:35 PM, Enos wrote:

Also do we have a definition of "rapidly"? How about a numeric graph or
table ("severity"/"max time for fix per badge"), e.g.

Rating | CVSSv2 | Bronze | Silver | Gold
---------+--------+----------+----------+---------
Critical | 9-10 | 14 days | 7 days | 3 days
High | 7-8.9 | 30 days | 14 days | 7 days
Medium | 4-6.9 | 60 days | 30 days | 14 days
Low | 1-3.9 | 90 days | 60 days | 60 days

Isn't this backwards?

I would expect *more* time for addressing critical issues because you
need embargoes, need to pre-notify people, and make sure that everyone
has updates ready on disclosure day. Some of us actually *test* the
bits we ship, and for core packages, this can take a week or longer.
And not all critical issues are buffer overflows with two-line fixes, so
even developing a patch might take a few days (not talking about
protocol development if things are really bad, but those issues tend to
be Important at most, not Critical).

Getting out a fix in 3 days (or less than a week) is only possible if
you don't do the embargo dance. For most vulnerabilities, this is
completely fine, but for Criticals and Importants, it may make sense to
aim for a more coordinated approach.

--
Florian Weimer / Red Hat Product Security

Florian Weimer wrote:

I would expect *more* time for addressing critical issues because you
need embargoes, need to pre-notify people, and make sure that everyone
has updates ready on disclosure day. Some of us actually *test* the
bits we ship, and for core packages, this can take a week or longer.
[...]

The current draft starts counting time when "a vulnerability becomes publicly known", i.e.

[...] A vulnerability becomes publicly known (for this purpose) once
it has a CVE with publicly released non-paywalled information [...]
or when [...] the information has been released to the public [...]

So most times you have enough time to do things properly. But a deadline must also be set when a project is first alerted, as the vulnerability may leak, be discovered by other parties, etc. .

As a side note, I would make mandatory for the basic badge:
- "push" security advisories (e.g. mailing list or RSS)
- specifying when the vulnerability was first introduced in production
- a list of *all* affected production versions

You bring solid arguments against tight deadlines for critical fixes, but from a different perspective, how long would users tolerate a trivially exploitable _critical_unauthenticated_remote_ vulnerability actively exploited in the wild? My maximum personal estimate for the top (gold) badge remains 3 days because not everybody has an IPS.

Small derogations could be me made while releasing temporary patches (e.g. those defending against available exploits but not fixing the vulnerability itself, or which are not tested against all unwanted side-effects). Again, two badge courses might be more realistic: code security and code quality are tightly related but have different priorities, so they cannot always be maximized concurrently.


Kind Regards
--
Enos

DISCLAIMER: what I write here is my own personal opinion; I do not have significant direct experience as OSS developer or project leader, though I act as main pentester, auditor, security advisor and incident response coordinator for about a hundred internal projects