Topics

Subject: First impressions on CII Best Practices and badges -- part 2


Kevin W. Wall
 

Hope you are ready for some more questions and comments. :)


1) One question asks for the CPE name and refers to https://nvd.nist.gov/cpe.cfm
I looked all over that page and still was unable to find how to request or
submit a CPE name to NIST. Do you have a more specific URL for the submittal?
(Or, short of that, I could use the CPE name that NIST used on 2 CVEs
related to the project. I'm just not sure they would always use that
same name in the future.)

2) This part of the questionnaire was a little nebulous I thought:

The information on how to contribute SHOULD include the requirements for
acceptable contributions (e.g., a reference to any required coding
standard). (URL required) [contribution_requirements]

Our README.md file has a one-liner about contributing...it just says
"just create a pull request". In my answer, I marked it as 'Unmet'
with this comment:
We generally only accept bug fixes, not new features (because as a
legacy project, we don't intend on adding new features). And we vet all
pull requests, including coding style of any contributions.
but I left it as 'Unmet' because I wasn't sure if you that's all you
were looking for or if you were looking for something else (a 'URL required'
is mentioned, so what URL would I use???).

3) I wanted to comment on the question about '[documentation_basics]'. I
marked this as 'met', but I almost cringed when I did so. Technically,
we have documentation and thus meet this, but much of the documentation
is outdated or simply very poor and/or usable to begin with. Certainly,
'documentation_basics' is a low water mark, but I think it would be helpful
to all people to somehow gauge the quality of their project's documentation.
That likely would require several questions, but for those of us who wish
to be honest about our projects, it will also help show us where we fall
short.

4) Question regarding 'vulerability_report_process'. I would like to use GitHub
Issues to report vulnerabilties, but at the same time, I would like a way
that GitHub supports this without making the vulnerability initially visible.
(We would make it visible after we had fixed it and made a patch available.)
Google Code issues had a way to do that and it was useful (not that people
followed it; sigh). If we can't use GitHub Issues, how do you suggest
that people report vulnerabilities without having them turn into potential
0days. (E.g., we've had people first announce what became 2 CVEs on our
public mailing list _despite_ instructions on how to report them securely
to our contributors.) Looking for suggestions here!

Well, that's all for tonight. Looking forward to reading your replies.

-kevin


--
Blog: http://off-the-wall-security.blogspot.com/ | Twitter: @KevinWWall
NSA: All your crypto bit are belong to us.


Daniel Stenberg
 

On Fri, 13 May 2016, Kevin W. Wall wrote:

The information on how to contribute SHOULD include the requirements
for acceptable contributions (e.g., a reference to any required
coding standard). (URL required) [contribution_requirements]

Our README.md file has a one-liner about contributing...it just says
"just create a pull request". In my answer, I marked it as 'Unmet'
with this comment:

We generally only accept bug fixes, not new features (because as a
legacy project, we don't intend on adding new features). And we vet all
pull requests, including coding style of any contributions.

but I left it as 'Unmet' because I wasn't sure if you that's all you were
looking for or if you were looking for something else (a 'URL required'
is mentioned, so what URL would I use???).
First, I think you're meeting the requirement already. Then, I think you'd be an even better open source citizen if the above comment would be visisble in association to your "just create a pull request" advice.

3) I wanted to comment on the question about '[documentation_basics]'. I
marked this as 'met', but I almost cringed when I did so. Technically,
we have documentation and thus meet this, but much of the documentation
is outdated or simply very poor and/or usable to begin with.
Would an outsider coming to your project call that as "having [enough] documentation"? Then I think met. If one wouldn't, then I'd say unmet.

This entire process is about grading your own efforts so of course there are grey scales in just about every question. Judgement calls if you like. And you're the one to make them.

4) Question regarding 'vulerability_report_process'. I would like to use GitHub Issues to report vulnerabilties,
I'm sure we are many who would, but due to the inability to ask for issues to get filed privately or hidden, we really SHOULD not.

This is actually one of the biggest issues (no pun intended) I have with hosting projects on github.

If we can't use GitHub Issues, how do you suggest that people report
vulnerabilities without having them turn into potential 0days.
I suggest providing a separate channel for (suspected) security related issues. Like an email address or a separate tracker, that is closed for the public first until someone in the trusted "security team" has had a chance to take a look at it.

--

/ daniel.haxx.se


David A. Wheeler
 

Kevin W. Wall:
1) One question asks for the CPE name and refers to https://nvd.nist.gov/cpe.cfm
I looked all over that page and still was unable to find how to request or
submit a CPE name to NIST. Do you have a more specific URL for the submittal?
It's there. I quote:
"Organizations interested in submitting CPE Names should contact the NVD CPE team at cpe_dictionary@nist.gov for help with the processing of their submission."

It's probably not as obvious as it should be, but that seems to be the most authoritative place to find the info.



2) This part of the questionnaire was a little nebulous I thought:
The information on how to contribute SHOULD include the requirements for
acceptable contributions (e.g., a reference to any required coding
standard). (URL required) [contribution_requirements]
Our README.md file has a one-liner about contributing...it just says
"just create a pull request". In my answer, I marked it as 'Unmet'
with this comment:
We generally only accept bug fixes, not new features (because as a
legacy project, we don't intend on adding new features). And we vet all
pull requests, including coding style of any contributions.
but I left it as 'Unmet' because I wasn't sure if you that's all you
were looking for or if you were looking for something else (a 'URL required'
is mentioned, so what URL would I use???).
That's fine. As Daniel Stenberg noted, "First, I think you're meeting the requirement already. Then, I think you'd be an even better open source citizen if the above comment would be visisble in association to your "just create a pull request" advice."

The URL would be either where to find out how to submit changes, or the change mechanism itself. So a URL for README.md or pull request mechanism would be fine.

3) I wanted to comment on the question about '[documentation_basics]'. I
marked this as 'met', but I almost cringed when I did so. Technically,
we have documentation and thus meet this, but much of the documentation
is outdated or simply very poor and/or usable to begin with. Certainly,
'documentation_basics' is a low water mark, but I think it would be helpful
to all people to somehow gauge the quality of their project's documentation.
That likely would require several questions, but for those of us who wish
to be honest about our projects, it will also help show us where we fall
short.
You'd be surprised how many OSS projects have *no* documentation at all :-(.
It's not just OSS; lots of software, including proprietary software, often has
outdated, poor, or not-clearly-usable documentation. We want it much *better*,
of course, but right now we're focusing on just the basics.

If you meet the criteria, then you meet it.

Frankly, if you have a vibrant community of people contributing, that community
can rapidly transform less-than-perfect documentation into pretty good stuff.
But if there's nothing at all, or it's difficult to contribute, then it's unlikely to get better.

*Nothing* built in this world is perfect. Instead, we're focusing on making sure the
project has the basics, and is welcoming to improvements.

4) Question regarding 'vulerability_report_process'. I would like to use GitHub
Issues to report vulnerabilties, but at the same time, I would like a way
that GitHub supports this without making the vulnerability initially visible.
Sadly, while tools like Bugzilla easily support this,
GitHub doesn't currently support this in its issue tracker. It's a common request, see:
https://github.com/isaacs/github/issues/37

If you use GitHub, and you want private vulnerability reports,
the best approach currently is to include information on how to
provide this information out-of-band. That works.
Feel free to add your vote to issue #37.

--- David A. Wheeler


Kevin W. Wall
 

On Mon, May 16, 2016 at 10:20 AM, Wheeler, David A <dwheeler@ida.org> wrote:
Kevin W. Wall:
1) One question asks for the CPE name and refers to https://nvd.nist.gov/cpe.cfm
I looked all over that page and still was unable to find how to request or
submit a CPE name to NIST. Do you have a more specific URL for the submittal?
It's there. I quote:
"Organizations interested in submitting CPE Names should contact the NVD
CPE team at cpe_dictionary@nist.gov for help with the processing of their
submission."

It's probably not as obvious as it should be, but that seems to be the
most authoritative place to find the info.
Thanks for that. I fired off an email to them. I'm honestly hoping that
it's not as a big of a pain in the ass to get a CPE issued as it is to
convince Mitre and them to accept a proposed CVE. (That was extremely
painful. It really shouldn't be THAT hard, especially when one is one of
the project co-owners.)

2) This part of the questionnaire was a little nebulous I thought:
The information on how to contribute SHOULD include the requirements for
acceptable contributions (e.g., a reference to any required coding
standard). (URL required) [contribution_requirements]
Our README.md file has a one-liner about contributing...it just says
"just create a pull request". In my answer, I marked it as 'Unmet'
with this comment:
We generally only accept bug fixes, not new features (because as a
legacy project, we don't intend on adding new features). And we vet all
pull requests, including coding style of any contributions.
but I left it as 'Unmet' because I wasn't sure if you that's all you
were looking for or if you were looking for something else (a 'URL required'
is mentioned, so what URL would I use???).
That's fine. As Daniel Stenberg noted, "First, I think you're meeting
the requirement already. Then, I think you'd be an even better open
source citizen if the above comment would be visisble in association to
your "just create a pull request" advice."

The URL would be either where to find out how to submit changes, or the
change mechanism itself. So a URL for README.md or pull request
mechanism would be fine.
Thanks Daniel and David for that suggestion. I've updated our README.md on
our 'develop' branch (which is out git default branch) accordingly.

3) I wanted to comment on the question about '[documentation_basics]'. I
marked this as 'met', but I almost cringed when I did so. Technically,
we have documentation and thus meet this, but much of the documentation
is outdated or simply very poor and/or usable to begin with. Certainly,
'documentation_basics' is a low water mark, but I think it would be helpful
to all people to somehow gauge the quality of their project's documentation.
That likely would require several questions, but for those of us who wish
to be honest about our projects, it will also help show us where we fall
short.
You'd be surprised how many OSS projects have *no* documentation at all :-(.
It's not just OSS; lots of software, including proprietary software, often has
outdated, poor, or not-clearly-usable documentation. We want it much *better*,
of course, but right now we're focusing on just the basics.

If you meet the criteria, then you meet it.
Yeah, well, while,
no documentation implies documentation sucks
unfortunately,
have documentation does NOT imply that documentation does not suck

While I will be the first to admit this is very subjective, I think we all
can agree that neither is "goodness / quality" of documentation is binary
condition (Met / Unmet), but rather a gray scale. I was only bemoaning the
fact that there is no way for those who wish to be honest and transparent
to our user communities to reflect that in the current badging process.
While I'm not sure how your BadgeApp would score such a gray scale result
(maybe a scale of, say, 1 to 4 or whatever) as it factors into the score, I
think having such a gradation would be useful for this and other questions
as well. Something like "Fully meets", "Mostly meets", "Somewhat meets",
"Unmet / Does not meet" would probably work. You would still have to get
"Fully meets" to get an overall "passing" grade.

Frankly, if you have a vibrant community of people contributing, that
community can rapidly transform less-than-perfect documentation into
pretty good stuff. But if there's nothing at all, or it's difficult to
contribute, then it's unlikely to get better.

*Nothing* built in this world is perfect. Instead, we're focusing on
making sure the project has the basics, and is welcoming to
improvements.
And I'm just sayin' that I think that having a sliding gray scale for this
and some of the other questions might allow us as FLOSS contributors to
better focus on where to allocate our skimpy resourdces. If someone looks
at it now, they see 'Met' and move on to something else, even though I
consider the lack of quality, up-to-date documentation one of the areas
where ESAPI needs the most help.

4) Question regarding 'vulerability_report_process'. I would like to use GitHub
Issues to report vulnerabilties, but at the same time, I would like a way
that GitHub supports this without making the vulnerability initially visible.
Sadly, while tools like Bugzilla easily support this,
GitHub doesn't currently support this in its issue tracker. It's a common request, see:
https://github.com/isaacs/github/issues/37

If you use GitHub, and you want private vulnerability reports,
the best approach currently is to include information on how to
provide this information out-of-band. That works.
Feel free to add your vote to issue #37.
Sigh. Okay, not sure it will help, but I've added my vote to the 700+
others who had already starred it.

I'll have to discuss this with the other project co-lead. I know what
he WANTS to do (set up Atlassian JIRA and use it), but he doesn't have
the time and I don't have the know-how so we'll have to come up with
something else.

-kevin
--
Blog: http://off-the-wall-security.blogspot.com/ | Twitter: @KevinWWall
NSA: All your crypto bit are belong to us.


David A. Wheeler
 

Kevin W. Wall [mailto:kevin.w.wall@gmail.com]:

While I will be the first to admit this is very subjective, I think we all can agree that neither is "goodness / quality" of documentation is binary condition (Met / Unmet), but rather a gray scale. I was only bemoaning the fact that there is no way for those who wish to be honest and transparent to our user communities to reflect that in the current badging process.
While I'm not sure how your BadgeApp would score such a gray scale result (maybe a scale of, say, 1 to 4 or whatever) as it factors into the score, I think having such a gradation would be useful for this and other questions as well. Something like "Fully meets", "Mostly meets", "Somewhat meets", "Unmet / Does not meet" would probably work. You would still have to get "Fully meets" to get an overall "passing" grade.
It's plausible ,but I don't think we should do it that way. Answering met/unmet is hard; figuring out the proper gradation for many questions is even harder. We're trying to limit the amount of time it takes to get a badge, and eventually automate a lot of it - gradations within questions would impede both goals.

I *do* agree that there's a need for levels, but we have a different plan for that. We intend to have multiple badge levels (eventually), and that would have some of the same effect. Even then we want to have clear yes/no answers for those harder requirements; then it'd be much less ambiguous whether or not a project met the requirements.

But there's only so much time in the day, and right now we're 100% focused on making it easy to get this one level & get a lot of projects on board. Once we do that, we'll have a lot more participation, and then it should be much easier to determine and refine those higher-level criteria.

--- David A. Wheeler