Topics

certification -vs- guide


Enos <temp4282138782@...>
 

Wheeler, David A wrote

[...] I suggest to deliberately ignore technical difficulties [...]
If the bar is set too high, most projects will not try.
[...]
It really will be a challenge to avoid "too high" AND "too
low"; our (partial) solution is to do discuss it in public. [...]
Absolutely. I was not referring to certifying projects, but rather to a former comment of yours about BadgeApp: as I interpreted it, you wrote that criteria requirements could be recommended rather than mandated because of difficulties in automating the corresponding checks.

Correct me if I'm wrong, but CII-badges are not aimed at a specific audience (e.g. CII-funded projects), so the bar for the entry-level badge ought to be lowered accordingly as the lowest (meaningful) level for the smallest project with the lowest security requirements.

In this context there should be no false expectations about different badges, because they measure technical controls rather than application security. When certifying, applications are not expected to outline their classification level or their security requirements, so end-users are left with no way to tell whether a badge is adequate for a specific application. Without complementary intelligence, CII-badges seems mostly useful to IT people and projects rather than to end users.


I'm happy to cite sources, I just want to limit what people *have* to
read. Again, we want people to actually *get* the badge; if it
requires too much reading, the number of projects that achieve it
will drop. [...]
For the very same reason, were I after the badge, I would hate spending time reading definitions (or causal relations) I am already familiar with, in fear of losing some other content, when instead I could just click a hyperlink if needed. As outlined in the background document, there are already numerous security guides out there.

I made my points and would not want to take too much of your precious time, so if it feels I am repeating myself feel free to close the discussion.


Thanks,
Kind Regards
--
Enos

p.s. there must be something wrong with the mailing list, as *again* I did not receive your message in my mailbox while I did receive others'


David A. Wheeler
 

Enos:
Absolutely. I was not referring to certifying projects, but rather to a former comment of yours about BadgeApp: as I interpreted it, you wrote that criteria requirements could be recommended rather than mandated because of difficulties in automating the corresponding checks.
Ah, sorry, that was NOT what I meant. If something is hard to automate, no big deal; a human just has to type in a justification. We'll check that the field isn't blank, and perhaps a few more sanity checks, but otherwise, people who care can review the text to see if they believe it.

We put in something RECOMMENDED if we recommend it, but for some reason aren't mandating it. Typically the reason is because for some non-trivial group it's hard to do.

I'll see if I can document that more clearly.

Correct me if I'm wrong, but CII-badges are not aimed at a specific audience (e.g. CII-funded projects), so the bar for the entry-level badge ought to be lowered accordingly as the lowest (meaningful) level for the smallest project with the lowest security requirements.
Yes, I think that's (mostly) fair.

In this context there should be no false expectations about different badges, because they measure technical controls rather than application security. When certifying, applications are not expected to outline their classification level or their security requirements, so end-users are left with no way to tell whether a badge is adequate for a specific application. Without complementary intelligence, CII-badges seems mostly useful to IT people and projects rather than to end users.
CII badges will not solve all possible problems, sure. My experience is that even if applications DO outline their security requirements, end users often can't reliably determine if the application will be adequate for a specific application. If the needs are complex, end users may need help selecting appropriate software.

But I think CII badges will still be useful. A badge will help OSS projects show to others that they're doing the basics, and make it easy for potential users to see that. In some cases pursuing a badge will help some OSS projects "get their house in order" to do the things they know they should do but haven't gotten around to it (really, does anyone argue AGAINST having test suites?).

p.s. there must be something wrong with the mailing list, as *again* I did not receive your message in my mailbox while I did receive others'
This time I am *only* replying to the "cii-badges" mailing list. If you're on the mailing list, you should receive a copy of this email. If you don't get it, let me know by private email and I will forward to the Linux Foundation (they run these servers).


Enos <temp4282138782@...>
 

Wheeler, David A wrote:

I'll see if I can document that more clearly.
No need thanks, I just misunderstood your initial post here.


[...] My experience is that even if applications DO outline their security requirements, end users often can't reliably determine if the application will be adequate for a specific application. If the needs are complex, end users may need help selecting appropriate software.
In the (distant) future you might consider attaching to the criteria a (short, optional and complementary) document recommending badge levels for standard application types. E.g. critical assets, secure communication and financial applications should aim for SILVER or higher, local applications only dealing with public data have no practical need to go beyond BRONZE, etc., if not for empowering users, at least as some kind of disclaimer.

Still it would be a pity if applications with low requirements and "perfect" security for their needs will never obtain gold badges, because they would be overkill for them. It's a bit like if gold medals were not assigned in women's 100 metres because men run a bit faster.
It would be ideal if badge names could somehow convey both their hierarchy and what they are suitable for, but maybe that's asking too much from a single word :P


But I think CII badges will still be useful. [...]
Most definitely!
Don't mind my remarks, I am just playing the devil's advocate.


[...] (really, does anyone argue AGAINST having test suites?).
In my country there's an unfortunate motto: "a law is made, a way around it is found". It may be not so much about *having* test suites as about what to make them do and how. I've seen "test suites" you people wouldn't believe (not to mention hardly any of them addressed security).


Thanks a lot for the discussion. I will come back in a few days proposing a list of additional controls for the criteria.

Kind Regards
--
Enos

p.s. I got your message in the inbox too this time, thanks.


David A. Wheeler
 

Enos:
In the (distant) future you might consider attaching to the criteria a (short, optional and complementary) document recommending badge levels for standard application types. E.g. critical assets, secure communication and financial applications should aim for SILVER or higher, local applications only dealing with public data have no practical need to go beyond BRONZE, etc., if not for empowering users, at least as some kind of disclaimer.
Still it would be a pity if applications with low requirements and "perfect" security for their needs will never obtain gold badges, because they would be overkill for them. It's a bit like if gold medals were not assigned in women's 100 metres because men run a bit faster.
It would be ideal if badge names could somehow convey both their hierarchy and what they are suitable for, but maybe that's asking too much from a single word :P
If we do, I think it's going to have to be in the future and probably distant future (as you suggest).

There was a lot of discussion in a meeting in Madrid about having non-hieararchical badge names. E.g., a "basic" badge, plus badges for "good crypto" or whatever that would have "basic" as a prerequisite. That doesn't really address the "what is recommended" issue though, indeed, it might be more complicate to give recommendations. And frankly, since we *want* people to strive to meet the criteria, having hierarchical names might encourage people to reach higher.

I don't see a reason that an application that had fewer needs *couldn't* achieve a gold badge. They might chose to not pursue it, but that's different. E.G., if "gold" required testing with 100% branch coverage, that is technically achievable by any application program. (It would be *tremendously* hard for a general-purpose kernel; I imagine that it'd be possible to do it by simulation, but it's less likely anyone would expend the effort.)


In my country there's an unfortunate motto: "a law is made, a way around it is found". It may be not so much about *having* test suites as about what to make them do and how. I've seen "test suites" you people wouldn't believe (not to mention hardly any of them addressed security).
I think THAT kind of thinking exists around the world, ESPECIALLY since a lot of the people we hope to get involved are hackers (in the sense of people who deeply know systems and can use them in novel ways). We can't completely solve the negative side of working around a system, especially since this badge is based on a *self-asserted* information. We can try to make rigorous criteria text, and we can create a fast update process to at least counter some of the "ways around". To be honest, I don't want to COMPLETELY eliminate that way of thinking; if the "way around" is a better way to do things, GREAT!

The current test suite requirements for basic/bronze are extremely minimal; they require that tests exist, not what they test. In my experience, even super lame test suites are still useful. When packaging programs I'd often add just one test for a program that originally had none; this turned out to be remarkably useful, since screwed-up packaging commands can lead to completely useless results that even trivial tests can detect. I expect that "silver" will require much more, if we create that later.

I will come back in a few days proposing a list of additional controls for the criteria.
Please do that soon! We hope to get the first draft of the criteria field implemented this week in BadgeApp, maybe we can slip it into this very early version.

p.s. I got your message in the inbox too this time, thanks.
Excellent.

--- David A. Wheeler