Topics

certification -vs- guide


Enos <temp4282138782@...>
 

Dear David and List,

The current criteria (v 0.0.4) includes best practices on both quality and security. It sets requirements for a (self) certification program, but also gives recommendations and detailed definitions.

From my (small) experience, compliance is either black, or it is white. It could include fuzzy requirements (for example dependent on the classification or scope of a system or project), but additional contents may be detrimental: a hybrid which is both a guide and a certification may not reap the full benefits of neither (although there could be such a thing as the best compromise for the largest audience).

RECOMMENDATIONS may trigger false expectations on certified projects. They could instead be turned into requirements for higher badges (making contents of all badges a recommendation for the badge below),
or they could depend on a project's classification or scope (as also other normal certification requirements).

DEFINITIONS and explanations are already available from specialized external resources and papers. In order to maintain focus and avoid duplicating information (which will require additional effort to be kept up-to-date), it may be best to only include concise requirements, only when essential, referencing trusted external resources as necessary. Specifically, to me the current sections on testing and patching deadlines appear slightly verbose.

SECURITY and quality within the document are matched into single badge levels. While it is being done intelligently, they wouldn't allow precisely estimating the value of projects with different levels of quality and security. Did you consider two distinct badge courses? Bronze security could become a requirement of silver quality, and vice-versa bronze quality a requirement of silver security.

The above issues are mostly aimed at brainstorming. Not even I think that all proposed modifications should be applied, but I do think that evaluating them now may be constructive. They are not nearly as important as core contents, but given their pervasiveness, if they are implemented, the earlier it is done, the less effort refactoring would require.

Thanks for reading. What do you think?


Kind Regards
--

Enos D'Andrea,
from Italy


David A. Wheeler
 

Enos D'Andrea:
The current criteria (v 0.0.4) includes best practices on both quality and security. It sets requirements for a (self) certification program, but also gives recommendations and detailed definitions.
From my (small) experience, compliance is either black, or it is white. It could include fuzzy requirements (for example dependent on the classification or scope of a system or project), but additional contents may be detrimental: a hybrid which is both a guide and a certification may not reap the full benefits of neither (although there could be such a thing as the best compromise for the largest audience).
I mostly agree. We want compliance requirements to be black or white, to the extent we can. That said, there will always be some criteria that are hard to make perfectly black-and-white. E.G., the requirement that the project website "succinctly describe what the software does (what problem does it solve?), in language that potential users can understand..." would be difficult to fully automate. In those cases my theory is that if there's a reasonable attempt that most people would accept as complying, that's good enough. For example, that criterion is create to deal with projects which say "We're the QZY project" without any hint of what it does; any reasonable attempt would be better than that, and it's hard to figure out if something is secure if you can't figure out what it does :-).

RECOMMENDATIONS may trigger false expectations on certified projects. They could instead be turned into requirements for higher badges (making contents of all badges a recommendation for the badge below), or they could depend on a project's classification or scope (as also other normal certification requirements).
I agree that they could turn into requirements for higher badges. However, I think it is MUCH better to include important recommendations, even if they're not required. While some projects may do just the minimum, I think many projects will do more... but only if it's recommended. I don't think the false expectation risk is high; the word "RECOMMENDED" is standard IETF terminology and it's basically the normal English meaning, so most readers shouldn't be confused.

Of course, this begs the question, "why have a badge?" The paper [Open badges for education: what are the implications at the intersection of open systems and badging?](http://www.researchinlearningtechnology.net/index.php/rlt/article/view/23563)
identifies three general reasons for badging systems, and I think all three reasons apply.

1. Badges as a motivator of behaviour. We hope that by identifying best practices, we'll encourage projects to implement those best practices if they do not do them already.
2. Badges as a pedagogical tool. Some projects may not be aware of some of the best practices applied by others, or how they can be practically applied. The badge will help them become aware of them and ways to implement them.
3. Badges as a signifier or credential. Potential users want to use projects that are applying best practices to consistently produce good results; badges make it easy for projects to signify that they are following best practices, and make it easy for users to see which projects are doing so.

So while the "RECOMMENDED" text is irrelevant as a signifier or credential, it definitely serves as a pedagogical tool. Most project participants *want* their projects to produce good results; we're just giving them information on how they can get there.

DEFINITIONS and explanations are already available from specialized external resources and papers. In order to maintain focus and avoid duplicating information (which will require additional effort to be kept up-to-date), it may be best to only include concise requirements, only when essential, referencing trusted external resources as necessary.
Specifically, to me the current sections on testing and patching deadlines appear slightly verbose.

I've recently tried to tighten them up a little bit. I'm happy to cite other resources (e.g., I do that with CVSS). The problem is that when you cite other sources, people have to track those down and read them too; if the burden is too high, people won't do it.

SECURITY and quality within the document are matched into single badge levels. While it is being done intelligently, they wouldn't allow precisely estimating the value of projects with different levels of quality and security. Did you consider two distinct badge courses? Bronze security could become a requirement of silver quality, and vice-versa bronze quality a requirement of silver security.
Yes, we've talked about having different courses. In particular, in Madrid there was a lot of talk about having a "basic" level, and then a set of specific badges (not a particular level) that required "basic" first. However, quality-not-including-security and security are rather interrelated. Also, we really want to keep things as simple as practical. So we've focused on creating just one badge level to start with, and then move from there.

The above issues are mostly aimed at brainstorming. Not even I think that all proposed modifications should be applied, but I do think that evaluating them now may be constructive. They are not nearly as important as core contents, but given their pervasiveness, if they are implemented, the earlier it is done, the less effort refactoring would require.
Thanks!! This is the *perfect* time for brainstorming.

--- David A. Wheeler


Blibbet <blibbet@...>
 

Hi, first post, just joined.

On 09/04/2015 11:53 AM, Wheeler, David A wrote:
Thanks!! This is the *perfect* time for brainstorming.
Unclear if I'm off-topic of this thread, but, please don't forget
firmware, and only focus on OS/apps.

A lot of firmware is open source these days. UEFI uses BSD-licensed
tianocore.org for common base, U-Boot is just as active as UEFI, and
coreboot has it's Libreboot and Chrome OS variants. For some, the issue
is removing blobs. For security, dealing with blobs is one issue, but
also dealing with security tech in firmware. Eg, Verified coreboot,
UEFI's Secure Boot, U-Boot has a similar option. Many of these work with
or w/o TPM, some work with TrustZone. OpenPOWER has very different
firmware, but parts are open source. Dealing with actual firmware, as
well as virtualized firmware in Xen/KVM/etc.

These are issues for VM software, FOSS-based OEMs, FOSS-friendly IBVs
(Independent BIOS vendors, like AMI, Insyde, Phoenix, Sage Engineering,
etc.).

Granted most is Linux-based, but FreeBSD currently supports UEFI. Having
them give a voice for their firmware [security] infrastructure needs
would be useful.

IMO, the MOST IMPORTANT firmware infrastructure issue needed is a CA
(Certificate Authority), for an alternative to Microsoft as the current
UEFI Forum CA. And having a Linux-friendly CA by itself probably won't
solve it, unless is an OS agnostic CA that UEFI Forum adopts. To get it
used, IMO, you need to bypass traditional IBVs and create a new FOSS
OS-centric IBV, which uses/requires this new CA. Then, you have a
complete firmware solution to offer non-Windows/non-Chrome OEMs. Today,
Linux OEMs normally use COTS BIOS, which means it comes with ACPI tables
with Windows executables embedded in them, Secure Boot that requires
Microsoft to sign any pre-OS boot tool (GRUB, any FOSS OS loader, etc.)
UEFI aside, coreboot and U-Boot both need this, both have similar PKI
issues now with their Verified/etc flavors of boots; see Sage
Engineering CEO's blog post on coreboot need a few years ago.

Today, Linaro is nearly an IBV for ARM, providing binaries of UEFI and
U-Boot for multiple ARM boards. We need a decentralized, community-based
IBV for FOSS OSes, in conjunction with a FOSS-friendly CA.

Then, while we wait for Open Hardware (eg, RISC-V), we have to work with
Intel/AMD/ARM vendors to reduce their blobs. AFAICT, AMD is doing pretty
well w/r/t blobs, on coreboot. Intel FSP (Firmware Support Package) is
main source of Intel blobs. Today, you need an NDA to get source to
modify those, like Purism and Sage Engineering do. Look at most recent
Purism blob, it sounds like they're trying to do a Free Software
alternative to FSP; not sure if possible, but a CII project should be to
try and do so, to help UEFI/coreboot/U-Boot.

I have some more notes on what's required, if this isn't too far
off-topic for CII....

Thanks,
Lee Fisher
RSS: http://firmwaresecurity.com/feed


David A. Wheeler
 

Lee Fisher:
Unclear if I'm off-topic of this thread, but, please don't forget firmware, and only focus on OS/apps.
Great point. To be honest, we haven't focused too much on firmware so far, so a pass specifically looking for problems would be very welcome. That said, we *have* tried to ensure that it's doable for kernels (such as the Linux kernel), so I have hopes that everything we've said so far is reasonable.

If you find a proposed criteria that is *NOT* appropriate for firmware, please post / create an issue / create a pull request.

IMO, the MOST IMPORTANT firmware infrastructure issue needed is a CA (Certificate Authority), for an alternative to Microsoft as the current UEFI Forum CA. And having a Linux-friendly CA by itself probably won't solve it, unless is an OS agnostic CA that UEFI Forum adopts....
I agree, but I think the "badging" work is the wrong forum for *that* problem. It may be a *CII* issue, conceivably, but it's not really a "badge".

... Then, while we wait for Open Hardware (eg, RISC-V), we have to work with Intel/AMD/ARM vendors to reduce their blobs.
Sounds fair, but the criteria are written to apply to open source software. If it's a binary-only blob, the badging criteria won't apply anyway.

not sure if possible, but a CII project should be to try and do so, to help UEFI/coreboot/U-Boot.
I think that'd be better addressed by creating a proposal for a new CII-funded project. This mailing list just focuses on the "badging" work for OSS projects.

--- David A. Wheeler


Blibbet <blibbet@...>
 

I agree, but I think the "badging" work is the wrong forum for *that*
problem. It may be a *CII* issue, conceivably, but it's not really
a "badge".
...
I think that'd be better addressed by creating a proposal for a new
CII-funded project. This mailing list just focuses on the "badging"
work for OSS projects.
Yeah, I think I was trying to target CII, not specifics of a badge
program, sorry. Is there a more appropriate CII list?

I need to go study on what a CII badge means, and how that applies to
firmware. I'll write up my CII firmware notes and figure out where is
best place to discuss this at CII.

Thanks,
Lee


Emily Ratliff <eratliff@...>
 

Hi Lee,

The more appropriate list is CII-discuss. As we discussed via email, I will continue to recommend that the UEFI Forum (for the CA topic) or the Open Compute (for the binary firmware blobs) be considered for this topic rather than CII.

https://lists.coreinfrastructure.org/mailman/listinfo/cii-discuss


Emily

On Fri, Sep 4, 2015 at 5:04 PM, Blibbet <blibbet@...> wrote:
> I agree, but I think the "badging" work is the wrong forum for *that*
> problem. It may be a *CII* issue, conceivably, but it's not really
> a "badge".
...
> I think that'd be better addressed by creating a proposal for a new
> CII-funded project.  This mailing list just focuses on the "badging"
> work for OSS projects.

Yeah, I think I was trying to target CII, not specifics of a badge
program, sorry. Is there a more appropriate CII list?

I need to go study on what a CII badge means, and how that applies to
firmware. I'll write up my CII firmware notes and figure out where is
best place to discuss this at CII.

Thanks,
Lee

_______________________________________________
CII-badges mailing list
CII-badges@...
https://lists.coreinfrastructure.org/mailman/listinfo/cii-badges


Enos <temp4282138782@...>
 

Wheeler, David A wrote:
there will always be some criteria that are hard to make perfectly
black-and-white. E.G., the requirement that the project website
"succinctly describe what the software does [...]"
I think it is MUCH better to include important recommendations, even
if they're not required.
This discussion touched different concepts:

1. clear requirements (the project MUST do X)
2. conditional requirements (if X then the project must do Y)
3. fuzzy requirements (the project MUST do X or Y)
4. recommendations (the project SHOULD do X)

You're actually right on including important recommendations. Then I would suggest to label or rephrase them as requirements if possible.


[...] would be difficult to fully automate. [...]
Some of the limits of automation may be partly overcome by asking the project owner to fill a form and then store it in the Github repository:

A1. The project have an official website [YES|NO]
A2. The project website include contact information [YES|NO]
B1. etc.

BadgeApp would then parse the form.

In any case I suggest to deliberately ignore technical difficulties while setting the project goals. It reminds me of projects lowering their data classification rating, just because they did not want to (or could not) implement the corresponding security controls (true story).


it's hard to figure out if something is secure if you can't figure
out what it does :-).
I agree, a definition of security being "to precisely define what to do, and do nothing else" (although with complex systems things get "a bit" more complicated). By the way, are the security and quality of the "business logic" within the scope of this project?


[...] many projects will do more... but only if it's recommended.
[...] Most project participants *want* their projects to produce
good results; we're just giving them information on how they can get
there.
I agree in principle, still in my experience
- most of those who go after certifications, want recognition
- most of those who want good results don't care (much) about badges

So what's being built here is a guide which also has badges and not the contrary? (i.e. good results are prioritized over their measurement)
While I couldn't agree more, this is not reflected in the project's title, which revolves around the word "badge".


[...] the word "RECOMMENDED" is standard IETF terminology and it's
basically the normal English meaning, so most readers shouldn't be
confused.
Readers may still somehow be confused, because projects will do different things: BusyGuy will do the bare minimum, while PerfectDude will implement all recommendations. Their systems will display the same badge, but will incorporate different security measures. While users will only demand the minimum from both projects, the same users will not be put in the position to take advantage from PerfectGuy's edge.

A workaround could be to associate each recommendation to a score (even if fixed), then print the total on the badge (e.g. BRONZE, 93%). On a side note, badging may include (optional) registration on a database, upon which a badge would be provided after impressing multiple watermarks (including the score) on the certifying project's name.

A disadvantage of recommendations is that they duplicate information between badges. For example control X may be recommended in BRONZE, but also required in SILVER. Those certifying for silver will have to read the same thing twice (unless it is stated that all recommendations are required for the level above).


[...] when you cite other sources, people have to track those down
and read them too; if the burden is too high, people won't do it.
Yes, but most will read the document while connected. You could prefer sources where topics are introduced by concise explanations (e.g. Wikipedia).


We've focused on creating just one badge level to start with, and
then move from there.
Nice choice, but in that case be careful not to set the first bar too high. People in CII seem to be extremely qualified. What for you are the basics, for some others are science fiction. I'm referring to
- complexity (use of many different technologies in a single project)
- security (e.g. projects only dealing with public data)
- size (controls requiring a lot of time to be setup)


[...] This is the *perfect* time for brainstorming.
Then brace yourself, after we're done with this thread another one is coming...

--
Enos


p.s. I did not receive your message in my mailbox (while I did receive mine and others' replies)


David A. Wheeler
 

Enos said:
This discussion touched different concepts:
1. clear requirements (the project MUST do X) 2. conditional requirements (if X then the project must do Y) 3. fuzzy requirements (the project MUST do X or Y) 4. recommendations (the project SHOULD do X)
You're actually right on including important recommendations. Then I would suggest to label or rephrase them as requirements if possible.
It sounds like we're agreeing. I completely agree that "MUST" statements are better that "SHOULD" or "RECOMMENDED" statements in a criteria set, when we can actually do that. If we can't (e.g., too difficult), then in the most important cases we can make them "SHOULD" or "RECOMMENDED", and then possibly add them as potential criteria for higher-level badges.


Some of the limits of automation may be partly overcome by asking the project owner to fill a form and then store it in the Github repository:
A1. The project have an official website [YES|NO]
A2. The project website include contact information [YES|NO]
BadgeApp would then parse the form.
Yes, we absolutely * are* asking project owners to fill in a form. However, we're doing that as a web app, instead of having them fill in something and putting it in a GitHub repo. That makes it much easier to create a good & simple UI, many projects don't use GitHub, and it also simplifies the process of looking across all badges (so we can learn from them).

In any case I suggest to deliberately ignore technical difficulties while setting the project goals. It reminds me of projects lowering their data classification rating, just because they did not want to (or could not) implement the corresponding security controls (true story).
All actions have a cost in terms of time / effort / money. If projects had infinite resources, then we could include mandates for all sorts of crazy things. Infinite resources seems to be in short supply. Our current intent is to create a badge that is not too difficult, with criteria that most people would agree are minimums, and then (hopefully) later create higher-level badges. It'll be easier to get people to pursue higher-level badges once they get the first one.

If the bar is set too high, most projects will not try. If the bar is set too low, the badge will have no real value (it will cause no change in projects, and users won't care if it's achieved). It will be challenging to determine where to set the bar; that's why the Linux Foundation is making the development process public, to enable public comment like this.

I agree, a definition of security being "to precisely define what to do, and do nothing else" (although with complex systems things get "a bit" more complicated).
Few software systems are defined that precisely, so it's hard to nail that down well.

By the way, are the security and quality of the "business logic" within the scope of this project?
If the business logic is implemented by software, I should think so.


[...] many projects will do more... but only if it's recommended.
[...] Most project participants *want* their projects to produce
good results; we're just giving them information on how they can get there.
I agree in principle, still in my experience
- most of those who go after certifications, want recognition
- most of those who want good results don't care (much) about badges
Happy to provide recognition. And if the best projects easily meet the badge requirements, well, that was to be expected right?

So what's being built here is a guide which also has badges and not the contrary? (i.e. good results are prioritized over their measurement) While I couldn't agree more, this is not reflected in the project's title, which revolves around the word "badge".
It's the criteria for a badge, but there are many reasons to create badges; see the criteria and background text.

[...] the word "RECOMMENDED" is standard IETF terminology and it's
basically the normal English meaning, so most readers shouldn't be confused.
Readers may still somehow be confused, because projects will do
different things: BusyGuy will do the bare minimum, while PerfectDude
will implement all recommendations. Their systems will display the same
badge, but will incorporate different security measures. While users
will only demand the minimum from both projects, the same users will not
be put in the position to take advantage from PerfectGuy's edge.

In this scenario, we're still better off, because in both cases they did more than they probably would have otherwise: BusyGuy at least did the bare minimum, and PerfectDude did all the should/recommended things as well. (Presuming each of those things are actually good things to do.) It's possible that a user may not realize that only MUST statements are required, but it's not all that likely. Also, we intend for people to be able to click on the badge and see the details provided by the project (which explains how they met the requirements). So users who care can see more detail, and users who only want the top-level will just see a badge (which is STILL more info than they had before).


A workaround could be to associate each recommendation to a score (even
if fixed), then print the total on the badge (e.g. BRONZE, 93%). On a
side note, badging may include (optional) registration on a database,
upon which a badge would be provided after impressing multiple
watermarks (including the score) on the certifying project's name.

The "% of options" is an interesting idea, though for *now* I'm more interested in knowing % towards getting the badge at all. We already plan to have the database.

A disadvantage of recommendations is that they duplicate information
between badges. For example control X may be recommended in BRONZE, but
also required in SILVER. Those certifying for silver will have to read
the same thing twice (unless it is stated that all recommendations are
required for the level above).

I expect we'll do the "all recommendations required for the previous level" thing. I think most projects won't instantly meet the higher-level requirements, so we need to make it easy to do things incrementally.

[...] when you cite other sources, people have to track those down
and read them too; if the burden is too high, people won't do it.
Yes, but most will read the document while connected. You could prefer sources where topics are introduced by concise explanations (e.g. Wikipedia).
I'm happy to cite sources, I just want to limit what people *have* to read. Again, we want people to actually *get* the badge; if it requires too much reading, the number of projects that achieve it will drop.

> We've focused on creating just one badge level to start with, and
> then move from there.
Nice choice, but in that case be careful not to set the first bar too
high. People in CII seem to be extremely qualified. What for you are the
basics, for some others are science fiction. I'm referring to
- complexity (use of many different technologies in a single project)
- security (e.g. projects only dealing with public data)
- size (controls requiring a lot of time to be setup)

Agreed. It really will be a challenge to avoid "too high" AND "too low"; our (partial) solution is to do discuss it in public so people can raise this.


[...] This is the *perfect* time for brainstorming.
Then brace yourself, after we're done with this thread another one is coming...
:-).

--- David A. Wheeler