A number of projects are close to a badge, but don’t (yet) quite have it.  Perhaps the criteria are too hard.  Perhaps the criteria are fine, but we’re asking people to make reasonable improvements (and thus we’re making things better by pressuring for change).  Or maybe some criteria are not adequately clear, and need some clarifications.  We can at least look at the dataset and see what’s going on.

So I did an analysis just now of all projects at 90% or more that have *not* achieved a badge, and extracted out *only* the criteria that *MUST* be met.  I present it below.

I’d like feedback.  My hope is that by doing this analysis we’ll get a better idea of what the issues are.

Anyway, the most-missed ones (with “Unmet” or “?”) in order, starting with most-unmet or unknown, are as follows:

["tests_are_added_status", 13, 5, 2, 0],

["sites_https_status", 14, 5, 1, 0],

["crypto_certificate_verification_status", 4, 0, 5, 11],

["vulnerability_report_process_status", 16, 2, 2, 0],

["test_policy_status", 16, 3, 1, 0],

["know_secure_design_status", 17, 0, 3, 0],

["know_common_errors_status", 17, 0, 3, 0],

["delivery_unsigned_status", 17, 1, 2, 0],

["static_analysis_status", 15, 2, 1, 2] …

The numbers are counts of met, unmet, unknown, and N/A.

Clearly the biggest issues are adding tests for new functionality, HTTPS for sites, crypto certificate verification, and documenting the vulnerability report process.  I don’t think those are unreasonable criteria, but clearly they’re the most common hurdles.  There are many possible actions.  We can keep the criteria as they are, make adjustments to the criteria, or perhaps find ways to help projects meet those criteria.  So, let’s think about it – with the end goal of improving FLOSS projects (especially their security).

Below is the full list, and the code to create it.

--- David A. Wheeler

=== FULL LIST (sorted MUSTs, totals for projects >=90% and <100% ===

[["tests_are_added_status", 13, 5, 2, 0], ["sites_https_status", 14, 5, 1, 0], ["crypto_certificate_verification_status", 4, 0, 5, 11], ["vulnerability_report_process_status", 16, 2, 2, 0], ["test_policy_status", 16, 3, 1, 0], ["know_secure_design_status", 17, 0, 3, 0], ["know_common_errors_status", 17, 0, 3, 0], ["delivery_unsigned_status", 17, 1, 2, 0], ["static_analysis_status", 15, 2, 1, 2], ["report_responses_status", 18, 1, 1, 0], ["vulnerability_report_private_status", 15, 0, 2, 3], ["vulnerability_report_response_status", 18, 1, 1, 0], ["crypto_working_status", 10, 2, 0, 8], ["crypto_password_storage_status", 3, 1, 1, 15], ["crypto_random_status", 7, 1, 1, 11], ["dynamic_analysis_fixed_status", 18, 2, 0, 0], ["documentation_interface_status", 19, 1, 0, 0], ["warnings_status", 16, 1, 0, 3], ["warnings_fixed_status", 17, 1, 0, 2], ["crypto_published_status", 11, 1, 0, 8], ["crypto_keylength_status", 7, 0, 1, 12], ["vulnerabilities_fixed_60_days_status", 19, 1, 0, 0], ["static_analysis_fixed_status", 16, 0, 1, 3], ["description_good_status", 20, 0, 0, 0], ["interact_status", 20, 0, 0, 0], ["contribution_status", 20, 0, 0, 0], ["floss_license_status", 20, 0, 0, 0], ["license_location_status", 20, 0, 0, 0], ["documentation_basics_status", 20, 0, 0, 0], ["discussion_status", 20, 0, 0, 0], ["repo_public_status", 20, 0, 0, 0], ["repo_track_status", 20, 0, 0, 0], ["repo_interim_status", 20, 0, 0, 0], ["version_unique_status", 20, 0, 0, 0], ["release_notes_status", 20, 0, 0, 0], ["release_notes_vulns_status", 20, 0, 0, 0], ["report_process_status", 20, 0, 0, 0], ["report_archive_status", 20, 0, 0, 0], ["build_status", 16, 0, 0, 4], ["test_status", 20, 0, 0, 0], ["crypto_floss_status", 12, 0, 0, 8], ["delivery_mitm_status", 20, 0, 0, 0], ["no_leaked_credentials_status", 20, 0, 0, 0]]

=== CODE ===

$ cat get-criteria-info

#!/bin/sh

rails console <<CONSOLE_END

results = []

# my_criteria = Project::ALL_CRITERIA_STATUS

my_criteria = Criteria.select { |c| c.must? }

puts my_criteria

my_criteria = my_criteria.map { |c| c.name.to_s + '_status' }

puts my_criteria

my_criteria.each do |criterion|

  data=Project.where('badge_percentage >= 90 AND badge_percentage < 100').select(criterion.to_s).group(criterion.to_s).unscope(:order).count

  results.append([criterion.to_s, data.fetch('Met',0),

    data.fetch('Unmet',0), data.fetch('?',0), data.fetch('N/A',0)])

end

results.sort! { |x,y| -(x[2]+x[3]) <=> -(y[2]+y[3]) }

puts 'criterion,Met,Unmet,?,N/A'

results.each do |row|

  puts row.join(',')

end

CONSOLE_END