[BUG] Recommendations are ineffective

[mbuechse]

What is a recommendation good for (as of now)?

1. it offers no guarantees, so users cannot rely on it;
2. operators don't get any points for achieving it;
3. if an environment achieves a recommendation, there is no telling whether this is intentional, and whether this will extend into the future or not.
4. they have a real cost because they make everything we do more complex; more testcases, more complicated formulas for evaluating compliance (where half the result is inconsequential), more complicated views in the compliance monitor;
5. our current assortment of recommendations is quite impenetrable: a long, heterogeneous list: some of them are strong recommendations bordering on requirement, some are kind proposals for operators with special interests. (Some rare recommendations are not represented as a test case.)

The first four items will never change, I presume.

(The fourth item could be alleviated by turning recommendations into requirements for a separate certificate scope or scope version; this way most of the machinery could be simplified, and the separate scope or scope version could be construed as secondary. Even so, the first two items still won't change.)

I heard that our goal with recommendations, as far as I can tell, is to document "good practice" so the ecosystem would evolve towards this direction. (However, who even takes not of the requirements? Usually, people are encouraged to just run the test suite and see if it passes. Once the requirements pass, people will leave it alone. And most implementation guides (like what we at FOCIS did for Yaook) usually omit the recommendations as well, because people don't have the patience or resources.

Can we prove this usecase, and if so, can we find an effective solution for it?

I don't want to get rid of recommendations per se, but if we don' find a way to make them effective, then yes, let's drop them (at least from certification).

[mbuechse]

Request for comment @fkr @depressiveRobot @jklare :-)

[mbuechse]

List of recommendations

To make it more concrete, here's a list of recommendations that we track using testcases (in some rare occasions, a recommendation might not be tracked by a testcase and either not tested at all or just cause some debug output to the log):

• (2 about entropy that ought to vanish with next scope version)
• scs-0102-prop-hash_algo: every public image has property hash_algo set
• scs-0102-prop-hypervisor_type: every public image has property hypervisor_type set
• scs-0102-prop-hw_rng_model: every public image has property hw_rng_model set
• scs-0102-prop-hotfix_hours: every public image has property hotfix_hours set
• (2 about property os_purpose that will be mandatory with next scope version)
• scs-0103-flavor-1v-4-10: flavor SCS-1V-4-10 is present
• (12 more of this type: "flavor with disk present")
• (3 more without disk, but mem >= 64 GiB)
• scs-0104-image-capi-2: CAPI image is present (no matter how recent)
• scs-0104-image-debian-12: Debian 12 image is present (ditto)
• scs-0114-encrypted-type: an encrypted volume type is present
• scs-0114-replicated-type: a replicated volume type is present
• scs-0116-presence: key-manager (Barbican) is present

Compliance status

• scs-0116-presence: 6 of 7 passing partners have it (regio-a doesn't); I still think it could be turned into a requirement;
• scs-0114-{encrypted,replicated}-type: NO ONE has it, no customer expects it, and the standard hesitated making them required because the aspect (encrypted, replicated) is encoded into the volume type's description, and the authors opted to wait for a "real" upstream solution (like a dedicated field);
• scs-0104-image-capi-2: 2 of 7 passing partners have it; so many of our partners use Gardener, so we should probably drop this (even more so since it doesn't imply any recency);
• scs-0104-image-debian-12: 7 of 7 passing partners have it; could be made a requirement;
• scs-0103-flavor-...:
  • 3 of 7 passing partners have all recommended flavors with disks;
  • 2 of 7 seem to have all three flavors with high mem, even though they are expressly not expected!
• scs-0102-prop-...: ALL partners have them; they could be made mandatory (wondering what value they use for hw_rng_model).

Proposal:

• find consensus for making scs-0116-presence mandatory,
• either drop standard on volume types (scs-0114) altogether or strengthen it a lot,
• drop CAPI image recommendation,
• make Debian stable mandatory (or drop),
• drop testcases for flavors with disks,
• make image properties mandatory

What's left are those higher-mem flavors. I get that we want to reduce fragmentation by suggesting kind of a "grid" for our partners to "snap onto", but these recommendations are too easily overlooked and maybe too weak for that. Again, I would probably drop them.

[mbuechse]

Please do comment @fzakfeld @chrisschwa @garloff :-)

[fkr]

Thanks @mbuechse for raising this topic. I like the approach of raising this and bringing it to everyones mind what recommendations are and your summary does that!

While reading through your thoughts, I did wonder about something:

One of the major points about recommendations is to propagate best practices, so that eventually those best practices become the new norm and we can promote them from recommendation to a requirement. However how could an encouragement for adopting recommendations (except for Karma Points) look like?

When I read again through your summary:

What is a recommendation good for (as of now)?

1. it offers no guarantees, so users cannot rely on it;
2. operators don't get any points for achieving it;
   [...]

this point struck me again. Maybe it makes sense to think about whether or not we could give points for following recommendations. The baseline would still be the required standards, but in order to drive adoption there needs to be some sort of reward.

[fkr]

Proposal:

• find consensus for making scs-0116-presence mandatory,
• either drop standard on volume types (scs-0114) altogether or strengthen it a lot,
• drop CAPI image recommendation,
• make Debian stable mandatory (or drop),
• drop testcases for flavors with disks,
• make image properties mandatory

I like your proposals and think they do make sense.
IF we strengthen the Volume Types A LOT that would something, where I think it would make sense to award it, IF providers follow a recommendation.

[fzakfeld]

I very much agree on the proposals. Simple things like image metadata can (at least in our view) be implemented without much effort.

Regarding the volume types - I think this is something where the IaaS offerings can differ from each other. Even something like encryption is a feature that not everyone supports (requires Barbican, which requires some secure secret store) and is also not always needed. Same would go for different storage tiers.

Also I don't like the idea of giving points here. We currently have a hand full of certified IaaS - does it really make sense to "rank" them on a fine scale?

[fkr]

Also I don't like the idea of giving points here. We currently have a hand full of certified IaaS - does it really make sense to "rank" them on a fine scale?

Sorry, I was easily misunderstood. It was not meant as a ranking - more as in a "Fleissternchen" ;) eg. a Mark that shows that the offering also implements the recommendations.

If we have recommendations that only apply to a specific set of offerings that this of course is also misleading - so far the train of thoughts was that all of the recommendations actually make sense to be implemented (otherwise they should not be recommendations - but this is also something where we might need to re-sharpen the picture).

[mbuechse]

The problem with the recommendations is that the number isn't really comparable. More recommendations isn't necessarily better, and achieving all of them (as in fkr's comment "implements the recommendations") may be theoretically possible, but that's not even our intention (see flavors with 64 Gigs or more of RAM).

[fkr]

The problem with the recommendations is that the number isn't really comparable. More recommendations isn't necessarily better, and achieving all of them (as in fkr's comment "implements the recommendations") may be theoretically possible, but that's not even our intention (see flavors with 64 Gigs or more of RAM).

Shows even more the importance of your initial point of making sure to define "what are recommendations" and thus make sure they're decoupled from the certification.

[mbuechse]

I suggest two things:

a) we proceed with my proposal regarding current recommendations; in particular, we check whether key-manager can be made mandatory (so hold off on stabilizing scs-0123-v2).
b) we remove recommendations from reporting; that is, they can remain in the standards texts, but they are no longer reported in the compliance monitor.

If we want to drive adoption, I suggest we do rallies where some of the operators pledge to implement a certain subset of recommendations, and then we hand out mini-certificates (badges) for that. This would solve another critical problem that recommendations currently have: it is totally unclear whether some environment achieves some recommendation on purpose or not, and there is no guarantee that the compliance will extend into the future.

[mbuechse]

Addendum regarding the rallies: this of course means more planning and more paperwork and just overall more work for everyone, particularly the assessment body, so I suppose the Forum has to decide whether to invest into this or not.