Skip to content

Scrap approval of test results, introduce color scale instead#1221

Open
mbuechse wants to merge 3 commits into
mainfrom
issue/1207
Open

Scrap approval of test results, introduce color scale instead#1221
mbuechse wants to merge 3 commits into
mainfrom
issue/1207

Conversation

@mbuechse

Copy link
Copy Markdown
Contributor

Show failed (some required testcase failed) as red, missing (some required testcase missing) as yellow, inconclusive (some required testcase aborted) as green with asterisk, passed (all required testcases passed) as green.

The reduces the probability of confusing people with false positives, while no longer requiring (unrealistically) labor-intensive approvals. Besides, the two views 'table_full' vs 'table' now only differ in that the former shows draft versions; similarly for 'details[_full]'.

Further work remains:

  • remove approval column from database
  • remove approval-related code from sql.py
  • remove endpoint for POSTing /results
  • simplify, consolidate, isolate business logic
  • etc.

We're doing this incrementally... for now, let's see whether this works at all.

@mbuechse mbuechse linked an issue Jun 17, 2026 that may be closed by this pull request
@mbuechse

Copy link
Copy Markdown
Contributor Author

I took the liberty to deploy this to the compliance monitor (after I tested it locally, of course). I found a defect, that I resolved. Now it should be good. Have a look!

BTW the views use the same color scale; the only difference is that the former shows draft versions.

This is the color scale:

COLOR_MAP = {
    -1: '🛑',  # fail
    None: '🟧',  # missing
    0: '✅*',  # inconclusive
    1: '✅',  # pass
}

I think the asterisk still needs to be explained, and it collides with the asterisk for draft versions. I'm sure we will get this ironed out.

@mbuechse

Copy link
Copy Markdown
Contributor Author

So, right now, with scs2, we have three testcases that couldn't be tested properly:

Screenshot from 2026-06-17 20-49-49

(the virtual machine didn't come up in time)

Which means: we don't know whether these testcases are actually satisfied or not.

Therefore, we now get the asterisk in the summary:

Screenshot from 2026-06-17 20-49-55

@mbuechse

Copy link
Copy Markdown
Contributor Author

Note: the asterisk for "draft" can probably just be dropped, because

a) we have the convention to call the draft "next", which is quite telling already,
b) the draft versions will only be shown in the "expert" view.

The double dagger for "deprecated" can probably be removed as well, because we have a convention to not show deprecated versions.

What remains is to put the legend of symbols at the top, and presto.

@mbuechse

Copy link
Copy Markdown
Contributor Author

After #1222, some subjects are missing the result for scs-0114-syntax-check, and so we see this:

Screenshot from 2026-06-18 19-41-51 Screenshot from 2026-06-18 19-42-04

@mbuechse

mbuechse commented Jun 20, 2026

Copy link
Copy Markdown
Contributor Author

Now the Sonobuoy tests on Noris take so long that they literally time out. Luckily, the reports got sent anyway (apparently, Zuul is kind of gentle with killing the process; I don't know how this works). Notice the unfinished test cases in the tables and the corresponding asterisks in the summary at the top

Screenshot from 2026-06-20 20-30-14

mbuechse added 3 commits June 22, 2026 23:23
Show failed (some required testcase failed) as red,
missing (some required testcase missing) as yellow,
inconclusive (some required testcase aborted) as green with asterisk,
passed (all required testcases passed) as green.

The reduces the probability of confusing people with false positives,
while no longer requiring (unrealistically) labor-intensive approvals.
Besides, the two views 'table_full' vs 'table' now only differ in that
the former shows draft versions; similarly for 'details[_full]'.

Further work remains:

- remove approval column from database
- remove approval-related code from sql.py
- remove endpoint for POSTing /results
- simplify, consolidate, isolate business logic
- etc.

We're doing this incrementally... for now, let's see whether this
works at all.

Signed-off-by: Matthias Büchse <matthias.buechse@alasca.cloud>
Signed-off-by: Matthias Büchse <matthias.buechse@alasca.cloud>
Signed-off-by: Matthias Büchse <matthias.buechse@alasca.cloud>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scrap approval of test results, introduce third state (yellow)

1 participant