Sample Platform · CI Correctness

False-Positive CI Status on Zero Comparison Rows

June 20265 min read

-- sample-platform#1119
SELECT COUNT(DISTINCT t.id) AS confirmed_false_positive
FROM test t JOIN test_progress tp ...
WHERE t.test_type = 'commit'
  AND crashes = 0 AND results = 0
  AND missing_comparison_rows > 0;
-- Result: 10  (876 affected)

-- fix: replace dual-count with get_test_results()

Background

The Sample Platform pushes GitHub commit statuses (SUCCESS/FAILURE) after running tests. Some commits were receiving SUCCESS incorrectly.

Root Cause

A dual-count logic flaw: when a test produced 0 comparison rows, the status calculation code interpreted this as 'no failures' and returned Status.SUCCESS. Zero rows should mean no comparisons ran — which is a failure state, not success.

The Fix

Replaced the flawed status derivation with get_test_results(), the same function used by the existing PR test path, which correctly handles the zero-row case.

Evidence

Production DB query identified 10 false-positive commits and 876 affected test run records. Cross-referenced with GitHub API — confirmed SUCCESS status was pushed for those specific commit SHAs.

Result

PR sample-platform#1119. A quietly lying CI status is worse than a failing one — this had been silently masking test infrastructure gaps.