The everyday problem
A scoring sheet can appear objective while rewarding features that the team will never use. Equal weighting makes a minor convenience count as much as a critical operational requirement. A high total score can therefore conceal a poor fit for the work that matters most.
A practical approach
Separate non-negotiable requirements from weighted preferences. Tie weights to the problem statement and document why each criterion matters. Preserve qualitative notes for important uncertainties rather than converting every unknown into an arbitrary numerical score.
Try this with your team
Review your five highest-weighted criteria. For each, describe the operational consequence if it is absent and the evidence needed to confirm that the vendor meets it.