Model evaluation is only as useful as the data behind its test cases. Before scoring outputs, confirm that the evaluation set represents the tasks, users, and failure modes the system will encounter.
Start with completeness, duplication, formatting consistency, and label quality. Missing fields, repeated examples, mixed schemas, and ambiguous labels can distort results before the model is even reviewed.
Check coverage across important categories, edge cases, languages, user groups, and time periods. A dataset can be internally clean while still leaving major production scenarios untested.
Review provenance and handling requirements as well. Teams should know where the data came from, whether it can be used for the intended purpose, and which sensitive fields require removal or controlled access.
Document every issue and remediation decision. A repeatable data-quality checklist makes later model comparisons more defensible and helps reviewers distinguish model failures from test-data failures.