ISO 19157: Data Quality Framework for AI Systems

ISO 19157 establishes essential principles for describing and evaluating data quality. Initially designed for geographic information but increasingly valuable in artificial intelligence (AI) systems. This international standard provides organizations with structured methodologies to assess how well datasets conform to specifications, ensuring that data quality meets the requirements for specific AI applications, including not just geographic datasets but broad data types.

As AI becomes more integrated into critical systems, the need for standardized quality assessment frameworks, such as those provided by this standard, has never been more urgent. The ISO 19157:2013 framework offers a robust foundation adaptable to ensure AI systems are built on reliable, high-quality data. By adhering to data quality measures, organizations can address specific dataset requirements and maintain sufficient quality standards.

​

The Six Core Data Quality Elements of ISO 19157

ISO 19157 defines six fundamental data quality elements, applicable to AI systems and contributing to an ISO-compliant quality framework:

1. Completeness: Evaluates whether all required data is present, accounting for both commission (excess data) and omission (missing data). In AI systems, completeness ensures training datasets include all necessary examples across the gamut of expected inputs, achieving minimum acceptable levels.
2. Logical Consistency: Ensures data follows expected patterns and relationships in AI applications, thereby preventing models from learning invalid correlations. At the structural level, conceptual consistency ensures alignment with the overarching data schema, while domain consistency enforces adherence to valid value ranges. On a technical level, format consistency confirms that data follows a uniform physical structure, and topological consistency validates the correctness of relational or spatial characteristics.
3. Positional Accuracy: Measures the accuracy of position features within a spatial reference system. For AI, this translates to precision in feature space positioning and the accuracy of numerical values, particularly important in geographical or gridded data applications.
4. Thematic Accuracy: Evaluates the accuracy of quantitative attributes and correctness of non-quantitative attributes. For AI systems, this ensures labels and classifications are correct—critical for supervised learning models and other AI applications.
5. Temporal Quality: Assesses the accuracy of temporal attributes and temporal relationships. In AI applications, this ensures time-series data maintains correct sequencing and relationships, supporting criteria in data quality reports.
6. Usability: Provides information about the dataset's suitability for a particular application. For AI systems, this helps determine if a dataset meets the intended use, supporting the content structure designed for particular application scenarios.

Applying ISO 19157 to AI Data Quality Assessment

The ISO 19157 data quality evaluation process involves four key steps that can be adapted to AI systems:

1. Specify data quality units: Define the scope of evaluation, identifying applicable quality elements for each AI dataset component.
2. Specify data quality measures: Determine appropriate metrics, like standardized data quality measures, to evaluate each quality element, including: percentage of missing values, number of duplicate instances, classification error rates, and distribution skewness measures.
3. Specify evaluation procedures: Define methods for quality assessment, which may be direct internal, direct external, or indirect.
4. Determine evaluation output: Perform the quality assessment, documenting results in standardized formats for complete transparency.

​

ISO 19157 and Emerging AI Standards

ISO 19157 principles are increasingly being included in emerging AI-specific standards, including:

• ISO/IEC 42001 for AI management systems.
• ISO/IEC 23053 for AI concepts and terminology.
• ISO 19178-1 for training data markup language, enhancing the ecosystem of AI lifecycle quality standards.

ISO 19157 and Training Data Markup Language for AI

Recent developments connect ISO 19157 principles with AI-specific standards. The Training Data Markup Language for Artificial Intelligence (TrainingDML-AI) incorporates this standard in data quality concepts to formalize and document training data, characterizing its content, metadata, quality, and provenance. It encompasses quality-related information, showcasing the relevance to artificial intelligence beyond geographic data.

Benefits and Governance Integration of ISO 19157

Implementing ISO 19157 principles for AI data quality provides a robust framework that transforms raw data into a reliable, governed asset. Organizations adopting these standards can anticipate: Enhanced transparency through standardized quality reporting, which provides a clear understanding of dataset limitations. This leads to improved decision-making, as high-fidelity quality information helps stakeholders determine if specific datasets are truly suitable for their intended AI applications.
The use of standardized quality descriptions also simplifies dataset comparison, allowing teams to evaluate multiple data sources side-by-side with ease. This consistency facilitates data sharing, ensuring that complete quality metadata promotes the safe reuse of data across different departments or external partners. Finally, these structured assessments are a critical tool for regulatory compliance, directly assisting organizations in meeting the strict data governance requirements of emerging laws like the EU AI Act.

​By integrating ISO 19157 principles into AI governance frameworks, organizations establish a systematic approach to data quality management. This integration provides a structured foundation for the systematic identification of data quality issues, ensuring that well-defined characteristics are addressed early in the development pipeline. A key advantage is the standardized documentation of quality assessment procedures, often utilizing XML schema implementations to ensure consistency across different technical environments. This leads to consistent reporting of quality metrics, which holds data producers accountable and ensures that stakeholders have a clear view of data integrity. Furthermore, this approach enables the traceability of quality issues throughout the entire AI lifecycle, enhancing the ability to audit and refine datasets as models evolve.

The versatility of the updated ISO 19157-1:2023 standard allows organizations to define domain-specific quality descriptors and measures while maintaining ISO compliance, essential for addressing domain-specific components.

Practical Implementation Steps

To implement its principles for AI data quality:

1. Establish quality requirements: Define minimum acceptable levels for each data quality element according to AI application needs.
2. Design quality assessment procedures: Develop standardized procedures for evaluating each quality element, ensuring acceptance testing protocols.
3. Implement quality monitoring: Integrate quality assessment into your data pipeline, focusing on the intended use and content structure.
4. Document quality results: Create standardized quality reports following ISO 19157 principles, incorporating data quality encoding standards.
5. Act on quality findings: Establish procedures for addressing identified quality issues effectively.

Challenges in Applying ISO 19157 to AI

While ISO 19157 provides a valuable framework, organizations may face challenges when applying it to AI systems:

• Adapting geospatial-focused measures to diverse AI data types.
• Handling the scale and complexity of modern AI datasets, including non-specific geographic data.
• Balancing comprehensive quality assessment with computational efficiency.
• Addressing quality aspects unique to AI, such as fairness and bias.

Organizations can overcome these challenges by leveraging the extensibility of ISO 19157-1:2023, which allows defining domain-specific quality components while maintaining compliance with the standard framework.

Embracing standardized frameworks like ISO 19157 helps organizations build reliable, trustworthy AI systems that deliver consistent value while minimizing risks. Ready to enhance your AI data quality management? Contact our experts to learn how ISO 19157 principles can strengthen your AI governance framework and support regulatory compliance.