ISO 19157: Data Quality Framework for AI Systems

ISO 19157 establishes essential principles for describing and evaluating data quality—initially designed for geographic information but increasingly valuable in artificial intelligence (AI) systems. This international standard provides organizations with structured methodologies to assess how well datasets conform to specifications, ensuring that data quality meets the requirements for specific AI applications, including not just geographic datasets but broad data types.

As AI becomes more integrated into critical systems, the need for standardized quality assessment frameworks, such as those provided by this standard, has never been more urgent. The ISO 19157:2013 framework offers a robust foundation adaptable to ensure AI systems are built on reliable, high-quality data. By adhering to data quality measures, organizations can address specific dataset requirements and maintain sufficient quality standards.

The Six Core Data Quality Elements of ISO 19157

ISO 19157 defines six fundamental data quality elements, applicable to AI systems and contributing to an ISO-compliant quality framework:

1. Completeness

Evaluates whether all required data is present, accounting for both commission (excess data) and omission (missing data). In AI systems, completeness ensures training datasets include all necessary examples across the gamut of expected inputs, achieving minimum acceptable levels.

2. Logical Consistency

Assesses adherence to logical rules of the data structure, including:

• Conceptual consistency: Conformance to the conceptual model and schema.
• Domain consistency: Adherence to value domains, essential for specific dataset analysis.
• Format consistency: Degree to which data is stored according to physical structure.
• Topological consistency: Correctness of explicit topological characteristics.

Logical consistency ensures data follows expected patterns and relationships in AI applications, thereby preventing models from learning invalid correlations.

3. Positional Accuracy

Measures the accuracy of position features within a spatial reference system. For AI, this translates to precision in feature space positioning and the accuracy of numerical values, particularly important in geographical or gridded data applications.

4. Thematic Accuracy

Evaluates the accuracy of quantitative attributes and correctness of non-quantitative attributes. For AI systems, this ensures labels and classifications are correct—critical for supervised learning models and other AI applications.

5. Temporal Quality

Assesses the accuracy of temporal attributes and temporal relationships. In AI applications, this ensures time-series data maintains correct sequencing and relationships, supporting criteria in data quality reports.

6. Usability

Provides information about the dataset's suitability for a particular application. For AI systems, this helps determine if a dataset meets the intended use, supporting the content structure designed for particular application scenarios.

Applying ISO 19157 to AI Data Quality Assessment

The ISO 19157 data quality evaluation process involves four key steps that can be adapted to AI systems:

1. Specify data quality units: Define the scope of evaluation, identifying applicable quality elements for each AI dataset component.
2. Specify data quality measures: Determine appropriate metrics, like standardized data quality measures, to evaluate each quality element, including:
• Percentage of missing values
• Number of duplicate instances
• Classification error rates
• Distribution skewness measures

• Specify evaluation procedures: Define methods for quality assessment, which may be direct internal, direct external, or indirect.
• Determine evaluation output: Perform the quality assessment, documenting results in standardized formats for complete transparency.

ISO 19157 and Training Data Markup Language for AI

Recent developments connect ISO 19157 principles with AI-specific standards. The Training Data Markup Language for Artificial Intelligence (TrainingDML-AI) incorporates this standard in data quality concepts to formalize and document training data, characterizing its content, metadata, quality, and provenance. It encompasses quality-related information, showcasing the relevance to artificial intelligence beyond geographic data.

As noted in the Open Geospatial Consortium's code sprint, ISO 19157-3 data quality measures are crucial for AI applications, especially within the context of JSON encoding of TrainingDML-AI.

Benefits of Implementing for AI Systems

Organizations implementing ISO 19157 principles for AI data quality can anticipate:

1. Enhanced transparency: Standardized quality reporting improves the understanding of dataset limitations.
2. Improved decision-making: Quality information helps determine if datasets are suitable for specific AI applications.
3. Simplified dataset comparison: Standardized quality descriptions facilitate easier comparison between multiple datasets.
4. Facilitated data sharing: Complete quality descriptions promote appropriate dataset sharing and reuse.
5. Regulatory compliance: Structured quality assessments assist in meeting emerging AI regulations like the EU AI Act.

ISO 19157 in the AI Governance Framework

Integrating ISO 19157 principles into AI governance frameworks provides organizations with a systematic approach to data quality management. This integration supports:

• Systematic identification of data quality issues and addressing well-defined characteristics.
• Standardized documentation of quality assessment procedures using XML schema implementation.
• Consistent reporting of quality metrics to ensure data producers maintain accountability.
• Traceability of quality issues throughout the AI lifecycle enhances the comprehensive list of quality aspects.

The versatility of the updated ISO 19157-1:2023 standard allows organizations to define domain-specific quality descriptors and measures while maintaining ISO compliance, essential for addressing domain-specific components.

Practical Implementation Steps

To implement its principles for AI data quality:

1. Establish quality requirements: Define minimum acceptable levels for each data quality element according to AI application needs.
2. Design quality assessment procedures: Develop standardized procedures for evaluating each quality element, ensuring acceptance testing protocols.
3. Implement quality monitoring: Integrate quality assessment into your data pipeline, focusing on the intended use and content structure.
4. Document quality results: Create standardized quality reports following ISO 19157 principles, incorporating data quality encoding standards.
5. Act on quality findings: Establish procedures for addressing identified quality issues effectively.

Challenges in Applying ISO 19157 to AI

While ISO 19157 provides a valuable framework, organizations may face challenges when applying it to AI systems:

• Adapting geospatial-focused measures to diverse AI data types.
• Handling the scale and complexity of modern AI datasets, including non-specific geographic data.
• Balancing comprehensive quality assessment with computational efficiency.
• Addressing quality aspects unique to AI, such as fairness and bias.

Organizations can overcome these challenges by leveraging the extensibility of ISO 19157-1:2023, which allows defining domain-specific quality components while maintaining compliance with the standard framework.

ISO 19157 and Emerging AI Standards

ISO 19157 principles are increasingly being included in emerging AI-specific standards, including:

• ISO/IEC 42001 for AI management systems.
• ISO/IEC 23053 for AI concepts and terminology.
• ISO 19178-1 for training data markup language, enhancing the ecosystem of AI lifecycle quality standards.

Practical Applications & Future Considerations

ISO 19157 provides a robust framework for data quality assessment, significantly enhancing AI system reliability. By adapting its principles to AI datasets, organizations can:

• Implement structured quality assessment procedures based on four conformance classes.
• Document quality characteristics in standardized formats, adding guidance for general procedures.
• Make informed decisions about dataset suitability and received data.
• Address quality issues systematically with a focus on comprehensive quality lists.

To implement ISO 19157 principles for your AI systems:

1. Review current data quality assessment practices.
2. Identify gaps between current practices and ISO 19157 requirements.
3. Develop a roadmap for implementing standardized quality assessment.
4. Explore how AI regulatory compliance requirements align with ISO 19157 principles.

Embracing standardized frameworks like ISO 19157 helps organizations build reliable, trustworthy AI systems that deliver consistent value while minimizing risks.

Ready to enhance your AI data quality management? Contact our experts to learn how ISO 19157 principles can strengthen your AI governance framework and support regulatory compliance.