ISTQB CT-AI: Testing AI-Based Systems

Testing AI systems requires adapting traditional testing concepts to handle the unique challenges these systems present. This article covers CT-AI syllabus chapters 6-8: ML Neural Networks and Testing, Testing AI-Based Systems Overview, and Testing AI-Specific Quality Characteristics.

You'll learn how traditional test levels apply to AI systems, how to handle non-deterministic behavior, how to test for AI-specific concerns like automation bias and concept drift, and specific approaches for neural network testing.

Chapter 6: Neural Networks and Testing

Neural networks power many modern AI applications, from image recognition to language models. Understanding their basic structure helps you design appropriate tests.

Neural Network Basics

What is a Neural Network?

A neural network is a computational model inspired by biological neurons. It consists of interconnected nodes (neurons) organized in layers that process information.

Basic structure:

• Input layer: Receives data features
• Hidden layers: Process information through weighted connections
• Output layer: Produces predictions

How it works:

1. Input data enters the input layer
2. Each neuron applies weights to inputs and an activation function
3. Information flows through hidden layers
4. The output layer produces final predictions

Deep Learning

Deep learning uses neural networks with many hidden layers. The "depth" allows learning increasingly abstract representations:

• Early layers might detect edges in images
• Middle layers might detect shapes
• Later layers might detect objects

Deep networks can learn complex patterns but require more data and computation.

Why Neural Networks Are Hard to Test

Opacity: You can't easily understand why a network made a specific prediction. The weights and activations don't map clearly to human-understandable concepts.

Complexity: Networks may have millions or billions of parameters. Traditional path coverage is meaningless with this many possible states.

Non-linearity: Small input changes can cause large output changes, making behavior hard to predict.

Emergent behavior: Networks can learn unexpected behaviors not explicitly programmed.

These characteristics require specialized testing approaches.

Coverage Criteria for Neural Networks

Traditional code coverage (statement, branch) doesn't apply meaningfully to neural networks. Alternative coverage criteria have been developed.

Neuron Coverage

Definition: The proportion of neurons that have been activated (output above a threshold) during testing.

Formula: Number of neurons activated / Total neurons

Intuition: If a neuron never activates during testing, you haven't tested the behavior it contributes to.

Limitations:

• High neuron coverage is easy to achieve
• Doesn't guarantee meaningful testing
• Ignores how neurons interact

Layer Coverage

Definition: Coverage measured at the layer level rather than individual neurons.

Approaches:

• Percentage of neurons activated per layer
• Distribution of activations within layers
• Comparison of activation patterns across inputs

k-Multisection Neuron Coverage

Definition: Divides each neuron's activation range into k sections and measures how many sections are covered.

Intuition: Tests should exercise neurons across their activation ranges, not just activate them.

Benefit: More discriminating than simple neuron coverage.

Neuron Boundary Coverage

Definition: Tests whether neurons are activated near their boundaries (just above and just below activation thresholds).

Intuition: Boundary behavior often reveals edge cases and potential problems.

Modified Condition/Decision Coverage for Neural Networks

Adapts traditional MC/DC concepts:

• Each neuron's activation should independently affect the output
• Tests should show that changing individual neuron activations changes predictions

Practical Neural Network Testing

Beyond coverage metrics, practical testing includes:

Input diversity: Test with varied inputs covering the expected input space.

Edge cases: Test with unusual but valid inputs.

Adversarial examples: Test with inputs designed to fool the network.

Comparison testing: Compare network outputs to known-correct outputs or simpler models.

Sanity checks: Verify the network produces sensible outputs (within expected ranges, consistent with domain knowledge).

Chapter 7: Testing AI-Based Systems Overview

This chapter adapts traditional testing concepts to AI systems.

Test Levels for AI

Traditional test levels (unit, integration, system, acceptance) still apply to AI systems, but with AI-specific considerations.

Unit Testing for AI

Traditional software: Test individual functions and classes in isolation.

AI systems: Test individual components of the ML pipeline:

• Data preprocessing functions
• Feature engineering code
• Model inference functions
• Postprocessing logic

Challenges:

• The "model" itself is difficult to unit test
• Data transformations may have subtle correctness requirements
• Feature engineering can introduce bugs

Integration Testing for AI

Traditional software: Test interactions between components.

AI systems: Test interactions between:

• Data sources and preprocessing pipelines
• Feature engineering and model
• Model and postprocessing
• ML system and application

Challenges:

• Data format mismatches between pipeline stages
• Feature definitions may drift between training and inference
• Model versions may be incompatible with application code

System Testing for AI

Traditional software: Test the complete integrated system.

AI systems: Test the complete AI system including:

• End-to-end prediction accuracy
• System performance under load
• Behavior across the full input space
• Quality characteristics (fairness, robustness)

Challenges:

• Defining "correct" system behavior for AI
• Coverage of the vast input space
• Testing emergent behaviors

Acceptance Testing for AI

Traditional software: Verify system meets business requirements.

AI systems: Verify the AI system meets:

• Business performance requirements
• Regulatory compliance
• User expectations
• Ethical standards

Challenges:

• Translating business requirements into testable metrics
• Validating fairness and ethical compliance
• Ensuring user trust calibration

The Test Oracle Problem

A fundamental challenge in AI testing is determining what's "correct."

What is the Test Oracle Problem?

A test oracle determines whether a test passed or failed. For traditional software, oracles compare actual outputs to expected outputs.

For AI systems, expected outputs are often unknown:

• What's the correct sentiment of a nuanced review?
• What's the correct next word in a sentence?
• What's the correct steering angle for an autonomous vehicle?

Without clear expected outputs, traditional pass/fail testing doesn't work.

Strategies for Addressing the Oracle Problem

Human evaluation: Experts review samples of outputs for correctness. Expensive and doesn't scale, but provides ground truth for important cases.

Statistical validation: Instead of verifying individual predictions, verify aggregate statistics (accuracy, precision, recall) meet requirements.

Metamorphic testing: Test relationships between inputs and outputs rather than specific expected values (covered in Chapter 9).

Reference models: Compare outputs to a trusted reference implementation.

Invariant testing: Verify properties that should always hold regardless of specific outputs (predictions within valid range, consistent with constraints).

Differential testing: Compare outputs of multiple models or model versions. Disagreements warrant investigation.

Automation Bias

Automation bias is a critical human factor concern in AI systems.

What is Automation Bias?

Automation bias is the tendency to over-rely on automated system outputs, accepting them without adequate critical evaluation.

Manifestations:

• Accepting incorrect AI recommendations without verification
• Not noticing AI errors
• Blaming AI failures on other factors
• Reduced vigilance when AI is present

Why Automation Bias Matters

Safety risks: In high-stakes domains (medical, aviation), blind trust in AI can cause serious harm.

Error propagation: AI errors that go unquestioned propagate through downstream decisions.

Accountability issues: Who is responsible when humans blindly follow incorrect AI advice?

Trust miscalibration: Users may trust AI more (or less) than appropriate.

Testing for Automation Bias Risk

Confidence communication: Test that the system communicates uncertainty appropriately. Users should know when predictions are uncertain.

Override mechanisms: Test that humans can easily override AI decisions. Verify overrides are logged and analyzed.

Explanation quality: Test that explanations help users evaluate predictions rather than just justify acceptance.

Edge case handling: Test how the system handles cases it shouldn't handle autonomously.

User interface: Evaluate whether the UI encourages appropriate scrutiny rather than blind acceptance.

Concept Drift and Continuous Testing

AI systems can degrade over time as the world changes.

Understanding Concept Drift

Concept drift: The relationship between inputs and outputs changes over time.

Examples:

• Fraud patterns evolve as fraudsters adapt
• Customer preferences shift with trends
• Medical best practices update with research
• Economic conditions change purchasing behavior

Impact: A model trained on historical data becomes less accurate as the concept drifts.

Detecting Concept Drift

Performance monitoring: Track prediction accuracy over time. Declining accuracy suggests drift.

Prediction distribution monitoring: Track the distribution of predictions. Unusual shifts suggest drift.

Feature distribution monitoring: Compare current feature distributions to training data. Significant changes suggest drift.

Statistical tests: Formal tests comparing distributions between time periods.

Responding to Concept Drift

Retraining: Update the model with recent data.

Sliding windows: Train only on recent data, discarding old data.

Ensemble approaches: Combine models trained on different time periods.

Online learning: Continuously update the model as new data arrives.

Continuous Testing for AI

Because AI systems can drift, testing must be continuous, not just pre-deployment:

Regular model evaluation: Periodically evaluate model performance on recent data.

Automated monitoring: Set up alerts for performance degradation.

A/B testing: Compare model versions in production.

Shadow deployment: Run new models alongside production models before switching.

Canary releases: Roll out changes gradually, monitoring for problems.

Chapter 8: Testing AI-Specific Quality Characteristics

This chapter provides practical approaches for testing AI quality characteristics.

Testing Self-Learning Systems

Some AI systems continue learning after deployment, which creates unique testing challenges.

What are Self-Learning Systems?

Online learning systems: Update models continuously as new data arrives.

Reinforcement learning systems: Learn from feedback during operation.

Personalization systems: Adapt to individual user behavior.

Testing Challenges

Moving target: The system changes as you test it.

Feedback loops: Testing can affect learning, which affects future behavior.

Emergent behavior: Learning may produce unexpected behaviors.

Reproducibility: The same test may produce different results at different times.

Testing Approaches

Snapshot testing: Freeze the model for testing, then release.

Behavior bounds: Verify behavior stays within acceptable bounds regardless of learning.

Learning rate testing: Test that learning proceeds at appropriate rates.

Stability testing: Verify the system doesn't diverge or become unstable.

Adversarial testing: Test that malicious inputs can't corrupt learning.

Rollback capability: Verify the system can revert to previous states.

Handling Non-Determinism

Many AI systems don't produce exactly the same output for the same input across runs.

Sources of Non-Determinism

Probabilistic outputs: Some models intentionally include randomness (sampling from distributions).

Floating-point operations: Different hardware or libraries may produce slightly different results.

Initialization randomness: Model training uses random initialization.

Environment variations: Caching, timing, and other environmental factors may affect results.

Testing Non-Deterministic Systems

Statistical testing: Instead of checking exact outputs, verify statistical properties (mean, variance, distribution).

Tolerance bands: Accept outputs within acceptable ranges rather than exact values.

Multiple runs: Run tests multiple times and analyze the distribution of results.

Seed control: When possible, fix random seeds for reproducibility during testing.

Property-based testing: Verify properties that should hold regardless of specific outputs.

Distribution testing: Verify that output distributions match expectations.

Setting Appropriate Tolerances

Too tight: Tests fail due to acceptable variation, creating false failures.

Too loose: Tests pass despite significant problems, creating false passes.

Calibration: Analyze typical variation to set appropriate thresholds.

Context-dependent: Tolerance depends on application requirements.

Testing Explainability

Verifying that explanations meet requirements.

What to Test

Availability: Are explanations provided when required?

Accessibility: Can users access explanations easily?

Understandability: Do target audiences understand the explanations?

Accuracy: Do explanations reflect actual model reasoning?

Consistency: Are similar predictions explained similarly?

Completeness: Do explanations cover relevant factors?

Testing Approaches

Automated checks: Verify explanations are generated and contain required elements.

User studies: Have representative users evaluate explanation quality.

Expert review: Have domain experts assess explanation accuracy.

Comparison testing: Compare explanations to known feature importance.

Manipulation testing: Verify that changing highlighted factors changes predictions.

Testing Fairness and Bias

Practical approaches for verifying fairness.

Defining Test Criteria

Before testing, define:

• Which groups to evaluate (protected attributes)
• Which fairness metrics apply (demographic parity, equalized odds, etc.)
• What thresholds are acceptable
• How to handle metric conflicts

Testing Approaches

Subgroup analysis: Calculate performance metrics for each protected group and compare.

Statistical testing: Use statistical tests to determine if differences are significant.

Counterfactual testing: Test if changing only protected attributes changes predictions.

Intersectionality testing: Test performance for intersections of protected groups (e.g., young women, elderly men).

Historical bias detection: Analyze training data for encoded historical bias.

Fairness Testing Challenges

Data availability: You may not have protected attribute data.

Proxy detection: Bias may operate through correlated features.

Metric conflicts: Different fairness metrics may conflict.

Context dependency: Appropriate fairness criteria depend on the application.

Testing Robustness

Verifying system resilience to varied conditions.

Input Robustness

Noise injection: Add random noise to inputs and measure performance degradation.

Missing data: Test with missing values and incomplete records.

Format variations: Test with different input formats and edge cases.

Out-of-distribution testing: Test with data outside the training distribution.

Adversarial Robustness

Known attacks: Test against documented adversarial attack types.

Perturbation testing: Test sensitivity to small input changes.

Boundary testing: Test at decision boundaries where small changes flip predictions.

Defense evaluation: If defenses are implemented, test their effectiveness.

Operational Robustness

Load testing: Test performance under high load.

Stress testing: Test behavior at system limits.

Recovery testing: Test recovery from failures.

Integration testing: Test robustness of integrations with other systems.

Robustness Metrics

Accuracy under noise: Performance with noisy inputs.

Perturbation threshold: How much perturbation causes prediction changes.

Recovery time: How quickly the system recovers from failures.

Degradation profile: How performance degrades with increasing stress.

---

Test Your Knowledge

---

---

Frequently Asked Questions

Frequently Asked Questions (FAQs) / People Also Ask (PAA)