Data normalization of historical parliamentary affairs and documents

Classification systems

Finland

Use case ID: 046

Author: Parliament of Finland

Date: 2 July 2024
 

Objective:

Automatically normalize parliamentary affairs and documents (2001–2014) to match the logical structure of newer data in order to make materials more findable, accessible, interoperable and reusable (FAIR) for search and application programming interfaces (APIs).

Actors:

  • Software vendor
  • Parliamentary business owners
  • Parliamentary IT office staff
  • Artificial intelligence (AI) tools

Prerequisites:

  • Database of parliamentary affairs and documents, 2001–2014 (SGML) 
  • Database of parliamentary affairs and documents, 2015–2024 (XML) 
  • Data and schema model for 2015–2024
  • Machine learning service trained on the data and schema model
  • Natural language processing (NLP) service that uses machine learning to identify insights and relationships in text

Scenario:

  1. The software vendor collects the data from the defined source databases, with the help of IT office staff. 
  2. The software vendor filters low-quality, incorrect and otherwise unwanted data from the data in the source database, and possibly derives the necessary missing information, with support from business owners.
  3. The software vendor stores the converted data in the actual data warehouse for further processing.
  4. The software vendor identifies key candidates for the relationship and selects the most suitable one as the basic key of the relationship in cooperation with business owners.
  5. The software vendor and IT office staff provide data to the AI tools for normalization (input), as well as learning material (expected output) against which the input is normalized.
  6. After AI normalization, the software provider implements the normalization check on a relation-by-relation basis. The normalization check requires the intervention (randomly or other technique) of business owners for assurance that the result is correct.

Alternate flows:

  • The data normalization work is done entirely manually.

Expected results:

  • Data in SGML format can be normalized to the newer XML schema.
  • Parliamentary affairs and documents 2001–2024 use the same data model and schema model.
  • Similar search factors can be applied to parliamentary affairs and documents, regardless of the year the material originates from.

Potential challenges:

  • Serious flaws in defining the tool’s presets in order to unify the data
  • Excessively imprecise results from automatic processing, requiring more manual work.
  • Insufficiently accurate automatic data processing, potentially leading to the need for data normalization to be re-evaluated, as the amount of manual work is likely to outweigh the benefits

Data requirements:

  • Accurate training material for data analysis

Integrations with other systems:

  • Machine learning service
  • NLP service 
  • Data modelling and analytics tools
  • Final operational database

Success metrics:

  • The data are reaching the third normal form (3NF).
  • Data content in the operational database is 98% correct.
  • As a result of automatic normalization, no more than 2% errors are found in parliamentary affairs and documents in relation to the amount of processed material.

 

The Use cases for AI in parliaments collection is published by the IPU’s Centre for Innovation in Parliament as part of the Parliamentary Data Science Hub’s project to create guidelines for AI governance in parliaments.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International licence. It may be freely shared and reused with acknowledgement of the author and the IPU. 

A use case describes how a system should work. It is used to plan, develop and measure implementation. A use case is not the same as a case study, which is a descriptive text of an actual project’s implementation. Please note that this use case is provided “as is” and neither the IPU nor the author accepts any responsibility for its use.

For more information about the IPU’s work on artificial intelligence, please visit www.ipu.org/AI or contact [email protected].