Retrieval of relevant documents for the lawmaking expert handling an MP’s demand

Use case ID: 020

Author: Chamber of Deputies of Brazil

Date: 14 June 2024

Objective:

Read the text of demands sent by MPs to the Legislative Consultancy Department for new bill drafting or research to support new bills, interpret this text, and present to the lawmaking expert the 12 (or more) most relevant documents (e.g. existing bills, previous demands) that are similar in subject to the demand.
Actors:

• MPs submitting demands
• Legislative Consultancy Department experts (lawmaking experts)
• AI system for retrieving relevant documents similar to the subject of an MP’s demand

Prerequisites:

• Database of textual documents to search, such as existing bills and previous demands
• Automated document indexing or vectorization process, including proper tokenization and semantic representation
• Database to store user-relevance feedback needed to improve future searches
• Integration with digital services commonly used by MPs and legislative consultants

Scenario:

1. The lawmaking expert selects an MP’s demand.
2. The AI system generates a query for document searching, focusing on the subject of the demand. This query can be enhanced with contextual information, including relevant laws and bill citations, to improve accuracy and relevance.
3. The AI ​​system searches the database for documents relevant to the generated query.
4. The AI system presents to the lawmaking expert a list of 12 (or more) documents, ranked by similarity to the subject of the demand. The ranking takes into account previous feedback on similar requests.
5. The lawmaking expert evaluates each proposed document and gives relevance feedback.

Alternate flows:

• When necessary, the lawmaking expert can inform the system about relevant documents not included in the list provided via the AI system interface.

Expected results:

• The process of searching for relevant documents to handle a given demand is more efficient.
• Less time is needed for law drafting.
• The relevance of AI system results is continuously enhanced through the accumulation of feedback over time.

Potential challenges:

• Ensuring continuous improvement of the AI system over time
• Encouraging users to provide relevance feedback

Data requirements:

• The content of laws and bills must be available in text format
• Adhere to the requirements applicable to the consulted documents, with particular attention to confidentiality
• Periodic verification of AI system performance

Integrations with other systems:

• Digital services commonly used by MPs and legislative consultants
• OCR tool for extracting text from images, PDFs and other formats
• Analytics and reporting tools

Success metrics:

• Recall: proportion of relevant documents that are successfully retrieved by the system
• Precision: proportion of retrieved documents that are relevant to the query
• R-precision: precision at the point where the number of retrieved documents equals the number of relevant documents for a given query
• Volume of searches conducted on the AI system
• Amount of relevance feedback provided by users of the AI system