Building a Generalized Framework for Analyzing Public Procurement Data from the Kapak Database

Public procurement systems generate large, complex datasets that can reveal corruption risks; however, analyzing these datasets is challenging due to unstructured formats, fragmented sources, and technical barriers. This paper introduced a data pipeline that streamlines the processing of procurement data from Ecuador's Official Public Procurement System (SOCE, for its acronym in Spanish), expanding on the Kapak project, which uses big data and data science to promote transparency. Specifically, Kapak implements a web crawler that periodically collects extensive procurement data from Ecuador's public procurement website SOCE. Nevertheless, its raw format-comprising base64encoded USHAY files, fragmented documents, and scattered JSON files-hinders effective analysis. To address this, our pipeline automates decoding, file reconstruction, and dataset consolidation, thereby ensuring that the data is ready for analysis. Subsequently, we evaluated, the pipeline on Reverse Electronic Auction (REA) documents, using TF-IDF and cosine similarity to detect patterns among high-risk procurement processes. The analysis specifically focused on technical specifications, bid documents, and stakeholder Question and Answer (Q&A) records. Overall, the pipeline offers a generalizable, modular framework that shifts focus from preprocessing to analysis. In addition, it supports integration with advanced Natural Language Processing (NLP) models, making it a valuable tool for corruption detection and public procurement oversight.