A SCALABLE LONGITUDINAL DATA MINING FRAMEWORK FOR PREDICTING ANTIRETROVIRAL THERAPY (ART) INTERRUPTION WITHIN THE NIGERIAN NATIONAL DATA REPOSITORY

Abstract

Interruption in Treatment (IIT) remains a major challenge to durable HIV epidemic control in Sub-Saharan Africa. Nigeria’s National Data Repository (NDR) contains tens of millions of longitudinal clinical records, yet traditional analytic methods are limited in scalability and predictive utility. We developed a scalable, temporally aware data mining pipeline to transform irregular, multi-year patient trajectories into a longitudinal feature matrix for high-throughput model training and near-real-time inference. The four-stage architecture includes distributed ingestion, longitudinal feature engineering, imbalance-aware ensemble training, and facility-level alerting. Analyses were conducted on a de-identified extract of 3,969,496 patient records, with strict privacy safeguards ensuring that no patient-level identifiers were retained. Model performance was assessed using AUC-ROC, PR-AUC, and sensitivity at fixed specificity thresholds. A baseline model using demographic covariates achieved an AUC-ROC of 0.773. Incorporating treatment-history signals increased discrimination to 0.992, though this was partly driven by a data-completeness confound: inactive patients had fewer follow-up entries. Scalability was demonstrated across datasets ranging from 10⁴ to 3.97 × 10⁶ records. This longitudinally aware framework enables a shift from retrospective IIT auditing to proactive, facility-level retention alerts. It is directly relevant to Nigeria’s progress toward the UNAIDS 95-95-95 targets and to other EMR-scale HIV programs in the region.