Development and Validation of a Mortality Risk Prediction Index Score for Adults Living with HIV and Multiple Chronic Comorbidities

Abstract

Introduction
Aging while living with HIV poses new challenges in clinical management, mainly due to the onset of multiple chronic comorbidities. Population-specific risk prediction indices considering comorbidities and other risk factors are essential to comprehensively characterise disease burden among PLWH. We developed and validated a mortality risk prediction index i to predict the risk of one-year all-cause mortality among people living with HIV (PLWH).

Methods
Participants were ≥18 years and had initiated antiretroviral therapy (ART) between 01/2001 and 12/2018, in British Columbia, Canada. The index date was randomly selected between one-year post-ART initiation and the end of the follow-up. Participants were followed for at least one year from the index date until 12/2019, the last contact date, or the date of death (all-cause), whichever came first. The MRPi included 18 physical/mental comorbidities, demographic and clinical variables, and ranged from 0 (no risk) to 100 (highest risk).

Results
The final model demonstrated the highest discrimination (c-statistic 0.8355, 95% CI: 0.8187-0.8523 in the training dataset and 0.7965, 95% CI: 0.7664-0.8266 in the test dataset). The comorbidities with the highest weights in the MRPi were substance use disorders, metastatic solid tumors and non-AIDs defining cancers. For example, for an MRPi of 30, the predicted one-year all-cause mortality was 0.2%, while an MRPi of 50 had a predicted mortality of 2.3%.

Conclusions
The MRPi provides a promising tool to assess the risk of short-term mortality among PLWH in the modern ART era that can inform clinical practice and health policy decisions.

Introduction

Following advancements in antiretroviral therapy (ART), there has been a shift in the prognosis of HIV infection from an acute, fatal illness to a manageable, yet complex, chronic condition [1]. As a result, the incidence of AIDS-defining events has declined [2], and the life expectancy of people living with HIV (PLWH) has improved approaching that of the general population [3, 4]. Consequently, many high-resource settings are experiencing a significant demographic shift among PLWH. For example, in British Columbia (BC), Canada, the proportion of diagnosed PLWH ≥50 years has increased from 47.0% in 2013 to 65% in 2022 [5, 6].

Aging while living with HIV poses new challenges in clinical management, mainly due to the onset of multiple chronic comorbidities. In BC, the proportion of PLWH with at least one comorbid condition, such as cardiovascular disease (CVD), chronic kidney disease (CKD), chronic obstructive pulmonary disease (COPD), non-AIDS-defining cancers (NADC), and chronic liver disease (CLD), is higher than their demographically similar HIV-negative counterparts [7]. Additionally, chronic inflammation, uncontrolled viremia, socioeconomic status, and lifestyle factors (e.g., substance use) exacerbate the risk of comorbidities among PLWH [1, 8, 9]. These chronic comorbidities are largely responsible for the continued excess mortality observed in this population, compared with the general population, despite the success of ART [10]. Clinical management of comorbidities may lead to drug-drug interactions and higher healthcare utilisation and costs, which represent an additional challenge for the individual and an added pressure for the healthcare system [11, 12]. Therefore, population-specific risk prediction weighted indices considering comorbidities and other risk factors are essential to comprehensively characterise disease burden among PLWH.

Previous validated weighted indices in the general population include the Charlson Comorbidity Index, Elixhauser Comorbidity Index, and the John Hopkins Adjusted Clinical Group Case-Mix System [13–15]. While these indices consider comorbid conditions in the general adult population, they are not appropriate to use among PLWH. However, there are few indices specifically for PLWH. In the past, risk prediction indices were derived from traditional biomarkers, such as HIV plasma viral load (pVL), CD4, and the presence of AIDS-defining events, all of which had limitations in their ability to accurately reflect the effects of HIV and ART on morbidity and mortality [16, 17]. In the modern ART era, the Veterans Aging Cohort Study (VACS) index 1.0 was derived based on HIV and non-HIV biomarkers and was later updated to the VACS index 2.0 with more variables and a superior ability to predict mortality and other adverse health outcomes [18–20]. Although evidence supports the accuracy of the VACS index in predicting outcomes in PLWH [21], some of its biomarkers (e.g., fibrosis-4, albumin, white blood count) are not routinely monitored by most ART programs and are not available in patient registries [19]. With the increasing use of administrative data in health research [22], a population-specific risk prediction weighted index relevant to this data source would be best for widespread applicability. Therefore, we sought to develop and validate a mortality risk prediction index (MRPi) to predict the risk of one-year all-cause mortality among PLWH in BC. Although long-term (e.g., five-year) mortality predictions can offer broader insights, there is a critical need for short-term (one-year) risk assessment in clinical practice. Many individuals with multiple comorbidities require urgent interventions, and clinicians often need to prioritise resources and treatment decisions based on the likelihood of near-term adverse outcomes. Therefore, a one-year mortality risk tool is particularly relevant for guiding immediate care strategies and ensuring timely support for those most at risk.

Methods

Data source

The Seek and Treat for Optimal Prevention of HIV/AIDS (STOP HIV/AIDS) cohort is a de-identified population-based cohort of all diagnosed PLWH in BC, followed between April 1^st, 1996, and March 31^st, 2020. The STOP HIV/AIDS cohort was formed through the annual linkage between the BC Centre for Excellence in HIV/AIDS’ Drug Treatment Program registry and other provincial administrative databases, as described in Supplementary Table S1 [23]. It is important to mention that, in BC, ART and related medical and laboratory monitoring are available free of charge to all PLWH residing in the province, which is likely to minimise any bias related to access to care in our analyses.

Study participants

Eligible participants met the following criteria: (i) initiated ART between January 1, 2001, and December 31, 2018, in BC; (ii) were ≥18 years old at the first ART date; (iii) had >1 pVL measurement between the first ART date and the end of follow-up; (iv) were ART naïve at baseline; (v) had a pVL ≥50 copies/mL at the first ART date; (vi) had >1 year of follow-up after ART initiation; (vii) had ≥5 years of administrative data before the randomly selected index date to ascertain comorbidity status [24]; and (viii) had >1 CD4 measurement between 1.5 years before and 6 months after the index date. The index date itself was chosen at least one year after the participant’s ART initiation (i.e., between one-year post-ART and the end of follow-up). Index dates were chosen randomly so that results from the final model would not be relative to a specific clinical visit date. We excluded PLWH who had documented structured treatment interruptions or participated in blinded trials during follow-up (see Supplementary Figure S1). These criteria were used to enhance the reproducibility of our study and minimise potential biases.

We followed all eligible participants for >1 year from the index date until i) December 31^st, 2019; ii) the last contact date (i.e., the last filled ART prescription refill date, the last available laboratory test date or the date of the last interaction with the healthcare system); or iii) the date of death (all-cause), whichever came first. Baseline pVL and CD4 nadir (lowest CD4) values were obtained within the above-mentioned window period. We omitted anyone missing a CD4 nadir measurement within the window period. Supplementary Figure S1 outlines the step-by-step process for inclusion in the final analytic sample.

Outcome and predictors

The outcome of interest was a one-year all-cause mortality probability. The mortality data was obtained from the BC Vital Statistics Agency database, available through STOP HIV/AIDS [25]. We chose chronic comorbidities: CVD, CLD, COPD, CKD, diabetes mellitus (DM), hypertension (HTN), substance use disorders (SUD), Alzheimer’s/dementia (AD/D), NADC, personality disorder (PD), schizophrenia (SCZ), mood/anxiety disorder (MAD), rheumatoid arthritis (RA), metastatic solid tumor (MST), osteoporosis (OP), asthma, peptic ulcer (PU), and osteoarthritis (OA), which are responsible for the excess mortality observed among PLWH despite the success of ART [7, 10]. These comorbidities were assumed to be irreversible once identified and considered present until the end of follow-up. In line with recent recommendations [26], we selected a broad list of 18 comorbidities that are most prevalent and clinically impactful among people living with HIV. We used administrative data definitions for each condition (see Supplementary Table S2), ensuring that we captured a wide spectrum of chronic comorbidities. We acknowledge that other conditions may also be relevant; future work with more comprehensive data could explore additional comorbidities.

The prevalence of each comorbidity was determined using a five-year lookback window from the index date [24], which is consistent with recommendations from Nanditha et al. [24], who demonstrated that longer windows substantially improve the accuracy of identifying chronic diseases in administrative data. Although a shorter window (e.g., two years) might have retained more participants, it risks missing previously diagnosed, but currently stable, conditions. Therefore, we opted for five years to minimise misclassification of comorbidities, aligning with other validated comorbidity indices. We acknowledge that this requirement reduced the eligible sample, but we believe it provides a more robust identification of chronic comorbidities. We ascertained diagnoses of these comorbidities based on the BC Ministry of Health (BC-MoH) case-finding algorithms or literature (where applicable) using the International Classification of Diseases (specifically the ninth revision with clinical modifications, and the Canadian tenth revision version) and Drug/Product Identification Numbers (Supplementary Table S2) [27]. Due to limitations inherent in administrative datasets, we were unable to capture the severity levels or staging of these chronic comorbidities. Consequently, comorbidities in our models were treated as binary variables (present vs. absent).

Other predictors included sex at birth (male, female), age (continuous), years on ART (i.e., years since ART initiation; continuous), pVL suppression (i.e., all pVL measurements <200 copies/mL; yes/no), CD4 nadir (<50, 50-199, 200-349, ≥350 cells/mm³; or continuous), and income assistance (yes/no), measured within the window period mentioned above. In multivariable analysis, we considered linear, quadratic, and cubic polynomial forms for age per 10-year increase, years on ART, and CD4 nadir per 100 cells/mm³, and we centred these variables at the median to control for multicollinearity. Due to a high proportion of missing data for ethnicity in our database, we used ‘income assistance’ as a proxy for socioeconomic status.

Statistical analyses

Categorical variables were expressed as counts and percentages while continuous variables were expressed as median values with 25^th to 75^th percentiles (Q1-Q3). In bivariable analyses, we compared categorical variables using the Chi-Square or Fisher’s exact test under the applicable conditions and continuous variables using the Wilcoxon rank-sum test [28]. We did not have an independent cohort for external validation. Thus, we performed internal validation using data-splitting (or split-sample) and a bootstrapping-based method and presented both ways. This method accounts for the over-optimism of using unadjusted bootstrapping results and adjusts the model for overfitting [29]. Cox proportional hazard regression models were used to derive the MRPi to predict one-year all-cause mortality probability. The coefficients of the multivariable models were presented as adjusted hazard ratios (aHRs) with 95% Wald Confidence Intervals (CI). Analyses were performed using SAS version 9.4 (SAS Institute, Inc. Cary, NC, USA) and R statistical software version 3.0.2 (R Foundation for Statistical Computing, Vienna, Austria). All p-values were double-sided, and significance was at the 5% level.

Model development

To improve readability and reduce redundancy, we provide a concise overview of the model-building steps here, while detailed derivation procedures are given in Supplementary Text S1.

Data-splitting method

We randomly split the sample into training and test datasets with a ratio 2:1 [29]. The training dataset was used for the model development and later validated on the test dataset. We built Cox proportional hazard regression models on the training dataset to assess the relationship between selected predictor variables and one-year all-cause mortality. We used a variable selection approach based on the Akaike information criterion and Type III p-value [30]. We considered the possibility that certain predictors (e.g., age, CD4 nadir, viral load suppression) might interact with comorbidities to influence one-year all-cause mortality. Accordingly, we tested these interaction terms in preliminary models. However, none of these interactions reached statistical significance or meaningfully improved model discrimination (Harrell’s c-statistic) or calibration. To maintain parsimony and interpretability, we, therefore, excluded interaction terms from the final model. We adopted a three-model approach to systematically assess the incremental predictive gain of each additional set of covariates. Model 1 included only comorbidities, Model 2 added age and sex at birth, and Model 3 further incorporated HIV-specific variables (e.g., plasma viral load suppression) and socioeconomic status (income assistance). This stepwise strategy allowed us to evaluate how much discrimination and calibration improved with each new set of predictors.

MRP index score derivation and mortality probability estimation

We used the coefficients derived from the chosen final predictive model (from the data-splitting method) to create the MRPi. First, we multiplied each coefficient by the value of the corresponding predictor to get a risk coefficient. Then, we summed the risk coefficients for each participant. Next, participants’ MRPi scores were calculated as shown in Supplementary Text S1. Scores range from 0 to 100 (higher values = higher outcome risk). Next, the predicted one-year all-cause mortality probability was computed using the baseline Cox survival function and the sum of risk coefficients [29]. To compute each participant’s one-year all-cause mortality probability, we applied the baseline Cox survival function to the sum of each individual’s risk coefficients. The full equation, including the baseline hazard and parameter estimates, is provided in Supplementary Text S1. Also, to facilitate clinical use, we developed both a web-based calculator and an Excel-based tool to estimate the MRPi and the corresponding one-year all-cause mortality probability. The web-based calculator provides a user-friendly interface, allowing clinicians to enter key patient characteristics to obtain an immediate mortality risk estimate. We have included the link to the web-based calculator and instructions for using the Excel tool in the Supplementary Data File 2 (Mortality Risk Prediction Calculator).

Model performance

We assessed the model’s performance through discrimination and calibration. For predictive discrimination, we calculated Harrell’s concordance statistic (c-statistic) [31]. The c-statistic ranges from 0.5 to 1.0, with 0.5 considered discrimination by chance alone, 0.70-0.79 considered good, and ≥0.80 excellent [32]. We calculated the 95% CI for each c-statistic, assuming a normal distribution [28]. The final predictive model was the model with the best discriminative ability. For calibration, we assessed the agreement between the predicted mortality probability and observed mortality probability using a calibration plot [29].

Data-splitting internal validation

The Cox model coefficients estimated from the training dataset were applied to the test dataset along with the estimation of the c-statistic. Calibration was performed as follows. We first predicted one-year all-cause mortality using our final Cox model derived from the training dataset [20]. Second, we binned the participants in the test dataset into ten subgroups based on a 5-point interval of their predicted mean MRPi [20]. Next, we estimated the observed one-year all-cause mortality probabilities using the Kaplan-Meier (KM) method and 95% CI for each subgroup. Finally, we graphically compared the predicted and the observed one-year all-cause mortality probabilities for the subgroups. The agreement between the predicted curve and the observed points indicates how well the model was calibrated.

Bootstrapping-based internal validation

The bootstrapping-based internal method estimates the future performance of the model on new participants by providing bias-corrected estimates when the model is applied to a new sample [29]. We carried out this intuitive approach to emphasise the internal validity of the MRPi. To perform the bootstrapping-based internal validation, we first built a Cox model using the overall dataset, and the best model was chosen based on its performance, as described above. We estimated c-statistics and predicted survival probabilities by applying the model to the full cohort. Then, we drew 400 bootstrap samples with replacement from the full cohort; for each bootstrap sample, we fitted a Cox model to measure its apparent performance. The optimism was estimated following the method by Harrell et al. [29, 31]. We then used the average of the optimism over all 400 bootstrap samples as a correction factor to estimate a validated performance (optimism-adjusted) measure by subtracting the estimated optimism from the apparent performance. For the calibration plot, we calculated and compared the observed and adjusted KM (y-axis) to the predicted (x-axis) one-year survival probabilities, where participants were binned into ten subgroups using deciles of their predicted one-year survival probabilities [29]. A 45-degree line with a slope equal to one and intercept equal to zero (perfect calibration) indicates how well the model was calibrated. Any deviation above or below this line implies the difference between the observed and predicted on year survival probabilities.

Secondary analyses

First, we compared participants included in this study with those excluded to address external validity. Next, we performed sex-based analyses to assess whether the performance of the MRPi varied by sex at birth (male vs. female). Specifically, we repeated the final model-building steps described above, stratifying by sex.

Results

Population characteristics

Table 1 shows the descriptive characteristics of the full cohort, comparing the training and test datasets at the index date. Among the 4,387 participants, 3,570 (81%) were male, 3,116 (71%) had a suppressed pVL, and 762 (18%) had a CD4 nadir <200 cells/mm³. The median age was 47 years (Q1-Q3: 38-54), follow-up time was 3.3 years (Q1-Q3: 1.4–6.3), and time on ART was 4.0 years (Q1-Q3: 2.0–6.0). MAD was the most prevalent comorbidity (2,154, 49%), followed by SUD (1,720, 39%), CLD (735, 17%), asthma (596, 14%), HTN (561, 13%), and the least prevalent, RA (38, 1%). Participants in the test dataset and training sets were very comparable, except for asthma prevalence at the index date where those in the test dataset were significantly more likely to have asthma than those in the training dataset (Table 1).

Table 1: Descriptive characteristics of the full cohort and the cohort participants included in training and test datasets for people living with HIV in British Columbia, Canada, from 2002 to 2019. Q1-Q3: 25^th – 75^th percentiles; ART: antiretroviral therapy; p-value significance is set at 0.05.

Variables	Full cohort (n = 4387) n (%)	Test dataset (n = 1463) n (%)	Training dataset (n = 2924) n (%)	p-value
Sex at birth
Female	817 (19)	284 (19)	533 (18)	0.3637
Male	3570 (81)	1179 (81)	2391 (82)
Death during the study period
No	3673 (84)	1238 (85)	2435 (83)	0.2740
Yes	714 (16)	225 (15)	489 (17)
Cardiovascular disease
No	4032 (92)	1355 (93)	2677 (92)	0.2456
Yes	355 (8)	108 (7)	247 (8)
Chronic kidney disease
No	4095 (93)	1362 (93)	2733 (93)	0.6883
Yes	292 (7)	101 (7)	191 (7)
Diabetes mellitus
No	4064 (93)	1362 (93)	2702 (92)	0.4460
Yes	323 (7)	101 (7)	222 (8)
Chronic liver disease
No	3652 (83)	1196 (82)	2456 (84)	0.0666
Yes	735 (17)	267 (18)	468 (16)
Non-AIDS-defining cancers
No	4010 (91)	1335 (91)	2675 (92)	0.8392
Yes	377 (9)	128 (9)	249 (8)
Chronic obstructive pulmonary disease
No	4098 (93)	1380 (94)	2718 (93)	0.0964
Yes	289 (7)	83 (6)	206 (7)
Hypertension
No	3826 (87)	1290 (88)	2536 (87)	0.1927
Yes	561 (13)	173 (12)	388 (13)
Substance use disorder
No	2667 (61)	863 (59)	1804 (62)	0.0893
Yes	1720 (39)	600 (41)	1120 (38)
Alzheimer’s/Dementia
No	4272 (97)	1432 (98)	2840 (97)	0.1697
Yes	115 (3)	31 (2)	84 (3)
Personality disorder
No	3978 (91)	1317 (90)	2661 (91)	0.3160
Yes	409 (9)	146 (10)	263 (9)
Schizophrenia
No	4124 (94)	1377 (94)	2747 (94)	0.8707
Yes	263 (6)	86 (6)	177 (6)
Mood/Anxiety disorder
No	2233 (51)	767 (52)	1466 (50)	0.1620
Yes	2154 (49)	696 (48)	1458 (50)
Rheumatoid arthritis
No	4349 (99)	1448 (99)	2901 (99)	0.5277
Yes	38 (1)	15 (1)	23 (1)
Metastatic solid tumor
No	4308 (98)	1436 (98)	2872 (98)	0.9703
Yes	79 (2)	27 (2)	52 (2)
Osteoporosis
No	4258 (97)	1426 (98)	2832 (97)	0.2954
Yes	129 (3)	37 (2)	92 (3)
Asthma
No	3791 (86)	1236 (84)	2555 (87)	0.0095
Yes	596 (14)	227 (16)	369 (13)
Peptic ulcer
No	4079 (93)	1358 (93)	2721 (93)	0.8228
Yes	308 (7)	105 (7)	203 (7)
Osteoarthritis
No	4124 (94)	1370 (94)	2754 (94)	0.5179
Yes	263 (6)	93 (6)	170 (6)
CD4 nadir, cells/mm3
<50	172 (4)	61 (4)	111 (4)	0.1779
50–199	590 (14)	202 (14)	388 (13)
200–349	895 (20)	271 (19)	624 (21)
350+	2730 (62)	929 (63)	1801 (62)
Plasma viral load suppression
No	1271 (29)	446 (31)	825 (28)	0.1266
Yes	3116 (71)	1017 (69)	2099 (72)
Income assistance
No	2799 (64)	932 (64)	1867 (64)	0.9508
Yes	1588 (36)	531 (36)	1057 (36)
Continuous covariates	Median (Q1-Q3)	Median (Q1-Q3)	Median (Q1-Q3)
Index year	2015 (2011-2017)	2015 (2011-2017)	2015 (2011-2017)	0.8439
Age at index date (years)	47 (38-54)	46 (38-54)	47 (39-55)	0.2169
Year of ART initiation	2009 (2006-2013)	2009 (2006-2013)	2009 (2006-2013)	0.3408
Years on ART at index date (years)	4.0 (2.0-6.0)	4.0 (2.0-6.0)	4.0 (2.0-6.0)	0.4012
CD4 nadir at index date (cells/mm3)	430 (250-610)	426 (250-620)	430 (250-300)	0.6017
Follow-up time at end of follow-up (years)	3.3 (1.4-6.3)	3.4 (1.4-6.4)	3.2 (1.3-6.3)	0.2933

Overall, 714 (16%) deaths were observed with a mortality rate of 38.18 per 1,000 PYs (95% CI: 35.46-41.06 per 1,000 PYs) by the end of follow-up, and 54.09 per 1,000 PYs (95% CI: 47.22-61.70 per 1,000 PYs) in the first year. The KM survival curve from the index date to the end of the study period is shown in Supplementary Figure S2.

Model development and performance

Table 2 summarises the multivariable-adjusted association of the predictors with one-year all-cause mortality and the c-statistics for the training and test datasets. Model 3 demonstrated superior discrimination (0.8355, 95% CI: 0.8187–0.8523 in the training dataset and 0.7965, 95% CI: 0.7664–0.8266 in the test dataset) of one-year all-cause mortality to Models 1 and 2, and it was chosen as the final predictive model. Among all predictors in Model 3, SUD (aHR 2.60, 95% CI: 2.08–3.27), MST (aHR 2.27, 95% CI: 1.47–3.50) and NADC (aHR 2.16, 95% CI: 1.63–2.86) had the strongest association with one-year all-cause mortality. Increasing CD4 nadir per 100 cells/mm³ and years on ART were associated with a reduced aHR (aHR 0.84 per 100 cells/mm³ increase, 95% CI: 0.80-0.88 and aHR 0.94 per one-year increase, 95% CI: 0.89–0.98, respectively) for one-year all-cause mortality. Other predictors significantly associated with higher one-year all-cause mortality risk are shown in Table 2. The unadjusted results for each variable in the training dataset are shown in Supplementary Table S3. The adjusted model (Supplementary Table S4) yielded bootstrapping-based adjusted c-statistic approximately the same as the unadjusted c-statistic and slightly lower than the c-statistic in the final predictive model, suggesting good discrimination (0.8227 [95% CI: 0.8077–0.8377] versus 0.8188 [95% CI: 0.8180–0.8196]).

Table 2: Cox proportional models showing adjusted hazard ratios based on the training dataset and Harrell’s c-statistics for the training and test datasets. ART: Antiretroviral Therapy; aHR: adjusted hazard ratio; CI: Confidence Interval; ^aLinear form of age centered at the median (47 years); ^bPolynomial forms of CD4 nadir centered at the median (430 cells/mm³); ^cLinear form of years on ART centered at the median (4 years).

Variables	Model 1 aHR (95% CI)	Model 2 aHR (95% CI)	Model 3 aHR (95% CI)
Cardiovascular disease (Ref: No)
Yes	1.93 (1.51-2.47)	1.88 (1.45-2.42)	1.89 (1.46-2.43)
Chronic liver disease (Ref: No)
Yes	1.79 (1.46-2.20)	1.72 (1.40-2.12)	1.72 (1.40-2.11)
Chronic kidney disease (Ref: No)
Yes	1.85 (1.43-2.38)	1.85 (1.43-2.40)	1.62 (1.25-2.11)
Chronic obstructive pulmonary disease (Ref: No)
Yes	2.02 (1.58-2.57)	1.84 (1.43-2.36)	2.06 (1.59-2.65)
Non-AIDS-defining cancers (Ref: No)
Yes	2.15 (1.64-2.82)	2.00 (1.51-2.64)	2.16 (1.63-2.86)
Substance use disorder (Ref: No)
Yes	2.92 (2.38-3.59)	3.04 (2.46-3.76)	2.60 (2.08-3.27)
Alzheimer’s/Dementia (Ref: No)
Yes	1.79 (1.30-2.47)	1.63 (1.17-2.26)
Metastatic solid tumor (Ref: No)
Yes	2.44 (1.59-3.76)	2.50 (1.62-3.86)	2.27 (1.47-3.50)
Diabetes mellitus (Ref: No)
Yes			1.28 (0.95-1.75)
Hypertension (Ref: No)
Yes		0.79 (0.61-1.03)	0.79 (0.60-1.04)
Sex at birth (Ref: Male)
Female		1.31 (1.06-1.61)	1.22 (0.99-1.50)
Income assistance (Ref: No)
Yes			1.33 (1.08-1.63)
aAge per 10-year increase		1.20 (1.09-1.33)	1.35 (1.22-1.49)
Plasma viral load suppression (Ref: Yes)
No			1.63 (1.31-2.03)
bCD4 nadir per 100 cells/mm3
Linear			0.84 (0.80-0.88)
Quadratic			1.01 (1.00-1.03)
cYears on ART
Linear			0.94 (0.89-0.98)
Quadratic			1.02 (1.00-1.04)
Cubic			0.99 (0.99-1.00)
Dataset	Harrell’s c-	Harrell’s c-	Harrell’s c-
	statistic	statistic	statistic
	(95% CI)	(95% CI)	(95% CI)
Training dataset	0.7766	0.7850	0.8355
	(0.7563-0.7969)	(0.7650-0.8050)	(0.8187-0.8523)
Test dataset	0.7538	0.7552	0.7965
	(0.7216-0.7859)	(0.7228-0.7876)	(0.7664-0.8266)

MRPi interpretation and risk calculator

The median MRPi in our population was 49.6 (Q1-Q3: 43.1-57.5). As the MRPi increased, the predicted one-year all-cause mortality also increased. For example, let us consider the hypothetical case of a male participant, 46 years old, on ART for four years with no chronic comorbidities, a CD4 nadir of 500 cells/mm³, and a suppressed pVL. This participant’s estimated MRPi was 41 and his one-year predicted all-cause mortality probability was 0.8%. If he was also diagnosed with SUD, his MRPi would increase to 49, and his one-year predicted all-cause mortality would rise to 2.0%. Alternatively, if his CD4 nadir dropped to 150 cells/mm³, his MRPi would be 47, and his predicted mortality would be 1.6%. Finally, if his pVL became unsuppressed, but his CD4 nadir remained the same, his MRPi would be 45, and his predicted mortality would be 1.2%.

Figure 1 depicts the calibration plot comparing predicted versus observed one-year all-cause mortality probabilities across the ten subgroups of predicted mean MRPi in the test dataset for the data-splitting internal validation method. When we applied the final Cox model to the test dataset, we found that the predicted and observed mortality were closely aligned (visually) (Supplementary Table S5) for MRPi <75 over the one-year observation period, which includes most of the data (1,437, 98%). The model overpredicted the probability of mortality for MRPi greater than or equal to 75. However, the less-than-perfect alignment was in the tail of the MRPi distribution, representing those individuals who were more likely to die. Therefore, the overall calibration plot suggests the model was well calibrated with good agreement between the predicted and observed mortality probabilities. Supplementary Table S5 summarises the data in Figure 1.

Figure 1: Predicted one-year all-cause mortality using our final Cox model vs observed Kaplan-Meier estimates of the one-year all-cause mortality probability by subgroups of the one-year mortality risk prediction index (MRPi) using the test dataset. The observed one-year mortality probability is shown with 95% confidence intervals. Solid lines reflect predicted mortality probability calculated using the test dataset (n=1463). MRPi ranges from 0-100; x-axis begins from 25 because the minimum score in our sample is 25.

Figure 2 (Supplementary Table S6) shows the calibration plot comparing the mean predicted and observed KM (unadjusted and adjusted) one-year survival probabilities using the full dataset (bootstrap-based internal validation). When the survival probability was <95%, there was a non-significant, modest difference between the predicted and observed one-year survival probabilities with overlapping 95% CI. Though there was a closer correspondence between the predicted and observed one-year survival as the one-year survival probability increased above 95%, some 95% CI did not cross the ideal line implying an underestimation of the predicted one-year survival probability in these subgroups. These inconsistencies were likely because the predicted survival probabilities in some subgroups were higher or lower depending on the sample size distribution. Note that we did not have observed survival probabilities <75% in this dataset. Overall, there was good agreement between the observed KM and predicted one-year survival probabilities suggesting a good calibration.

Figure 2: Observed Kaplan-Meier estimates vs predicted estimates of one-year survival probabilities using the full dataset. KM; Kaplan-Meier. The observed one-year survival probability is shown with 95% confidence intervals. A solid line is drawn to join the two unadjusted observed survival points. The distance between a 45-degree (dotted) line with slope 1 and intercept 0 and the points reflects the difference between the observed and predicted survival probabilities. We did not have any observed survival probability <75%.

Secondary analyses

The excluded participants were significantly more likely to be younger, and have longer median years on ART and lower mortality (Supplementary Table S7). Results of the sex-based model-building are shown in Supplementary Figures S7–S10 and Tables S8–S11. Females in the final analytic sample were younger than males. About 572 (70%) females versus 2084 (58%) males were <50 years (Supplementary Figure S3). Sex-based analyses indicated that females tended to have a higher comorbidity burden and slightly higher MRPi scores than males. However, the overall predictive performance of the MRPi (c-statistic) remained high in both subgroups. Multimorbidity was more common among females (602, 74%), and males (958, 27%) were more likely to have no comorbidity at the index date (Supplementary Figure S4). The most prevalent comorbidities were SUD, MAD and CLD for both males and females (Supplementary Figure S5). In addition, females experienced two times the burden of SUD and CLD compared with males (66% versus 33%, and 30% versus 14%, respectively). Supplementary Figure S6 shows the distribution of the MRPi by sex at birth. A total of 473 (13%) males versus 36 (4%) females had an MRPi between 30 and 39, while 225 (28%) females versus 448 (13%) males had an MRPi between 60 and 69.

Discussion

We developed and validated an index score to predict the one-year all-cause mortality probability among PLWH using information routinely available in administrative health data. Our results showed that accounting for age, sex at birth, comorbidities, income assistance, and HIV-related markers in a single model provided the best discrimination of one-year all-cause mortality probability. By adjusting for the key variables in a single index, this study underscores the compound effects of HIV, ART, and chronic comorbidities on all-cause mortality among the aging PLWH. In this contemporary index, comorbidities were the strongest predictors of one-year all-cause mortality among PLWH. In addition, females had a higher comorbidity burden and MRPi than males in this cohort, highlighting the need to address the drivers of their sub-optimal clinical outcomes.

The recently updated VACS 2.0 index [19], which included age, body mass index and other clinical biomarkers to estimate all-cause mortality, has been validated in large North American cohorts (c-statistic 0.819, 95% CI: 0.815-0.823) [20]. However, it contains laboratory and clinical data to identify the presence of comorbidities which are not uniformly available in clinical or administrative datasets. In such circumstances, the MRPi is an alternative index with strong discrimination (c-statistic 0.8355, 95% CI: 0.8187-0.8523). Thus, once the MRPi is externally validated, it can become a valuable new tool to characterise short-term mortality among PLWH.

The final predictors in our model represent common factors which have been shown to contribute to an increase in the health burden of PLWH [7, 10]. The presence of comorbidities such as SUD, MST, and NADC had the largest association with one-year all-cause mortality probability among all predictors assessed. SUD was also strongly associated with an increased one-year mortality probability consistent with a previous study [33]. A diagnosis of SUD was associated with drug overdose and other adverse health outcomes (e.g., high viremia and premature mortality) [34–36]. It is worth mentioning that, since 2017, BC has had record-setting mortality rates related to the illicit drug toxicity crisis, which may further exacerbate the short-term mortality probability associated with SUD [37]. Our index was developed using data before the illicit drug toxicity crisis in BC, so our mortality risk prediction is unlikely to be biased by this crisis. Similar to previous indices, NADC increased the mortality probability in our population [13, 14, 38]. There is evidence that aging, lifestyle factors (e.g., high tobacco use) and HIV-specific factors (e.g., duration of HIV infection and immune status) increase the risk of NADC in PLWH [39, 40].

Age is a crucial social determinant of health, making our MRPi relevant to the growing population of aging PLWH. We also found that being female was associated with increased mortality risk. Of note, females in our population were more likely to have a history of injection drug use, a higher overdose-related mortality, and they faced inequities in access and adherence to HIV care, which further compounded their mortality risk [34, 36, 41]. A high CD4 nadir at the index date was a protective factor in the MRPi while the reverse was true for unsuppressed pVL. We have shown that by keeping all participant characteristics constant, the MRPi and predicted mortality changed as the CD4 nadir and pVL improved or worsened indicating the direct impact of clinical management on mortality. The association between high CD4 and mortality has been attributed to improved immunologic response in PLWH [42]. Unsuppressed pVL and its association with mortality can be explained by longer exposure to inflammation, which increases the risk of chronic comorbidities and, therefore, mortality [1]. Also, unsuppressed pVL is associated with poor retention to care, which may lead to poor health outcomes [43, 44]. We found that years on ART was significantly associated with a lower mortality risk in this study, and it improved the performance of the MRPi. ART use is associated with pVL suppression and CD4 improvement [42], which were common in over half of the study participants; it is, thus, associated with a lower mortality risk. However, our study’s relatively short median time on ART may also partly explain the observed lower mortality risk. Our separate sex-based analyses confirmed that females in our cohort had a slightly higher comorbidity burden, which aligns with previous studies of aging among PLWH. While our final model performed well in both sexes, future research might explore additional sex-specific risk factors.

Our study has some potential limitations. First, administrative data are not primarily collected for use in research. Thus, to identify the comorbidities used in our study, we relied on the Canadian Chronic Disease Surveillance System case definitions, supplemented by input from epidemiology and medical care experts familiar with BC-specific claims-related policies, as well as published literature [7, 10, 24]. Second, because we used administrative data, we could not assess the clinical severity of comorbidities; however, this limitation did not prevent our index from adequately predicting mortality, which is in line with previous reports indicating that administrative data can yield models with strong discrimination and calibration [45]. Third, the median follow-up time in our study was less than five years due to selecting a random index date for each participant, which necessarily shortened the observation period for some participants. Consequently, the MRPi is most applicable for short-term (one-year) mortality risk rather than long-term prediction. We also note that, although participants initiated ART from 2001 to 2018, we extended follow-up through 2019, thereby capturing many modern changes in treatment practices. Nonetheless, ART continues to evolve; this could limit generalisability to individuals who started therapy more recently or outside of BC’s universal healthcare setting. Fourth, although we initially sought to include ethnicity, the high proportion of missing data precluded its use; instead, we used income assistance as a proxy for socioeconomic status, which may not fully capture the full dimension of this social determinant of health. Fifth, although we considered 18 comorbidities widely recognised in the literature, there may be additional conditions relevant to PLWH that we did not capture. As more detailed or specialised data become available, future iterations of this risk index could expand the range of comorbidities included. Finally, while our use of data-splitting and bootstrapping supports the internal validity of our model, predictions for highly comorbid individuals with high mortality risk were inevitably more uncertain, given the smaller sample size in this subgroup.

Conclusions

The MRPi provides a promising tool to assess the risk of short-term mortality among PLWH in the modern ART era. In addition, these findings highlight the need for integrated HIV care models that suit the evolving healthcare needs of aging PLWH while reflecting the intersectionality of HIV and chronic diseases.

Funding

JSGM is supported with grants paid to his institution by BC Ministry of Health, Health Canada, Canadian Institutes of Health Research, Public Health Agency of Canada, Genome Canada, Genome BC, Vancouver Coastal Health and VGH Foundation. VDL is funded by a grant from the Canadian Institutes of Health Research (PJT-148595), and the Canadian Foundation for AIDS Research (CANFAR Innovation Grant – 30-101).

Acknowledgements

The authors thank all the participants included within STOP HIV/AIDS, the British Columbia Centre for Excellence in HIV/AIDS, the BC Ministry of Health, and the institutional data stewards for granting access to the data.

Disclaimer

All inferences, opinions, and conclusions drawn in this manuscript are those of the authors and do not reflect the views or policies of the data stewards.

Conflict of interest

JSGM received institutional grants provided by Gilead Sciences Inc, Janssen, Merck Sharp & Dohme LLC, and ViiV Healthcare. VDL received honoraria to present at the 2023 CROI (Conference on Retroviruses and Opportunistic Infections) ViiV Healthcare Ambassador Program. The other authors declare that they have no conflict of interest.

Ethics statement

This study received approval from the University of British Columbia Ethics Review Committee at the St Paul’s Hospital, Providence Health Care site (H18-02208). The usage of administrative data was approved by data stewards. Due to the use of anonymised administrative data, informed consent was not required for this study.

Data availability

The British Columbia Centre for Excellence in HIV/AIDS (BC-CfE) is prohibited from making individual-level data available publicly due to provisions in our service contracts, institutional policy, and ethical requirements. To facilitate research, we make such data available via data access requests. Some BC-CfE data is not available externally due to prohibitions in service contracts with our funders or data providers. Institutional policies stipulate that all external data requests require collaboration with a BC-CfE researcher. For more information, please contact Mark Helberg, Senior Director, Internal and External Relations and Strategic Development: mhelberg@bccfe.ca.

Author’s contribution

Concept and design: VDL; Acquisition, analysis, or interpretation of data: BTT, NF, HN, KD, JZ, JT, VDL; Drafting of the manuscript: BTT, VDL; Critical revision of the manuscript for important intellectual content: BTT, NF, HN, JZ, SE, KD, JT, KAS, RB, JSGM, VDL; Administrative, technical, or material support: RB, JSGM, VDL. All authors have read and approved the final manuscript.

References

1. Deeks SG, Lewin SR, Havlir DV. The end of AIDS: HIV infection as a chronic disease. The Lancet. 2013;382(9903):1525-33. https://doi.org/10.1016/S0140-6736(13)61809-7

2. Monforte AD, Sabin CA, Phillips A, Sterne J, May M, Justice A, et al. The changing incidence of AIDS events in patients receiving highly active antiretroviral therapy. Archives of Internal Medicine. 2005;165(4):416-23. https://doi.org/10.1001/archinte.165.4.416

3. Teeraananchai S, Kerr S, Amin J, Ruxrungtham K, Law M. Life expectancy of HIV-positive people after starting combination antiretroviral therapy: a meta-analysis. HIV Medicine. 2017;18(4):256-66. https://doi.org/10.1111/hiv.12421

4. Samji H, Cescon A, Hogg RS, Modur SP, Althoff KN, Buchacz K, et al. Closing the gap: increases in life expectancy among treated HIV-positive individuals in the United States and Canada. PLoS ONE. 2013;8(12):e81355. https://doi.org/10.1371/journal.pone.0081355

5. British Columbia Centre for Excellence in HIV/AIDS. HIV Monitoring Quarterly Report for British Columbia. Fourth Quarter 2013 2013 [Available from: https://bccfe.ca/sites/default/files/uploads/publications/centredocs/bc-monitoring-report-13q4-updated-2015-jan-20.pdf.

6. British Columbia Centre for Excellence in HIV/AIDS. HIV Monitoring Semi-Annual Report For British Columbia. Fourth Quarter 2022 2023 [Available from: https://stophivaids.ca/qmr/2022-Q4/#/bc.

7. Nanditha NGA, Paiero A, Tafessu HM, St-Jean M, McLinden T, Justice AC, et al. Excess burden of age-associated comorbidities among people living with HIV in British Columbia, Canada: a population-based cohort study. BMJ Open. 2021;11(1):e041734. https://doi.org/10.1136/bmjopen-2020-041734

8. Sokoya T, Steel HC, Nieuwoudt M, Rossouw TM. HIV as a cause of immune activation and immunosenescence. Mediators of Inflammation. 2017;2017. https://doi.org/10.1155/2017/6825493

9. Sereti I, Altfeld M. Immune activation and HIV: an enduring relationship. Current Opinion in HIV and AIDS. 2016;11(2):129. https://doi.org/10.1097/COH.0000000000000244

10. Nanditha NGA, Zhu J, Wang L, Kopec J, Hogg RS, Montaner JSG, et al. Disability-adjusted life years associated with chronic comorbidities among people living with and without HIV: Estimating health burden in British Columbia, Canada. PLOS Glob Public Health. 2022;2(10):e0001138. https://doi.org/10.1371/journal.pgph.0001138

11. Lachaine J, Baribeau V, Lorgeoux R, Tossonian H. Health care resource utilization and costs associated with HIV-positive patients with comorbidity versus HIV-negative patients with comorbidity. Value in Health. 2017;20(9):A791. https://doi.org/10.1016/j.jval.2017.08.2323

12. Nachega JB, Hsu AJ, Uthman OA, Spinewine A, Pham PA. Antiretroviral therapy adherence and drug-drug interactions in the aging HIV population. AIDS. 2012;26 Suppl 1:S39-53. https://doi.org/10.1097/QAD.0b013e32835584ea

13. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. Journal of Chronic Diseases. 1987;40(5):373-83. https://doi.org/10.1016/0021-9681(87)90171-8

14. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Medical Care. 1998:8-27. https://doi.org/10.1097/00005650-199801000-00004

15. Weiner JP, Starfield BH, Steinwachs DM, Mumford LM. Development and application of a population-oriented measure of ambulatory care case-mix. Medical Care. 1991:452-72. https://doi.org/10.1097/00005650-199105000-00006

16. Justice AC, Feinstein AR, Wells CK. A new prognostic staging system for the acquired immunodeficiency syndrome. New England Journal of Medicine. 1989;320(21):1388-93. https://doi.org/10.1056/nejm198905253202106

17. Egger M, May M, Chêne G, Phillips AN, Ledergerber B, Dabis F, et al. Prognosis of HIV-1-infected patients starting highly active antiretroviral therapy: a collaborative analysis of prospective studies. The Lancet. 2002;360(9327):119-29. https://doi.org/10.1016/s0140-6736(02)09411-4

18. Justice AC, McGinnis K, Skanderson M, Chang C, Gibert C, Goetz M, et al. Towards a combined prognostic index for survival in HIV infection: the role of ‘non-HIV’biomarkers. HIV Medicine. 2010;11(2):143-51. https://doi.org/10.1111/j.1468-1293.2009.00757.x

19. Tate JP, Sterne JA, Justice AC, Study VAC, Collaboration ATC. Albumin, white blood cell count, and body mass index improve discrimination of mortality in HIV-positive individuals. AIDS (London, England). 2019;33(5):903. https://doi.org/10.1097/qad.0000000000002140

20. McGinnis KA, Justice AC, Moore RD, Silverberg MJ, Althoff KN, Karris M, et al. Discrimination and Calibration of the Veterans Aging Cohort Study Index 2.0 for Predicting Mortality Among People With Human Immunodeficiency Virus in North America. Clinical Infectious Diseases. 2022;75(2):297-304. https://doi.org/10.1093/cid/ciab883

21. Justice AC, Modur S, Tate JP, Althoff KN, Jacobson LP, Gebo K, et al. Predictive accuracy of the Veterans Aging Cohort Study (VACS) index for mortality with HIV infection: a North American cross cohort analysis. Journal of Acquired Immune Deficiency Syndromes (1999). 2013;62(2):149. https://doi.org/10.1097/QAI.0b013e31827df36c

22. Qiao S, Li X, Olatosi B, Young SD. Utilizing Big Data analytics and electronic health record data in HIV prevention, treatment, and care research: a literature review. AIDS Care. 2021:1-21. https://doi.org/10.1080/09540121.2021.1948499

23. Heath K, Samji H, Nosyk B, Colley G, Gilbert M, Hogg RS, et al. Cohort Profile: Seek and Treat for the Optimal Prevention of HIV/AIDS in British Columbia (STOP HIV/AIDS BC). International Journal of Epidemiology. 2014;43(4):1073-81. https://doi.org/10.1093/ije/dyu070

24. Nanditha NGA, Dong X, McLinden T, Sereda P, Kopec J, Hogg RS, et al. The impact of lookback windows on the prevalence and incidence of chronic diseases among people living with HIV: an exploration in administrative health data in Canada. BMC Medical Research Methodology. 2022;22(1):1. https://doi.org/10.1186/s12874-021-01448-x

25. BC Ministry of Health. [creator] 2017. Vital Statistics Deaths. BC Ministry of Health [publisher]. Data extract. BC Vital Statistics Agency (2017) [Available from: https://www2.gov.bc.ca/gov/content/health/conducting-health-research-evaluation/data-access-health-data-central.

26. McGinnis KA, Justice AC, Marconi VC, Rodriguez-Barradas MC, Hauser RG, Oursler KK, et al. Combining Charlson comorbidity and VACS indices improves prognostic accuracy for all-cause mortality for patients with and without HIV in the Veterans Health Administration. Front Med (Lausanne). 2023;10:1342466. https://doi.org/10.3389/fmed.2024.1532350

27. British Columbia PharmaCare for health professionals. Pharmacies. Product Identification Numbers (PINS) 2023 [Available from: https://www2.gov.bc.ca/gov/content/health/practitioner-professional-resources/pharmacare/pharmacies/product-identification-numbers.

28. McDonald JH. Handbook of biological statistics. Baltimore (US): Sparky House Publishing; 2009.

29. Harrell Jr FE. Regression Modeling Strategies [Internet]. 2nd edition. New York (US): Springer-Verlag; 2015. [cited 2023 March 20].

30. Lima VD, Le A, Nosyk B, Barrios R, Yip B, Hogg RS, et al. Development and validation of a composite programmatic assessment tool for HIV therapy. PLoS One. 2012;7(11):e47859. https://doi.org/10.1371/journal.pone.0047859

31. Harrell Jr FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine. 1996;15(4):361-87. https://doi.org/ 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4

32. Pencina MJ, D’Agostino RB, Sr. Evaluating Discrimination of Risk Prediction Models: The C Statistic. JAMA. 2015;314(10):1063-4. https://doi.org/10.1001/jama.2015.11082

33. Nigussie F, Alamer A, Mengistu Z, Tachbele E. Survival and Predictors of Mortality Among Adult HIV/AIDS Patients Initiating Highly Active Antiretroviral Therapy in Debre-Berhan Referral Hospital, Amhara, Ethiopia: A Retrospective Study. HIV AIDS (Auckl). 2020;12:757-68. https://doi.org/10.2147/HIV.S274747

34. Salters KA, Parent S, Nicholson V, Wang L, Sereda P, Pakhomova TE, et al. The opioid crisis is driving mortality among under-served people living with HIV in British Columbia, Canada. BMC public health. 2021;21(1):1-8. https://doi.org/10.1186/s12889-021-10714-y

35. Cohn SE, Jiang H, McCutchan JA, Koletar SL, Murphy RL, Robertson KR, et al. Association of ongoing drug and alcohol use with non-adherence to antiretroviral therapy and higher risk of AIDS and death: results from ACTG 362. AIDS Care. 2011; 23(6):775-85. https://doi.org/10.1080/09540121.2010.525617

36. St-Jean M, Dong X, Tafessu H, Moore D, Honer WG, Vila-Rodriguez F, et al. Overdose mortality is reducing the gains in life expectancy of antiretroviral-treated people living with HIV in British Columbia, Canada. International Journal of Drug Policy. 2021;96:103195. https://doi.org/10.1016/j.drugpo.2021.103195

37. British Columbia Coroners Service. Illicit drug toxicity deaths in BC: January 1, 2012–August 31, 2022. Vancouver: Ministry of Public Safety & Solicitor General; 2022.

38. Quan H, Li B, Couris CM, Fushimi K, Graham P, Hider P, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. American Journal of Epidemiology. 2011;173(6):676-82. https://doi.org/10.1093/aje/kwq433

39. Deeken JF, Tjen ALA, Rudek MA, Okuliar C, Young M, Little RF, et al. The rising challenge of non-AIDS-defining cancers in HIV-infected patients. Clin Infect Dis. 2012;55(9):1228-35. https://doi.org/10.1093/cid/cis613

40. Chiao EY, Coghill A, Kizub D, Fink V, Ndlovu N, Mazul A, et al. The effect of non-AIDS-defining cancers on people living with HIV. The Lancet Oncology. 2021;22(6):e240-e53. https://doi.org/10.1016/S1470-2045(21)00137-6

41. Carter A, Min JE, Chau W, Lima VD, Kestler M, Pick N, et al. Gender inequities in quality of care among HIV-positive individuals initiating antiretroviral treatment in British Columbia, Canada (2000-2010). PLoS One. 2014;9(3):e92334. https://doi.org/10.1371/journal.pone.0092334

42. Moore DM, Harris R, Lima V, Hogg B, May M, Yip B, et al. Effect of baseline CD4 cell counts on the clinical significance of short-term immunologic response to antiretroviral therapy in individuals with virologic suppression. J Acquir Immune Defic Syndr. 2009;52(3):357-63. https://doi.org/10.1097/QAI.0b013e3181b62933

43. Teixeira da Silva DS, Luz PM, Lake JE, Cardoso SW, Ribeiro S, Moreira RI, et al. Poor retention in early care increases risk of mortality in a Brazilian HIV-infected clinical cohort. AIDS Care. 2017;29(2):263-7. https://doi.org/10.1080/09540121.2016.1211610

44. Yehia BR, French B, Fleishman JA, Metlay JP, Berry SA, Korthuis PT, et al. Retention in care is more strongly associated with viral suppression in HIV-infected patients with lower versus higher CD4 counts. J Acquir Immune Defic Syndr. 2014;65(3):333-9. https://doi.org/10.1097/QAI.0000000000000023

45. Rothberg MB, Pekow PS, Priya A, Zilberberg MD, Belforti R, Skiest D, et al. Using highly detailed administrative data to predict pneumonia mortality. PLoS ONE. 2014;9(1):e87382. https://doi.org/10.1371/journal.pone.0087382