|Year : 2020 | Volume
| Issue : 6 | Page : 1273-1280
|Validation of acute kidney injury prediction scores in critically ill patients
Ahmed Mohamed Zahran1, Yasser Ibrahim Fathy2, Asmaa Esmail Salama2, Mohamed Esam Alebsawi2
1 Department of Internal Medicine, Nephrology Unit, Faculty of Medicine, Menoufia University, Menoufia, Egypt
2 Department of Critical Care, Faculty of Medicine, Menoufia University, Menoufia, Egypt
Click here for correspondence address and email
|Date of Web Publication||29-Jan-2021|
| Abstract|| |
Prediction of acute kidney injury (AKI) in critically ill patients allows prompt intervention that improves outcome. We aimed for external validation of two AKI prediction scores that can be bedside calculated. A prospective observational study included patients admitted to medical and surgical critical care units. Performance of two AKI prediction scores, Malhotra score and acute kidney injury prediction score (APS), was assessed for their ability to predict AKI. The best cutoff point for each score was determined by Youden index. Area under the receiving operation characteristic curve, sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were used to assess performance of each score. Univariate and multivariate regression analyses were done to detect the predictability of AKI. Goodness-of-fit and kappa Cohen agreement tests were done to show whether the expected score results fit well and agree with the observed results. AKI prevalence was 37.6%. The best cutoff values were 5 and 4 for Malhotra score and APS, respectively. Area under the curve for Malhotra 5 was 0.712 and for APS 4 was 0.652 with nearly similar sensitivity and specificity. Regression analysis demonstrated that Malhotra 5 was the independent predictor of AKI. Goodness-of-fit test showed significant results denoting lack of fit between the scores and the actual results. Kappa test showed moderate agreement for Malhotra 5 and fair agreement for APS 4. Both scores showed moderate performance for AKI prediction. Malhotra 5 showed better performance compared to APS 4. Multicenter international study is warranted to develop a universal model that can predict AKI in critically ill patients.
|How to cite this article:|
Zahran AM, Fathy YI, Salama AE, Alebsawi ME. Validation of acute kidney injury prediction scores in critically ill patients. Saudi J Kidney Dis Transpl 2020;31:1273-80
|How to cite this URL:|
Zahran AM, Fathy YI, Salama AE, Alebsawi ME. Validation of acute kidney injury prediction scores in critically ill patients. Saudi J Kidney Dis Transpl [serial online] 2020 [cited 2022 Jun 27];31:1273-80. Available from: https://www.sjkdt.org/text.asp?2020/31/6/1273/308336
| Introduction|| |
Acute kidney injury (AKI) is a serious condition characterized by rapid deterioration of kidney function and is highly prevalent in critically ill patients and associated with higher risk of morbidity and mortality., The prevalence of AKI in critically ill patients varies greatly from 15% up to more than 50% depending on risk factors and comorbidities., AKI diagnosis depends on criteria defined by Kidney Disease: Improving Global Outcomes (KDIGO) which mainly dependent on serum creatinine (SCr). A rise in SCr marks an already advanced stage of kidney injury in addition in elderly and malnourished patients; the rise in SCr might be minimal and considerably delayed. Several biochemical markers were identified for early detection of AKI; however, their use is still practically limited as it is not clear when they should be measured, and it is unlikely that a single biomarker is able to indicate different pathogenetic mechanisms and different causes of AKI. As early detection of AKI in critically ill patients is crucial and remains a challenge, a clinical risk prediction scores for AKI in adult critically ill patients in intensive care unit (ICU) setting can help for prevention, early diagnosis and targeted interventions. Several prediction scores have been developed to allow early intervention with improved outcome.,,,,,,, The aim of this work is to evaluate the validity of two AKI prediction scores in our critically ill patients for their accuracy to predict development of AKI.
| Subjects and Methods|| |
This is a prospective observational study that included patients admitted to medical and surgical critical care units, Menoufia University hospitals, Menoufia Governorate, Egypt. This study followed the ethical standards of our hospitals and approved by the ethical committee. Informed consent was obtained from all patients or their first-degree relatives. Patients admitted to critical care units during period from January 2018 to December 2018 without AKI at admission were enrolled in this study. The following patients were excluded from the study:
- Age <18 years
- Patients admitted with AKI according to KDIGO criteria (5) with known baseline creatinine before admission. [Patients with high SCr (known CKD) but within 10% change from known baseline creatinine in last 3 months were included]
- Patients with high SCr without known baseline creatinine before admission
- Patients died within 24 h of admission
- Patient with cardiothoracic trauma or cardiothoracic surgery
- Obstetric and gynecological admissions.
All patients underwent thorough history taking, clinical examination, and laboratory investigations. Baseline data were collected at the time of admission. Patients were followed during their ICU stay till endpoint either discharge or death. Patients who developed AKI according to KDIGO criteria were identified.
Risk prediction score developed by Malhotra et al (Malhotra score) and acute kidney injury prediction score (APS) developed by Forni et al were evaluated in our cohort to determine the performance of each score in predicting AKI. Malhotra scores were developed depending on chronic and acute variables with a maximum score of 21 points. Cutoff point of >5 was chosen by Malhotra et al to be optimum to define high-risk patients to develop AKI based on best sensitivity and specificity. APS was developed to predict AKI of acute medical admission within seven days depending on clinical variables at the time of admission with a maximum score of 14 points. In our cohort, we identified the best cutoff value for each score after constructing area under receiving operation characteristic (ROC) curve for each score as a total and choosing the point with best sensitivity and specificity depending on Youden index (sensitivity + (1 – specificity). We identified that the best cutoff value was 5 for Malhotra (Malhotra 5) and 4 for APS (APS 4). All risk variables were identified according to appendix associated with Malhotra score and according to Forni et al.
| Statistical Analysis|| |
Data were analyzed using IBM SPSS Statistics for Windows version 24.0 (IBM Corp., Armonk, NY, USA), EpiCalc 2000 version 1.02, and Microsoft Excel. Numerical data were expressed as mean ± standard deviation while categorical data were expressed as number and percentage. ROC curve was constructed to detect area under the curve (AUC) for evaluation of good discrimination of AKI. The higher the values, the greater the performance of the score. Sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), positive predictive value (PPV), negative predictive value (NPV), and accuracy were done to evaluate the performance of prediction scores. Univariate and multivariate logistic regression analyses were done to determine which score can predict the occurrence of AKI. Goodness-of-fit test and Cohen’s kappa test were done to detect whether the expected score results fit well and agree with the observed results. Significant difference detected by goodness-of-fit test can be interpreted as the studied score did not fit and agree with actual results. Kappa test was interpreted as poor agreement if ≤0.2, fair agreement 0.21–0.4, moderate agreement 0.41–0.6, good agreement 0.61–0.8, and perfect agreement if ≥0.81. P ≤0.05 was considered statistically significant.
| Results|| |
This prospective observational study included 210 critically ill patients admitted to medical and surgical ICUs, Menoufia University Hospitals, Egypt. The prevalence of AKI was 37.6%. Patient demographics are shown in [Table 1]. Youden test demonstrated that the best cutoff value for AKI prediction is 5 points for Malhotra score and 4 points for APS. AUCs demonstrated by ROC curves for Malhotra 5 showed better results (0.721) when compared to APS 4 (0.652) [Figure 1]. Sensitivity, specificity, PLR, NLR, PPV, NPV, and accuracy of Malhotra 5 and APS 4 to predict AKI are shown in [Table 2]. Univariate binary logistic regression analysis for Malhotra 5 and APS 4 showed that each score can significantly predict the occurrence of AKI with odd ratio 6.11 and 3.58, respectively, while multivariate binary logistic regression for both scores demonstrated that Malhotra 5 is the significantly predictor of AKI occurrence [Table 3]. Goodness-of-fit test showed a significant difference of Malhotra 5 with P = 0.018 and APS 4 with P = 0.000 when compared to actual results denoting that both scores did not fit well with the observed results. Cohen kappa agreement demonstrated fair agreement for APS 4 and moderate agreement for Malhotra 5 [Table 3].
|Figure 1: Area under the receiver operating characteristic curve for Malhotra 5 and APS 4 for AKI prediction.|
AUC: Area under the curve, CI: Confidence interval, APS: Acute kidney injury prediction score.
Click here to view
|Table 2: Sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, positive predictive value, and accuracy of the studied predictor scores.|
Click here to view
|Table 3: Logistic regression analysis, Goodness of fit test and kappa agreement of the studied predictor scores.|
Click here to view
| Discussion|| |
AKI is a severe and frequent complication in patients admitted to ICUs and is associated with high mortality rates. Early identification of patients high-risk to develop AKI provides an opportunity to develop strategies for the prevention, early diagnosis, and treatment of AKI. This prospective observational study represents an external validation of AKI prediction scores that can be easily calculated bedside. The prevalence of AKI in our study was 37.6%. The prevalence of AKI varies with definition used in addition of urine output criteria. In our work, AKI was diagnosed according to KDIGO criteria depending on SCr only like the way the studied prediction scores were developed. Both scores demonstrated better sensitivity at lower cutoff values while specificity improved with higher cutoff values. In our cohort, Malhotra 5 AUC was 0.712 which is lower than that demonstrated by Malhotra development group [University of California San Diego (UCSD) 0.79] and validation group (Mayo clinic 0.81). We could not identify in literature any other works that externally validate Malhotra score. The APS 4 showed an AUC of 0.652 which is lower than that demonstrated by Malhotra 5. Forni et al demonstrated an AUC of 0.72 for APS as a total of the development group and 0.76 when internally validated their score in a group of 60 patients. Forni et al did not show a cutoff point for their score. APS as a total was externally validated by Hodgson et al and they found AUC of 0.65 for a medical admission with hospital-acquired AKI and known baseline SCr and AUC of 0.66 for surgical admission with hospital-acquired AKI and known baseline creatinine which are very close to our findings.
We identified a sensitivity of 72.15% and specificity of 70.23% for Malhotra 5 which is nearly similar to that found by Malhotra development group (UCSD) 74% and 72%, respectively, however, Mayo clinic validation group showed a sensitivity of 63% which is lower than our results and specificity of 85% which is better than our finding. APS 4 showed a sensitivity of 70.89% and specificity of 59.54%. Unfortunately, Forni et al did not show any cutoff points, sensitivity, or specificity for their score, however, Hodgson et al when externally validated APS found that sensitivity of 34% and specificity of 82% at cutoff point 5 but when reduced cutoff point to 3 demonstrated a sensitivity of 82% and specificity of 37% denoting that sensitivity improved with lower cutoff points while specificity improved with higher cutoff points. In this study, Malhotra score 5 showed better predictability of AKI compared to APS 4, however, still both scores did not show good agreement with the actual results denoting poor-to-average performance in predicting AKI in critically ill patients. In general, most of the AKI predicting scores were developed to predict AKI in specific situation like post coronary angiography,,,,, post cardiothoracic surgery,, post chemotherapy, post trauma,, and in relation to liver diseases.,, There are some reports that show prediction of AKI in hospital admission and in critically ill patients admitted to ICU, and most of them were internally validated.,,,,, AKI prediction models showed heterogeneous data regarding risk prediction as diabetes was included in PAS, however, it was not a risk predictor in Malhotra score. In addition, most of the prediction models were developed in a single-center study and some of these models are difficult to be calculated bedside. Furthermore, all risk prediction models mainly depend on area under the ROC curve for good discrimination which may be good for diagnostic purposes but not be an optimal in assessing models that predict future risk or stratify individuals into risk categories. Our study has the advantage of being an external validation of two AKI prediction scores that can be bedside calculated. Our ethnic group is completely different from the derivation score cohort that may allow generalizability of these scores. We assess the performance of AKI prediction scores with multiple statistical methods. On other hand, our study has the following limitations. The study included relatively small number of patients. The APS score was derived from acute medical admission while our patients included acute surgical admission, however, APS was externally validated in medical and surgical admission with the same performance; in addition, majority of our patients were acute medical admissions.
| Conclusions|| |
Malhotra 5 and APS 4 AKI prediction scores showed moderate performance, however, both did not show good agreement with actual observed data. Malhotra 5 showed better performance compared to APS 4. Both scores can help to identify high-risk patients, as with lower scores, patients are less likely to develop AKI while the higher the score the more likely to develop AKI that may allow better prevention and targeted actions. A prospective international multicenter study with different ethnicity is recommended to develop a universal model that can predict AKI in critically ill patients in ICU. This model should be easy calculated depending on different clinical and laboratory variables and it should be multicenters with different ethnic groups validated.
Conflict of interest None declared.
| References|| |
Neyra JA, Mescia F, Li X, et al. Impact of acute kidney injury and CKD on adverse outcomes in critically ill septic patients. Kidney Int Rep 2018;3:1344-53.
Flechet M, Güiza F, Schetz M, et al. AKI predictor, an online prognostic calculator for acute kidney injury in adult critically ill patients: Development, validation and comparison to serum neutrophils gelatinase-associated lipocalin. Intensive Care Med 2017;43: 764-73.
Sileanu FE, Murugan R, Lucko N, et al. AKI in low-risk versus high-risk patients in intensive care. Clin J Am Soc Nephrol 2015; 10:187-96.
Hoste EA, Bagshaw SM, Bellomo R, et al. Epidemiology of acute kidney injury in critically ill patients: The multinational AKI-EPI study. Intensive Care Med 2015;41:1411-23.
Kidney Disease: Improving Global Outcomes (KDIGO). Acute Kidney Injury Work Group. KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney Inter Suppl 2012;2:1-138.
Tavakoli R, Lebreton G. Biomarkers for early detection of cardiac surgery-associated acute kidney injury. J Thorac Dis 2018;10:S3914-8.
Beker BM, Corleto MG, Fieiras C, Musso CG. Novel acute kidney injury biomarkers: Their characteristics, utility and concerns. Int Urol Nephrol 2018;50:705-13.
Malhotra R, Kashani KB, Macedo E, , et al. A risk prediction score for acute kidney injury in the intensive care unit. Nephrol Dial Transplant 2017;32:814-22.
Forni LG, Dawes T, Sinclair H, et al. Identifying the patient at risk of acute kidney injury: A predictive scoring system for the development of acute kidney injury in acute medical patients. Nephron Clin Pract 2013; 123:143-50.
Mohamadlou H, Lynn-Palevsky A, Barton C, et al. Prediction of acute kidney injury with a machine learning algorithm using electronic health record data. Can J Kidney Health Dis 2018;5:1-9.
Ho KM, Harahsheh Y. Predicting contrast-induced nephropathy after CT pulmonary angiography in the critically ill: A retrospective cohort study. J Intensive Care 2018; 6:3.
Jorge-Monjas P, Bustamante-Munguira J, Lorenzo M, Heredia-Rodríguez M et al. Predicting cardiac surgery-associated acute kidney injury: The CRATE score. J Crit Care 2016;31:130-8.
Jiang W, Teng J, Xu J, et al. Dynamic predictive scores for cardiac surgery-associated acute kidney injury. J Am Heart Assoc 2016;5: e003754.
Grimm JC, Lui C, Kilic A, et al. A risk score to predict acute renal failure in adult patients after lung transplantation. Ann Thorac Surg 2015;99:251-7.
Rueggeberg A, Boehm S, Napieralski F, et al. Development of a risk stratification model for predicting acute renal failure in orthotopic liver transplantation recipients. Anaesthesia 2008;63:1174-80.
Neyra JA, Leaf DA. Risk prediction model for acute kidney injury in critically ill patient. Opus in progru. Nephron 2018;140:99-104.
Maccariello E, Valente C, Nogueira L, et al. SAPS 3 scores at the start of renal replacement therapy predict mortality in critically ill patients with acute kidney injury. Kidney Int 2010;77:61-6.
Bevc S, Ekart R, Hojs R. The assessment of acute kidney injury in critically ill patients. Eur J Intern Med 2017;45:54-8.
Koeze J, Keus F, Dieperink W, van der Horst IC, Zijlstra JG, van Meurs M. Incidence, timing and outcome of AKI in critically ill patients varies with the definition used and the addition of urine output criteria. BMC Nephrol 2017;18:70.
Hodgson LE, Dimitrov BD, Roderick PJ, Venn R, Forni LG. Predicting AKI in emergency admissions: An external validation study of the acute kidney injury prediction score (APS). BMJ Open 2017;7:e013511.
Çınar T, Karabağ Y, Ozan Tanık V, Çağdaş M, Rencüzoğulları İ, Öz A. The investigation of TIMI risk index for prediction of contrast-induced acute kidney injury in patients with ST elevation myocardial infarction. Acta Cardiol 2019;25:1-8.
Ibrahim NE, McCarthy CP, Shrestha S, et al. A clinical, proteomics, and artificial intelligence-driven model to predict acute kidney injury in patients undergoing coronary angiography. Clin Cardiol 2019;42:292-8.
Elhosseiny S, Akel T, Mroue J, Tathineni P, El Sayegh S, Lafferty J. The value of adding red cell distribution width to mehran risk score to predict contrast-induced acute kidney injury in patients with acute coronary syndrome. Cureus 2018;10:e2911.
Araujo GN, Pivatto Junior F, Fuhr B, et al. Simplifying contrast-induced acute kidney injury prediction after primary percutaneous coronary intervention: The age, creatinine and ejection fraction score. Cardiovasc Interv Ther 2018;33:224-31.
Guo BL, Ouyang FS, Yang SM, et al. Development of a preprocedure nomogram for predicting contrast-induced acute kidney injury after coronary angiography or percutaneous coronary intervention. Oncotarget 2017;8: 75087-93.
Che M, Wang X, Liu S, et al. A clinical score to predict severe acute kidney injury in chinese patients after cardiac surgery. Nephron 2019; 21:1-10.
Ranucci M, Aloisio T, Cazzaniga A, Di Dedda U, Gallazzi C, Pistuddi V. Validation of renal-risk models for the prediction of non-renal replacement therapy cardiac surgery-associated acute kidney injury. Int J Cardiol 2018;272:49-53.
Cheng DH, Lu H, Liu TT, Zou XQ, Pang HM. Identification of risk factors in high-dose methotrexate-induced acute kidney injury in childhood acute lymphoblastic leukemia. Chemotherapy 2018;63:101-7.
Haines RW, Lin SP, Hewson R, et al. Acute kidney injury in trauma patients admitted to critical care: Development and validation of a diagnostic prediction model. Sci Rep 2018; 8:3665.
Harrois A, Soyer B, Gauss T, et al. Prevalence and risk factors for acute kidney injury among trauma patients: A multicenter cohort study. Crit Care 2018;22:344.
Gameiro J, Agapito Fonseca J, et al. Prediction of acute kidney injury in cirrhotic patients: A new score combining renal, liver and inflammatory markers. Int J Nephrol Renovasc Dis 2018;11:149-54.
Lee HC, Yoon SB, Yang SM, et al. Prediction of acute kidney injury after liver transplanttation: Machine learning approaches vs. logistic regression model. J Clin Med 2018; 7:428.
Park MH, Shim HS, Kim WH, et al. Clinical risk scoring models for prediction of acute kidney injury after living donor liver transplantation: A retrospective observational study. PLoS One 2015;10:e0136230.
Matheny ME, Miller RA, Ikizler TA, et al. Development of inpatient risk stratification models of acute kidney injury for use in electronic health records. Med Decis Making 2010;30:639-50.
Bhatraju PK, Zelnick LR, Katz R, et al. A Prediction model for severe AKI in critically Ill adults that incorporates clinical and biomarker data. Clin J Am Soc Nephrol 2019;14: 506-14.
Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928-35.
Ahmed Mohamed Zahran
Department of Internal Medicine, Nephrology Unit, Faculty of Medicine, Menoufia University, Menoufia
Source of Support: None, Conflict of Interest: None
[Table 1], [Table 2], [Table 3]
| Article Access Statistics|
| Viewed||1240 |
| Printed||8 |
| Emailed||0 |
| PDF Downloaded||164 |
| Comments ||[Add] |