Abstract
-
Background
- Prompt differentiation between ischemic stroke (IS) and hemorrhagic stroke (HS) is critical because their treatment strategies fundamentally differ. While neuroimaging is essential, clinical decision-making often begins before imaging is completed, and conventional clinical scores have shown inconsistent performance. The objective of this study was therefore to develop and externally validate a machine-learning model that supports HS vs. IS subtype suspicion at emergency department (ED) presentation using only clinical variables.
-
Methods
- We conducted a retrospective multicenter cohort study of 2,998 adult patients with a final diagnosis of acute IS or HS treated at three comprehensive stroke centers (July 2020–January 2024). Patients from hospitals A and B comprised the development/internal validation cohort (n=2,418), while patients from hospital C served as an independent external validation cohort (n=580). An extreme gradient boosting (XGBoost) algorithm was trained using four-fold cross-validation, and feature contributions were assessed using Shapley additive explanation (SHAP) values.
-
Results
- Internal validation showed an area under the receiver operating characteristic curve (AUROC) of 0.937 (95% CI, 0.922–0.950) with a sensitivity 0.828, specificity of 0.932, and accuracy of 0.905. Independent external validation yielded an AUROC of 0.841 (95% CI, 0.792–0.883) with a sensitivity 0.758, specificity of 0.789, and accuracy of 0.783. SHAP analysis identified headache and higher National Institutes of Health Stroke Scale item 1a (level of consciousness) as factors increasing the model output toward HS, whereas atrial fibrillation shifted predictions toward IS.
-
Conclusions
- A clinical variable-only model can support early HS vs. IS subtype suspicion at ED presentation among patients managed in an acute-stroke pathway without requiring laboratory tests. Performance decreased on independent external validation, suggesting potential site-related differences and the need for prospective evaluation and calibration. Stroke mimics were not included and should be addressed in future studies.
-
Key Words: classification; clinical decision support systems; hemorrhagic stroke; ischemic stroke; machine learning
INTRODUCTION
Stroke is a major worldwide neurological disorder associated with high mortality and long-term disability [1]. Depending on the underlying pathology, stroke is classified into ischemic stroke (IS) and hemorrhagic stroke (HS). Although HS accounts for nearly one-quarter of all strokes [2], it carries a much worse prognosis with a 30-day mortality of approximately 40% compared to IS [3]. Rapid and accurate classification of stroke subtypes is therefore crucial for determining appropriate treatment strategies immediately upon the patient’s arrival to the emergency department (ED). Early recognition of HS versus IS can help direct patients to hospitals equipped for thrombolysis or emergency neurosurgery, decrease the time to reperfusion therapy in IS patients [4,5], and enable prompt blood-pressure management and neurosurgical preparation in HS patients [6]. Therefore, accurate discrimination of stroke subtypes before imaging plays a crucial role in acute stroke care.
Notably, many treatment-relevant decisions and preparations begin during the initial ED evaluation, often before definitive imaging results are available. Although neuroimaging remains the gold standard for stroke subtyping [7,8], imaging workflows and resources vary across settings, and clinicians may need to initiate time-sensitive management steps while awaiting imaging. Therefore, auxiliary tools based on immediately obtainable clinical information may support early risk stratification at ED presentation without delaying standard imaging-based pathways. Most patients undergo non-contrast computed tomography (CT) to exclude hemorrhage, followed by computed tomography angiography or computed tomography perfusion when ischemia is suspected, which prolongs diagnosis and burdens clinical resources [9]. Accordingly, the need exists for supportive tools that can assist subtype suspicion early in the ED course.
Conventional clinical scores, such as the Siriraj Stroke Score and Guy’s Hospital Stroke Score [10], which are based on variables such as headache, vomiting, blood pressure, and consciousness, have shown inconsistent performance across populations. Many patients fall into indeterminate ranges and symptoms, such as headache and impaired consciousness, which are not specific to HS. Consequently, these scores have had limited clinical utility in guiding critical decisions [11].
Machine learning (ML) offers an alternative approach to addressing these limitations, and several ML-based studies have attempted stroke subtype classification [12]. However, most studies have relied on relatively small, single-center datasets, raising concerns of overfitting and limited generalizability [13,14]. Some have also required laboratory markers that are not immediately available at presentation, limiting their feasibility for real-time use [15]. Another challenge is class imbalance: even models with high accuracy have often shown unacceptably low sensitivity for HS. For example, Wu et al. [16] reported accuracy near 95% in imbalanced datasets but sensitivity as low as 0.10. Considering the prognostic impact of HS, these findings underscore the need for models that are optimized to detect it.
While some studies have pursued multiclass classification of large-vessel occlusion, intracranial hemorrhage, subarachnoid hemorrhage, and IS, this classification approach limits sensitivity optimization for HS. Moreover, few studies have focused specifically on binary classification of HS versus IS [17,18]. Therefore, in this study, a ML model was developed and externally validated. The proposed model rapidly classifies HS and IS using only clinical variables routinely obtainable immediately upon ED arrival, without requiring imaging or laboratory data. To enhance clinical utility, we further applied a threshold optimization strategy to maximize HS sensitivity, presenting a reproducible tool to support acute stroke decision-making. This study focused only on patients with a final diagnosis of IS or HS. Evaluations of unselected ED populations, including stroke mimics, are required in future prospective studies.
MATERIALS AND METHODS
The study was conducted in accordance with the Declaration of Helsinki. It was approved by the Institutional Review Boards of hospital A (Chungbuk National University Hospital; No. 2025-06-23; July 17, 2025), hospital B (Dong-A University Hospital; No. 25-148; September 22, 2025), and hospital C (Seoul National University Bundang Hospital; No. 2506-981-107; June 16, 2025). Informed consent was waived owing to the retrospective study design.
Study Population and Dataset
This multicenter retrospective cohort study utilized electronic health records from three tertiary referral centers that provide specialized stroke care. A total of 2,998 patients were included: hospital A (n=1,370), hospital B (n=1,048), and hospital C (n=580). The patient selection flow is shown in Figure 1. The enrollment periods were July 2020–December 2023 for hospital A, January 2023–December 2023 for hospital B, and August 2023–January 2024 for hospital C. Patients from hospitals A and B (n=2,418) constituted the development and internal validation cohort, while patients from hospital C (n=580) served as an independent external validation cohort.
The inclusion criteria were adults (≥18 years) presenting to the ED within 24 hours of stroke symptom onset and diagnosed with acute stroke. The exclusion criteria were: (1) traumatic brain injury; (2) stroke mimics such as brain tumor or migraine; and (3) cases with substantial missing values in key predictor variables. The primary outcome was the classification of stroke subtype. HS included intracranial hemorrhage, intraventricular hemorrhage, and subarachnoid hemorrhage confirmed on official imaging reports by board-certified neuroradiologists.
The baseline characteristics included demographics (age, sex, smoking status, body mass index, onset-to-arrival time); clinical symptom of headache; medical history (cardiovascular disease, hypertension, diabetes, hyperlipidemia, atrial fibrillation [AF], prior stroke); medication history (antihypertensive agents, antiplatelet agents, anticoagulants, antidiabetic agents, lipid-lowering agents); point-of-care glucose; vital signs (systolic and diastolic blood pressure); and detailed National Institutes of Health Stroke Scale (NIHSS) item scores.
ML Approaches to Stroke Subtype Classification
Data from hospitals A and B were used for model development, whereas data from hospital C were completely separated for external validation. The development dataset was further divided into training and internal validation sets in a 4:1 ratio. Stratified four-fold cross-validation was performed to enhance the robustness of internal validation with stratified sampling applied to preserve class distribution. Class imbalance was addressed using the synthetic minority oversampling technique (SMOTE) [19], which was applied only to the training folds within each cross-validation iteration. An overview of the cross-validation procedure is presented in Figure 2.
We compared the performances of four ML algorithms: logistic regression, random forest, categorical boosting (CatBoost), and extreme gradient boosting (XGBoost). Among these, XGBoost demonstrated the best overall performance and was therefore chosen as the primary model for stroke subtype classification [20]. XGBoost is a decision tree–based gradient-boosting algorithm that has shown high efficiency and predictive accuracy when applied to structured medical data. The algorithm incorporates regularization techniques to prevent overfitting and leverages optimized parallel computation to accelerate training. In addition, it can effectively handle missing values and capture nonlinear feature interactions, which is particularly advantageous for clinical datasets.
Hyperparameter optimization was performed using the Optuna Bayesian optimization framework [21,22]. This method applies Bayes’ theorem to iteratively identify the optimal values for the objective function. The best-performing hyperparameters were selected using the Optuna search strategy; they are listed in Table 1. In this configuration, “learning rate” controls the step size for weight updates, balancing the speed and accuracy of training. “N estimators” refers to the number of boosting rounds (trees) that directly affect the model complexity. “Max depth” determines the maximum depth of each tree and influences the level of detail captured. “Min child weight” specifies the minimum sum of instance weights required in a child node, serving as a regularization parameter. “Subsample” and “colsample bytree” control the proportion of samples and features used for training each tree, respectively, thereby reducing overfitting risk. “Reg lambda” (L2 regularization term) and “reg alpha” (L1 regularization term) penalize large weights to improve generalization. “Gamma” sets the minimum loss reduction required for further partitioning a leaf node. Finally, “eval metric” is set to the area under the receiver operating characteristic curve (AUROC), which effectively measures the model’s ability to discriminate between IS and HS.
All model implementation and analyses were performed in Python 3.11 using scikit-learn (v1.6.1), XGBoost (v3.0.2), CatBoost (v1.2.7), and Optuna (v4.4.0), with computations executed on a Windows 10 Pro workstation equipped with an 11th Gen Intel Core i7-11700K CPU (3.60 GHz) and 32 GB of RAM without GPU acceleration. The trained pipeline, together with cross-validation metadata, was serialized with Joblib to ensure reproducibility.
Methods for Training and Internal Validation
To validate the reliability and consistency of the proposed ML model, we conducted internal validation using four-fold cross-validation with the hospital A and B datasets [23]. Of the datasets, 20% was allotted for testing while maintaining the class distribution; the remaining 80% was divided into four equal folds with balanced proportions of stroke patients. In each iteration, 25% of the data was used for validation and 75% for training.
Since the training dataset was imbalanced (1,791 IS and 627 HS patients), we applied SMOTE to generate a balanced dataset with equal numbers of IS and HS patients. We subsequently standardized numerical features such as body mass index, glucose, systolic blood pressure, and diastolic blood pressure using StandardScaler, which transforms data to have zero mean and unit variance according to the formula:
where z is the standardized value and x is the raw value. This normalization reduces the influence of differences in scale among variables and improves the stability of model training.
For internal validation, the model performance was evaluated on the A and B datasets using predictions based on an optimally determined threshold. For each fold, the optimal threshold was selected to maximize the sum of the sensitivity and specificity, ensuring the best balance between correctly identifying true positives and true negatives.
Performance Evaluation
The model performance was evaluated using four metrics: accuracy, sensitivity, specificity, and AUROC [24,25]. The metrics were calculated as follows:
where TP, FP, TN, and FN respectively denote true positive, false positive, true negative, and false negative. Sensitivity reflects the model’s ability to correctly identify positive cases, whereas specificity reflects its ability to correctly identify negative cases. AUROC summarizes the trade-off between sensitivity and specificity across all threshold settings, ranging from 0 to 1, with higher values indicating better discriminatory performance. For all models, AUROC values were calculated along with their 95% CIs.
Furthermore, we compared the sensitivity, specificity, and accuracy of the XGBoost model with those of the other ML algorithms to evaluate their relative performances. In addition to these discrimination metrics, model calibration was assessed using calibration curves and quantified by the Brier score, which measures the mean squared difference between predicted probabilities and the actual outcomes [26]. A lower Brier score indicates better calibration, and calibration plots provide a visual assessment of the agreement between predicted probabilities and observed event rates.
Method of External Validation
To evaluate the model’s generalizability, its performance was assessed on an independent external dataset. Preprocessing procedures—including data imputation and scaling—together with the model parameters identified during training were applied without modification. The performance was evaluated using the same metrics as those used in the internal validation. This external validation was designed to provide an unbiased estimate of the model’s robustness in an independent clinical setting.
Relative Feature Importance
We employed Shapley additive explanation (SHAP) to assess the relative importance of the input features used in this model [27]. SHAP originates from game theory and provides a systematic framework for interpreting ML models by quantifying the contribution of each feature to an individual prediction. In this study, we conducted a global feature importance analysis to evaluate the overall influence of each variable on the model.
Statistical Analysis
Statistical analysis was conducted to compare the IS and HS groups in the dataset and to assess the association between each feature and the stroke subtype. The dataset included only patients presenting with stroke, which resulted in clinical variables that did not follow a normal distribution. Therefore, non-parametric statistical tests were applied. Numerical features were described using medians and interquartile ranges, whereas categorical features were analyzed using counts and percentages. For categorical features, we employed the chi-square test or Fisher’s exact test, while the Mann-Whitney U-test was applied for continuous variables [28,29]. A two-sided P-value <0.05 was considered statistically significant.
In addition, we performed a comparison of the demographic and clinical characteristics between the development cohorts and the external validation cohort. This analysis was conducted to examine how potential differences in patient populations and clinical practices might have contributed to the observed performance discrepancies in the external validation.
RESULTS
Statistical Analysis
The baseline characteristics of the development cohort are summarized in Table 2. Among 2,418 patients in the development cohort, 627 (25.9%) had HS, which is consistent with the prevalence reported in clinical practice [2]. Compared with HS patients, those with IS were older (median, 70 vs. 64 years; P<0.001), more often men (60.0% vs. 49.4%, P<0.001), and more likely to be current smokers (34.6% vs. 23.3%, P<0.001). Onset-to-arrival time was substantially longer in IS (median, 180 vs. 72 minutes; P<0.001). Headache was strongly associated with HS (56.5% vs. 5.0%, P<0.001). Hypertension, diabetes, hyperlipidemia and AF were more prevalent in IS (all P<0.001) as was prior use of antihypertensive, antiplatelet, anticoagulant, and lipid-lowering medications. HS patients had higher systolic and diastolic blood pressure and glucose levels at presentation (all P<0.001).
As summarized in Supplementary Table 1, significant differences were observed in nearly all items of the NIHSS between patients with IS and those with HS. Patients with HS exhibited markedly greater impairments across most items. Supplementary Table 2 presents the demographic and clinical characteristics of the development and external cohorts. Significant differences are observed in age, body mass index, onset-to-arrival time, and key medical histories, such as hyperlipidemia and AF. Specifically, the development cohort was younger in age (69 years vs. 71 years), had a shorter onset-to-arrival time (140.0 vs. 342.0 minutes), and had a lower prevalence of hyperlipidemia (32.8% vs. 43.1%). Furthermore, the overall prevalence of headache was significantly higher in the development cohort (18.36% vs. 14.31%, P=0.025). Other clinical features such as hypertension, diabetes, and stroke subtypes showed no significant differences. The NIHSS distribution also differed between cohorts, particularly for NIHSS item 1a (P<0.001), indicating case-mix variability between the development and external cohorts (Supplementary Table 2).
Internal Validation
The four-fold cross-validation results of the XGBoost model are illustrated in Figure 3. In terms of AUROC, the model achieved a maximum value of 0.948. The quantitative performance metrics calculated from the confusion matrix are summarized in Table 3. Overall, the model demonstrated an average accuracy of 0.905, sensitivity of 0.828, specificity of 0.932, and AUROC of 0.937 (95% CI, 0.922–0.950). As shown in Figure 4, this model outperformed the other ML models in terms of AUROC. Detailed performance results of all models are summarized in Table 4. The calibration plots using both uniform and quantile binning (Figure 5) are close to the ideal baseline, indicating good agreement between the predicted and observed probabilities. The Brier score is 0.071, suggesting that the model was well calibrated.
External Validation
To assess the generalizability of the model, the optimal hyperparameters and the mean threshold derived from internal cross-validation folds were applied. In the external validation dataset, the model achieved an accuracy of 0.783, sensitivity of 0.758, specificity of 0.789 and AUROC of 0.841 (95% CI, 0.792–0.883), demonstrating moderate discrimination in the independent cohort (Figure 6). Given the baseline differences between cohorts (Supplementary Table 2), we additionally performed a supplementary site-adaptation analysis by updating the model using 20% of the external cohort for adjustment, with the remaining 80% reserved for testing. The ROC curves comparing the internal validation, independent external validation, and this updated model are shown in Supplementary Figure 1. In this site-adaptation setting (i.e., not an independent external validation), discrimination improved to AUROC of 0.875.
Feature Importance Using SHAP
To more effectively elucidate the contribution of each feature to the model’s predictions, we visualized SHAP values. Figure 7 shows a SHAP summary plot for the training dataset. SHAP analysis identified headache, NIHSS item 1a (level of consciousness), and AF as the three most influential features. Headache and higher NIHSS item 1a scores contributed toward HS prediction, whereas the presence of AF contributed toward IS prediction.
DISCUSSION
The most important finding of this study was that HS and IS can be differentiated through meaningful discrimination using a limited set of routinely available clinical variables at ED presentation without requiring laboratory tests. Because early management priorities differ substantially between HS and IS, a tool such as the one proposed may support early subtype suspicion while standard imaging-based pathways proceed, rather than replacing neuroimaging-based diagnosis.
In independent external validation, the proposed model demonstrated moderate discrimination (AUROC, 0.841); however, its performance decreased compared with internal validation (AUROC, 0.937). This gap suggests potential case-mix differences and site-related dataset shifts across institutions, underscoring the need to evaluate transportability when applying the model to new clinical settings.
Several factors may have contributed to the observed decrease in external performance. Even within comprehensive stroke centers, differences in patient severity distribution, symptom documentation (e.g., headache), timing metrics, and routine assessment practices (including item-level NIHSS scoring) can alter the joint distribution of predictors and their relationship to outcomes. Such heterogeneity is a common challenge for clinical prediction models and supports explicit calibration assessment and site-specific performance monitoring. Accordingly, the baseline characteristics differed between the development and external cohorts (Supplementary Table 2), including onset-to-arrival time, headache prevalence, hyperlipidemia, and NIHSS/NIH1a distributions, which could have plausibly influenced predictor–outcome relationships and calibration across institutions.
As a supplementary analysis, we evaluated a simple model-updating strategy using a small subset of the target-site data (20%) for adjustment, with the remaining 80% reserved for testing. This updating process improved discrimination (AUROC, 0.875); however, because target-site data were used for updating, these results should be interpreted as site adaptations rather than independent external validations. Together, these findings suggest that local calibration may help mitigate dataset shifting. Nonetheless, prospective evaluation remains essential prior to clinical implementation. Future work should assess how such updating affects both discrimination and calibration in prospective settings. The corresponding ROC curves comparing the internal validation, independent external validation, and site-adaptation analysis are provided in Supplementary Figure 1.
The model’s discrimination fell within ranges reported in prior subtype prediction studies and recent systematic reviews [11]. Two methodological strengths likely accounted for this performance. First, the use of a large multicenter dataset (>2,900 patients) mitigated the limitations of single-center studies that are often biased by local population characteristics or institutional protocols, supporting broader transportability [12-14]. Notably, the decrease in performance in independent external validation underscores the presence of site-related dataset shifts and the need for local evaluation and calibration. Second, to maximize clinical practicality, we excluded laboratory test variables that require additional processing time. By relying only on routinely obtainable clinical information at ED arrival, this model addresses a key limitation of prior studies and strengthens its feasibility in time-sensitive stroke workflows [15]. Direct head-to-head comparisons with conventional scores (e.g., the Siriraj Stroke Score) would require calculation within the same cohort and are an important direction for future work.
One of the notable strengths of this study is that it was based on real-world cohorts collected from three tertiary stroke centers. Unlike artificially balanced datasets, our cohort reflected the true distribution of IS and HS cases encountered in clinical practice, where HS patients are inevitably underrepresented. This imbalance presented a methodological challenge, as class imbalance is known to compromise model performance. Nevertheless, by applying appropriate strategies to address this imbalance, the model achieved clinically meaningful sensitivity for HS while maintaining overall discrimination, despite class imbalance. This point demonstrates that the proposed model was developed and validated not in an artificially controlled environment, but in cohorts that reflect routine acute-stroke practices, thereby strengthening its potential relevance to real-world implementation.
The core clinical significance of this study lies in its sensitivity-focused strategy for HS, achieved by adjusting the model’s probability threshold. Although the model cannot replace neuroimaging, which remains essential for determining thrombolytic eligibility, it can function as a valuable risk stratification tool to reduce diagnostic uncertainty during the initial ED evaluation while imaging workflows proceed. Most predictive model studies have emphasized overall accuracy [16], whereas in acute stroke care, the priority is rapid identification and exclusion of HS to expedite appropriate imaging and management. This trade-off is not intended to replace definitive treatment decisions; rather, it is intended to complement standard care without delaying imaging or reperfusion pathways. Notably, any use of the model should run in parallel with routine care and should not delay CT imaging or reperfusion workflows. For example, in settings where imaging or transfer decisions are competing for limited resources, patients predicted to be at high risk of HS could be prioritized for rapid CT scanning or considered for early neurosurgical preparation, thereby contributing to safer and more efficient acute stroke workflows. Conversely, for patients predicted to be at low risk of HS, the model may assist clinicians in anticipating potential eligibility for thrombolysis and preparation for timely reperfusion therapy, while still requiring imaging confirmation before treatment. Given that timely reperfusion therapy is strongly associated with better outcomes in real-world Korean registry data, tools that reduce early diagnostic uncertainty and streamline front-end triage may have practical value in time-critical stroke pathways [30].
The principal predictive features identified by SHAP analysis—headache, NIHSS item 1a, and AF—are consistent with clinically established indicators highlighted in both traditional scoring systems such as the Siriraj score and recent ML studies [9,11]. Headache emerged as the strongest predictor, with higher headache values shifting the model output toward HS, consistent with elevated intracranial pressure in HS. Similarly, impairment in NIHSS item 1a reflects the clinical reality that HS frequently presents with reduced consciousness due to increased intracranial pressure or a hematoma mass effect, and higher NIHSS item 1a scores contribute positively to HS prediction. In contrast, the association of AF with IS prediction is clinically plausible, given its role as a leading cause of embolic stroke, and AF contributes toward IS prediction in the model. Collectively, these findings indicate that the model captures clinically meaningful patterns rather than mere statistical associations, thereby enhancing the interpretability of its predictions. Nevertheless, explanatory consistency does not eliminate the need for prospective validation in new clinical settings.
This study had several limitations. First, its retrospective design inevitably introduced limitations in data quality. For example, important information such as anticoagulant use may have been incompletely captured. Although anticoagulant therapy critically influences stroke subtype classification, accurate medication histories are often difficult to obtain in the ED setting and may not always be objectively verified at ED arrival. Even when these data are available through insurance records after admission, they could confound the intended early application of the model. NIHSS assessments performed in emergency situations may also be subject to inter-rater variability. Nevertheless, NIHSS remains an internationally standardized tool that can be rapidly applied without specialized equipment. The model is not intended to delay CT imaging or add assessments beyond routine acute-stroke workflows. Thus, future prospective studies should explicitly evaluate workflow feasibility and time-to-imaging impact. Importantly, this study incorporated item-level NIHSS data rather than the total score, enabling a more granular characterization of neurological deficits. Patients with HS consistently demonstrated more severe impairments across most NIHSS items compared with IS patients, and this item-level approach likely contributed to the improved performance of the proposed model.
Second, the model’s predictive performance can be further improved. While this study employed the XGBoost algorithm, future research could explore deep-learning approaches, which are well suited to modeling complex nonlinear relationships. Moreover, this study reported only average performance across the entire cohort without subgroup analyses for specific populations, such as patients with minor stroke or very elderly individuals. The deliberate minimization of predictor variables to ensure feasibility in the early ED setting may also have constrained the model’s performance. For instance, previous studies have suggested that features such as pupillary abnormalities or seizure occurrence could further improve HS prediction [31,32]. In addition, simplified models using fewer predictors should be evaluated to reduce the operational burden in high-throughput ED settings.
Third, the possibility of selection bias must be acknowledged. Since this study included only patients with stroke confirmed by neuroimaging, the model’s ability to differentiate stroke mimics—conditions frequently encountered in real-world emergency settings—was not assessed. This exclusion may have led to overestimation of the model’s performance, particularly with respect to specificity, which may have limited its applicability to unselected ED populations. Addressing this limitation will require a two-stage classification framework: the first stage would discriminate true stroke from stroke mimics, followed by application of the proposed subtype classification model. The design of such a first-stage model could build on existing research in this area [33]. However, future work should also incorporate validation in cohorts that include both confirmed strokes and mimics, such as prospective stroke code registries, to more accurately reflect real-world performance and ensure patient safety. In particular, such analyses will be essential to evaluate whether the model can reduce unnecessary transfers or imaging while avoiding delays in reperfusion therapy. Accordingly, the present results should be interpreted as decision support for HS vs IS subtype suspicion among patients clinically managed within an acute stroke pathway, rather than stroke-versus-mimic triage.
Fourth, this study had regional limitations, as it was based exclusively on Korean patient data. Since stroke risk factors and clinical presentations vary across races and regions, further validation is warranted to confirm its reproducibility in other populations. Large-scale international external validation will be needed in collaboration with multinational research groups and emerging techniques such as federated learning, which allows joint model training without sharing raw data. This approach could provide a feasible solution under strict privacy regulations. Performance monitoring and calibration strategies should also be considered when implementing the model across healthcare systems.
Fifth, since this study was based on retrospective data, prospective studies are essential to determine whether the proposed model can improve real-time clinical decision-making and ultimately enhance patient outcomes in real-world clinical practice. In particular, integration in electronic health record systems, evaluation of clinical acceptance, and usability assessment in the rapidly changing ED environment will be crucial for determining the model’s effects on reducing treatment delays, guiding therapeutic choices, and facilitating efficient blood-pressure management. Prospective studies should also predefine safety endpoints (e.g., time-to-imaging, time-to-reperfusion) and evaluate calibration over time.
While this study primarily focused on ED presentation, any extension beyond the ED should be considered cautiously. Although NIHSS assessment is traditionally performed by trained neurologists, prior studies have demonstrated that stroke unit nurses and coordinators can reliably administer item-level NIHSS assessments with good agreement [34,35]. With further expansion of structured training programs, ED triage nurses and paramedics may be able to perform simplified NIHSS evaluations [36]. However, such applications would require evaluation of site-specific calibration and prospective validation in unselected populations (including stroke mimics) to ensure that the model use does not delay imaging or reperfusion workflows. Only after these conditions are met could the model be considered for broader use, potentially including pre-hospital decision support to inform referral pathways within acute stroke systems of care [37].
In this study, we developed a stroke subtype classification model using a large multicenter dataset and evaluated its generalizability in an independent external cohort. By incorporating a sensitivity optimization strategy for HS through probability threshold adjustment while preserving clinically meaningful discrimination, the model addresses key limitations of prior approaches. These strengths support its potential as a clinically practical decision-support tool to assist early subtype suspicion assessment and facilitate timely referral within acute stroke systems of care while standard imaging workflows proceed. Ultimately, this model may contribute to improving the safety and efficiency of acute stroke care. However, because stroke mimics were not included in this study, and the model performance varied across institutions, prospective validation in unselected ED populations (including mimics), assessment of calibration and workflow impact (e.g., time-to-imaging), and broader external validation across diverse populations are required before clinical implementation. Any future extension beyond the ED, including pre-hospital decision support, should be considered only after these conditions are met and should not delay imaging or reperfusion pathways.
KEY MESSAGES
▪ A multicenter machine-learning model using routinely available clinical variables at emergency department (ED) presentation distinguished hemorrhagic from ischemic stroke without requiring laboratory tests.
▪ The model maintained discrimination in an independent external cohort, although performance decreased compared with internal validation, suggesting potential site-related case-mix differences and the need for calibration in new settings.
▪ The model was designed for decision support for early subtype suspicion in an acute-stroke pathway; evaluation in unselected ED populations, including stroke mimics, is required.
NOTES
-
CONFLICT OF INTEREST
Dong-Ick Shin is an editorial board member of the journal but was not involved in the peer reviewer selection, evaluation, or decision process of this article. No other potential conflicts of interest relevant to this article were reported.
-
FUNDING
This work was supported by Chungbuk National University (2021) and by the Technology Management Center (Technology Licensing Office) project through Commercialization Promotion Agency for R&D Outcomes, funded by the Korean government (Ministry of Science and ICT; grant No. 2710006818)
-
ACKNOWLEDGMENTS
None.
-
AUTHOR CONTRIBUTIONS
Conceptualization: JHJ, KSY. Methodology: SP, KSY. Formal analysis: JWK, SP, KSY. Data curation: JWK, SP, KSY. Visualization: JWK. Project administration: KSY. Funding acquisition: SP, KSY. Writing - original draft: JWK, JHJ, SP, KSY. Writing - review & editing: JHJ, MKH, KHK, SP, DIS, KSY. All authors read and agreed to the published version of the manuscript.
SUPPLEMENTARY MATERIALS
Supplementary materials can be found via https://doi.org/10.4266/acc.004925.
Supplementary Table 2.
Comparison of demographic and clinical characteristics between the development cohorts and the external validation cohort
acc-004925-Supplementary-Table-2.pdf
Supplementary Figure 1.
Comparison of receiver operating characteristic curves for internal validation, external validation before fine-tuning, and external validation after fine-tuning.
acc-004925-Supplementary-Figure-1.pdf
Figure 1.Patient inclusion flow diagram (pooled) from the stroke registries of three hospitals. Patients arriving within 24 hours of symptom onset were included; 358 patients were excluded due to missing data, leaving 2,998 in the final analytic cohort.
Figure 2.Workflow of the stratified four-fold cross-validation performed for internal validation.
Figure 3.Receiver operating characteristic curves (A) and confusion matrices (B) of the extreme gradient boosting (XGBoost) model using four-fold cross-validation. Rows represent actual classes (0: ischemic stroke, 1: hemorrhagic stroke); columns represent predicted classes. Red dots indicate the optimal thresholds derived from Youden’s J statistic. AUROC: area under the receiver operating characteristic curve.
Figure 4.Receiver operating characteristic curves comparing the performances of different models. Values are presented as AUROC±standard error derived from cross-validation. AUROC: area under the receiver operating characteristic curve.
Figure 5.Calibration curves for the proposed prediction model. The calibration plot compares the predicted probabilities with the observed outcomes. The solid diagonal line represents the ideal reference (perfect calibration).
Figure 6.Comparison of receiver operating characteristic curves between internal and external validation. AUROC: area under the receiver operating characteristic curve.
Figure 7.Shapley additive explanation (SHAP) importance of features in the extreme gradient boosting (XGBoost) model. DBP: diastolic blood pressure; SBP: systolic blood pressure.
Table 1.Hyperparameters of the XGBoost model
|
Hyperparameter |
Set value |
|
Learning rate |
0.1012 |
|
n_estimators |
367 |
|
Max depth |
8 |
|
Min child weight |
2.5144 |
|
Subsample |
0.7096 |
|
Colsample bytree |
0.6446 |
|
Reg lambda |
0.0045 |
|
Reg alpha |
0.0029 |
|
Gamma |
1.6900 |
|
Eval metric |
AUROC |
Table 2.Descriptive statistics of stroke patients in the development cohort
|
Variable |
Total (n=2,418) |
Ischemic stroke (n=1,791) |
Hemorrhagic stroke (n=627) |
P-value |
|
Demographics |
|
|
|
|
|
Age (yr) |
69 (59–79) |
70 (61–80) |
64 (53–74) |
<0.001 |
|
Sex (male) |
1,385 (57.3) |
1,075 (60.0) |
310 (49.4) |
<0.001 |
|
Smoking |
765 (31.6) |
619 (34.6) |
146 (23.3) |
<0.001 |
|
BMI (kg/m2) |
23.6 (21.5–25.8) |
23.7 (21.6–25.7) |
23.4 (20.8–26.0) |
0.130 |
|
Onset to arrival time (min) |
140.0 (60.0–434.5) |
180.0 (69.0–523.0) |
72.0 (54.0–180.0) |
<0.001 |
|
Symptom |
|
|
|
<0.001 |
|
Headache |
444 (18.4) |
90 (5.0) |
354 (56.5) |
<0.001 |
|
Medical history |
|
|
|
|
|
Cardiovascular disease |
274 (11.3) |
207 (11.6) |
67 (10.7) |
0.600 |
|
Hypertension |
1,465 (60.6) |
1,137 (63.5) |
328 (52.3) |
<0.001 |
|
Diabetes |
701 (29.0) |
576 (32.2) |
125 (19.9) |
<0.001 |
|
Hyperlipidemia |
793 (32.8) |
706 (39.4) |
87 (13.9) |
<0.001 |
|
Atrial fibrillation |
450 (18.6) |
435 (24.3) |
15 (2.4) |
<0.001 |
|
Stroke |
|
|
|
<0.001 |
|
None |
1,940 (80.2) |
1,416 (79.1) |
524 (83.6) |
|
HS |
92 (3.8) |
46 (2.6) |
46 (7.3) |
|
IS |
386 (16.0) |
329 (18.4) |
57 (9.1) |
|
Medication |
|
|
|
|
|
Antihypertensive agents |
1,202 (49.7) |
957 (53.4) |
245 (39.1) |
<0.001 |
|
Antiplatelet agents |
586 (24.2) |
493 (27.5) |
93 (14.8) |
<0.001 |
|
Anticoagulant |
215 (8.9) |
184 (10.3) |
31 (4.9) |
<0.001 |
|
Antidiabetic agents |
561 (23.2) |
465 (26.0) |
96 (15.3) |
<0.001 |
|
Lipid–lowering agents |
663 (27.4) |
573 (32.0) |
90 (14.4) |
<0.001 |
|
Blood laboratory |
|
|
|
<0.001 |
|
Glucose (mg/dl) |
140 (116–182) |
136 (114–175) |
154 (125–198) |
<0.001 |
|
Vital signs |
|
|
|
|
|
SBP (mm Hg) |
155 (134–178) |
150 (132–170) |
163 (140–196) |
<0.001 |
|
DBP (mm Hg) |
88 (76–100) |
85 (75–97) |
93 (80–106) |
<0.001 |
Table 3.Performance of the XGBoost model evaluated by four-fold cross-validation
|
Accuracy |
Sensitivity |
Specificity |
AUROC |
|
Fold 1 |
0.882 |
0.921 |
0.869 |
0.948 |
|
Fold 2 |
0.897 |
0.800 |
0.930 |
0.928 |
|
Fold 3 |
0.899 |
0.856 |
0.913 |
0.937 |
|
Fold 4 |
0.894 |
0.888 |
0.897 |
0.939 |
|
Average |
0.905 |
0.828 |
0.932 |
0.937 |
Table 4.Performance of the CatBoost, random forest, logistic regression, and XGBoost
|
Accuracy |
Sensitivity |
Specificity |
AUROC (95% CI) |
|
CatBoost |
0.881 |
0.752 |
0.927 |
0.902 (0.857–0.922) |
|
Random forest |
0.885 |
0.839 |
0.902 |
0.925 (0.889–0.954) |
|
Logistic regression |
0.872 |
0.816 |
0.892 |
0.908 (0.869–0.938) |
|
XGBoost |
0.905 |
0.828 |
0.932 |
0.937 (0.922–0.950) |
REFERENCES
- 1. World Health Organization [WHO]. Cardiovascular diseases (CVDs). Fact sheet [Internet]. WHO 2025 [cited 2026 Feb 19]. Available from: https://www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds)
- 2. Hong KS, Bang OY, Kang DW, Yu KH, Bae HJ, Lee JS, et al. Stroke statistics in Korea: part I. Epidemiology and risk factors: a report from the Korean stroke society and clinical research center for stroke. J Stroke 2013;15:2-20.ArticlePubMedPMC
- 3. An SJ, Kim TJ, Yoon BW. Epidemiology, risk factors, and clinical features of intracerebral hemorrhage: an update. J Stroke 2017;19:3-10.ArticlePubMedPMCPDF
- 4. Dhillon PS, Singh N, Ospel JM, Roozenbeek B, Goyal M, Hill MD, et al. Pre-hospital stroke triage and research: challenges and opportunities. Cerebrovasc Dis 2025;54:282-8.ArticlePubMedPDF
- 5. Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Vamvakidis NC, Becker K, et al. Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2019;50:e344-418.ArticlePubMed
- 6. Hemphill JC, Greenberg SM, Anderson CS, Becker K, Bendok BR, Cushman M, et al. Guidelines for the management of spontaneous intracerebral hemorrhage: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2015;46:2032-60.ArticlePubMed
- 7. Dhamija RK, Donnan GA. The role of neuroimaging in acute stroke. Ann Indian Acad Neurol 2008;11:S12-23.PubMedPMC
- 8. Nour M, Liebeskind DS. Brain imaging in stroke: insight beyond diagnosis. Neurotherapeutics 2011;8:330-9.ArticlePubMedPMCPDF
- 9. Latchaw RE, Alberts MJ, Lev MH, Connors JJ, Harbaugh RE, Higashida RT, et al. Recommendations for imaging of acute ischemic stroke: a scientific statement from the American Heart Association. Stroke 2009;40:3646-78.ArticlePubMed
- 10. Mwita CC, Kajia D, Gwer S, Etyang A, Newton CR. Accuracy of clinical stroke scores for distinguishing stroke subtypes in resource poor settings: a systematic review of diagnostic test accuracy. J Neurosci Rural Pract 2014;5:330-9.ArticlePubMedPMC
- 11. Weir CJ, Murray GD, Adams FG, Muir KW, Grosset DG, Lees KR, et al. Poor accuracy of stroke scoring systems for differential clinical diagnosis of intracranial haemorrhage and infarction. Lancet 1994;344:999-1002.ArticlePubMed
- 12. Garg R, Oh E, Naidech A, Kording K, Prabhakaran S. Automating ischemic stroke subtype classification using machine learning and natural language processing. J Stroke Cerebrovasc Dis 2019;28:2045-51.ArticlePubMed
- 13. Daidone M, Ferrantelli S, Tuttolomondo A. Machine learning applications in stroke medicine: advancements, challenges, and future prospectives. Neural Regen Res 2024;19:769-73.ArticlePubMedPMC
- 14. Zhang S, Wang J, Pei L, Liu K, Gao Y, Fang H, et al. Interpretable CNN for ischemic stroke subtype classification with active model adaptation. BMC Med Inform Decis Mak 2022;22:3.ArticlePubMedPMCPDF
- 15. Abedi V, Misra D, Chaudhary D, Avula V, Schirmer CM, Li J, et al. Machine learning-based prediction of stroke in emergency departments. Ther Adv Neurol Disord 2024;17:17562864241239108.ArticlePubMedPMCPDF
- 16. Wu Y, Fang Y. Stroke Prediction with Machine Learning Methods among Older Chinese. Int J Environ Res Public Health 2020;17:1828.ArticlePubMedPMC
- 17. Uchida K, Kouno J, Yoshimura S, Kinjo N, Sakakibara F, Araki H, et al. Development of machine learning models to predict probabilities and types of stroke at prehospital stage: the Japan urgent stroke triage score using machine learning (JUST-ML). Transl Stroke Res 2022;13:370-81.ArticlePubMedPDF
- 18. Yoshida Y, Hayashi Y, Shimada T, Hattori N, Tomita K, Miura RE, et al. Prehospital stroke-scale machine-learning model predicts the need for surgical intervention. Sci Rep 2023;13:9135.ArticlePubMedPMCPDF
- 19. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 2002;16:321-57.ArticlePDF
- 20. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016 Aug 13-17; San Francisco, CA, USA. ACM; 2016. p. 785-94.
- 21. Akiba T, Sano S, Yanase T, Ohta T, Koyama M. Optuna: a next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2019 Aug 4-8; Anchorage, AK, USA. ACM; 2019. p. 2623-31.
- 22. Shekhar S, Bansode A, Salim A. A comparative study of hyper-parameter optimization tools. In: Proceedings of the 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE); 2021 Dec 8-10; Brisbane, Australia. IEEE; 2021. p. 1-6. Article
- 23. Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence; 1995 Aug 20-25; Montreal, QC, Canada. Morgan Kaufmann Publishers; 1995. p. 1137-43.
- 24. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982;143:29-36.ArticlePubMed
- 25. Hicks SA, Strümke I, Thambawita V, Hammou M, Riegler MA, Halvorsen P, et al. On evaluation metrics for medical applications of artificial intelligence. Sci Rep 2022;12:5979.ArticlePubMedPMCPDF
- 26. Dimitriadis T, Gneiting T, Jordan AI. Evaluating probabilistic classifiers: reliability diagrams and score decompositions revisited. arXiv [Preprint]. 2020 [cited 2026 Feb 19]. Available from: https://doi.org/10.48550/arXiv.2008.03033. Article
- 27. Lundberg SM, Lee SI. A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems; 2017 Dec 4-9; Long Beach, CA, USA. Curran Associates Inc.; 2017. p. 4765-74.
- 28. McHugh ML. The chi-square test of independence. Biochem Med (Zagreb) 2013;23:143-9.ArticlePubMedPMC
- 29. Jung SH. Stratified Fisher's exact test and its sample size calculation. Biom J 2014;56:129-40.ArticlePubMed
- 30. Kim M, Lee JS, Lee SJ, Park SY, Seo J, Hong JM, et al. Effectiveness of intravenous thrombolysis in patients with large-vessel occlusion receiving endovascular treatment in South Korea. Acute Crit Care 2025;40:282-92.ArticlePubMedPMCPDF
- 31. Osman M, Stutzman SE, Atem F, Olson D, Hicks AD, Ortega-Perez S, et al. Correlation of objective pupillometry to midline shift in acute stroke patients. J Stroke Cerebrovasc Dis 2019;28:1902-10.ArticlePubMed
- 32. Derex L, Rheims S, Peter-Derex L. Seizures and epilepsy after intracerebral hemorrhage: an update. J Neurol 2021;268:2605-15.ArticlePubMedPDF
- 33. Maeng JY, Sung J, Kim GH, Kim JW, Yum KS, Park S, et al. Machine learning-based diagnostic model for stroke in non-neurological intensive care unit patients with acute neurological manifestations. Sci Rep 2024;14:29610.ArticlePubMedPMCPDF
- 34. Comer AR, Templeton E, Glidden M, Bartlett S, D'Cruz L, Nemati D, et al. National Institutes of Health Stroke Scale (NIHSS) scoring inconsistencies between neurologists and emergency room nurses. Front Neurol 2022;13:1093392.ArticlePubMed
- 35. Larsen K, Jæger HS, Hov MR, Thorsen K, Solyga V, Lund CG, et al. Streamlining acute stroke care by introducing National Institutes of Health Stroke Scale in the emergency medical services: a prospective cohort study. Stroke 2022;53:2050-7.ArticlePubMedPMC
- 36. Purrucker JC, Härtig F, Richter H, Engelbrecht A, Hartmann J, Auer J, et al. Design and validation of a clinical scale for prehospital stroke recognition, severity grading and prediction of large vessel occlusion: the shortened NIH Stroke Scale for emergency medical services. BMJ Open 2017;7:e016893. ArticlePubMedPMC
- 37. Kothari RU, Pancioli A, Liu T, Brott T, Broderick J. Cincinnati Prehospital Stroke Scale: reproducibility and validity. Ann Emerg Med 1999;33:373-8.ArticlePubMed
Citations
Citations to this article as recorded by
KSCCM
, Jin-Heon Jeong2,*
PubReader
ePub Link
Cite
XML Download
