Cross-Dataset Feature Stability Selection for Generalizable Phishing URL Detection: A Comparative Study of Classical Ensembles and Deep Learning (Published)
Phishing remains a vector for credential theft, and URL-based detection offers a fast first line of defense. We present a leakage-free comparative study of machine learning models for phishing URL detection on 549,346 URLs, testing whether in-distribution gains generalize to two independently collected datasets. Among classical models, XGBoost performed best (F1 = 0.8838), stacking improved it (F1 = 0.8951), and a character-level CNN (CharCNN) led in-distribution (F1 = 0.9604); tuning narrowed but did not close the gap (F1 = 0.9295). On external datasets, performance collapsed: CharCNN’s AUC-ROC fell below chance (0.271) on one, and all models fell to F1 0.44-0.48 on the other. We propose Cross-Dataset Feature Stability Selection, retaining 66 features important across three datasets; it costs 3.3 F1 points in-distribution but yields an 11.9-point gain on one external dataset, with a small trade-off on the second. We argue that cross-dataset evaluation should be standard in phishing-detection studies.
Keywords: CharCNN, Ensemble learning, SHAP, URL classification, XGBoost, phishing detection