Prior studies on VPN traffic classification using the ISCX VPN-nonVPN dataset report high accuracy (often above 90%) under conventional random train-test splits. This study investigates whether such performance reflects genuine generalization. After deduplication and 20-seed application-disjoint evaluation, Random Forest’s accuracy falls from 92.3% to between 35% and 67% when tested on withheld, unseen application categories, and for four of seven categories falls at or below a trivial majority-class baseline. Three independent sanity checks confirm this gap is genuine rather than a data-leakage artifact. A feature ablation study shows the gap is driven by category-specific feature interactions rather than a single overfitting mechanism.As a secondary, partial mitigation, we propose Category-Invariance-Guided Feature Selection (CIFS), a lightweight feature-selection method that removes application-revealing signal from features. On Random Forest, CIFS significantly improves accuracy in two of seven categories after Bonferroni correction (BROWSING, MAIL). However, an identical evaluation on Decision Tree shows no significant benefit and one significant harm, indicating CIFS’s benefit is classifier-specific rather than universal.
Keywords: Feature Selection, Random Forest, VPN traffic classification, generalization gap, machine learning
