E-Commerce Fraud Prediction
Ranks e-commerce orders by fraud risk under a strict time split, built for a pickup and return point (PARP) abuse setting.
Overview
The question is how well a model can rank orders by fraud risk when it only learns from the past. I use the labelled part of the IEEE-CIS Fraud Detection data from Vesta: 571,887 card transactions, 3.42% fraud in train, 4.12% in valid and 3.44% in test. No public labelled data exists for pickup and return point (PARP) abuse, so the models are built on card transaction fraud and the return side stays a design.
Train covers day 1 to 111, valid day 111 to 134, a 7 day gap follows, and test is day 141 to 182. Every choice was made on valid or on forward windows. History features use strictly earlier rows and no labels, and no absolute time features are used.
How the data is split
Train covers day 1 to 111 (385,712 rows, 3.42% fraud), valid day 111 to 134 (68,067 rows, 4.12%), then a 7 day gap, then test day 141 to 182 (118,108 rows, 3.44%). Model choices used six 16 day forward windows inside train and valid.
Time split on the transaction day axis, then the six selection windows. Counts from results/final/metrics.json and the window rule in the project repo.
What I built
- A time split with a 7 day gap, so no feature or label from the test period reaches training.
- Past only history features on card and address keys, which bring the final model to 483 features, with a fix for missing keys and a class weight of 28.
- Reference models on the same split: logistic regression, an isolation forest and a simple rule.
- A LightGBM model with five seeds, each retrained on train + valid (453,779 rows) for the round count that early stopping found on valid, combined by rank average.
- Two selection rounds, p4 and p5, on six 16 day forward windows with five seeds, a paired day block bootstrap and Holm correction (ADR-005).
- A decision layer with review capacity, a cost policy, isotonic calibration and SHAP, and a design for the return side.
Results
The final model, final_bag5_refit, reaches a test PR-AUC of 0.6234. The test period was viewed 6 times while the model was developed, each candidate fixed before its look, so 0.6234 is a comparative number and not a fresh estimate. The gain over the five seed bag comes from recency: the retrained seeds see the 14 most recent labelled days that early stopping had held out. A model trained on the same rows as final without a hold out ties final (+0.0016), so this is a recency effect and not a general case against early stopping.
| Model | What it is | Test PR-AUC | Paired gain of final_bag5_refit (95% interval) | Top 1% precision |
|---|---|---|---|---|
| final_bag5_refit | 5 seeds retrained on train + valid, rank average | 0.6234 | reference model | 92.0% |
| final_bag5 | 5 seeds trained on train, rank average | 0.5662 | +0.0572 [0.0320, 0.0891] | 89.2% |
| final | one seed, 483 features, class weight 28 | 0.5605 | +0.0629 [0.0380, 0.0937] | 88.2% |
Final model against the two models before it
Test PR-AUC is 0.6234 for final_bag5_refit, 0.5662 for final_bag5 and 0.5605 for final. The paired gain over final_bag5 is +0.0572 [0.0320, 0.0891] and over final +0.0629 [0.0380, 0.0937].
Test period, 118,108 orders, fraud rate 3.44%. Source: results/final_bag5_refit/test_view.json and the other two metrics files in the project repo.
Six looks at the same test period
Test PR-AUC in the order the test period was opened: baseline_lgbm 0.5232, lgbm_fe1 0.5166, lgbm_fe2 0.5628, final 0.5605, final_bag5 0.5662, final_bag5_refit 0.6234.
Each candidate was fixed before its look. Later looks are comparisons, not fresh estimates. Whiskers are 95% day block intervals where the project computed them.
Five seeds of the final model
The five refit seeds score 0.6233, 0.6255, 0.6234, 0.5885 and 0.6257 on test; seed 45 stopped early at 669 rounds. Their rank average scores 0.6234, the mean of the single seeds 0.6173.
Test PR-AUC per seed with the round count early stopping found on valid. Seed 45 stopped early at 669 rounds; a bag of five tolerates one such seed, a single seed deployment would not. Source: results/final_bag5_refit/test_view.json.
How the final model was chosen
Earlier rounds
The improvement round screened eleven feature blocks against a valid PR-AUC of 0.6950; the best gained +0.0028 and the round selected none. Graph features, a GNN member, XGBoost and CatBoost members, blends and stacks, and drift pruning did not beat final on valid either. In round p4, seed bagging added +0.0068 over a single seed in 6 of 6 windows and was selected. A no identity segment blend passed with +0.0024 but was not taken to test.
Round p5, Stage A
Nine ideas were screened against final, alone and mixed with it. Only retraining on train + valid stood out by a wide margin.
Round p5, Stage A: nine ideas screened
Six of nine ideas advanced. refit_x100 gained +0.0567 PR-AUC against final when used alone and won 6 of 6 windows; the next best, fastrp_past, gained +0.0023 and won 3 of 6.
Change in PR-AUC against final over six 16 day forward windows, solo and mixed with final. Source: results/p5/stage_a.json.
Round p5, Stage B
The ideas that advanced were confirmed window by window. The refit bag won every window against the final bag.
Round p5, Stage B: confirmation per window
The refit bag beat the final bag by +0.0556 [0.0489, 0.0618] in 6 of 6 windows. Under labels 14 days later, refit_lag14 still beat final_lag14 by +0.0431 [0.0353, 0.0486] in 5 of 5 windows.
PR-AUC per window for the reference and the candidate, with the pooled difference and its day block interval. Source: results/p5/stage_b.json.
Labels that arrive late
Real labels reach a fraud team late, so I repeated the comparison with labels delayed by 14 days.
When labels arrive 14 days later
With labels 14 days later, final loses 0.0415 and refit 0.0485 PR-AUC on average over windows w2 to w6 (final 0.5923 to 0.5508, refit to 0.5939 under the delay); refit still leads under the same delay.
Seed 42 PR-AUC per window for final, refit and their _lag14 versions (refit without delay comes from Stage A). Source: results/p5/stage_a.json and results/p5/*_lag14.
Decision layer
The decision layer was computed for the single seed final, not for the five seed refit bag. At the top 1% of orders the review queue is about 28 alerts a day. Ranking by p x amount beats ranking by p at every review fee.
Review capacity (single seed final)
Single seed final, whole test period: the top 1% of orders is 1,182 alerts, about 28 a day, with precision 88.2%, recall 25.7% and 15.1% of fraud amount. The top 5% reaches precision 41.6%, recall 60.5% and 55.8% of fraud amount.
Share of orders sent to review (k), computed on the test period for the single seed final, not for the five seed bag. Source: results/final/metrics.json.
Cost policy (single seed final)
Single seed final: ranking by p x amount saves more than ranking by p at every review fee, 74.8% against 70.3% at fee 2, 68.3% against 59.8% at fee 5 and 53.3% against 41.1% at fee 15.
Thresholds set on valid; amounts are IEEE-CIS currency units, so savings are relative.
Isotonic calibration, fitted on the first half of valid, brings the test ECE to 0.0031 and the Brier score to 0.0211 (raw scores: 0.0230 and 0.0236). The strongest SHAP features are the count features C1, C13, C14 and C11, followed by D1n.
Return and PARP side
- The label would be a return or not received claim that was later confirmed as abuse, scored at claim time.
- Features stay past only: the customer's return rate so far, days from delivery to claim, claims in the last 30 days, distinct customers per pickup point, device and address use in the last 7 to 30 days, and a mismatch between recipient and buyer.
- Training uses only matured labels. The real data path was not run; ASOS GraphReturns is the candidate, but its label is a return, not fraud.
- Held orders go to human review with reason codes. The model does not decline orders automatically (GDPR Art. 22, Swiss FADP Art. 21; not legal advice).
Limits
- The test period was viewed 6 times. 0.6234 compares candidates and is not a fresh estimate.
- The gain is recency. With labels 14 days later,
finalloses 0.0415 and refit 0.0485 PR-AUC on average over windows w2 to w6; refit still leads under the same delay. - Each selection window has three day blocks, so intervals rest on few blocks.
- Seed noise is about 0.005 to 0.015 PR-AUC, and IEEE-CIS has no device id and unnamed currency units for amounts.
- The PARP part is a design without labelled data.