Approach
One alert, one row: summarise the history, then let cross-validation decide
The transaction table is relational: 6,987,663 training transactions belong to 14,000 alerts, and the target lives on the alert. We therefore summarise each alert's own history into one row of features and model at the alert level. No statistic is ever computed on the test set, and every fitted quantity (thresholds, normalisation, feature selection, calibration) is fitted inside the training folds.
Ideas were borrowed from past Kaggle competitions with the same structure (Home Credit, AmEx, Elo, IEEE-CIS). Each was first tested on this data; in total 123 experiments were run, each accepted or rejected by the same rule.
- Explore the history behind each alert and record every finding as a measured fact or a labelled assumption.
- Aggregate each alert's transactions into ~470 candidate features in 16 families, plus scores from a transaction-level model.
- Judge every family with repeated stratified 5-fold cross-validation (3 fixed repeats) and one acceptance rule: keep a change only if the mean AUC rises by at least 0.0010 and it rises in all three repeats.
- Tune, add models to a rank-average ensemble under the same rule, calibrate on training predictions, and predict the hidden alerts.
The data
Four files, one relational key
| File | One row per | Rows | Columns |
|---|---|---|---|
| train_signals.csv | alert | 14,000 | signal_id · signal_sanasi (alert date) · eskalatsiya (1 escalated / 0 dismissed) |
| test_signals.csv | alert | 6,000 | signal_id · signal_sanasi |
| train_transactions.parquet | transaction | 6,987,663 | signal_id · tranzaksiya_vaqti (timestamp) · kirim_chiqim (in / out) · tranzaksiya_turi (karta card, bank_otkazmasi bank transfer, naqd cash, xalqaro international) · miqdor_indeksi (standardised amount) |
| test_transactions.parquet | transaction | 3,027,575 | same columns |
No missing valuesin any column of any file; every alert has at least one transaction.
No shared transactionsno transaction appears under two alerts, so alerts cannot be linked into customers.
180-day windoweach history ends at the alert date; 5.1 % of alerts have less than 170 days.
Single-transaction alerts139 alerts have exactly one transaction.
Type mixcard 53.8 %, bank transfer 39.4 %, cash 6.3 %, international 0.46 %.
Direction75.7 % incoming (kirim), 24.3 % outgoing (chiqim).
Train and test alerts are interleaved over the same 24 months
Test makes up 28.2–31.4 % of the alerts in every month, and the alert IDs are mixed the same way. The split is random, so a stratified random K-fold reproduces the test situation; a time-based split would measure extrapolation into the future that the test does not require.
train alertstest alerts
Data table
About 461 transactions behind a typical alert
Median 461 transactions (test 460). Escalated alerts have slightly more (AUC of the count alone 0.531). The spike at 1 is the 139 single-transaction alerts.
dismissedescalatedtest
Data table
Nearly every alert has the full 180-day window
Days from the first transaction to the alert date. 5.1 % of alerts have less than 170 days of history.
Data table
Target
17.2 % escalated, with no drift over two years
The monthly escalation rate is consistent with a constant
Monthly rates range from 13.9 % to 21.2 %, all within sampling noise of the overall 17.2 % (χ² test against a constant rate, p = 0.094). Date features were still tested (F6).
monthly rate with 95 % intervaloverall rate
Data table
Transaction behaviour
Activity fades before the alert, then a burst lands in its last minutes
Transactions per alert per day, and the final burst
Activity rises to about 3.2 transactions a day four months before the alert and falls to about 1 a day in the last week. Then 39 transactions on average arrive in a burst, 90 % of them within 2.6 minutes of 00:00 on the alert date. Escalated alerts are 4–9 % more active at every lag, but the burst size alone is nearly uninformative (AUC 0.514).
dismissedescalated
days before the alert date (burst excluded)
last 10 minutes before 00:00 of the alert date, 10-second bins
Data table
Incoming versus outgoing, and the mix of transaction types, barely differ by outcome
Per-alert shares, non-burst history. Escalated alerts use slightly more cash and international transfers; every share stays below AUC 0.53 on its own.
dismissedescalated
Data table
One global amount scale, a different level for each transaction type
miqdor_indeksi is one global z-score (mean -0.13, sd 0.98). International transfers are largest, then cash, bank transfers and cards; outgoing exceeds incoming within each type. Each type has a hard floor and cap: the only repeated values in the column are the caps.
incoming (kirim)outgoing (chiqim)
Data table
Time of day carries no signal
Timestamps are spread evenly over the day (busiest/quietest hour ratio 1.017) with the same profile for both outcomes.
dismissedescalated
Data table
Escalated vs dismissed
Escalated alerts move smaller amounts, especially by bank transfer
Density of escalated relative to dismissed, by amount percentile
For each type and direction, amounts are split into 20 equal-count bands; a value above 1 means the band is over-represented in escalated alerts. Bank transfers fall steadily from about 1.15 in the smallest band to about 0.72 in the largest, in both directions. Cards show a weaker version, cash is flat, and international transfers are noisy (few transactions).
incoming (kirim)outgoing (chiqim)no difference (ratio 1)
Data table
Every single feature is weak on its own
ROC-AUC of single per-alert aggregates on 14,000 training alerts. The strongest, the mean outgoing bank-transfer amount, reaches 0.426 (lower when escalated). Amount levels of different types move together (correlation 0.75–0.87), which suggests one customer-level amount factor.
Data table
…but the effects are steady and monotone
Escalation rate by feature decile with 95 % intervals; the grey line is the overall rate. Weak but monotone effects add up in a multivariate model.
Data table
From findings to features
Most ideas did not survive the acceptance rule — the ones that did: F1 · Amount statistics, F7 · Contrasts, fi · Cell interactions, exp-aggregates
Every family was added to the current best set one at a time and scored on the same 3 × 5 folds; Δ is the change in out-of-fold ROC-AUC in each repeat. Round 1 compared most families against F1+F2 and round 2 re-tested the rejected ones against F1+F2+F7. Phase 3 added five new ideas (kNN target mean, class density ratio, histograms, rank transforms, cell interactions); once the interactions were accepted, every family rejected so far was re-tested against the new base, and leave-one-out on the final set removed F2. Hyper-parameter tuning (three variants), seed averaging and monotone constraints did not pass the rule, so every row uses the same LightGBM settings.
| Family | What it measures | Motivated by | Δ AUC r42 / r43 / r44 | Mean Δ | Decision |
|---|---|---|---|---|---|
| F1 · Amount statistics | count, share, mean, sd, min, quantiles, max of the amount index in 15 cells (all; in/out; 4 types; 8 type × direction), non-burst history | amounts are the strongest signal | — | — | ✓ baseline |
| F2 · Volume and rhythm · baseline, removed by leave-one-out | active days, transactions per day, busiest day, gaps between transactions, history length, recency | escalated alerts are 4–9 % more active | -0.0012 / -0.0016 / -0.0006 | -0.0011 | ✗ removed |
| fi · Cell interactions · phase 3 | amount level of each cell minus the alert's overall level, outgoing − incoming, bank − card, share × level | trees need many splits to form differences of two features | +0.0049 / +0.0032 / +0.0058 | +0.0046 | ✓ accepted |
| exp-aggregates | mean of exp(σ·amount), σ ∈ {1, 1.5}, per cell | Kaggle Elo: work on de-standardised amounts | -0.0019 / -0.0005 / +0.0016 | -0.0003 | ✗ rejected |
| exp-aggregates · round 2 (base F1+F2+F7) | mean of exp(σ·amount), σ ∈ {1, 1.5}, per cell | Kaggle Elo: work on de-standardised amounts | -0.0021 / -0.0008 / -0.0009 | -0.0013 | ✗ rejected |
| exp-aggregates · phase 3 re-check (base with fi) | mean of exp(σ·amount), σ ∈ {1, 1.5}, per cell | Kaggle Elo: work on de-standardised amounts | +0.0012 / +0.0022 / +0.0008 | +0.0014 | ✓ accepted |
| F7 · Contrasts | last transaction, last-10 mean vs history, 7/30-day vs history, burst amount rank inside own history | Kaggle AmEx: last − mean features | +0.0008 / +0.0009 / +0.0024 | +0.0014 | ✓ accepted |
| Type-normalised amounts | amount z-scored within type × direction (fitted on the training fold), averaged per alert | one amount level across types | -0.0001 / -0.0000 / +0.0022 | +0.0007 | ✗ rejected |
| Type-normalised amounts · round 2 (base F1+F2+F7) | amount z-scored within type × direction (fitted on the training fold), averaged per alert | one amount level across types | -0.0019 / -0.0001 / +0.0006 | -0.0005 | ✗ rejected |
| Type-normalised amounts · phase 3 re-check (base with fi) | amount z-scored within type × direction (fitted on the training fold), averaged per alert | one amount level across types | +0.0014 / +0.0009 / -0.0010 | +0.0004 | ✗ rejected |
| PCA amount level | first two principal components of the 8 cell means (fitted on the training fold) | cell means correlate 0.75–0.87 | -0.0008 / +0.0003 / -0.0003 | -0.0003 | ✗ rejected |
| PCA amount level · round 2 (base F1+F2+F7) | first two principal components of the 8 cell means (fitted on the training fold) | cell means correlate 0.75–0.87 | -0.0019 / -0.0009 / +0.0012 | -0.0005 | ✗ rejected |
| PCA amount level · phase 3 re-check (base with fi) | first two principal components of the 8 cell means (fitted on the training fold) | cell means correlate 0.75–0.87 | +0.0011 / +0.0006 / +0.0005 | +0.0007 | ✗ rejected |
| Threshold shares | share of an alert's transactions below q20 / above q80 / above q95 of their cell (thresholds fitted on the training fold) | the escalated/dismissed density ratio | -0.0011 / -0.0001 / +0.0000 | -0.0004 | ✗ rejected |
| Threshold shares · round 2 (base F1+F2+F7) | share of an alert's transactions below q20 / above q80 / above q95 of their cell (thresholds fitted on the training fold) | the escalated/dismissed density ratio | -0.0016 / -0.0032 / +0.0002 | -0.0015 | ✗ rejected |
| Threshold shares · phase 3 re-check (base with fi) | share of an alert's transactions below q20 / above q80 / above q95 of their cell (thresholds fitted on the training fold) | the escalated/dismissed density ratio | -0.0001 / +0.0003 / +0.0005 | +0.0002 | ✗ rejected |
| F4 · AML behaviour | pass-through (out within 1/24/72 h of an in with a similar amount), cash-out after a transfer-in, in/out balance | AML typologies | -0.0010 / -0.0003 / -0.0002 | -0.0005 | ✗ rejected |
| F4 · AML behaviour · round 2 (base F1+F2+F7) | pass-through (out within 1/24/72 h of an in with a similar amount), cash-out after a transfer-in, in/out balance | AML typologies | -0.0030 / -0.0020 / -0.0021 | -0.0024 | ✗ rejected |
| F4 · AML behaviour · phase 3 re-check (base with fi) | pass-through (out within 1/24/72 h of an in with a similar amount), cash-out after a transfer-in, in/out balance | AML typologies | -0.0035 / -0.0006 / -0.0010 | -0.0017 | ✗ rejected |
| F3 · Burst | size, direction and type mix, amounts of the final burst and its deviation from history | the burst before every alert | +0.0007 / +0.0002 / +0.0009 | +0.0006 | ✗ rejected |
| F3 · Burst · round 2 (base F1+F2+F7) | size, direction and type mix, amounts of the final burst and its deviation from history | the burst before every alert | -0.0018 / -0.0031 / -0.0016 | -0.0022 | ✗ rejected |
| F3 · Burst · phase 3 re-check (base with fi) | size, direction and type mix, amounts of the final burst and its deviation from history | the burst before every alert | -0.0017 / +0.0003 / +0.0003 | -0.0004 | ✗ rejected |
| F5 · Time windows | count, outgoing share, mean/max amount by type in 5 windows (1, 7, 30, 30–90, 90–180 days) | Kaggle Home Credit: windows relative to the decision date | +0.0001 / -0.0018 / +0.0014 | -0.0001 | ✗ rejected |
| F5 · Time windows · round 2 (base F1+F2+F7) | count, outgoing share, mean/max amount by type in 5 windows (1, 7, 30, 30–90, 90–180 days) | Kaggle Home Credit: windows relative to the decision date | -0.0021 / -0.0024 / +0.0001 | -0.0015 | ✗ rejected |
| F5 · Time windows · phase 3 re-check (base with fi) | count, outgoing share, mean/max amount by type in 5 windows (1, 7, 30, 30–90, 90–180 days) | Kaggle Home Credit: windows relative to the decision date | -0.0006 / -0.0028 / -0.0010 | -0.0015 | ✗ rejected |
| F6 · Alert date | month, weekday, year | monthly rate varies 14–21 % | +0.0002 / -0.0003 / -0.0002 | -0.0001 | ✗ rejected |
| F6 · Alert date · phase 3 re-check (base with fi) | month, weekday, year | monthly rate varies 14–21 % | -0.0000 / -0.0016 / -0.0013 | -0.0010 | ✗ rejected |
| kNN target mean · phase 3 | escalation rate among the 50/200/500 nearest training alerts in the amount-level space (training rows leave-one-out) | Kaggle Home Credit 1st place | -0.0018 / -0.0012 / -0.0027 | -0.0019 | ✗ rejected |
| kNN target mean · phase 3 re-check (base with fi) | escalation rate among the 50/200/500 nearest training alerts in the amount-level space (training rows leave-one-out) | Kaggle Home Credit 1st place | -0.0020 / -0.0018 / -0.0023 | -0.0020 | ✗ rejected |
| Class density ratio · phase 3 | log(p_escalated / p_dismissed) of each amount inside its cell (20 bins, fitted without the alert's own label), averaged | naive-Bayes view of the density ratio chart | -0.0029 / -0.0000 / +0.0001 | -0.0009 | ✗ rejected |
| Class density ratio · phase 3 re-check (base with fi) | log(p_escalated / p_dismissed) of each amount inside its cell (20 bins, fitted without the alert's own label), averaged | naive-Bayes view of the density ratio chart | -0.0014 / -0.0006 / +0.0005 | -0.0005 | ✗ rejected |
| Amount histograms · phase 3 | share of an alert's transactions in each of 10 decile bins per cell (edges fitted on the training fold) | full distribution shape instead of 5 quantiles | +0.0013 / -0.0026 / +0.0005 | -0.0003 | ✗ rejected |
| Amount histograms · phase 3 re-check (base with fi) | share of an alert's transactions in each of 10 decile bins per cell (edges fitted on the training fold) | full distribution shape instead of 5 quantiles | +0.0008 / +0.0005 / -0.0000 | +0.0004 | ✗ rejected |
| Rank / Yeo-Johnson amounts · phase 3 | mean and sd of within-cell ECDF rank, mean of Yeo-Johnson-transformed amount (fitted on the training fold) | robust and variance-stabilised levels | -0.0023 / -0.0014 / +0.0001 | -0.0012 | ✗ rejected |
| Rank / Yeo-Johnson amounts · phase 3 re-check (base with fi) | mean and sd of within-cell ECDF rank, mean of Yeo-Johnson-transformed amount (fitted on the training fold) | robust and variance-stabilised levels | -0.0002 / -0.0007 / -0.0026 | -0.0012 | ✗ rejected |
| Transaction-level model | nested GPU XGBoost scoring single transactions; 9 aggregates of its scores per alert | Kaggle AmEx 1st place | -0.0010 / -0.0012 / -0.0012 | -0.0011 | ✗ rejected |
Measured, not assumed: scored on its own, F1 reaches 0.6446, practically the whole baseline. Several rejected families carry signal by themselves (Threshold shares 0.621, exp-aggregates 0.618, Time windows 0.611, Burst 0.562) yet add less than the threshold on top of F1, so their information overlaps with the amount statistics. The cell-interaction gain comes from the differences between cell levels: removing them changes AUC by -0.0045, removing the products by +0.0005. Null-importance feature selection was tested and rejected.
Validation & models
From 0.6443 to 0.6528 cross-validated ROC-AUC
Why a random stratified split. Train and test alerts are interleaved over the same months and the escalation rate is stable, so a random split reproduces the test situation. A time split (as in Kaggle IEEE-CIS, where test lies after train) would answer a different question.
Why no group split. No transaction is shared between alerts and there is no customer key, so there is nothing to group by. Each alert's transactions stay inside its fold because the model sees one row per alert.
Acceptance rule. Model-seed noise alone moves AUC by up to 0.0008 per repeat, so a change is kept only if the mean over three repeats improves by at least 0.0010 and every repeat improves. The rule was fixed before modelling and applied to every experiment, including ensemble members.
Repeated stratified 5-fold, 3 repeats
Blue: validation fold. Every fitted statistic (thresholds, normalisation, PCA, feature selection, the model) sees only the other four folds. Hyper-parameters, the feature-selection threshold and the spline penalty were chosen on separate tuning repeats (seeds 100 and 101), never on the three scoring repeats.
Cross-validated ROC-AUC by model
Out-of-fold ROC-AUC, mean ± sd over the three repeats. After the cell-interaction features were added, neither XGBoost, CatBoost nor the spline logistic regression improved the LightGBM under the rule (the spline model, accepted as a second ensemble member earlier, now lowered the score), so the submission is the single LightGBM. Platt scaling fitted on training out-of-fold predictions turns the score into probabilities without changing the ranking; test predictions are mapped through the training out-of-fold distribution, so nothing is fitted on test.
Data table
What the final LightGBM relies on
Share of total split gain, averaged over the fold models of the final LightGBM. The three largest: cash incoming level − overall level (6.3 %), level: bank transfer − card (4.7 %), bank transfer outgoing level − overall level (4.3 %). Top-20 by family: F1 · Amount statistics 12, fi · Cell interactions 7, F7 · Contrasts 1.
Data table
Conclusions
What we learned
Amount level is the signal. Escalated alerts move smaller amounts, most clearly in bank transfers (density ratio about 1.15 in the smallest band, 0.72 in the largest). Per-type amount levels are correlated (0.75–0.87), and F1 alone reproduces the baseline score.
Relative levels matter. The largest single gain (+0.0046) came from differences between the amount levels of an alert's own cells, such as outgoing versus incoming bank transfers or each cell versus the alert's overall level. Trees can in principle build such differences, but here giving them explicitly paid off in all three repeats.
Volume and timing add little. Escalated alerts are 4–9 % more active over the whole window; the volume family helped at first (+0.0012) but became redundant once the interaction features were in (removing it then gained +0.0011). The burst family gained only +0.0006, time of day shows no difference between the classes, and date features were rejected.
Weak features, careful validation. No single aggregate we scored exceeds AUC 0.574; three rounds of hyper-parameter tuning won on the tuning repeats but not on the scoring repeats. The final model reaches 0.6528 ± 0.0009 out-of-fold ROC-AUC.
Assumptions, not facts. That miqdor_indeksi is a z-scored log amount, that the floors and caps are clipping by the data generator, and that the burst is a generation artefact rather than the triggering activity are interpretations; the data cannot confirm them.
Synthetic data. The organisers generated and localised the data, so the patterns describe the generator, not real banking behaviour.
One submission. There is no leaderboard feedback; every decision rests on cross-validation, which is why the acceptance rule is strict.
Reproduce
Everything runs from the repository
The pipeline code and the reproducible notebook are in the repository. Put the four data files into data/, create the environment, then run the notebook: it rebuilds the features from the raw files, trains the final model on the fixed folds, checks the scores against the reference values and writes the submission file (about a minute on a laptop CPU).
python -m venv .venv && .venv/Scripts/pip install -r requirements.txt jupyter nbconvert --to notebook --execute notebooks/FB203632_pipeline.ipynb --output FB203632_pipeline.ipynb