ECM3175 · University of Exeter · 2025–26
Stock Market Direction Prediction
Benchmarking LSTM, Transformer, XGBoost, MLP, and Logistic Regression on 23 years of daily equity data — can feature-engineered machine learning beat chance on the S&P 500?
0
Years of Data
0k+
Daily Observations
0
Engineered Features
0
ML Architectures
Programme
Electrical and Electronic Engineering BEng (Hons)
Module
ECM3175 — Dissertation
Research Question
Can standard ML models beat chance on daily price direction?
Daily equity markets are famously noisy. This dissertation asks whether supervised learning pipelines trained on price-derived features alone can extract a statistically significant directional signal from S&P 500 (SPY) next-day returns.
The study uses a dual-task design — identical features and splits feed both a binary direction classifier and a continuous return regressor, enabling direct comparison of the two paradigms on the same data.
Fat-tail analysis confirms the data is non-Gaussian: extreme returns are 6× to 5,000× more frequent than a Gaussian model predicts, motivating loss functions and model choices that are robust to heavy tails.
Top-line Result
50.0%
Balanced accuracy · XGBoost classifier
+3.6 pp above chance · not statistically significant (McNemar χ² = 2.34, p = 0.126)
Chronological Data Split
Feature Engineering
54 features, zero look-ahead bias.
Computed for 16 equity symbols
All features are constructed with strict chronological ordering across seven categories spanning price, momentum, volatility, and volume. Cross-asset context features from QQQ, IWM, EEM, and USO augment the core SPY signal.
Returns & Momentum
71d/log returns, lagged Δ (t−1 to t−10)
Lagged Prices
5Close[t−1] through Close[t−5]
Trend / SMA
8SMA 5/10/20/50d · price-to-MA ratios
Volatility
3Rolling return σ over 5, 10, 20 days
Volume
6Volume Δ, rolling means, vol/MA ratios
Technical Indicators
8RSI-14 · MACD + signal · Bollinger Bands
Intraday Spreads
4H−L range · O−C · H−close · L−close
Data Characteristic
Heavy-tailed return distribution
SPY daily returns are emphatically non-Gaussian. Empirical tail probabilities far exceed Gaussian predictions — a key motivation for choosing robust model architectures and evaluation metrics. The effect is concentrated entirely in the tails: at 2σ the empirical rate is barely above Gaussian, showing the excess kurtosis comes from rare extreme moves rather than the bulk of the distribution.
|r| > 2σ
1.11×
vs. Gaussian
|r| > 3σ
6.02×
vs. Gaussian
|r| > 4σ
105.28×
vs. Gaussian
|r| > 5σ
5,368.84×
vs. Gaussian
Methodology
Five architectures, one benchmark.
All deep learning models share the same training infrastructure: Adam optimiser with ReduceLROnPlateau decay, early stopping at patience 15, and gradient clipping at max norm 1.0.
Click a card to inspect · pin two cards to compare
XGBoost
max_depth 6 · train-only imputation · gain-based splits
Bal. Acc
0.536
R²
-0.0625
Transformer
1 encoder layer · d_model 32 · 2 heads · seq len 20
Bal. Acc
0.516
R²
-0.0119
MLP
128 → 64 → 1 · ReLU · Dropout 0.5 · BCEWithLogits / MSE
Bal. Acc
0.519
R²
+0.0098
Logistic Regression
Median imputation · standardisation · L2 regularisation
Bal. Acc
0.497
R²
-0.0667
LSTM
1 layer · hidden 32 · seq 20 · Dropout 0.5 · grad clip
Bal. Acc
0.484
R²
+0.0073
Empirical Results
Test-set performance.
Evaluated on held-out data from 2019–01–01 to 2021–03–30 (565 trading days), unseen during training or hyperparameter selection.
| Model | Bal. Acc ↑ | AUC-ROC ↑ | Precision | Recall | F1 |
|---|---|---|---|---|---|
| XGBoost | 0.536 | 0.545 | 0.616 | 0.518 | 0.563 |
| Transformer | 0.516 | 0.526 | 0.588 | 0.985 | 0.737 |
| MLP | 0.519 | 0.504 | 0.594 | 0.735 | 0.657 |
| Logistic Regression | 0.497 | 0.498 | 0.571 | 0.183 | 0.277 |
| LSTM | 0.484 | 0.507 | 0.565 | 0.488 | 0.524 |
| Majority Baseline | 0.500 | — | 0.581 | 1.000 | 0.735 |
| Persistence Baseline | 0.469 | — | 0.555 | 0.555 | 0.555 |
Test split: 2019–01–01 to 2021–03–30 · 565 observations · greyed rows = baselines
Proof of Work
Outputs from the dissertation.
All figures generated with matplotlib and seaborn during model training and evaluation — click any to expand.
7 figures from the dissertation · click any to expand · generated with matplotlib + seaborn
Academic Contribution
Where do these results sit on the Efficient Market Hypothesis?
The semi-strong form of EMH holds that only truly non-public information can generate alpha. Our results suggest the reality lies just past that boundary — directional signal is detectable, but the magnitude of future moves is not.
This dissertation
Semi-Strong EMH
thin direction edge · magnitude unpredictable
01
A thin, unconfirmed directional edge
XGBoost's 3.6 pp edge above chance is not statistically significant (McNemar's χ² = 2.34, p = 0.126) — real and consistent across seeds, but indistinguishable from the majority baseline by this test.
02
Magnitude remains noise
All regression models converge near R² ≈ 0. Predicting how much the market will move is essentially equivalent to guessing — consistent with the semi-strong form of EMH.
03
Deep learning offers no decisive edge
LSTM and Transformer match but do not beat XGBoost in classification. The directional signal is thin enough that architectural complexity adds no useful inductive bias.
Conclusions
Six key findings.
01
A numerical edge, not a statistical one
XGBoost achieves 53.6% balanced accuracy — a 3.6 pp edge above random chance, consistent across seeds — but McNemar’s test shows it is not statistically distinguishable from the majority baseline (χ² = 2.34, p = 0.126).
02
Magnitude is essentially unpredictable
All regression models converge near R² ≈ 0, barely clearing the historical-mean baseline. The absolute size of next-day moves is dominated by noise.
03
Direction beats magnitude
Classifiers outperform their regression counterparts on identical data — binary directional structure is detectable even when continuous magnitude is not.
04
No decisive deep-learning advantage
LSTM and Transformer match but do not consistently beat XGBoost in classification. In regression, MLP edges out by a slim R² margin of 0.0025.
05
Transformer over-predicts ‘up’
Recall of 98.5% signals class-imbalance overfitting — the model learns to predict ‘up’ unconditionally rather than discovering genuine directional signal.
06
Semi-strong EMH broadly supported
Directional edges are real but thin. Bollinger Band positions and moving-average ratios dominate feature importance, consistent with mean-reversion microstructure.
Reproducibility
Stable across random seeds.
All deep learning models were evaluated across five random seeds (42, 123, 7, 2024, 99). Balanced accuracy remains tight, confirming results are not driven by lucky weight initialisations.
Statistical Significance · McNemar's Test (α = 0.05)
- Logistic Reg. vs Majority — significant, but worse than baseline (χ² = 12.23, p = 0.0005)
- LSTM vs Majority — significant, but worse than baseline (χ² = 9.96, p = 0.0016)
- LSTM vs XGBoost — significant, in favour of XGBoost (χ² = 4.0, p = 0.046)
- XGBoost vs Majority — not significant (χ² = 2.34, p = 0.126)
- MLP vs Majority — not significant (p = 0.267)
- Transformer vs Majority — not significant (p = 0.211)
Technical Stack
Built with.
Top Predictive Features · XGBoost Gain
Mean-reversion signals (Bollinger Bands, MA ratios) dominate — consistent with known equity microstructure.
Reflection
What I'd do differently.
What worked: XGBoost's robustness on tabular data confirmed the literature — gradient boosted trees genuinely suit engineered financial features better than recurrent or attention architectures in the low-data regime.
What I'd fix: The Transformer's recall collapse was avoidable. Weighted cross-entropy or focal loss tuned during validation would likely have corrected this; I identified it post-hoc. Next time: monitor per-class metrics during training, not just aggregate loss.
What's next: Alternative data (sentiment, options flow, macro regime indicators), longer historical windows, and live backtesting through the LEAN engine to measure real-world slippage against the theoretical edge.
Get in touch
Interested in the methodology, results, or potential applications? I'm open to conversations about ML in finance, quantitative research, and software engineering roles.
Contact me