Extreme Weather Event Classifier

Classifying tropical cyclones and atmospheric rivers from climate data, with models written from scratch in NumPy. Final project of IFT 3395 (Université de Montréal, 2021), built as a Kaggle competition entry.

Contents: Summary · What I learned · Repository structure · How to run · Results

Summary

This project is my entry to the IFT 3395 – Fondements de l'apprentissage machine Kaggle competition (Université de Montréal, fall 2021), done as a team of two.

The task: given atmospheric measurements for one place and time (latitude, longitude, time and 16 climate variables such as pressure, temperature and humidity), classify the observation as

• 0 – standard conditions,
• 1 – tropical cyclone, or
• 2 – atmospheric river.

The data is a small subset of ClimateNet: 120 locations, 47,760 training points (1996–2010) and 7,320 test points (2011–2013). Predictions on the test set were submitted to Kaggle and scored on a public leaderboard.

The constraints: the logistic regression had to be written from scratch (only the standard library, NumPy, scipy.sparse and pandas allowed), and at least one other model had to be tried and compared. We implemented three classifiers from scratch with NumPy — multinomial logistic regression, a soft Parzen-window (RBF kernel) classifier and a Naive Bayes classifier with kernel density estimates — plus three preprocessing strategies, and documented the results in a report (in French, see report/report.pdf).

The original course instructions are in instructions.pdf (French) and instructions.en.md (English translation).

What I learned

Machine learning

• Writing multinomial logistic regression from scratch: softmax, cross-entropy, deriving the gradient, vectorizing it with matrix operations, ridge regularization (bias excluded), step-size schedules and a convergence-based stopping rule.
• Non-parametric density estimation: Parzen windows with an RBF kernel, used both as a classifier and as the class-conditional densities of a Naive Bayes model.
• Model selection: train/validation splits, grid search over hyperparameters, retraining on train + validation with the best setting, and reading validation curves in terms of under/overfitting.

Data and features

• Feature engineering from domain knowledge: a sinusoidal seasonal feature from month and hemisphere, wind speed from its zonal/meridional components, and density-based spatial and temporal "event centre" distance features.
• Feature selection with the Variance Inflation Factor (VIF) to remove multicollinear variables, and standardization.

Engineering and teamwork

• Designing a small, composable preprocessing pipeline and abstract Model / Preprocessor interfaces so models and preprocessing strategies can be mixed and compared.
• Working in a competition setting (limited submissions per day, public leaderboard), collaborating in a team, and writing a technical report.

Repository structure

.
├── README.md
├── instructions.pdf         # Competition instructions (French)
├── instructions.en.md       # English translation of the instructions
├── run.py                   # Interactive CLI: train a model, predict on the test set, write a submission CSV
├── run_analysis.py          # VIF-based feature selection analysis
├── core.py                  # Preprocessing pipelines (none / standardize / manual / VIF), CSV reading and writing
├── preprocess.py            # Preprocessor building blocks: RemoveColumn, UnaryTransform, MapColumns, Standardize
├── classes.py               # Abstract Model / Preprocessor interfaces and the preprocessing + train/val/test split pipeline
├── lib.py                   # Logging, one-hot helpers, seasonal encoding, distance and progress-bar utilities
├── data/                    # train.csv and test.csv from the Kaggle competition
├── models/
│   ├── logistic_reg.py      # Multinomial logistic regression (softmax, ridge, gradient descent) with grid search
│   ├── parzen.py            # Soft Parzen-window classifier with RBF kernel
│   ├── naive_bayes.py       # Naive Bayes with Parzen densities and spatio-temporal feature engineering
│   ├── sample_models.py     # Baselines: always predict class 0, random predictions
│   └── main.py              # Earlier standalone prototype (one-vs-rest logistic regression), not used by run.py
├── out/                     # Predictions and plots generated when running the code (gitignored)
└── report/
    ├── report.pdf           # Final report (French)
    ├── Figure_1.png ... Figure_4.png
    └── (LaTeX source)

How to run

Requirements: Python 3 and the following packages.

pip install "numpy<2" matplotlib            # to run the models
pip install pandas plotly statsmodels       # only for run_analysis.py

NumPy 2.x is not supported: the code uses np.infty, which was removed in NumPy 2.0.

Setup: create the output folder:

mkdir out

Train a model and write predictions

python run.py

The menu is in French. Pick a model, then a preprocessing method, by typing its number (anything else quits):

Model		Preprocessing
0	Logistic regression	0	None
1	Parzen window, RBF kernel	1	Standardization
2	"Modèle 0" (always predicts class 0)	2	Standardization + manual feature design
3	Random model	3	Standardization + VIF feature selection

The model is trained with a hyperparameter search on a validation split, then predicts the test set. Predictions are written to out/<ModelName> (<timestamp>).csv in the Kaggle submission format (S.No,LABELS).

Naive Bayes has its own entry point (the matching menu entry in run.py is not wired up):

python -c "from models.naive_bayes import run_naive_bayes; run_naive_bayes()"

It shows the validation-cost curve and writes out/BayesNaif.csv.

VIF analysis (iteratively removes the feature with the highest VIF until all are below 5):

python run_analysis.py

A note on runtime: the hyperparameter searches are exhaustive and written with plain Python loops (396 combinations for logistic regression, 24 values of σ for Parzen), so a full run on the real dataset is slow. For a quick try, shorten the LOGREG_* lists in models/logistic_reg.py and SIGMA_VALUES in models/parzen.py.

Results

Model	Result
Multinomial logistic regression (ridge)	0.74207 (Kaggle public score)
Parzen window, RBF kernel	0.72213 (Kaggle public score)
Naive Bayes with Parzen densities	≈ 0.66 accuracy

The best preprocessing overall was VIF feature selection followed by standardization. The manual approach (seasonal feature, merged wind components, hand-picked removals) followed by standardization did worse.

Hyperparameter curves

Validation error of the logistic regression against the ridge weight λ, for three step-size schedules η(t). Decaying steps stay around 20% error whatever λ is, while a constant step degrades quickly as λ grows (from about 19% to 40%).

Same model against the convergence threshold ε. Smaller ε is better, and η(t) = 1 / (1 + ln(t + 1)) gives the lowest error (about 19.5%).

Parzen window with RBF kernel: 0–1 validation loss against the kernel width σ. The minimum is around σ ≈ 0.6 (about 13% error); a very small σ overfits and a large σ underfits.

Naive Bayes: validation cross-entropy against the kernel-width setting. Per feature, σ = (max − min) / s, and the x-axis shows s. The best value is around s = 50.

Takeaways from the report

• Naive Bayes struggles because features are not independent. Even after replacing latitude/longitude with distances to the main cyclone and river "centres", the independence assumption stays costly. With a too-narrow kernel the model over-predicts the rare classes (1,067 and 1,234 predictions for classes 1 and 2, against 438 and 982 real examples); with a too-wide kernel it predicts class 0 for everything.
• Validation and Kaggle scores disagreed. The Parzen validation curve suggested about 0.85 accuracy, yet the public score was 0.72, and logistic regression showed the same gap. We first suspected too little validation data, but a 1/10 train/validation ratio did not fix it, so the cause stayed unexplained.
• What we would do differently (from the report's discussion):
  • Analyze results as we go instead of writing the report at the end; our experiments lacked structure and rigor.
  • Keep U850 and V850 separate rather than merging them into a wind speed, since the direction carries information about cyclone types.
  • Try UBOT/VBOT as a baseline wind to subtract from U850/V850.
  • Use one σ per feature in the Naive Bayes model.

The full analysis (in French) is in report/report.pdf.