AI & Machine Learning Topic
Evaluate and Improve an ML Model Without Fooling Yourself
A model can score well in a notebook and fail after release. The test may be too easy, the metric may hide important mistakes or the real data may differ. Evaluation means checking the model against the job it must do.
Keep three roles separate
Training data fits the model. Validation data, or cross-validation inside the training portion, helps choose settings. The final test set checks the chosen process. Repeatedly choosing settings from the test score makes that test less independent.
A random split is not always right. Use a time-based split for predicting future events. Use suitable group splits when related rows must stay together. Prepare data inside the validation process so scalers and feature selection cannot see held-out rows.
Choose a metric from the mistake
Accuracy is the share of correct class predictions. If only one percent of cases are faults, always saying no fault is 99% accurate and still misses every fault.
- Precision: of the positive predictions, how many were correct?
- Recall: of the real positive cases, how many were found?
- F1: a combination of precision and recall. It does not include every business cost.
- Confusion matrix: counts of correct and confused classes.
For regression, mean absolute error gives the average absolute miss in the target's units. Root mean squared error gives larger misses more weight. R-squared can be negative on held-out data; it is not a percentage accuracy score.
Tune inside training data
Install scikit-learn and save evaluation.py. This complete example chooses one setting by three-fold cross-validation, then evaluates once on held-out Iris rows.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.model_selection import GridSearchCV, StratifiedKFold, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=42)
pipeline = Pipeline([("scale",StandardScaler()),
("model",LogisticRegression(max_iter=500))])
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
search = GridSearchCV(pipeline, {"model__C":[0.1,1.0,10.0]}, cv=cv, scoring="f1_macro")
search.fit(X_train, y_train)
prediction = search.predict(X_test)
print("Chosen settings:", search.best_params_)
print("Confusion matrix:\n", confusion_matrix(y_test,prediction))
print(classification_report(y_test,prediction,zero_division=0))
Run python evaluation.py. Read the chosen C value, confusion matrix and per-class report. C controls regularisation strength for this classifier: smaller values mean stronger regularisation. The search is small on purpose.
What cross-validation can and cannot do
Cross-validation repeats training and validation on different folds. It gives a better view of variation than one convenient split. It still needs a split method that matches the task. Related users, future data or duplicate records can make every fold misleading.
Report the data size and how it was split. Compare with a simple baseline. Look at errors for relevant groups and rare cases. Do not claim certainty from a tiny difference between two scores.
Improve the process before making it larger
Inspect wrong predictions. Check label quality, missing fields and train-serving differences. Add a useful feature only if it is available at prediction time. Try simple changes before a much larger model. Keep a record of the dataset, code, settings and metric for each experiment.
If a probability drives a decision, check calibration and choose thresholds with the right costs. Test latency and resource use as well as predictive quality. A score alone cannot show whether an API is usable.
Practice
Write down which errors matter most for a spam filter and a fault detector. Explain why they may need different thresholds. Then name one source of leakage that a pipeline cannot fix, such as related customers placed across splits.
Official references
Reviewed on 25 October 2026. Use documentation for your installed versions.
Comments (0)
No comments yet. Be the first to comment!