Authors: Augusto Cerqua, Marco Letta, Gabriele Pinto
studying (ML) has gained a central function in economics, the social sciences, and enterprise decision-making. Within the public sector, ML is more and more used for so-called prediction coverage issues: settings the place policymakers goal to establish items most liable to a adverse consequence and intervene proactively; as an example, concentrating on public subsidies, predicting native recessions, or anticipating migration patterns. Within the personal sector, comparable predictive duties come up when companies search to forecast buyer churn, or optimize credit score danger evaluation. In each domains, higher predictions translate into extra environment friendly allocation of sources and simpler interventions.
To realize these targets, ML algorithms are more and more utilized to panel information, characterised by repeated observations of the identical items over a number of time intervals. Nonetheless, ML fashions weren’t initially designed to be used with panel information, which function distinctive cross-sectional and longitudinal dimensions. When ML is utilized to panel information, there’s a excessive danger of a delicate however significant issue: information leakage. This happens when info unavailable at prediction time by accident enters the mannequin coaching course of, inflating predictive efficiency. In our paper “On the Mis(Use) of Machine Studying With Panel Information” (Cerqua, Letta, and Pinto, 2025), lately printed within the Oxford Bulletin of Economics and Statistics, we offer the primary systematic evaluation of knowledge leakage in ML with panel information, suggest clear tips for practitioners, and illustrate the implications by way of an empirical software with publicly out there U.S. county information.
The Leakage Downside
Panel information mix two buildings: a temporal dimension (items noticed throughout time) and a cross-sectional dimension (a number of items, corresponding to areas or companies). Normal ML observe, splitting the pattern randomly into coaching and testing units, implicitly assumes unbiased and identically distributed (i.i.d.) information. This assumption is violated when default ML procedures (corresponding to a random cut up) are utilized to panel information, creating two fundamental varieties of leakage:
- Temporal leakage: future info leaks into the mannequin in the course of the coaching section, making forecasts look unrealistically correct. Moreover, previous info can find yourself within the testing set, making ‘forecasts’ retrospective.
- Cross-sectional leakage: the identical or very comparable items seem in each coaching and testing units, that means the mannequin has already “seen” many of the cross-sectional dimension of the information.
Determine 1 exhibits how totally different splitting methods have an effect on the chance of leakage. A random cut up on the unit–time degree (Panel A) is essentially the most problematic, because it introduces each temporal and cross-sectional leakage. Options corresponding to splitting by items (Panel B), by teams (Panel C), or by time (Panel D), mitigate one sort of leakage however not the opposite. Because of this, no technique utterly eliminates the issue: the suitable selection is dependent upon the duty at hand (see beneath), since in some circumstances one type of leakage is probably not an actual concern.
Determine 1 | Coaching and testing units underneath totally different splitting guidelines
Two Sorts of Prediction Coverage Issues
A key perception of the examine is that researchers should clearly outline their prediction objective ex-ante. We distinguish two broad courses of prediction coverage issues:
1. Cross-sectional prediction: The duty is to map outcomes throughout items in the identical interval. For instance, imputing lacking information on GDP per capita throughout areas when just some areas have dependable measurements. The perfect cut up right here is on the unit degree: totally different items are assigned to coaching and testing units, whereas all time intervals are saved. This eliminates cross-sectional leakage, though temporal leakage stays. However since forecasting shouldn’t be the objective, this isn’t an actual difficulty.
2. Sequential forecasting: The objective is to foretell future outcomes primarily based on historic information—for instance, predicting county-level earnings declines one yr forward to set off early interventions. Right here, the right cut up is by time: earlier intervals for coaching, later intervals for testing. This avoids temporal leakage however not cross-sectional leakage, which isn’t an actual concern because the identical items are being forecasted throughout time.
The fallacious method in each circumstances is the random cut up by unit-time (Panel A of Determine 1), which contaminates outcomes with each varieties of leakage and produces misleadingly excessive efficiency metrics.
Sensible Tips
To assist practitioners, we summarize a set of do’s and don’ts for making use of ML to panel information:
- Select the pattern cut up primarily based on the analysis query: unit-based for cross-sectional issues, time-based for forecasting.
- Temporal leakage can happen not solely by way of observations, but in addition by way of predictors. For forecasting, solely use lagged or time-invariant predictors. Utilizing contemporaneous variables (e.g., utilizing unemployment in 2014 to foretell earnings in 2014) is conceptually fallacious and creates temporal information leakage.
- Adapt cross-validation to panel information. Random k-fold CV present in most ready-to-use software program packages is inappropriate, because it mixes future and previous info. As a substitute, use rolling or increasing home windows for forecasting, or stratified CV by items/teams for cross-sectional prediction.
- Make sure that out-of-sample efficiency is examined on actually unseen information, not on information already encountered throughout coaching.
Empirical Utility
For example these points, we analyze a balanced panel of three,058 U.S. counties from 2000 to 2019, focusing completely on sequential forecasting. We take into account two duties: a regression downside—forecasting per capita earnings—and a classification downside—forecasting whether or not earnings will decline within the subsequent yr.
We run tons of of fashions, various cut up methods, use of contemporaneous predictors, inclusion of lagged outcomes, and algorithms (Random Forest, XGBoost, Logit, and OLS). This complete design permits us to quantify how leakage inflates efficiency. Determine 2 beneath studies our fundamental findings.
Panel A of Determine 2 exhibits forecasting efficiency for classification duties. Random splits yield very excessive accuracy, however that is illusory: the mannequin has already seen comparable information throughout coaching.
Panel B exhibits forecasting efficiency for regression duties. As soon as once more, random splits make fashions look much better than they are surely, whereas right time-based splits present a lot decrease, but lifelike, accuracy.
Determine 2 | Temporal leakage within the forecasting downside
Panel A – Classification activity
Panel B – Regression activity
Within the paper, we additionally present that the overestimation of mannequin accuracy turns into considerably extra pronounced throughout years marked by distribution shifts and structural breaks, such because the Nice Recession, making the outcomes significantly deceptive for coverage functions.
Why It Issues
Information leakage is greater than a technical pitfall; it has real-world penalties. In coverage purposes, a mannequin that appears extremely correct throughout validation might collapse as soon as deployed, resulting in misallocated sources, missed crises, or misguided concentrating on. In enterprise settings, the identical difficulty can translate into poor funding choices, inefficient buyer concentrating on, or false confidence in danger assessments. The hazard is very acute when machine studying fashions are supposed to function early-warning programs, the place misplaced belief in inflated efficiency may end up in expensive failures.
In contrast, correctly designed fashions, even when much less correct on paper, present trustworthy and dependable predictions that may meaningfully inform decision-making.
Takeaway
ML has the potential to rework decision-making in each coverage and enterprise, however provided that utilized appropriately. Panel information provide wealthy alternatives, but are particularly susceptible to information leakage. To generate dependable insights, practitioners ought to align their ML workflow with the prediction goal, account for each temporal and cross-sectional buildings, and use validation methods that forestall overoptimistic assessments and an phantasm of excessive accuracy. When these rules are adopted, fashions keep away from the entice of inflated efficiency and as an alternative present steering that genuinely helps policymakers allocate sources and companies make sound strategic selections. Given the speedy adoption of ML with panel information in each private and non-private domains, addressing these pitfalls is now a urgent precedence for utilized analysis.
References
A. Cerqua, M. Letta, and G. Pinto, “On the (Mis)Use of Machine Studying With Panel Information”, Oxford Bulletin of Economics and Statistics (2025): 1–13, https://doi.org/10.1111/obes.70019.
