One Ticker, Two Markets: Non-Stationarity and Hidden RegimesE-mini S&P 500 FuturesCME_MINI_DL:ES1!EdgeToolsPart 3 of 5: The Signal Book Across the past three decades SPY spent roughly 62% of its trading days in an environment that realised about 11% annualised volatility and compounded at some 17% a year, and the remaining 38% in one that realised about 27% and went essentially nowhere. Same ticker, same thirty years, two economically different assets. Which of the two a backtest is actually describing is a question that almost never gets asked, and it is the question this part is about. Parts 1 and 2 built two measuring tools. Part 1 asked whether a score forecasts anything at all and settled on the time-series information coefficient (IC), with the teaching signal, a z-score mean reversion on SPY, scoring about 0.076 against the next five day return, Part 2 showed that a shelf full of indicators can collapse into barely two independent forecasts once the correlations between the scores are measured honestly. Both rested on an assumption neither part examined, and it is the one attacked here: that the thirty year sample is a single stable machine, so that a number measured across all of it describes how the market behaves in general. It does not. The 0.076 is an average taken across booms, panics, grinding bull markets and violent crashes, as though the rules generating returns never changed. They change. There are long stretches where a signal earns its keep and long stretches where the same signal does nothing, and the single-sample average blends the two into a figure that describes neither. Two results follow from taking that seriously, and neither is comfortable. The teaching signal turns out to forecast better in the calm regime than in the turbulent one, which is the reverse of what most mean-reversion traders assume; and a regime model fitted the way most of them are fitted disagrees with an honest real-time version on roughly one day in eleven, concentrated precisely at the turns where the call would have mattered. Before we combine forecasts into a book in Part 4, we need a language for the fact that the market is not one machine but several, and that we rarely know which one is running today. The average is a fiction you can be honest about Start with a word. A process is stationary when its statistical behaviour, the average return, the volatility, the way today relates to tomorrow, stays fixed through time. Non-stationarity is simply the denial of that: the statistics of returns do not stay put. In plain terms, the market you are trading this quarter may not be the market you backtested on, even on the same ticker. Every trader already knows this in their body. Volatility clusters. You get months where daily moves are a fraction of a percent and the tape barely breathes, and then a cluster of days where five-percent swings arrive back to back. The precise version of that intuition is that returns are non-stationary in their second moment: the variance itself moves, and it moves in persistent blocks rather than at random. What makes the split in the opening paragraph more than a curiosity is how long each environment lasts. These are not alternating days that wash out over a quarter; the calm state runs for something like five months at a time and the turbulent one for about three, which is long enough that a strategy can be launched, evaluated and abandoned entirely inside a single regime. That split is not eyeballed either, and neither are the durations; both come out of the model built in the next section. What we see and what we do not Here is the awkward part. We observe prices and returns. We do not observe a label stamped on each day that reads calm or crisis. The environment is real and it drives behaviour, but it is hidden: it has to be inferred from the observations it leaves behind. This is the distinction the rest of the part turns on. The observation is what you can see, the daily return and how choppy the recent tape has been. The hidden state, or regime, is the unobserved mode the market is in, which shapes those observations without ever announcing itself. A regime is not a candlestick pattern and not a headline; it is a latent setting that changes the distribution returns are drawn from. Inferring it is guesswork with structure, and the tool that formalises the guesswork is the hidden Markov model. A hidden Markov model, without the intimidation A hidden Markov model (HMM) is a way to describe data that is generated by an unseen state which switches over time. Strip away the machinery and it has three ingredients, all of which map onto things a trader already believes: A small number of hidden states. Here, two: calm and turbulent. Chances of staying or switching. Each day the market either stays in its current state or jumps to the other, with fixed probabilities. This is the "Markov" part: tomorrow's state depends on today's, not on the whole history. How observations look inside each state. Within calm, returns are small and tightly clustered; within turbulent, they are large and widely spread. Each state carries its own return-and-volatility profile. The staying-and-switching probabilities live in a small table called the transition matrix. Fitted to SPY, it is strongly persistent. Figure 1: The fitted two-state model. On any given day the market stays calm with probability 99.1% and turns turbulent with only 0.9%; once turbulent, it stays so with probability 98.5%. Those stay-probabilities imply average spells of about 109 trading days of calm and about 67 of turbulence. The reason regimes are worth modelling at all is written in those numbers. Because the market stays put roughly 99% of the time states are sticky, a calm day is overwhelmingly followed by another calm day. Persistence is what makes the current state informative about the near future, and it is what turns "volatility clusters" from a vague observation into something you can estimate. A memoryless coin flip between calm and turbulent every day would be useless a process that stays in a regime for months is not. Fitting such a model means choosing the transition probabilities and the two return profiles so that the observed history is as likely as possible under the model. There is no closed-form answer, so the parameters are learned by iterative guessing and updating. Start with a rough split, compute how well it explains the data, nudge the parameters to explain it better, repeat until the improvement stalls. That is the whole idea behind the estimation algorithm, and the three questions an HMM is built to answer are worth stating in words, because they are the same three questions every regime user asks (Rabiner, 1989): how likely is the observed data under a given model, what is the most likely sequence of hidden states behind it, and how do we learn the parameters in the first place? Optional detail: the transition matrix and expected spell length. The 2x2 matrix P holds the probability of moving from each state to each state, with rows summing to one. Fitted here: P(calm -> calm) = 0.991 P(calm -> turbulent) = 0.009 P(turb -> calm) = 0.015 P(turb -> turbulent) = 0.985 If a state persists with probability p each day, the expected length of one uninterrupted spell is 1 / (1 - p) days: 1 / (1 - 0.991) is about 109 days of calm, 1 / (1 - 0.985) about 67 days of turbulence. The "most likely path" of hidden states through the data is recovered by the Viterbi algorithm, which is just an efficient way to pick the single best sequence rather than scoring every possible one. What the model actually outputs Feed SPY's daily returns and a rolling volatility measure into this model and it does not hand back a trade. For each day it returns a state probability: how likely it is, given what has been observed, that the market is in each regime. Plain version: a number between zero and one saying how sure the model is that today belongs to the stormy regime rather than the calm one. Precise version: the posterior probability of each hidden state on a given day, conditional on the observed returns. On the worst days of March 2020 that turbulent probability sits at essentially one; through the quiet grind of 2017 it stays near zero. Placed in the plane of daily return against recent volatility, the two regimes occupy two different neighbourhoods, and the numbers quoted at the top of this article are simply the two clouds described one at a time. Figure 2: Every trading day since 1993 placed by its recent volatility (horizontal) and that day's return (vertical), coloured by the model's probability that the day is turbulent. Calm days cluster at low volatility with a mild upward drift; turbulent days spread out to the right, with far larger swings in both directions and no drift to speak of. The same information laid out through time shows how the two states take turns. Figure 3: Estimated regime probabilities on SPY across the full sample. The orange band is the probability of turbulence; it swells around the dot-com unwind, the 2008 crisis, the 2011 downgrade, the 2020 crash and the 2022 selloff, and shrinks to almost nothing through the long calm stretches between them. Two honest caveats belong right here. The bands are close to zero or one most of the time because the two states are so well separated in volatility that the model is rarely undecided that crispness is a property of this simple two state setup, not a claim of certainty. And the model occasionally flags a single violent day inside an otherwise calm stretch as turbulent, because a one-day volatility spike genuinely looks like the turbulent distribution. The regime label is a statement about the return environment, not a verdict on the next trade. The teaching signal forecasts better in calm markets than in chaos Now connect this back to Parts 1 and 2. If the market really is two environments, the mean-reversion signal from Part 1 need not forecast equally well in both. Split the days by their estimated regime and recompute the time-series IC of the −z(20) score against the next five-day return. Figure 4: The Part 1 teaching signal's IC, measured on all days, on calm days only, and on turbulent days only. Its forecast power is not constant: it is higher in the calm regime (0.092) than in the turbulent one (0.062), with the all-days figure of 0.076 sitting in between. The result runs the wrong way round for anyone who assumes mean reversion pays best in chaos, when prices overshoot violently. The score's rank association with future returns is stronger in the calm regime (IC 0.092 across 5,207 days) than in the turbulent one (IC 0.062 across 3,215 days), with the pooled 0.076 landing between them, which means the headline number from Part 1 was never a description of either environment. It was a weighted blend of the two, and a blend can hide a signal that quietly fails in exactly the environment you most need it to survive. Whether that particular gap reflects a genuine property of the signal or merely the noise of dividing a small edge across sub-samples is not settled by one split, and this is worth saying plainly rather than burying: with an IC of 0.062 measured over 3,215 days the turbulent estimate is statistically distinguishable from zero, yet the difference between 0.092 and 0.062 is not something a single split can establish as real. The claim being made here is the structural one, that a pooled IC is a blend and has to be decomposed before it can be trusted, and not the directional one about which regime favours reversion. A second caution keeps the figure from being oversold, and it is the more serious of the two. The regime labels used to split the days come from a model fitted on the whole history, so they already know where the crises were. Conditioning on them is a descriptive exercise, useful for understanding the signal, not a real-time trading result. That distinction is the subject of the next section, and it is where most regime work quietly cheats. The look-ahead trap that most regime backtests fall into Here is the trap that makes regime models look better on paper than they can ever be live. Fit the model once on the entire sample and it has, in effect, read the last page of the book before labelling the first. The estimate for any given day is allowed to lean on what happened afterwards. Look-ahead bias in regime fitting is exactly that: using observations from after date t to decide the state at date t. The full-sample fit does it by construction, because the smoothing step that produces those clean probability bands passes information backwards through time. It is not wrong for describing history, but it is not available to anyone standing in the present. The honest alternative is a causal estimate: at each day, use only the data up to that day. Refit the model periodically on an expanding window, never on the future, and filter forward one day at a time. Doing that on SPY, after a three-year warm-up and refitting twice a year, the real-time estimate and the full-sample estimate disagree on the hard call, calm versus turbulent, on about 9% of days. Figure 5: The probability of turbulence around two crises, estimated two ways. Orange is the full-sample fit that is allowed to use the future; blue is the real-time filter that never sees past the current day. Around the obvious core of each crisis the two agree, because the signal is unmistakable; they part company at the edges, where the turns are ambiguous and only hindsight makes them look sharp. The two lines agreeing through the depths of 2008 and March 2020 is not reassuring, it is expected: when volatility triples, no model needs the future to notice. The disagreement lives at the turning points, the run-up and the recovery, precisely the moments a trader would most want a regime call and precisely where the real-time version is least sure. Any backtest that filters trades by a full-sample regime label is quietly borrowing that hindsight, and the edge it reports is not one that was available in real time. Roughly one day in eleven is the honest tax on not knowing the future, and it is charged where it hurts. There is a concrete way to check whether your own work is paying it. Take any rule you have that switches on a regime, a volatility state, a trend filter fitted once over the whole history, and ask what the filter would have said had it only seen data up to each day. If the answer required a parameter estimated on the full sample, the reported edge includes something you could not have known. Refitting causally and comparing the two label series costs an afternoon and tells you the size of the borrowing, whether it is nine percent of days or considerably worse. What regimes do and do not tell you A regime model answers the third research question of this series, after Part 1's "does this score forecast anything?" and Part 2's "how many independent forecasts do I really have?": is the market in a state where my forecasts are even likely to behave the way I measured? On SPY the answer is that the environment flips between two economically distinct states, each lasting months, and that the demo signal's forecast power is not the same in both. It does not tell you: when the next switch will happen; the model estimates the current state, it does not forecast the transition that a regime colour is a trade; diagnosis of the environment is upstream of any entry rule, exactly as a positive IC was in Part 1 that two states are the right number; that is a modelling choice made here for clarity, not a fact about markets which regime genuinely favours mean reversion; the gap measured here is descriptive, and one split cannot settle it how to weight several forecasts once you know the regime, which is the next problem History is not one long sample; it is a sequence of regimes that never introduce themselves. A two-state model on SPY splits thirty years into a calm environment worth about 11% volatility and a turbulent one worth about 27%, and a signal that looks steady on average can be strong in one and weak in the other. The moment you let the model see the future, its regime calls sharpen and stop being real; the discipline is to estimate the present using only the past. You should now be able to explain Non-stationarity: the statistics of returns, especially volatility, do not stay fixed, so an average measured over decades can describe no single period. Observed versus hidden: we see returns; the regime that generates them is a hidden state we infer, not something the market prints. An HMM is a small model of a switching hidden state, with staying-and-switching probabilities and a return profile per state, learned by iterative fitting. A state probability is a diagnosis of the environment, not an entry signal, and the same forecast can carry different IC in different regimes. Why a pooled IC has to be decomposed: 0.076 across all days was a blend of 0.092 in calm and 0.062 in turbulence, and it described neither. Look-ahead in regime fitting: a full-sample fit uses the future to label the past, a causal fit does not, and the two disagreed on about 9% of days here. Next: Combining Weak Edges Under Risk and Turnover Constraints Three parts in, we can measure whether a forecast has power, count how many independent forecasts we truly hold, and judge whether the environment is one where those forecasts are likely to work. What remains is the problem every desk actually faces: given a handful of weak, correlated, regime-dependent edges, how do you weight them into a single book without letting estimation error, trading costs and turnover eat the entire advantage? Part 4 turns to combination, where the enemy is no longer measurement but the cost of acting on what you measured. References Ang, A. and Timmermann, A. (2012) 'Regime changes and financial markets', Annual Review of Financial Economics Hamilton, J.D. (1989) 'A new approach to the economic analysis of nonstationary time series and the business cycle', Econometrica Rabiner, L.R. (1989) 'A tutorial on hidden Markov models and selected applications in speech recognition', Proceedings of the IEEE