Abstract
The U.S. stock market is one of the largest and most complex marketplaces in the global financial system. Over the past several decades, this market has evolved at multiple structural and temporal scales. New exchanges became active, and others stopped trading, regulations have been introduced and adapted, and technological innovations have pushed the pace of trading activity to blistering speeds. These developments have supported the growth of a rich machine-trading ecology that leads to qualitative differences in trading behavior at human and machine time scales. We conduct a longitudinal analysis of comprehensive market data to quantify nonstationary dynamics throughout this system. We quantify the relationship between fluctuations in the number of active trading venues and realized opportunity costs experienced by market participants. We find that information asymmetries, in the form of quote dislocations, predict market-wide volatility indicators. Lastly, we uncover multiple micro-to-macro level pathways, including those exhibiting evidence of self-organized criticality.
1. Introduction
The U.S. National Market System (NMS) is composed of adaptive agents with goals and strategies for achieving them [1–3]. The NMS and its various participants continue to coevolve: trading strategies adapted, publicly traded companies are listed and delisted, regulations change, and stock exchanges launch and cease operations. This coevolution may yield emergent phenomena, such as bubbles and crashes [4–6].
The motivation for our study has two primary components. The theoretical component encompasses recent extensions to prevailing financial theory: from the Efficient Market Hypothesis to the Adaptive Market Hypothesis [1]. Market efficiency is tightly coupled to various aspects of the market ecology; namely, the number of competitors, the frequency and magnitude of profit opportunities, and the adaptability of market participants. The empirical component encompasses recent studies to characterize bubbles and crashes [6] and market inefficiencies [7, 8]. Central to both components, and core to our study, are nonstationary dynamics arising from information asymmetries as observed via a comprehensive order flow dataset.
The NMS has changed in many ways during our study period, which spans Q3 2009 through Q2 2017. Aside from the commonly observed fluctuations in asset price and volatility [9–12], we show that quote dislocations [7, 8] display nonstationary behavior that correlate with several market-wide volatility measures. Further up the hierarchy of market complexity, the topology of NMS infrastructure has changed. At the beginning of our study period, NMS infrastructure was primarily located in Carteret and Weehawken New Jersey. In 2010, NYSE relocated their exchanges to a data center in Mahwah, and in 2016, IEX began operating an exchange in Weehawken. We show that changes in the topology of the NMS are associated with opportunity costs realized by NMS participants, even though the most rapidly fluctuating trading venues execute a comparatively small portion of all trades.
We briefly outline the structure of the NMS, which is summarized in Figure 1. The NMS is composed of many interacting components, including data centers, exchanges, traders, regulators, and multiple layers of regulations. The data centers that host the computing infrastructure of the NMS are located in north-eastern corner of New Jersey. Carteret houses the NASDAQ family of exchanges, Mahwah the NYSE family, and Secaucus the BATS and DirectEdge families, while Weehawken is home to the Investors Exchange (IEX). Traders that aim to minimize the latency of their connection to the NMS infrastructure often colocate their trading servers in the same data centers as the exchanges they often interact with. These data centers are connected by communication channels built with state-of-the-art technology, including fiber optic cables, laser free space optics, and millimeter wave wireless.
Market participants connected to the NMS can subscribe to a variety of data feeds offered by an exchange (direct feed), a Securities Information Processor (SIP feed), or a data vendor (third party feed). Direct feeds give traders access to top of book information (e.g., quote and trade messages), depth of book information (e.g., add, modify, and cancel messages), and administrative notifications (e.g., trading halt and order imbalance messages). SIPs are a market utility that aggregate information from exchanges, which introduces a small amount of latency relative to direct feeds but provide important market-wide signals. SIPs offer top of book information via a trade and quote (TAQ) feed, the National Best Bid and Offer (NBBO; an important market-wide best price signal), and Limit-Up/Limit-Down bands (LULD; a system to dampen excessive volatility). Trade-through protections mandate that trades execute at a price that is at least as good as the NBBO, giving it an importance beyond a simple price signal for traders.
Central to the structure of the NMS is the differentiation between National Securities Exchanges, or “lit” venues, and alternative trading systems (ATS), also known as “dark” venues, which are governed by different regulations and are subject to different reporting requirements [13]. National Securities Exchanges are required to provide quotes to the SIP, resulting in active participation in a market wide price discovery process. ATS are only required to issue quotes in extraordinary circumstances [13] and thus tend to impact price discovery indirectly.
The arrangement of data centers and exchanges has been fairly dynamic since NSEs began operating electronic trading systems. The transition to electronic trading was sparked by Nasdaq’s acquisition of Inet in 2005 (originally located in Carteret) and was quickly followed by NYSEs acquisition Archipelago in 2006 (originally located in Weehawken). Since that time, the NMS has grown to include 12 NSEs and roughly 40 ATS distributed across four geographic locations. Births, deaths, technical glitches, and unplanned market events caused exchanges to switch on and off a total of 86 times during our study period, excluding normally scheduled outages (e.g. holidays).
We extend the analyses presented in Tivnan et al. [7] and Ring et al. [8] to cover more than 3000 trading symbols over a period of roughly 8 calendar years. We identify correlations between data feed information asymmetries, opportunity costs associated with those asymmetries, and trading venue fluctuations. While some of these correlations may be relatively easy to infer, they lead us to a less intuitive result. Metrics derived from data feed information asymmetries have forward predictive power with respect to several volatility measures. We also discuss long-term trends in each of the studied quantities and implications those trends may have.
2. Related Work
Our work aims to quantify and investigate aspects of market efficiency in the NMS. The prevailing theory is the Efficient-Market Hypothesis (EMH), which was popularized in its current form by [14] and supported by many including Bachelier [15], Mandelbrot [16], and Samuelson [17]. EMH states that asset prices reflect available information, and EMH is usually presented as having three forms. The three forms of EMH are differentiated by the information that each form claims is incorporated into asset prices. The strong form claims that asset prices incorporate all public and private information, the semistrong form only considers publicly available information, and the weak form only considers historical prices. Regardless of which form we inspect, the connection between the EMH, excess profits, and random walks follows from the definition. If the current price of an asset already accounts for a piece of information, then it should not have any bearing on future prices. If that information has no bearing on future prices, trading based on that information should not provide any excess returns.
Though the intuition behind the EMH is widely accepted, objections have arisen regarding its various formalizations. Several works have indicated that perfectly efficient markets are unlikely to exist, due to the cost of obtaining and acting upon relevant information, thus calling into question the realism of the strong form of EMH [18, 19]. The theoretical justification for the connection between EMH and random walk models has been contested, with some works supporting the connection [14, 17] and others refuting it [20, 21]. Likewise, a body of empirical work with conflicting findings exists. Empirical works in support of the EMH often focus on events where markets quickly react to information [22–26] or develop and apply statistical tests of efficiency [27–29]. Conversely, many studies have found evidence that tests for efficiency are intermittently passed at best [30–32] or that price anomalies arise consistently [33, 34].
Sustained controversy over the EMH suggests that market efficiency is not necessarily a static property and has led to the rise of alternative theories, such as the Adaptive Markets Hypothesis (AMH) [35]. With findings covering efficiency dynamics in emerging markets [36], established markets [37–39], and interasset variation [40], recent empirical work has largely supported the AMH. Rather than testing for the existence or absence of market efficiency, tools that quantify aspects of efficiency are needed to better understand the dynamics suggested by the AMH. Ding et al. [41] and Bartlett and McCrary [42] investigated quote dislocations between pairs of data feeds and provided insights into potentially pervasive information asymmetries. Wah [43] analyzed arbitrage opportunities, shining a light on the potential gross profits of certain trading strategies. Ahmed and Satchell [44] developed a bubble detection method that allows for effective interasset comparisons. Godfrey [45] studied a measure of efficiency based on the relative profits of ideal passive and active traders.
3. Methods
We analyzed data from Thesys Technologies [46] that contained every message from each SIP and direct feed in a unified data format. Consisting of ∼40 PB of data, this dataset covered trading activity from ∼2008 to present. Thesys collected this data using hardware colocated in the Carteret datacenter and applied a s-resolution timestamp to each message on receipt to mitigate clock synchronization issues. Thesys served as the data provider for the SEC’s Market Information Data Analytics System (MIDAS) until 2019, when MayStreet acquired some of its assets and assumed its role as the MIDAS data provider.
Using Dislocation Segments (DSs) and Realized Opportunity Costs (ROC) associated with those DSs, as described in Tivnan et al. [7] and Ring et al. [8], we cataloged information asymmetries in the NMS and estimated their impacts. A DS is an information asymmetry between a pair of data feeds that is caused by quote price discrepancies. We follow the previous work in considering prices displayed by the SIP NBBO and a synthetic Direct Best Bid and Offer (DBBO). A DS begins when the price of the National Best Bid (NBB) or Offer (NBO) does not match its counterpart in the DBBO. An active DS ends if the prices converge, the relationship of the price divergence changes (i.e., NBO DBO DBO NBO or vice versa), or the trading day ends. For each DS, we track the starting time, duration, maximum price divergence, and minimum price divergence. DSs are calculated independently for each trading symbol and each side of the market.
Trades that execute during DSs may incur opportunity costs, since routing and timing decisions may differ based on specifics of the available price signals. We quantify the Realized Opportunity Cost (ROC) for each trade as the number of shares traded the total price difference between the NBBO and DBBO quotes on the appropriate side of the market. We only consider trades executed at one of the prices displayed by the prevailing NBBO. This provides a conservative estimate of total opportunity costs and focuses on trades that were likely informed by the NBBO.
Our analysis covers trading activity from June 1, 2009, to May 31, 2017, and all trading symbols that were included in the Russell 3000 at any point during that period. By covering both an extensive time period and a broad population of trading symbols, we aim to capture both longitudinal and cross-sectional relationships. We focus on the relationship between DSs and volatility of security prices as well as that between fluctuations in the number of exchanges and ROC. We primarily investigate direct ROC, opportunity costs that occur when a trade executes at a price displayed by the NBBO and that price was better for the active order than the corresponding price displayed by the DBBO. These are the costs that may be experienced by a participant that is an exclusive subscriber to direct feeds and does not consider SIP information when placing trades, or one who only considers lit liquidity.
Our methods build upon and extend those used by Ding et al. [41], Bartlett and McCrary [42], Tivnan et al. [7], Ring et al. [8], and to a lesser extent Wah [43]. Specifically, Ding et al. [41] studied similar quote divergences and used data collected from two distinct data feeds (i.e., SIP and direct), and that data was collected via a single, static observer. We advocate for this approach over that used by Bartlett and McCrary [42], which relied solely on SIP data, since this approach avoids potential issues with timestamp synchronization. Though our dislocation methodology and data quality are very similar to previous studies [7, 8, 41], the scope of our dataset is several orders of magnitude larger. Ding et al. [41] study dislocations in 24 securities across five exchanges and a single SIP for 16 days of trading. Tivnan et al. [7] expand this scope to consider dislocations and ROC in 30 securities across 13 exchanges and two SIPs for a year of trading. Ring et al. [8] go further, covering dislocations and ROC in more than 2900 securities for the same year. In this study, our analysis includes dislocations and ROC in more than 3000 securities across 14 exchanges and two SIPs spanning eight years of trading. Beyond extending the scope of these previous studies, we also apply new analytical tools, such as spectral methods and Granger causality, which we detail in the following section.
4. Results
Changes in NMS infrastructure had a material effect on observed direct ROC, which can be seen in Figure 2. The number of active exchanges ranged between a minimum of and a maximum of during the study period, with a total of fluctuations. The time series of venue fluctuations features long periods of stationarity interleaved with shorter periods of bursty activity.
We model this burstiness using an exponential Hawkes process of the form , where . When fit to the time series of venue fluctuations, this model has a branching ratio of , suggesting that if the underlying dynamics are accurately characterized by the model, 86% of venue fluctuations are endogenous and may occur as a result of another venue fluctuation. This endogeneity of bursty behavior, coupled with a pink noise power spectral density , suggests that the dynamics of trading venue fluctuations may be driven by self-organized criticality [47, 48]. There is an inverse correlation between a venue’s total percent of within-study active days and the number of times that venue fluctuated on and off (). This relationship is only partially explained by the birth and death of venues. A plurality of fluctuations are due to thinly traded venues switching on and off many times over relatively short periods without officially ceasing operations. ROC exhibits quasi-stationary spectral behavior throughout the study period, as seen in Figure 3, with power spectrum well-fit by . However, the level values of ROC are negatively correlated with the number of active trading venues (). This indicates that the spectral characterization of ROC as simply colored noise does not capture the full extent of its dynamics. Granger-causality analyses conducted in both potential causality directions (venues ROC and ROC venues) highlight significant and persistent causality from venues to ROC (venues ROC). We show the relationship between the time lag and the significance of the test results in the upper left panel of Figure 2. By displaying the test significance for a range of we avoid arbitrary thresholds and unintended multiple comparisons. Additionally, Figure 4 shows that ROC occurring across disjoint groups of assets has nontrivial auto- and cross-correlation at long time lags, providing evidence of long memory in the generating process for ROC in addition to venue fluctuations.
(a)
(b)
Beyond their relationship with trading venue fluctuations, we are also interested in the longitudinal dynamics of dislocations, which can occur any time two or more trading venues and two information feeds are present [7]. In particular, we wish to understand how the prevalence of dislocations may have changed over time, along with how the duration and magnitude of dislocations may have developed over our period of study. Time series and time-decoupled statistics of these statistics are displayed in Figure 5. For each trading symbol under study and each trading day , we compute the total count and average duration of dislocations. From these, we calculate two indicators of expected dislocation intensity: and . Both indicators display quasi-stationary behavior over the study period (, ). However, displays seasonal behavior over the range of time studied, exhibiting large peaks on trading days adjacent to major U.S. federal holidays. Rank-intensity diagrams for and , sorted by both date (modulo year) and trading symbol, are displayed in Figure 6.
(a)
(b)
is correlated with volatility statistics and has the ability to predict future volatility. We obtained three different volatility measures—the VIX volatility index, and 30-day midpoint and forward volatility of the Russell 3000 index (trading symbol: RUA). Midpoint volatility over time periods is defined as the standard deviation of log returns for , while forward volatility is the same measure computed for . For each volatility measure, we computed Pearson correlations between it and , finding significant correlation for all three combinations (, , ). Since forward volatility is noncausal, correlation significantly greater than zero implies that has some ability to predict forward volatility. Marginal distributions of volatility measures and our indicators are both well-fit by the lognormal family of distributions. Thus, we apply the SABR model [49], which assumes a time-dependent lognormal distribution whose volatility parameter varies following a geometric Brownian motion with zero-drift. The bottom three panels of Figure 5 highlight the correlation between and volatility. This, coupled with lognormal marginal distributions, suggests that the pairs have similar generative mechanisms.
Quantile-based and distributional analysis of dislocation duration, shown in Figure 7, reveals complexity in the evolutionary dynamics of the NMS beyond what is seen in the time series of mean dislocation duration. The time series of median dislocation duration shows relatively smooth decay toward lower durations, punctuated by sharp drops that are likely due to upgrades in SIP technology [50, 51]. Distributions of dislocation duration quantiles have gradually drifted lower by nearly an order of magnitude. In contrast, distributions of mean dislocation duration have remained almost constant—expected due to their spectral characterization above—while distributions of dislocation duration standard deviations have generally increased over the study period. These observations indicate that the “typical” dislocation is becoming shorter, while the incidence of rare, long dislocations is growing more prevalent.
Emergent properties, such as realized opportunity cost and volatility, are driven by microlevel interactions among agents. These actions generate order flow, such as quotes and trades, which can then be used to predict macrolevel statistics. We have mainly focused on two micro-to-macro pathways, venue fluctuation to ROC and dislocation statistics to volatility. However, both dislocation and venue fluctuation statistics have predictive power with respect to ROC and volatility. We fit ordinary least squares models of the formwhere is one of ROC, VIX, mid-volatility, or forward volatility. All variables were normalized as . The design variables differing trades and differing traded value are control variables that measure, respectively, the number of trades that occurred during dislocations and the total value (share price number of shares) exchanged by differing trades. Results of these regressions are summarized in Tables 1–4. The fraction of explained variance is high for both ROC () and VIX () and moderate for both mid () and forward () volatility. As detailed above, a greater number of trading venues is associated with lower ROC and volatility statistics in all models (, for all ), while a greater number of dislocations is associated with higher ROC and volatility statistics in all models (, for all ). Average dislocation duration does have a significant () positive effect on ROC () and VIX () but not on mid (, ) and forward (, ) volatility. These regression analyses are summarized by Figures 8 and 9.
(a)
(b)
5. Discussion
Our results indicate coherent nonstationarity in the U.S. National Market system. The topology of the system exhibits long periods of static behavior interspersed with short windows of rapid fluctuation in the number of active trading venues. This fluctuation, though infrequent, has a pronounced effect on dislocations and realized opportunity costs. Namely, realized opportunity costs decrease as the number of venues increases, while an increase in the number of venues Granger-causes future realized opportunity cost.
In addition to topological considerations, we examine the properties of dislocation segments longitudinally. Quantile- and moment-based analyses of distributions of dislocation segment durations uncover complimentary narratives on the evolution of the NMS. Distributions of 25th, 50th, and 75th percentiles of dislocation length show a near-monotone decrease toward lower values, indicating that market efficiency has generally improved over time. On the other hand, distributions of mean and standard deviation of dislocation duration remain nearly constant over the study period. In conjunction with decreasing quantile statistics, this implies an increase in the probability of large dislocation events.
We investigate indicators composed of aggregate dislocation statistics, which are positively correlated with multiple measures of current and future volatility. These indicators also highlight seasonal effects, including well documented irregularities that occur near U.S. holidays [52–55]. Linking the venue, ROC and dislocation, volatility analyses, we find that venue fluctuation and dislocation statistics predict both ROC and price volatility of the Russell 3000 in least-squares regression models.
Though our work extends an existing line of inquiry to an unprecedented scale, it is not without limitations. Dislocation Segments, the primary unit under study, capture a specific form of information asymmetry and thus only capture a portion of the greater notion of market efficiency. Realized Opportunity Costs, which are derived from DSs and estimate their economic impact, might not provide an exhaustive estimate of economic impact. A more precise measure would likely account for available liquidity during each DS and a probabilistic estimate of whether a trader could access that liquidity. Additionally, neither DSs nor ROC are numerically well bounded, which may complicate comparisons between data from different assets or time periods. Finally, though the scope of our data allows us to investigate an impressive amount of cross-sectional variation and longitudinal dynamics, the level of aggregation needed to summarize and display the results may mask additional structures of interest.
6. Conclusion
We summarize the three salient aspects of our contributions to measure nonstationarity in the US National Market System (NMS). First, we identify predictive relationships between various market measures, to include dislocations predicting future volatility. Second, our findings provide evidence of increasing market efficiency via decreases in dislocation durations, coupled with increased tail risk of anomalous dislocations. Third, we uncover multiple micro-to-macro pathways, to include those exhibiting evidence of self-organized criticality.
Our methods, and their limitations, unearth several fruitful avenues for future work. There are opportunities to improve or extend the calculation and application of Dislocation Segments and Realized Opportunity Costs, as noted in our discussion of limitations. In contrast to our analysis, which investigated coarse-grained and large-scale dynamics, it may be informative to investigate the behavior of Dislocation Segments and Realized Opportunity Costs in close proximity to market anomalies such as the Flash Crash or the more recent activity in Game Stop. While we noted several alternative methods for quantifying market (in) efficiencies in our discussion of related work, this list of alternatives is not exhaustive. Therefore, a comparative study of various quantitative (in) efficiency measures would give researchers and policy makers a deeper understanding of their relative merits. Additionally, existing quantitative (in)efficiency measures each have limitations that present opportunities for new measures that fill different niches or to subsume existing measures.
While our findings highlight nonstationarity in the NMS throughout our period of study, the NMS continues to evolve. Notably, three new National Securities Exchanges became active in 2020 (Members Exchange (MEMX), Long-Term Stock Exchange (LTSE), and MIAX Pearl Equities (MIAX)). To better understand the dynamics of the current NMS, and how coevolutionary adaptations may impact those dynamics, we advocate for additional investigations of micro-behavior and micro-to-macro mechanisms. Principled models that consider market microstructure theory [56–58] and account for agent heterogeneity seem promising. The quantities observed in this study, dislocations and realized opportunity costs, along with their noted relationships with various econometrics, can serve as empirical targets for the next generation of market models.
Data Availability
Our data was originally provided by Thesys Technologies (https://www.thesystech.com/). Some components of Thesys have since been acquired by MayStreet (https://maystreet.com/). These data are publicly available as a commercial product.
Disclosure
All opinions and remaining errors are the sole responsibility of the authors and do not reflect the opinions nor perspectives of their affiliated institutions nor those of any funding agencies.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this article.
Acknowledgments
The authors are grateful for discussions with Anshul Anand, James Bagrow, David Bringle, Eric Budish, Carl Burke, Peter Carrigan, Bill Gibson, Matthew Koehler, Blake LeBaron, Matthew McMahon, Mark Phillips, Mark Rosenthal, Wade Shen, David Slater, Jonathan Smith, and Jason Veneman. C.V.O., J.H.R., D.R.D., and B.F.T. were supported by the Defense Advanced Research Projects Agency (DARPA). C.M.D. was supported in part by a gift from the Massachusetts Mutual Life Insurance Company.