▶ Full Post Text
As of this year, the dataset from the world’s most prolific equity benchmark (officially launched in 1957) [goes back a full century to 1926](https://www.sensiblefinancial.com/how-has-the-sp-500-performed-over-the-last-98-years/). I thought now might be a good time to revisit its merit since one of the most tediously repetitive questions across all investing subs is whether the S&P 500 alone (eg VOO) is a sufficient investment, or perhaps if a *total* US market fund like VTI would be better. The S&P 500 is considered the standard to which the industry is held; it has been recommended for average investors by the great Warren Buffett, and many say small caps are mostly “junk” anyway, so why would you invest in the total market? This question is so common that r/Bogleheads had a pinned post to address it, and it even has its own satirical subreddit: [r/VOOorVTI](https://www.reddit.com/r/VOOorVTI/). And what makes the question particularly tiresome is the fact that whether you choose an S&P 500 fund or a total US market fund will likey make no meaningful difference in your long-term returns (or in your life, for that matter). The question is barely worth the brain cells used to dwell on it, yet some people describe themselves as struggling to choose. But do those investors grappling with the question understand the history and merits of each index? Join me on a deep dive to review them (so I can just link to this post and never write out another explanation again).
**In this post**
* In 1926, Poor’s Publishing and the bond rating company Standard Statistics created a 90-stock composite index in an attempt to track and report the daily returns of the US stock market [more broadly than the Dow Jones Industrials Average](https://www.spglobal.com/spdji/en/research-insights/index-literacy/the-sp-500-and-the-dow/) which only included 12 stocks at the time
* This index would gradually be expanded to 500 stocks and become the Standard & Poor’s 500 Index, launched in 1957, with returns tracked up to the minute
* The explicit purpose of these indexes was to gauge the returns of the total US stock market, but by using only a representative sample of stocks to make it easier to calculate back when this work was done by hand
* The S&P 500 index quickly became the preferred benchmark for US equity mutual funds
* But we have had indexes of the total US market (all the stocks, not just the arbitrary number 500) for more than 50 years now
* And we have had index *funds* tracking these total US market indexes for more than 30 years now
* If you seek passive exposure to the total US stock market, it would be more effective to use a modern total stock market index fund, not a primitive sampling index methodology from a century ago just because it’s more familiar or popular
* Regardless of which representative US stock market index fund you choose, the returns are going to be so similar that it doesn’t warrant deep deliberation. This is probably the least important decision you could make as an investor. But if you really want to scrutinize it, read on…
**Early History of Dow Jones and Standard & Poor’s**
Public stock markets have been around for nearly four centuries, but it wasn’t until the dawn of the 20th century that there was any reliable data source to know the overall performance and direction of the stock market as a whole. Wall Street Journal co-founder Charles Dow was among the first to attempt to index market returns with a daily Transportation Average in 1884, not coincidentally during the heyday of the railroad bubble that would burst a decade later (note that a preoccupation with indexing the returns of the day’s hot themes and innovative growth sectors is a trend as old as stock investing). Later in 1896, Dow and his statistician partner Edward Jones (not [THAT Edward Jones](https://www.reddit.com/r/Bogleheads/comments/swiy7n/what_grade_would_you_give_to_edward_jones_as_a/)) began publishing a daily Industrials Average of 12 representative companies across major sectors. This index was primitively weighted by the share price of each stock such that the rather arbitrary datapoint of stock share price determined the weighting of each company. The “Dow Jones Industrial Average" was expanded to 30 companies in 1928 (hey, another growth bubble period) and is still used as a market indicator to this day, despite it being such an obsolete methodology (for all I know it may have originally been tabulated by the light of flame).
Meanwhile, as stock investing reaches feverish excitement in the Roaring Twenties, demand for market data became stronger and, consequently, a more profitable enterprise. In 1923, Poor’s Publishing - known for publishing a railroad investing guide in the 19th century - joined forces with the bond rating company Standard Statistics to publish a weekly market indicator index of 233 stocks in 26 groups, using a base-weighted aggregate technique tabulating growth from the base year. In 1926, they created a separate 90-stock composite price index (50 industrials, 20 railroads, and 20 utilities), while the 233-stock base-weighted index was re-based to 1926 prices. In 1928, the more manageable 90-stock index was then calculated and published daily, and, eventually, hourly. This is why many academic backtests only go back to 1926 or 1928 - it is the oldest point at which we have daily or weekly index data that meets rigorous academic standards (although monthly US stock return data was later compiled back to 1871). In 1941, the two companies merge to form Standard & Poor’s, the index of 233 companies is expanded to 416, and both indexes are based to 1939.
**The S&P 500 is born**
By 1957, the 416 company index had grown to 500 “leading” companies (425 industrials, 60 utilities, and 15 railroads) listed on the New York Stock Exchange, comprising about 90% of that exchange by weight. Thanks to the use of early calculating computers (ie calculators), this new larger index could now be tabulated every minute and linked to the 90 Stock Composite to provide a daily record of index returns back to 1928. As was intended, this 80-90% sample of the total market by weight proved to be extremely close to tracking the relative returns of the US market as a whole, without having to calculate the returns of the thousands of smaller stocks which would have been too cumbersome (mind you, it was still being recorded by hand). Based on its success, this 80-90% market sample methodology would continue to be used by S&P to this day in markets or sectors where small stocks may be insignificant or illiquid and thus not desirable for trading operations.
The “S&P 500” proved to be a vast improvement on the DJIA, not just because it was a more diversified sample with far more companies, but also because it used market capitalization weighting instead of share price weighting. While both indexes were commonly cited in the media to gauge the direction of the market (as they still are today), the S&P 500 became THE equity market benchmark for measuring asset manager performance, especially in the growing mutual fund industry. Retail investors may pay more attention to the Dow, but the S&P 500 earned itself an undeniable reputation in the finance sector for being the benchmark of choice. As a result of its popularity, familiarity, accessibility, benchmark status, and manageable sample size, the S&P 500 was the natural choice for the first major public index fund, Jack Bogle’s flagship Vanguard 500 Fund VFINX (although at first the fund could only afford to own 280 of the stocks). State Street also used the S&P 500 index for the launch of the first major ETF index fund (SPY) in 1993.
The S&P 500 index composition is managed by a committee at Standard & Poor's (the US Index Committee). In 1988, the S&P 500 would remove the limit on the number of companies by industry, allowing sector weights to float and for substitutions between categories. Per S&P: “the selection process for the S&P 500 is governed by quantitative criteria—including financial viability, public float, adequate liquidity, and company type—that determine whether a security is eligible for inclusion. The committee’s role is to choose among those eligible stocks, taking sector representation into account. Among the key requirements are that a company has a sizeable enough market capitalization to qualify as a large-cap stock. It also must have sufficient float, or percentage of shares available for public trading.” Many people consider the S&P 500 to be “passsive” but as you can see that is [something of a myth](https://www.optimizedportfolio.com/passive-is-a-myth/).
**Growth of indexes for benchmarking**
The 1950’s and 1960’s were a heady time for academic study of markets and developing investing theories which laid the groundwork for the Boglehead investing philosophy. In a period of less than 20 years, you have the emergence of Modern Portfolio Theory (Harry Markowitz, 1952), the Capital Asset Pricing Model (Bill Sharpe et al, 1961-66) and the Efficient Market Hypothesis (Eugene Fama et al, 1964-70). It’s also an exciting time for the development of indexing. The Center for Research in Securities Pricing (CRSP) is founded at the University of Chicago in 1960 to tabulate monthly returns for common stocks by size decile back to 1926 for academic research. In 1968, Capital International begins publishing international developed market indexes which are later licensed by, and co-branded with, Morgan Stanley in 1986 to become the MSCI indexes. MSCI’s EAFE (Europe, Australia, and Far East) Index gives us solid backtesting for international stock returns to 1968. Like the S&P 500 for US stocks, the MSCI EAFE becomes the preeminent benchmark for international markets. Benchmarks are increasingly important as more investors are piling into mutual funds (including international ones at a time when those markets were significantly outperforming the US market), and since virtually all mutual funds are actively-managed until index funds grow in popularity in the 1990’s, benchmarking is critical for evaluating manager performance.
The NASDAQ Exchange was founded in the US in 1971 as the first fully electronic, computerized stock exchange. This attracted a lot of financial firms looking for opportunities to get an edge with computerized trading, and incidentally, the overwhelming majority of firms to list on the NASDAQ wound up being financial companies. The NASDAQ offered a Composite Index from the start but it was so dominated by financial firms that in 1985 they created a separate index of the non-financial companies listed on the exchange called the NASDAQ 100. It is an accident of history but, based on the timing of its creation and the subsequent popularity of the exchange for listing technology companies launching in the dotcom era of the 1990’s, the NASDAQ 100 became the preeminent benchmark for the US technology and telecommunications sector and got its own ETF (QQQ) in 1999.
As indexes become fully computerized, the Wilshire 5000 is launched by Wilshire Associates in 1974 and is really the first index to offer comprehensive cap-weighted returns for the entire investable US market (it had roughly 4,700 stocks when formed but 5,000 sounded better). In 1984 in London, FTSE launches it’s 100 UK index and the Russell Company launches US indexes: the Russell 3000 (a comprehensive US market fund), Russell 1000 (large caps) and Russell 2000 (small caps) which becomes the preeminent US small caps benchmark. All these indexes with catchy but arbitrary big numbers ending in zeros (there’s also MSCI 300, 450, 750, 1750, and 2,500, as well as S&P 100, 400, and 600) are useful for benchmarking returns, but the S&P 500 remained the only one with an index fund tracking it which you could invest in for more than 15 years starting in 1976. Eventually, indexing would become such big business that competition would beget consolidation: FTSE ended up acquiring Russell to make FTSE Russell indexes while Dow Jones acquired Wilshire indexes and later combined with Standard & Poor’s to become S&P Dow Jones. As of 2017, there are more stock indexes than there are stocks!
**Vanguard launches total US market index**
Back to the 1990’s bull market… this is another major period of advancement in stock market theory and index investing, notably with the publishing of the Fama-French 3-Factor Model in 1992 and the founding of Dimensional Fund Advisors. But this is also the year that Vanguard explodes with the launch of numerous new index funds to complement their flagship 500 Fund from 1976: their “extended market fund” using the Wilshire 4500 index in 1987 which holds all the US stocks NOT in the 500 index, and their Total Bond Market Fund using the US Aggregate Index in 1986 (now owned by Bloomberg and known as “The Agg”). Thanks to newer stock style indexes from S&P and Russell, Vanguard puts out growth funds, value funds, large and small cap funds, and an international index fund (VGTSX, at first using the MSCI EAFE). But perhaps most importantly, they launched VTSMX - the Vanguard Total Stock Market Index fund - tracking the Wilshire 5000 index. As Jack Bogle described, **this single fund would now “enable investors to make a commitment to the entire stock market, which I consider as the full fruition of the index fund concept.”**
These low cost index funds pioneered by Vanguard then made it possible for average investors to create portfolios that fully realize the aforementioned theories developed in the 1960’s and 1970’s (ICAPM, Merton 1973), namely that passive, cap-weighted total market exposure offers roughly the optimal return-on-risk as determined by the market, it serves as a reasonable proxy for the “market portfolio” of all investable assets in the world, and can be calibrated to any investor’s goals, risk tolerance, and timeline by adjusting the percentage of fixed income assets. Fandom for Vanguard grows and a group of “Die-Hards” on the Morningstar internet forums would become “The Bogleheads”, popularizing the Three-Fund Portfolio of the total US stock market, total international stock market, and total bond market. Vanguard would become the largest brokerage in the world by AUM in 2023, incidentally the same year that the net assets invested in index funds would eclipse those of actively-managed funds.
Interestingly, Vanguard has fine-tuned their total market funds over the years, switching between similar indexes which may have lower licensing fees, be more effective at tracking, or have a preferable methodology for other reasons. In 2004, Dow Jones took over the Wilshire 5000 index which Vanguard had been using for their Total Stock Market Index fund and, surely not by coincidence (cost?), in 2005 Vanguard switched it to tracking the MSCI Broad Market Index which “only” covers about 99.5% of the total market. But in 2010-12, CRSP began launching and licensing real-time indexes including the US Total Market Index (known academically as the CRSP 1-10, representing all 10 deciles of the US stock market by size). In a flurry of sweeping moves in 2013, Vanguard switched the Total Stock Market Index Fund again, this time over to the CRSP US Total Market Index (which includes more microcaps) where it remains today. The Total International Stock Market Fund was moved from an MSCI index to the FTSE Global All Cap ex US index.
**How low will you go?**
For any total market index fund, the manager must determine the smallest practicable market cap of company they are willing to invest in, informing the size of the investable market they are aiming to achieve exposure to. Even most “total market” index funds won’t capture absolutely ALL of the public companies in a market because when they get small enough, there are simply not enough floating shares available to buy and those stocks become functionally illiquid. The S&P 500 chooses to set its floor at roughly the 500th of the “leading” 500 companies (among the very largest in the market). There is nothing eimpirally significant about the number 500 stocks except that it is particularly catchy branding for us primates using a base 10 counting system (see: Fortune 500, Indy 500, Fiat 500, 500 Club, etc).
VTI, Vanguard’s total market index fund holds about 3,500 stocks with a goal of holding all the publicly-traded stocks in the US (a number that varies considerably over time) at market cap weight. In practice, VTI probably holds about 99.9x% of them because there may be a new IPO they haven’t added yet, or a microcap company so small and illiquid they can’t reasonably find sellers of the shares they would need to buy to meet the market cap weight. And when you look up the holdings, you will often find VTI owns MORE stocks than the index, perhaps because it may have a few companies that have folded, been acquired, or were otherwise de-listed from the index, and the fund hasn’t officially unloaded those shares yet.
This is meant to be a reminder that, although owning the entire market is the empirical objective of Boglehead-style investing, you can’t ever truly achieve that with 100% daily accuracy. You cannot own an index, you can only own shares of a fund which tracks an index, and there are bound to be some very minor discrepancies at the margins (including cash holdings for fund operations). The CRSP total market index which VTI tracks seeks to own all the stocks in the US market. The S&P 500 index includes only the 500 leading US companies since 1957, and considers that roughly 80% US market ownership by weight to be sufficient to simulate the returns of the total market.
**So what’s the difference in returns?**
The reason I’ve taken this long-winded walk through the history of indexing is mainly to illustrate that there’s nothing so special about the S&P 500 or any index that makes it clearly a superior choice or guaranteed to perform better than any other alternative, and the number 500 is fairly arbitrary. It is simply one relatively old total US stock market benchmark index among several others, past and present, using a sampling methodology and certain inclusion criteria.
In practice, there should be little to no difference in returns between the S&P 500 and the total US market since the S&P 500 was explicitly designed to closely track the total market. And it turns out it has done a remarkably good job at this - over the last 54 years, the S&P 500 average annual returns are [within 0.01-0.02% of the total US market](https://www.portfoliovisualizer.com/backtest-asset-class-allocation?s=y&sl=58xUpq2DT2G0wY3xPATslu), while taking turns outperforming by small margins. That’s not even “noise”, that is a functionally identical result over tens of thousands of trading days. For all the arguments that the S&P 500 is a superior index because of its rigorous inclusion criteria avoiding the “junk” in small cap stocks, that hasn’t mattered. Liekwise, for all the arguments about the importance of including small cap diversification and the higher returns of small caps as a group, excluding them has had no penalty in returns. These are distinctions in methodology with no meaningful difference in outcomes. Not only that, the returns of the S&P 500 are [also nearly identical](https://archbridge.com/insights/head-to-head-dow-vs-sp-500-and-the-shocking-results/) to those of the even more primitive Dow Jones Industrial Average with only 30 hand-picked stocks.
As the [Boglehead investment philosophy](https://www.bogleheads.org/wiki/Bogleheads%C2%AE_investment_philosophy) is based on the pillar of [wide diversification to approximate the total market](https://web.archive.org/web/20170628221046/https://www.norstad.org/finance/total.html), clearly a total market index fund including small caps is, on principle, a better fulfillment of that objective. But if you feel more comfortable using a sampling index of 1,000 or 500 or 250 or even just 30 stocks, that’s fine too. Just pick one, invest early and often, tune out the noise, and stay the course!