One Day While Changing a Diaper...

Do you spend too much time worrying about money? Are you anxious about investing for your retirement, your kids' education or just a rainy day?

As a self-employed writer - married to another self-employed writer - I got to thinking about how to invest our savings a few years ago. It was the middle of the dot-com bust-up, and our mutual funds had lost much of their value. Yes, our advisor never called us either. I learned about technical analysis - the study of market charts - and started writing about investing for the Montreal Gazette, the largest English-language newspaper in the province of Quebec.

One day, I heard about something called the Commitments of Traders Reports. Some analysts swore by these little-known reports, saying they could help determine future market direction.

SMART MONEY AND DUMB

The COT reports give data on trillions of dollars of futures and options holdings in over 90 markets - everything from crude oil to the U.S. dollar, gold and the NASDAQ. The reports are issued every week by the U.S. Commodity Futures Trading Commission. They can be downloaded for free from the CFTC's website each Friday at 3:30 p.m. (Eastern Standard Time).

But the data in its raw form is hard to make use of - or even understand. Each weekly report gives the total position held in each market by three groups of traders.

First are the commodity producers. Analysts call them the "smart money." Also known as commercial hedgers or just "the commercials," they are seen as the folks with the best market information. When they are increasing their positions in crude oil, for example, it's probably a good time to get some energy stocks.

Then there is the dumb money - the "non-commercial" category. These are the large speculative investment funds, usually called the "large speculators" or simply the "large specs" - or just the "dumb money." They are trend-followers. Analysts say they are usually wrong at market extremes. If they are buying oil, it's probably a good time to sell your energy stocks.

The third group is the "non-reportable" category. These guys are seen as the really dumb money. They are the small-time traders who apparently don't know what they're doing at all and don't provide any meaningful market information. Or so say most analysts.

WHAT TO DO?

To make any sense of all this data, analysts say, you have to compare each group's position in a market to what it held last week. So that's what I started to do.

I wrote down the weekly positions for all three groups for a couple of dozen markets - the S&P 500, the NASDAQ, crude oil, gold, silver, the U.S. dollar, the Yen, the 10-year Treasury and others.

But then I was stumped. What should I do with all this information? The price of gold, for example, rarely went up just because the commercials had bought more gold or because the large specs had sold it. And sometimes, the commercials and large specs were both buying the NASDAQ at the same time. What would I do then?

The data looked interesting, but there didn't seem to be any clear way to use it.

LOOKING FOR TRENDS

After doing this for a while, I started to download the data into Microsoft Excel and make charts out of it to see if I could spot trends. I still couldn't get much use out of it. I tried to find correlations between the COT data and underlying cash prices. I couldn't find much of any use.

(One bizarre discovery I made was that some equity indexes have very high correlations with subsequent COT data many months later. However intriguing, that didn't seem like very useful information for investing.)

To my knowledge - despite many studies by economists and statisticians - no one has found any correlations you can bank money on and made this research public. (If I'm wrong on that, let me know. I'd love to see the study.)

Just then, a couple of fascinating books came out by COT gurus Larry Williams (who wrote the classic book on the subject Trade Stocks and Commodities with the Insiders) and Floyd Upperman (author of Commitments of Traders). Both analysts suggested the COT data was most useful when the commercials or large specs had accumulated historically extreme positions.

In other words, if the commercials have a multi-year high net long NASDAQ position, it's probably a good time to buy. This was a major breakthrough in public understanding of the previously difficult-to-interpret COT data.

I put my own twist on this idea and started to think about ways of testing what happened when traders hit certain extreme positions. I would sometimes watch the commercial hedgers or large specs accumulate record positions for weeks and even months, without any change in the underlying markets. Relying solely on such "signals" would probably lose lots of money.

The same problem exists with many technical analysis tools. A market can be overbought or oversold for weeks without a change in trend. You can be right, but if your timing is wrong, well, you won't be trading for long.

As a result, some COT analysts caution against relying solely on the data to time trades. They suggest the data is most useful as a guidepost for possible future market turns, but it must be combined with technical analysis for a signal.

Here was another problem that occurred to me: Let's say the commercial traders are indeed the smart money. When do you buy? When the commercials start accumulating (i.e. when they start getting less net short or more net long)? Or only when their bullishness has peaked?

The dilemma is this: If it's the former case, then aren't we often buying near a market top, right after the commercials have had on their most bearish position? But if it's latter, then aren't we really trading opposite to the commercials?

A REVELATION

While I was on paternity leave with our second baby, I had one of those revelations that sometimes comes when changing a poopy diaper or sitting in a rocking chair well past midnight with a sleeping baby on your shoulder.

Why not use technical analysis to study the COT data? There must be a way, I thought, to examine rigorously how extreme COT positions impact subsequent cash prices. Could the tools of technical analysis help?

I looked again at the S&P 500 data going back to 1995 (when the weekly COT data first came out for free in electronic form). I wanted to see what happened to the index after traders acquired an extreme net position. I defined "extreme" as two standard deviations or more from the 27-week moving average.

The results were pretty exciting. An extreme net position usually led to returns that beat the market. For example, if you bought the S&P 500 index when the commercials were at an extreme net long - or when the large specs or small traders were at an extreme net short - your returns over the next weeks and months were usually better on average than if you had just bought the market at any random moment.

For example, the return for the subsequent week was 0.6 percent, compared to the S&P 500's average one-week return of 0.2 since 1995. For the subsequent three weeks, the average return was 2.5 percent, compared to 0.5 percent for the S&P 500. Over the subsequent 10 weeks, the average return was 6.6 percent, compared to 1.7 percent for the S&P 500 since 1995.

I kept checking further and further after the signal to see what the optimal holding period would be. I found that the superior returns continued for up to 40 weeks after the extreme position was first registered - as far out as I measured.

And there were superior returns in most time frames for both the large specs - and even the usually-ignored small traders.

What it meant was the data could be the basis for a trading system after all.

TIMING SYSTEM

There were still a bunch of questions to resolve. Which group of traders should I follow? What if they give contradictory signals? And I still needed to figure out the best way to measure an extreme position in the first place, since I had picked the two standard deviations and 27-week moving average arbitrarily. Were better results possible with another definition of "extreme"?

Using Excel, I started to test returns with various combinations of moving averages and standard deviations. I was stunned at the results. With many of the combinations, a simple switching system of buying the S&P 500 when the commercials were at an extreme net long position and selling when they were at an extreme net short would have given market-beating profits. The returns were in some cases more than double those of buying and holding the index.

The results were just as interesting fading - or trading opposite to - the large specs. But the biggest surprise was that the best results came from fading the small traders - the guys everyone was ignoring.

I found a way to automate the testing with help from a clever journalist colleague, Mike Gordon, who is an expert in computer-assisted reporting.

MARKET-BEATING

In every index or commodity I analyzed, I found that a market-beating trading system could be found. I discovered that it was possible to increase profitability, in some cases, by delaying the trade for one or more weeks after a signal was given.

Also, I found the most profitable results came from using not the net number of contracts of each group of traders, but rather their net-percentage-of-open-interest position. This makes sense, I believe, because of the explosion of futures and options trading in recent years. The net-percentage-of-open-interest numbers have probably preserved their internal consistency better during this time.

The best news for me is that the system requires only one or two trades a week at most - and often none at all. It can certainly be exhilarating to swing or day trade. It feels like you can actually behold the entire world - all its people, their anxieties, their hopes, the events that shape their lives - in motion at once, like a raging and crashing ocean. It can be so preoccupying, it may turn into a struggle to focus on anything else.

But with my COTs Timer system, I need no longer fret about daily market ups and downs. All I have to do is spend just a few minutes each week downloading the latest COT data and executing any trades, then sit back and relax. Hopefully, by a beach somewhere with the bambinos!

AR: Note that since writing this in mid-2008, I've added extensive new steps to my backtesting system. A detailed description of my backtesting is available on my FAQs page.