The Weights You Can Hold

Lesson 72 priced a book in which every position was the same size and left this page the question of how much goes in each. Here is the answer the standard machinery gives, on lesson 71’s four rules over the same twenty-nine moves: 0.746, 0.078, minus 0.672 and 0.848. It cuts the standard deviation of the day by 9.72 per cent, which on four positions of two per cent each is 0.71 percentage points of the day. It also asks you to short two thirds of one of your own rules and to carry 2.35 dollars of position for every dollar in the account. Forbid the short and the answer becomes 0.711 and 0.289 with two rules at zero, which keeps 80 per cent of the cut, holds one dollar of position per dollar of account, and beats the unconstrained answer on your own data until you have sixty days of it.

Prerequisites: Lesson 72, for the equally weighted book this page reweights, lesson 71, for the four rules and their six correlations, and lesson 12, for the cost that scales with what you trade.

What the objective actually contains

Minimum variance is one line. Take the covariance matrix of the things you are holding, invert it, multiply by a column of ones, and divide by the sum so the weights add to one. That is the whole method, it has a closed form, and any spreadsheet will do it on four rules in a minute.

w = Σ⁻¹1 / (1′Σ⁻¹1)

Read what is in it and, more importantly, what is not. There are no expected returns, which is deliberate: lesson 67 needed 589 trades to settle whether a tenth of an R was real, so an estimate of what each rule will earn is the least trustworthy number in the building, and an objective that needs it inherits that. There are no costs. There is no constraint. The formula will return a negative weight without comment, and negative means short.

The inputs are the four rules lesson 71 picked, a 2-bar average against a 5, a 3 against a 10, a 5 against a 20 and an 8 against a 30, over the twenty-nine moves on which every rule in lesson 63’s grid is defined. Their six correlations are lesson 71’s, from 0.6254 to 0.9648. Their standard deviations are close to each other: 0.6015, 0.7104, 0.7278 and 0.7082 in the units the rules return.

Four weightings are worth putting side by side. Equal weights are what lesson 72 assumed. Inverse variance is what most people mean by risk parity when they are in a hurry: weight by one over each rule’s variance and ignore the correlations entirely. Then the minimum-variance answer with the short forbidden, and the same answer with nothing forbidden.

Weighting	Weights	Position per dollar	Day’s standard deviation	Cut
Equal	0.25, 0.25, 0.25, 0.25	1.000	0.62719	—
Inverse variance	0.320, 0.230, 0.219, 0.231	1.000	0.61520	1.91%
Minimum variance, no shorting	0.711, 0, 0, 0.289	1.000	0.57813	7.82%
Minimum variance	0.746, 0.078, −0.672, 0.848	2.345	0.56624	9.72%

Start with the size of the argument. The whole distance from the top row to the bottom is 0.06095, against a starting 0.62719. Everything portfolio construction has to say about this book, from the crudest rule to the exact optimum computed with the answers in hand, is worth 9.72 per cent of the day’s standard deviation. At two per cent a position and lesson 71’s average correlation of 0.7695 for these four, equal weights make the day 7.276 per cent and the optimum makes it 6.568 per cent. The whole prize is seven tenths of a percentage point a day. It is worth having and it is not worth a great deal of trouble, and knowing which of those it is before you start is most of what this page is for.

Read the second row next, because it is the one people actually run. Inverse variance collects 1.91 per cent of the available 9.72, which is a fifth. It fails because the four standard deviations are nearly the same, so weighting by them barely moves anything, and the whole of the prize on this book lives in the correlations. A weighting that ignores correlations on a set of rules whose correlations run from 0.63 to 0.96 is solving a problem this book does not have.

Then the two that matter. The unconstrained answer takes another 1.90 percentage points off the day, moving the cut from 7.82 per cent to 9.72, and it buys that with a short leg of 0.672 and a gross position of 2.345 dollars for every dollar of account. The constraint costs a fifth of the prize and removes all of the leverage.

Almost nobody has computed both. The unconstrained number is what every textbook and every library function returns by default, the constrained one is four lines further on, and the difference between them is the difference between a portfolio you can describe and a portfolio you can hold.

Module 9’s card gains a third column, and it is the weight this page gives each rule, written twice: once as the optimiser returns it and once with the short forbidden. Lesson 75 is where the whole card gets spent at once.

What twenty-nine days can pay for

The weights above came out of ten numbers — four variances and six covariances — estimated from twenty-nine observations. That is not obviously too few, and it is not obviously enough, so measure it rather than arguing about it.

Treat the covariance measured above as if it were the truth. Draw T observations from it, fit the optimiser to the draw as though that were all you had, then score the resulting weights against the truth you started from. The gap between the score and the true optimum is the estimation error, and it is the only thing this experiment contains.

Observations	Standard deviation the fitted weights actually deliver	Share of the prize captured	Beats the no-shorting answer
29	0.59971	45.1%	21.7%
60	0.58154	74.9%	50.1%
125	0.57312	88.7%	82.9%
250	0.56969	94.3%	—
1,000	0.56708	98.6%	—

The first row is the honest verdict on the weights this page opened with. Fitted on twenty-nine observations, the unconstrained optimiser delivers 0.59971 rather than the 0.56624 it prints, which is 45.1 per cent of the prize rather than all of it. It still beats equal weights, and comfortably: on this book the optimiser is not a mistake and the folk claim that dividing by N always wins is not what the arithmetic says.

The fourth column is the finding. Fitted on twenty-nine observations, the unconstrained answer beats the constrained one 21.7 per cent of the time. Four times in five, the weights you cannot hold are also the weights that lose. It takes sixty observations to make it a coin flip and 125 before the unconstrained answer wins five times in six, and 125 daily observations is six months of closes.

Now look at the weights themselves rather than at what they deliver. Resample the twenty-nine days with replacement four thousand times, refit, and read the middle ninety of a hundred results:

2-and-5, fitted at 0.746: lands between 0.377 and 1.081.

3-and-10, fitted at 0.078: lands between minus 0.916 and 0.635, and is negative 44.7 per cent of the time.

5-and-20, fitted at minus 0.672: lands between minus 1.504 and minus 0.011.

Gross position, fitted at 2.345: median 2.795, and above three dollars per dollar of account in 31.9 per cent of resamples.

The standard deviation those weights deliver is stable. The weights are not. A number that is 0.078 on your data and negative in nearly half of its own resamples is not a decision about a rule, it is the residue of ten estimates fighting over twenty-nine days, and printing it to three figures is the part that misleads.

Which is where the constraint earns its keep, and it does more than protect you from leverage. Forbidding negative weights is arithmetically close to admitting the covariance is measured badly and pulling it toward something simpler: shrink this book’s covariance halfway toward the single average correlation lesson 72 used and the optimiser returns a gross position of 1.076 and a standard deviation of 0.57831, against the constraint’s 1.000 and 0.57813. Two different admissions of the same ignorance, arriving within two ten-thousandths of each other.

And the constrained answer sets 5-and-20 to zero, which is the rule lesson 71 told you to delete for having the highest correlation to the rest. Two different objectives, one measured on correlations and one on variance, pick the same rule to remove.

So run the optimiser with the short forbidden, and take the 7.82 per cent it can hold over the 9.72 it cannot.

What this does not settle

That minimum variance is the objective. It is the objective that needs the least, not the one you want. Nothing in it knows which rule makes money, so a rule that is quiet and worthless collects weight for being quiet. Every threshold in this course applies to each rule before this page’s weights apply to the set: lesson 64’s bar, lesson 67’s test, lesson 70’s filter condition. Weighting a book of four dead rules to minimum variance produces a very stable way to lose money.

That the 9.72 per cent is free. Gross position goes from one dollar to 2.345, and every cost that scales with what you trade scales with that: lesson 12’s spread on the way in and out, lesson 59’s impact on the way in, lesson 68’s capacity on the size you can hold at all. The objective contains none of them, so it will happily buy seven tenths of a percentage point of daily standard deviation with a doubling of the bill. On the fund in lesson 12 that trade is obviously worth it and on the micro cap it is obviously not, and the formula cannot tell them apart because it was never shown the quotes.

That the simulation measures the whole error. It does not, and it flatters the optimiser by construction. The truth it draws from is the same covariance the twenty-nine days produced, so what the table prices is estimation error and nothing else. It says nothing about the covariance being wrong in the first place, which is the larger problem and the next page’s.

That four rules is the hard case. It is the easy one. Ten parameters from twenty-nine observations is uncomfortable; twenty rules is 210 parameters from the same twenty-nine days, and at twenty-nine rules the measured matrix stops having an inverse at all, so the optimiser does not return a bad answer, it fails to return one. The number of parameters grows as the square of the number of rules and the data does not, which is why every professional treatment of this problem is about shrinking, constraining or factoring the matrix rather than inverting what you measured.

That the weights are a decision about the rules. They are a decision about this book on these days. Add a fifth rule and all four weights move, because the objective has no notion of a rule’s own merit, only of its contribution to a particular set. The card’s third column is therefore the least portable thing on it, which is why it is written twice and why the second writing is the one to act on.

And the concession that costs most: this page treated the covariance as one object measured on one window, and it is not one object. Split the same twenty-nine moves in half and the average correlation between the four rules is 0.5966 in the first fifteen and 0.8800 in the last fourteen, and the weights fitted on the first half are 0.448, 0.273, minus 0.498 and 0.777 rather than 0.746, 0.078, minus 0.672 and 0.848. In the last fourteen it is worse than unstable: three of the four rules hold the same position on every one of those days, their correlations are exactly one, and the covariance matrix has no inverse at all. Lesson 74 starts there, with the half of your own data on which the machinery on this page does not run.

Problems

1. Weight your own book two ways. Take the daily returns of the rules you run, build the covariance matrix in a spreadsheet, and compute the minimum-variance weights with and without a floor of zero. Half an hour, and you end holding two sets of weights and one number, the difference in the day’s standard deviation between them, which is what the short leg is being paid.
2. Price the prize before you chase it. Compute the standard deviation of your book at equal weights and at the unconstrained optimum, and take the ratio. Ten minutes, and you end holding one number, the most that reweighting can ever do for you, which you compare against the 9.72 per cent on this page and against what your costs would rise by at the optimum’s gross position.
3. Resample your own weights. Draw your own days with replacement, refit the weights, and do it two hundred times. An evening, and you end holding one number for each rule, the share of resamples in which its weight is negative, and any rule whose share is near half has not been given a weight by your data.

Sources. Harry Markowitz, “Portfolio Selection” (The Journal of Finance, 1952), for the objective this page computes and for the observation, made at the outset and usually forgotten, that its inputs are estimates. Richard O. Michaud, “The Markowitz Optimization Enigma: Is Optimized Optimal?” (Financial Analysts Journal, 1989), for the argument that the optimiser maximises the errors in its inputs along with everything else, which is the second table read in one sentence. Ravi Jagannathan and Tongshu Ma, “Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps” (The Journal of Finance, 2003), for the result that forbidding short positions is equivalent to shrinking the covariance matrix, which is why the constrained answer and the shrunk one land in the same place here. Victor DeMiguel, Lorenzo Garlappi and Raman Uppal, “Optimal Versus Naive Diversification: How Inefficient Is the 1/N Portfolio Strategy?” (The Review of Financial Studies, 2009), for the estimation windows an optimised portfolio needs before it beats equal weights, which is the third column of the second table done properly and at scale.

Educational only. Trading involves substantial risk of loss. Not financial advice. Past performance does not guarantee future results.