Skip to main content

Do you see the Matrix? Derivation of Linear chi^2 minimization

This blog post is primarily for my Ay117 students. However, if you've ever wondered where chi-squared minimization comes from, here's my derivation.

Figure 1: Either a scene from The Matrix or the hallway in your astronomy building.
Yesterday in class we reviewed the concept of "chi-squared minimization," starting with Bayes' Theorem
$P(\{a\} | {d}) \propto P(\{d\} | {a}) P(\{a\})$
In other words, if we wish to assess the probability of a hypothesis that is expressed in terms of the parameters $\{a\}$ conditioned on our data $\{d\}$, we first calculate how likely we were to get our data under the hypothesis (first term on the right), and multiply this "likelihood" by our prior notion that a given set of parameters is representative of the truth.

Supposing that we have data that are independent from one another, and normally distributed, then our likelihood term can be written
$P(\{d\} | {a}) = \prod_i \frac{1}{\sqrt{2\pi \sigma^2}} \exp{\left[\frac{1}{2}\left( \frac{y_i - f(x_i)}{\sigma_i}\right)^2\right]}$
As for the priors, we'll make the fast and loose assumption that they are constant ($P(a_0) = P(a_1) = ... = {\rm const}$). It is computationally advantageous to compute the log-likelihood
$l = \ln{P(\{d\} | {a})} = C - \frac{1}{2} \sum_{i=0}^{N-1} \left[ \frac{y_i - f(x_i)}{\sigma_i}\right]^2 = C - \frac{1}{2} \chi^2$
Since our goal is to find the parameters that maximize the likelihood, this is equivalent to maximizing the log-likelihood, which is in turn equivalent to minimizing that $\chi^2$ thingy.

For the specific problem of fitting an Mth-degree polynomial, $f(x_i) = \sum_{j=0}^{M-1}a_j x_i^j$, and this results in a linear system of equations that can be solved for the best-fitting parameters.

In class, I got my notation all scrambled, and I neglected the measurement uncertainties $\sigma_i$. My bad! Here's what should have appeared on the board (worked out this morning over breakfast, so be sure to check my work!).


To be clear, the "weights" are $w_i = 1/\sigma_i^2$. Zooming in on the key part:


The first problem of the next Class Activity will be to write a function that takes abscissa and ordinate values, and the associated uncertainties, and computes the best-fitting coefficients for a polynomial of arbitrary dimension $M$. 

See also David Hogg's excellent line-fitting tutorial.

Comments

Popular posts from this blog

On the Height of J.J. Barea

Dallas Mavericks point guard J.J. Barea standing between two very tall people (from: Picassa user photoasisphoto).

Congrats to the Dallas Mavericks, who beat the Miami Heat tonight in game six to win the NBA championship.

Okay, with that out of the way, just how tall is the busy-footed Maverick point guard J.J. Barea? He's listed as 6-foot on NBA.com, but no one, not even the sports casters, believes that he can possibly be that tall. He looks like a super-fast Hobbit out there. But could that just be relative scaling, with him standing next to a bunch of extremely tall people? People on Yahoo! Answers think so---I know because I've been Google searching "J.J. Barea Height" for the past 15 minutes.

So I decided to find a photo and settle the issue once and for all.

I started by downloading a stock photo of J.J. from NBA.com, which I then loaded into OpenOffice Draw:


I then used the basketball as my metric. Wikipedia states that an NBA basketball is 29.5 inches in circumfe…

Finding Blissful Clarity by Tuning Out

It's been a minute since I've posted here. My last post was back in April, so it has actually been something like 193,000 minutes, but I like how the kids say "it's been a minute," so I'll stick with that.
As I've said before, I use this space to work out the truths in my life. Writing is a valuable way of taking the non-linear jumble of thoughts in my head and linearizing them by putting them down on the page. In short, writing helps me figure things out. However, logical thinking is not the only way of knowing the world. Another way is to recognize, listen to, and trust one's emotions. Yes, emotions are important for figuring things out.
Back in April, when I last posted here, my emotions were largely characterized by fear, sadness, anger, frustration, confusion and despair. I say largely, because this is what I was feeling on large scales; the world outside of my immediate influence. On smaller scales, where my wife, children and friends reside, I…

The Force is strong with this one...

Last night we were reviewing multiplication tables with Owen. The family fired off doublets of numbers and Owen confidently multiplied away. In the middle of the review Owen stopped and said, "I noticed something. 2 times 2 is 4. If you subtract 1 it's 3. That's equal to taking 2 and adding 1, and then taking 2 and subtracting 1, and multiplying. So 1 times 3 is 2 times 2 minus 1."

I have to admit, that I didn't quite get it at first. I asked him to repeat with another number and he did with six: "6 times 6 is 36. 36 minus 1 is 35. That's the same as 6-1 times 6+1, which is 35."

Ummmmm....wait. Huh? Lemme see...oh. OH! WOW! Owen figured out

\begin{equation}
x^2 - 1 = (x - 1) (x +1)
\end{equation}

So $6 \times 8 = 7 \times 7 - 1 = (7-1) (7+1) = 48$. That's actually pretty handy!

You can see it in the image above. Look at the elements perpendicular to the diagonal. There's 48 bracketing 49, 35 bracketing 36, etc... After a bit more thought we…