Skip to main content

Do you see the Matrix? Derivation of Linear chi^2 minimization

This blog post is primarily for my Ay117 students. However, if you've ever wondered where chi-squared minimization comes from, here's my derivation.

Figure 1: Either a scene from The Matrix or the hallway in your astronomy building.
Yesterday in class we reviewed the concept of "chi-squared minimization," starting with Bayes' Theorem
$P(\{a\} | {d}) \propto P(\{d\} | {a}) P(\{a\})$
In other words, if we wish to assess the probability of a hypothesis that is expressed in terms of the parameters $\{a\}$ conditioned on our data $\{d\}$, we first calculate how likely we were to get our data under the hypothesis (first term on the right), and multiply this "likelihood" by our prior notion that a given set of parameters is representative of the truth.

Supposing that we have data that are independent from one another, and normally distributed, then our likelihood term can be written
$P(\{d\} | {a}) = \prod_i \frac{1}{\sqrt{2\pi \sigma^2}} \exp{\left[\frac{1}{2}\left( \frac{y_i - f(x_i)}{\sigma_i}\right)^2\right]}$
As for the priors, we'll make the fast and loose assumption that they are constant ($P(a_0) = P(a_1) = ... = {\rm const}$). It is computationally advantageous to compute the log-likelihood
$l = \ln{P(\{d\} | {a})} = C - \frac{1}{2} \sum_{i=0}^{N-1} \left[ \frac{y_i - f(x_i)}{\sigma_i}\right]^2 = C - \frac{1}{2} \chi^2$
Since our goal is to find the parameters that maximize the likelihood, this is equivalent to maximizing the log-likelihood, which is in turn equivalent to minimizing that $\chi^2$ thingy.

For the specific problem of fitting an Mth-degree polynomial, $f(x_i) = \sum_{j=0}^{M-1}a_j x_i^j$, and this results in a linear system of equations that can be solved for the best-fitting parameters.

In class, I got my notation all scrambled, and I neglected the measurement uncertainties $\sigma_i$. My bad! Here's what should have appeared on the board (worked out this morning over breakfast, so be sure to check my work!).


To be clear, the "weights" are $w_i = 1/\sigma_i^2$. Zooming in on the key part:


The first problem of the next Class Activity will be to write a function that takes abscissa and ordinate values, and the associated uncertainties, and computes the best-fitting coefficients for a polynomial of arbitrary dimension $M$. 

See also David Hogg's excellent line-fitting tutorial.

Comments

Popular posts from this blog

An annual note to all the (NSF) haters

It's that time of year again: students have recently been notified about whether they received the prestigious NSF Graduate Student Research Fellowship. Known in the STEM community as "The NSF," the fellowship provides a student with three years of graduate school tuition and stipend, with the latter typically 5-10% above the standard institutional support for first- and second-year students. It's a sweet deal, and a real accellerant for young students to get their research career humming along smoothly because they don't need to restrict themselves to only advisors who have funding: the students fund themselves!
This is also the time of year that many a white dude executes what I call the "academic soccer flop." It looks kinda like this:


It typically sounds like this: "Congrats! Of course it's easier for you to win the NSF because you're, you know, the right demographic." Or worse: "She only won because she's Hispanic."…

Culture: Made Fresh Daily

There are two inspirations for this essay worth noting. The first is an impromptu talk I gave to the board of trustees at Thatcher School while I was visiting in October as an Anacapa Fellow. Spending time on this remarkable campus interacting with the students, faculty and staff helped solidify my notions about how culture can be intentionally created. The second source is Beam Times and Lifetimes by Sharon Tarweek, an in-depth exploration of the culture of particle physics told by an anthropologist embedded at SLAC for two decades. It's a fascinating look at the strange practices and norms that scientists take for granted.
One of the stories that scientists tell themselves, whether implicitly or explicitly, is that science exists outside of and independent of society. A corollary of this notion is that if a scientific subfield has a culture, e.g. the culture of astronomy vs. the culture of chemistry, that culture is essential rather than constructed. That is to say, scientific c…

The Bright Line is not Monotonic

The anthology of myths commonly known as America rests upon the notion that history is linear. In the past people in this country ignorantly did bad things to other people. But thanks to the passage of time, we can now "let the past to be the past," because today we live in a time when things have gotten much better. Furthermore, any problem that our society faces in the present will inevitably be solved as "the old guard" dies off and a new generation of better people takes their place. 
Of course this story isn't told so simply or explicitly. But the assumption lurks beneath the other stories we, as Americans, tell ourselves and each other. The myth certainly undergirds the notion that racism is a thing of the past, and that today we inhabit a "post-racial" world in which all people, regardless of race have equal access to betterment, dignity and happiness. We are lulled into beliving that at some point in the mid to late 1960's, a wise reveren…