Thursday, January 6, 2011
What's Missing?
You may have noticed that there is no term for the deviation along the direction of the line. The reason is that any error is included in the position along the line. And there are no error estimates for the position of the center of the distribution and the angle of the eigenvector. The error estimate for determining the position of a point is a function of the standard deviations and the number of data points. One would also expect that the error in the direction of the line would be of the order of angle of a right triangle whose height is the standard deviation normal to the line and base is the maximum length of the fitted line from the center as the base. The results would depend on the error distribution and could be checked by numerical computations. By assuming a center and direction for the line and an error model and then comparing the estimates of the fit with the initial assumptions one gets feedback on the method used to fit the data.
Wednesday, January 5, 2011
Vectors, Moments and Forms
Variance is the principal fundamental form in statistics. The model for the first fundamental form that of the length of a vector and is defined in terms of an inner product. A moment is basically the idea behind the balance which involves the weighted sum of distances. The moments used in Physics and Engineering are associated with exterior products. It also seems natural to associate moments with the deviations of statistics.
The condition for a minimum is associated with a nonchanging value of the variance in least squares fits. The numerical computations seem to suggest that there may be more general truths associated with variances and moments. It's something to ponder on.
Supplemental: Consider for example the process of normalization in statistics.
The condition for a minimum is associated with a nonchanging value of the variance in least squares fits. The numerical computations seem to suggest that there may be more general truths associated with variances and moments. It's something to ponder on.
Supplemental: Consider for example the process of normalization in statistics.
Tuesday, January 4, 2011
Derivation for Linear Fit in nD
In nD a line is an intersection of n-1 planes so we can let the total variance be the sum of variances for the deviations from each plane. The distance of closest approach to the origin for each plane is then a projection of the average of Δx and we get a number of terms for the variance similar to that in two dimensions.
For the changes of the e's to be independent they would have to be perpendicular to each other. Each e is then an eigenvector of the covariance matrix.
For the changes of the e's to be independent they would have to be perpendicular to each other. Each e is then an eigenvector of the covariance matrix.
Monday, January 3, 2011
Equation for the Eigenvalues in 2D
In two dimensions one can derive a simple formula for the angle of the eigenvector of a matrix if it is symmetric as in the case of the covariance matrix.
Since the equation is quadratic there are two solutions and one can substitute these into the equation for V to find the eigenvalues.
In three dimensions the initial equation in the derivation is that of a plane whose parametric equation is the composition of two independent unit vectors and corresponding parameters. One can use the eigenvectors corresponding to the two largest eigenvalues. The deviations from the plane are in the direction normal to it which corresponds to the smallest eigenvalue. For a line the parametric equation is the same as in two dimensions but there are deviations in the two directions normal to that of the line.
Since the equation is quadratic there are two solutions and one can substitute these into the equation for V to find the eigenvalues.
In three dimensions the initial equation in the derivation is that of a plane whose parametric equation is the composition of two independent unit vectors and corresponding parameters. One can use the eigenvectors corresponding to the two largest eigenvalues. The deviations from the plane are in the direction normal to it which corresponds to the smallest eigenvalue. For a line the parametric equation is the same as in two dimensions but there are deviations in the two directions normal to that of the line.
Connection with the Covariance Matrix
The matrix used to find the eigenvector for the "Balanced Least Squares" fit is the covariance matrix without averaging. One gets the expected value of a quantity by taking the average of its sum. The proof essentially states that the direction of the line giving the best fit is an eigenvector of the covariance matrix of the data.
I forgot to mention that the reason for using the difference between a point on the line and a data point for the deviation is that under the circumstances the fitted line is a function of the data. The justification is that the data maps onto the line. The point of view is that the derived rule is a product of experience. This is less hypothetical than assuming a rule, deriving its consequences and correlating the results with experience in order to justify the rule.
Supplemental note: Data reduction is a special case of e pluribus unum.
I forgot to mention that the reason for using the difference between a point on the line and a data point for the deviation is that under the circumstances the fitted line is a function of the data. The justification is that the data maps onto the line. The point of view is that the derived rule is a product of experience. This is less hypothetical than assuming a rule, deriving its consequences and correlating the results with experience in order to justify the rule.
Supplemental note: Data reduction is a special case of e pluribus unum.
Sunday, January 2, 2011
Derivation for the "Balanced Least Squares" Fit
The following is a short proof of the condition for the direction of the "Balanced Least Squares" fit line in 2D. One starts with a general equation for a line. The deviations are defined as the relative differences between a line through a data point parallel to the fitted line and the origin. x is a 2 by n matrix consisting of the data points. Like the deviations that I used for the previous fits these are from the data point to a point on the line which is opposite to the standard definition which is usually used for residuals.
Since V and its terms are scalars one can transpose them without changing the result. The formulas for the deviations and the variance turn out to be quite simple.
Since e is a unit vector and its changes are perpendicular to it the conclusion to be drawn is that the direction to the closest point of the fitted line to the origin is an eigenvector of the indicated matrix.
The eigenvector corresponding to the smallest eigenvalue is chosen for the direction to the point of closest approach to the origin. The one for the larger eigenvalue corresponds to the direction of the fitted line. So one can find parametric equations for the line with the best fit.
For a linear fit in nD the smallest eigenvalue will not necessarily be along a line through the origin. If the data is spread out well enough there will be a largest eigenvalue and the direction of the point of closest approach to the origin can be found by eliminating the direction of the fitted line from the direction to the center of the distribution of data points.
Since V and its terms are scalars one can transpose them without changing the result. The formulas for the deviations and the variance turn out to be quite simple.
Since e is a unit vector and its changes are perpendicular to it the conclusion to be drawn is that the direction to the closest point of the fitted line to the origin is an eigenvector of the indicated matrix.
The eigenvector corresponding to the smallest eigenvalue is chosen for the direction to the point of closest approach to the origin. The one for the larger eigenvalue corresponds to the direction of the fitted line. So one can find parametric equations for the line with the best fit.
For a linear fit in nD the smallest eigenvalue will not necessarily be along a line through the origin. If the data is spread out well enough there will be a largest eigenvalue and the direction of the point of closest approach to the origin can be found by eliminating the direction of the fitted line from the direction to the center of the distribution of data points.
An Alternative Least Squares Data Fit
When fitting a straight line to a set of data using Ordinary Least Squares (OLS) one often finds that the resulting fit deviates slightly from the expected "best fit". An example of a "misfit" can be seen in the image below. The data for the example was found by randomly selecting a line, points on it and then applying a random normal error to the data points. The fit appears more skewed because the scales are not equal.
An OLS fit seeks to minimize the squares of the vertical deviations from the line. A plot of these deviations for the fit shows that they are larger near the center of the distribution of errors and smaller at larger distances away from the center along the x-axis. Apparently OLS favors minimizing the deviations of points with extreme x distances from the center of the error distribution. If one computes the moment which corresponds to the sum of the "weights" δy at distance Δx from the center of the distribution one finds that it is zero. Similarly, if one exchanges the directions x and y one finds that this moment is quite large since the δxs were ignored. One also finds that the moment about the origin is zero and that the moment of normal deviations of points along the fitted line is quite large.
If one does a least squares fit using simultaneously both the x and y deviations from the line one gets a more balanced fit. This is probably closer to what one would get by drawing a straight line through the data by hand using a straight edge.
This "Balanced Least Squares" method has a moment which is zero for normal deviations along the line using distances from the center of the data. The moments for deviations normal to the x and y axis tend to cancel themselves out resulting in small values which appear to be the same.
It can shown that the direction of the fitted line is an eigenvector of a matrix which is the product of a matrix deviations Δr of the data points from the center of the distribution and its transpose and that the fitted line passes through the center of the data points. An eigenvector, e, of matrix, M, has the property that M e = m e, that is, the vector produced by multiplying a matrix by one of its eigenvectors has the same direction as the original eigenvector but the magnitude may be subject to change.
An OLS fit seeks to minimize the squares of the vertical deviations from the line. A plot of these deviations for the fit shows that they are larger near the center of the distribution of errors and smaller at larger distances away from the center along the x-axis. Apparently OLS favors minimizing the deviations of points with extreme x distances from the center of the error distribution. If one computes the moment which corresponds to the sum of the "weights" δy at distance Δx from the center of the distribution one finds that it is zero. Similarly, if one exchanges the directions x and y one finds that this moment is quite large since the δxs were ignored. One also finds that the moment about the origin is zero and that the moment of normal deviations of points along the fitted line is quite large.
If one does a least squares fit using simultaneously both the x and y deviations from the line one gets a more balanced fit. This is probably closer to what one would get by drawing a straight line through the data by hand using a straight edge.
This "Balanced Least Squares" method has a moment which is zero for normal deviations along the line using distances from the center of the data. The moments for deviations normal to the x and y axis tend to cancel themselves out resulting in small values which appear to be the same.
It can shown that the direction of the fitted line is an eigenvector of a matrix which is the product of a matrix deviations Δr of the data points from the center of the distribution and its transpose and that the fitted line passes through the center of the data points. An eigenvector, e, of matrix, M, has the property that M e = m e, that is, the vector produced by multiplying a matrix by one of its eigenvectors has the same direction as the original eigenvector but the magnitude may be subject to change.
Subscribe to:
Posts (Atom)