The matrix used to find the eigenvector for the "Balanced Least Squares" fit is the covariance matrix without averaging. One gets the expected value of a quantity by taking the average of its sum. The proof essentially states that the direction of the line giving the best fit is an eigenvector of the covariance matrix of the data.
I forgot to mention that the reason for using the difference between a point on the line and a data point for the deviation is that under the circumstances the fitted line is a function of the data. The justification is that the data maps onto the line. The point of view is that the derived rule is a product of experience. This is less hypothetical than assuming a rule, deriving its consequences and correlating the results with experience in order to justify the rule.
Supplemental note: Data reduction is a special case of e pluribus unum.
Monday, January 3, 2011
Sunday, January 2, 2011
Derivation for the "Balanced Least Squares" Fit
The following is a short proof of the condition for the direction of the "Balanced Least Squares" fit line in 2D. One starts with a general equation for a line. The deviations are defined as the relative differences between a line through a data point parallel to the fitted line and the origin. x is a 2 by n matrix consisting of the data points. Like the deviations that I used for the previous fits these are from the data point to a point on the line which is opposite to the standard definition which is usually used for residuals.
Since V and its terms are scalars one can transpose them without changing the result. The formulas for the deviations and the variance turn out to be quite simple.
Since e is a unit vector and its changes are perpendicular to it the conclusion to be drawn is that the direction to the closest point of the fitted line to the origin is an eigenvector of the indicated matrix.
The eigenvector corresponding to the smallest eigenvalue is chosen for the direction to the point of closest approach to the origin. The one for the larger eigenvalue corresponds to the direction of the fitted line. So one can find parametric equations for the line with the best fit.
For a linear fit in nD the smallest eigenvalue will not necessarily be along a line through the origin. If the data is spread out well enough there will be a largest eigenvalue and the direction of the point of closest approach to the origin can be found by eliminating the direction of the fitted line from the direction to the center of the distribution of data points.
Since V and its terms are scalars one can transpose them without changing the result. The formulas for the deviations and the variance turn out to be quite simple.
Since e is a unit vector and its changes are perpendicular to it the conclusion to be drawn is that the direction to the closest point of the fitted line to the origin is an eigenvector of the indicated matrix.
The eigenvector corresponding to the smallest eigenvalue is chosen for the direction to the point of closest approach to the origin. The one for the larger eigenvalue corresponds to the direction of the fitted line. So one can find parametric equations for the line with the best fit.
For a linear fit in nD the smallest eigenvalue will not necessarily be along a line through the origin. If the data is spread out well enough there will be a largest eigenvalue and the direction of the point of closest approach to the origin can be found by eliminating the direction of the fitted line from the direction to the center of the distribution of data points.
An Alternative Least Squares Data Fit
When fitting a straight line to a set of data using Ordinary Least Squares (OLS) one often finds that the resulting fit deviates slightly from the expected "best fit". An example of a "misfit" can be seen in the image below. The data for the example was found by randomly selecting a line, points on it and then applying a random normal error to the data points. The fit appears more skewed because the scales are not equal.
An OLS fit seeks to minimize the squares of the vertical deviations from the line. A plot of these deviations for the fit shows that they are larger near the center of the distribution of errors and smaller at larger distances away from the center along the x-axis. Apparently OLS favors minimizing the deviations of points with extreme x distances from the center of the error distribution. If one computes the moment which corresponds to the sum of the "weights" δy at distance Δx from the center of the distribution one finds that it is zero. Similarly, if one exchanges the directions x and y one finds that this moment is quite large since the δxs were ignored. One also finds that the moment about the origin is zero and that the moment of normal deviations of points along the fitted line is quite large.
If one does a least squares fit using simultaneously both the x and y deviations from the line one gets a more balanced fit. This is probably closer to what one would get by drawing a straight line through the data by hand using a straight edge.
This "Balanced Least Squares" method has a moment which is zero for normal deviations along the line using distances from the center of the data. The moments for deviations normal to the x and y axis tend to cancel themselves out resulting in small values which appear to be the same.
It can shown that the direction of the fitted line is an eigenvector of a matrix which is the product of a matrix deviations Δr of the data points from the center of the distribution and its transpose and that the fitted line passes through the center of the data points. An eigenvector, e, of matrix, M, has the property that M e = m e, that is, the vector produced by multiplying a matrix by one of its eigenvectors has the same direction as the original eigenvector but the magnitude may be subject to change.
An OLS fit seeks to minimize the squares of the vertical deviations from the line. A plot of these deviations for the fit shows that they are larger near the center of the distribution of errors and smaller at larger distances away from the center along the x-axis. Apparently OLS favors minimizing the deviations of points with extreme x distances from the center of the error distribution. If one computes the moment which corresponds to the sum of the "weights" δy at distance Δx from the center of the distribution one finds that it is zero. Similarly, if one exchanges the directions x and y one finds that this moment is quite large since the δxs were ignored. One also finds that the moment about the origin is zero and that the moment of normal deviations of points along the fitted line is quite large.
If one does a least squares fit using simultaneously both the x and y deviations from the line one gets a more balanced fit. This is probably closer to what one would get by drawing a straight line through the data by hand using a straight edge.
This "Balanced Least Squares" method has a moment which is zero for normal deviations along the line using distances from the center of the data. The moments for deviations normal to the x and y axis tend to cancel themselves out resulting in small values which appear to be the same.
It can shown that the direction of the fitted line is an eigenvector of a matrix which is the product of a matrix deviations Δr of the data points from the center of the distribution and its transpose and that the fitted line passes through the center of the data points. An eigenvector, e, of matrix, M, has the property that M e = m e, that is, the vector produced by multiplying a matrix by one of its eigenvectors has the same direction as the original eigenvector but the magnitude may be subject to change.
Wednesday, November 17, 2010
New Man vs Old Man
The old man clings to the past but tends to lose his grip on it over time. He is set in his ways and a ghost of what once was.
The new man is somewhat of a tyro who wants more but lacks perspective and so has a limited world view. He tends to be more adaptive but suffers from a lack of restraint due to an uncritical nature. Oversimplification leaves him prone to error.
Caesar was a new man who did not respect bounds and ended up with delusions of grandeur. Cicero, also a new man, recognized the need for moderation and balance. He was a rare breed who tried to see both sides but was not fully accepted by either side.
The Renaisance saw a re-emergence of the new man and the beginnings of the Reformation. The spirit of self-determination gained ascendency in northern Europe. In America too there was a break brought about by its treatment as a protectorate by a remote government. America was the New World and its people sought a new beginning. The Enlightenment with its progressive viewpoint had spurred change but the excesses of the French Revolution and the rise of Napoleon raised a lot of doubts.
Perhaps the best approach is that of the "regenerative receiver". He too is no longer satisfied with predetermined solutions and seeks a better picture by reworking the problem. In the process he acquires some knowledge of what is missing. Additional creativity allows one to go beyond what has already been accomplished. A broader perspective takes one beyond the here and now. But the wise man is also interested in bounds.
The new man is somewhat of a tyro who wants more but lacks perspective and so has a limited world view. He tends to be more adaptive but suffers from a lack of restraint due to an uncritical nature. Oversimplification leaves him prone to error.
Caesar was a new man who did not respect bounds and ended up with delusions of grandeur. Cicero, also a new man, recognized the need for moderation and balance. He was a rare breed who tried to see both sides but was not fully accepted by either side.
The Renaisance saw a re-emergence of the new man and the beginnings of the Reformation. The spirit of self-determination gained ascendency in northern Europe. In America too there was a break brought about by its treatment as a protectorate by a remote government. America was the New World and its people sought a new beginning. The Enlightenment with its progressive viewpoint had spurred change but the excesses of the French Revolution and the rise of Napoleon raised a lot of doubts.
Perhaps the best approach is that of the "regenerative receiver". He too is no longer satisfied with predetermined solutions and seeks a better picture by reworking the problem. In the process he acquires some knowledge of what is missing. Additional creativity allows one to go beyond what has already been accomplished. A broader perspective takes one beyond the here and now. But the wise man is also interested in bounds.
Jefferson on Education
Jefferson believed that public education was essential for a republic to function properly. The following passages are from The Portable Thomas Jefferson,
To George Wythe (Paris, Aug 13, 1786): "...I think by far the most important bill in our whole code is that of the diffusion of knowledge among the people. No other sure foundation can be divised for the preservation of freedom, and happiness."
Report of the Commissioners for the University of Virginia (Aug 4, 1818),
"...The objects of this primary education determine its character and limits. These objects would be,
To enable him to calculate for himself, and to express and preserve his ideas, his contracts and accounts in writing,
To improve, by reading, his morals and faculties;
To understand his duties to his neighbors and country, and to discharge with competence the functions confided to him by either;
To know his rights; to exercise with order and justice those he retains; to choose with discretion the fiduciary of those he delegates; and to notice their conduct with diligence, with candor, and judgment;
And in general, to observe with intelligence and faithfulness all the social relations under which he shall be placed."
"...Some good men, and even of respectable information, consider the learned sciences as useless acquirement; some think that they do not better the condition of man; and others that education, like private and individual concerns, should be left to private individual effort; not reflecting that an establishment embracing all the sciences which may be useful and even necessary in the various vocations of life, with the buildings and apparatus belonging to each, are far beyound the reach of individual means, and must either derive existence from public patronage, or not exist at all. This would leave us then without those callings which depend on education, or send us to other countries to seek the instruction they require. But the Commissioners are happy in considering the statute under which they are assembled as proof that the Legislature is far from the abandonment of objects so interesting. They are sensible that the advantages of well-directed education, moral, political and economical, are truly above all estimate. Education generates habits of application, of order, and the love of virtue; and controls, by the force of habit, any innate obliquities in our moral organization. We should be far, too, from the discouraging persuasion that man is fixed, by the law of his nature, at a given point; that his improvement is a chimera, and the hope delusive of rendering ourselves wiser, happier or better than our forefathers were. As well might it be urged that the wild and uncultivated tree, hitherto yielding sour and bitter fruit only, can never be made to yield better; yet we know that the grafting art implants a new tree on the savage stock, producing what is most estimable both in kind and degree. Education, in like manner, engraphs a new man on the native stock, and improves what in his nature was vicious and perverse into qualities of virtue and social worth. And it cannot be but that each generation succeeding to the knowledge acquired by all those who preceeded it, adding to it their own acquisitions and discoveries, and handing the mass down for successive and constant accumulation, must advance the knowledge and well-being of mankind, not infinitely, as some have said, but indefinitely, and to a term which no one can fix and forsee. Indeed, we need look back half a century, to times which many now living remember well, and see the wonderful advances in the sciences and arts which have been made within that period. Some of these have rendered the elements themselves subservient to the purposes of man, have harnessed them to the yoke of his labors, and effected the great blessings of moderating his own, of accomplishing what was beyond his feeble force, and extending the comforts of life to a much enlarged circle, to those who had before known its necessaries only. That these are not the vain dreams of sanguine hope, we have before our eyes real and living examples."
To George Wythe (Paris, Aug 13, 1786): "...I think by far the most important bill in our whole code is that of the diffusion of knowledge among the people. No other sure foundation can be divised for the preservation of freedom, and happiness."
Report of the Commissioners for the University of Virginia (Aug 4, 1818),
"...The objects of this primary education determine its character and limits. These objects would be,
To enable him to calculate for himself, and to express and preserve his ideas, his contracts and accounts in writing,
To improve, by reading, his morals and faculties;
To understand his duties to his neighbors and country, and to discharge with competence the functions confided to him by either;
To know his rights; to exercise with order and justice those he retains; to choose with discretion the fiduciary of those he delegates; and to notice their conduct with diligence, with candor, and judgment;
And in general, to observe with intelligence and faithfulness all the social relations under which he shall be placed."
"...Some good men, and even of respectable information, consider the learned sciences as useless acquirement; some think that they do not better the condition of man; and others that education, like private and individual concerns, should be left to private individual effort; not reflecting that an establishment embracing all the sciences which may be useful and even necessary in the various vocations of life, with the buildings and apparatus belonging to each, are far beyound the reach of individual means, and must either derive existence from public patronage, or not exist at all. This would leave us then without those callings which depend on education, or send us to other countries to seek the instruction they require. But the Commissioners are happy in considering the statute under which they are assembled as proof that the Legislature is far from the abandonment of objects so interesting. They are sensible that the advantages of well-directed education, moral, political and economical, are truly above all estimate. Education generates habits of application, of order, and the love of virtue; and controls, by the force of habit, any innate obliquities in our moral organization. We should be far, too, from the discouraging persuasion that man is fixed, by the law of his nature, at a given point; that his improvement is a chimera, and the hope delusive of rendering ourselves wiser, happier or better than our forefathers were. As well might it be urged that the wild and uncultivated tree, hitherto yielding sour and bitter fruit only, can never be made to yield better; yet we know that the grafting art implants a new tree on the savage stock, producing what is most estimable both in kind and degree. Education, in like manner, engraphs a new man on the native stock, and improves what in his nature was vicious and perverse into qualities of virtue and social worth. And it cannot be but that each generation succeeding to the knowledge acquired by all those who preceeded it, adding to it their own acquisitions and discoveries, and handing the mass down for successive and constant accumulation, must advance the knowledge and well-being of mankind, not infinitely, as some have said, but indefinitely, and to a term which no one can fix and forsee. Indeed, we need look back half a century, to times which many now living remember well, and see the wonderful advances in the sciences and arts which have been made within that period. Some of these have rendered the elements themselves subservient to the purposes of man, have harnessed them to the yoke of his labors, and effected the great blessings of moderating his own, of accomplishing what was beyond his feeble force, and extending the comforts of life to a much enlarged circle, to those who had before known its necessaries only. That these are not the vain dreams of sanguine hope, we have before our eyes real and living examples."
Jefferson's Esteem for Classical Literature
Jefferson's esteem for classical literature can be seen in the following passage from a letter to his nephew and ward Peter Carr (Paris, Aug 19, 1785),
"...I advise you to begin a course of antient history, reading everything in the original and not in translations. First read Goldsmith's history of Greece. This will give you a digested view of that field. Then take up antient history in the detail, reading the following books in the following order. Herodotus. Thucydides. Xenophontis hellenica. Xenophontis Anabasis. Quintus Curtius. Justin. This shall form the first stage of your historical reading, and is all I need mention to you now. The next will be of Roman history. From that we will come down to Modern history. In Greek and Latin poetry, you have read or will read at school Virgil, Terence, Horace, Anacreon, Theocritus, Homer. Read also Milton's paradise lost, Ossian, Pope's works, Swift's works in order to form your style in your own language. In morality read Epictetus, Xenophontis memorabilia, Plato's Socratic dialogues, Cicero's philosophies..." -The Portable Thomas Jefferson
"...I advise you to begin a course of antient history, reading everything in the original and not in translations. First read Goldsmith's history of Greece. This will give you a digested view of that field. Then take up antient history in the detail, reading the following books in the following order. Herodotus. Thucydides. Xenophontis hellenica. Xenophontis Anabasis. Quintus Curtius. Justin. This shall form the first stage of your historical reading, and is all I need mention to you now. The next will be of Roman history. From that we will come down to Modern history. In Greek and Latin poetry, you have read or will read at school Virgil, Terence, Horace, Anacreon, Theocritus, Homer. Read also Milton's paradise lost, Ossian, Pope's works, Swift's works in order to form your style in your own language. In morality read Epictetus, Xenophontis memorabilia, Plato's Socratic dialogues, Cicero's philosophies..." -The Portable Thomas Jefferson
Tuesday, November 16, 2010
Adams, Jefferson and Coke
In his response to the letter of General Brattle in the Boston Gazette in 1773 on the Independence of the Judiciary, John Adams cites Lord Coke who wrote The Institutes of the Laws of England which became a standard text on common law in both England and America. Coke was removed as Chief Justice for his refusal to submit to the king's command. He was later influential as an opponent to the king championing the rule of law and playing a pivotal role in the writing of the Petition of Right.
Thomas Jefferson also appears to have been familiar with Coke. In a letter to John Page dated December 25, 1762, he writes,
"However, whatever misfortunes may attend the picture or lover, my hearty prayers shall be, that all the health and happiness which Heaven can send may be the protion of the original, and that so much goodness may ever meet with what may be the most agreeable in this world, as I am sure it must be in the next. And now, although the picture be defaced, there is so lively an image of her imprinted in my mind, that I shall think of her too often, I fear, for my peace of mind; and too often, I am sure, to get through old Coke this winter; for God knows I have not seen him since I packed him up in my trunk in Williamsburg. Well, Page, I do wish the Devil had old Coke, for I am sure I never was so tired of an old dull scoundrel in my life...But the old fellows say we must read to gain knowledge, and gain knowledge to make us happy and admired. Mere jargon!"
Thomas Jefferson also appears to have been familiar with Coke. In a letter to John Page dated December 25, 1762, he writes,
"However, whatever misfortunes may attend the picture or lover, my hearty prayers shall be, that all the health and happiness which Heaven can send may be the protion of the original, and that so much goodness may ever meet with what may be the most agreeable in this world, as I am sure it must be in the next. And now, although the picture be defaced, there is so lively an image of her imprinted in my mind, that I shall think of her too often, I fear, for my peace of mind; and too often, I am sure, to get through old Coke this winter; for God knows I have not seen him since I packed him up in my trunk in Williamsburg. Well, Page, I do wish the Devil had old Coke, for I am sure I never was so tired of an old dull scoundrel in my life...But the old fellows say we must read to gain knowledge, and gain knowledge to make us happy and admired. Mere jargon!"
Subscribe to:
Posts (Atom)