Stats for Spikes-Goodhart’s Law and Correlation

“When a measure becomes a target, it ceases to be a good measure.”

-Goodhart’s Law as Summarized by Anthropologist Marilyn Strathern

Ever since I read Marilyn Strathern’s summary of Goodhart’s Law, I was intrigued by the truth that it revealed across all human activities.

The short sentence packs a punch. We  have become ultra-focused on measuring results, partly to CYA, partly to gain an understanding on how our activities are progressing. We are convinced of the dire need to measure all variables every step along the way whether they are measurable or not. We can't seem to be doing anything without having to check the results. Ever since the business world became aware of the importance of  statistics for decision making, our society has decided to take measurement of everything that we can measure even if the information is of dubious importance. The idea is to take the data now and we can figure out what the numbers mean later.

The act of making the measurement the target also violates the statistical adage: “correlation does not mean causation,” most of us who have been exposed to Statistical Process Control (SPC) have had that adage drilled into our heads. We take measurements because it gives us a good estimation of how what we are measuring is  performing, it serves as a performance monitor, whether it is performing as we want. We have determined that the variable or variables that we are measuring has a certain statistical correlation to our desired results, it indicates whether we are on the right trajectory. The mistake that we make, which is where we violate Goodhart’s Law, is that we assume the correlation we observe between measurement and result automatically infers causation; making the threshold measurement a target is a simple and natural leap in logic, something we do without critical thought.

It is not a large leap, although it is a fatal leap in many instances.

In the process of leaping from treating a measurement, an estimate of the process, as a target,  we allow human nature to take over. In our eagerness to hit the target, a target that we believe to be causally related to accomplishing our final goal, we make that fatal error of assuming the correlation is causation. We are very much stuck in the If A then B line of thinking when we equate correlation to causation. Even as we ignore the other variables. People assume that the complex system being measured is linear and directly related to the desired result and that the relationship is one-to-one. So much so that we put blinders on, we have our eyes on the prize, and we have “laser” focus.

Reality, however, is multivariable, more than we can fathom or remember. Each of the variables could potentially be related to all the other variables; additionally,  some are more correlated than others.

In my engineering career I saw numerous instances of people making decisions which violates this very principle, even though they have been trained in Statistical Process Control. They were warned about “correlation does not equal causation” yet they either had not internalize the lesson or they ignored it completely because of wishful thinking.

The managers at a company I was working at realized that they were going to miss their June delivery goal. As the end of June straddles the weekend leading into the Fourth of July, they made the decision to pay the workers extra to work through the Fourth of July holiday and included all the products that were delivered over that weekend into the ledger as June delivery, which made their July delivery data abysmal. The production goal is based on an estimate of the capability of the manufacturing facility, the amount of workers, the difficulty of the manufacturing task, and the consistency of the supply chain. The production goals are estimates, not hard targets, because there are too many variables and natural variances with comes with the estimates. Eventually,  estimates will regress to the mean over the long term. By tweaking the number of days in the reporting period to buttress the production estimates, they are guaranteeing that the later estimates will not be met. In this case the regression happened immediately, although not all tweaking would have the same result.

In another case, most large manufacturing companies have adapted project management and project management tools to monitor progress. We were made aware through weekly updates of our manufacturing process status, mainly with regard to two variables: schedules and expenditures through the use of Schedule and Cost Performance Index (SPI and CPI). These are quantitative measures that are tracked and reported by the project managers. They measure the status and are compared to the  estimates that were created in an open loop fashion at the beginning of the project. Changing those SPI and CPI to reflect evolving challenges in real time are not rare events, all with the permission of the customers. Yet, at every weekly meeting, senior management will inevitably succumb to the temptation of cheating their schedules, forcing overtime, taking short cuts to meet the amorphous indices; indices that were determined many months in advance of the present, indices that were determined without knowledge of the evolving challenges. They do this in their eagerness to show the higher ups their ability to “Make things happen,” and then to be rewarded for their “get it done” attitude. The good project managers would counter these short-sighted whims, refraining from compromising the quality of the process.

There are many other instances where both “correlation does not equal causation” adage and Goodhart’s Law are ignored.

·       The US News and World Report college ranking, where universities treat the rankings as targets, they go to great lengths to game the data (measurements) which affects their rankings by treating the rankings as a target, even though they know that the factual meaning of the ranking is ambiguous.

·       In athletics, people have tried to quantify measures to identify athletic talent. In the case of the NFL combines, they run specific drills and measure each athlete’s performance in those drills. Those drills do not measure the potential draftees’ prowess  for playing the game, it just gives the coaches and team management a skewed set of  measurements, because trainers and strength coaches have successfully culled the accumulated data over time and have created customized workouts to train the athletes to meet the threshold of what is believed to be the ideal physical measurements as revealed by the specific drills. Those who come closest to the pre-determined threshold are more likely to be drafted and sign for large bonuses. These measurements do not guarantee success as a professional, yet the focus on that set of targets persists. The prime example is Tom Brady, he was the 199^th pick in the sixth round in the 2000 draft.

Yet the NFL was able to change their thought process when they stopped using the Wonderlic test as a measure of sports intelligence; maybe it is because that test is abysmal in predicting professional success.

It is also in sports that we see steadfast refusal to fall into the trap of ignoring the Goodhart’s Law and “correlation does not equal causation” adage.

There are important statistics and measurements, in all sports, the successful coaches view them as partial measures of the total team performance. The successful coaches understand that sports are messy, complicated, and interrelated, to try to infer performance from such pristine and single dimensional data is foolish. They understand that focusing on only one statistic, or statistics from one aspect of the total game gives them only a partial picture. Using only on-base percentage in baseball, ace to error ratio in volleyball, points in the paint in basketball, total rushing yards in football, etc. really does not indicate that the team will win, they are just one statistic out of many. It is up to the coach to understand the interaction of all the changing data in the context of the game.

Nature also plays tricks on the measurements because there will always be the unmeasurable factors which skews the total picture of each game.

The point of all this is not that we should not make measurements, or that data is inherently skewed. On the contrary, measurements are critical to our understanding the unknown, it gives us a means of judging whether the complex system that we are working with is behaving as we hope. The key to gaining knowledge and making good decisions has more to do with asking questions about our assumptions and beliefs before we start drawing inference. Data and information does not give humans the tools to predict outcome; humans infer the predictions based on their experiences and humans will draw wrong inferences more often than the correct inferences if left alone.

Questions need to be asked, and often. More importantly, those asked questions must address our proclivity to assume that correlation is causation, and our desire to make measurements a target.

 

 

 

 

Ruminations-On Historical Accuracy of the King James Bible

I recently posted a meme on my Facebook page, it made light of the way the King James version of the New Testament bible came into being. The meme itself did not skimp on historical facts, it did neglect to mention what one of my friends noted: All English translations were originally from Greek, which was translated from either Hebrew or the original Aramaic. The meaning of many Aramaic words was guessed at. So basically, the words read today are translations of translations of translations. In other words, a massive game of telephone, over many years.  

Here is the meme:

·       The King James bible was completed by 8 members of the Church of England in 1611.

·       There are no original texts to translate. The oldest manuscripts were written hundreds of years after the last apostle died. There are over 8,000 of these manuscripts, no two are alike.

·       The King James translators used none of these, they edited the previous translations to create a version that their king and Parliament would approve.

·       21^st century Christians believe ‘The Word of God’ is a book edited in the 17^th century from the 16^th century translations of 8,000 contradictory copies of the 4^th century scrolls that claims to be the copies of lost letter written in the first century.

·       And the snarky remark that drove people crazy: “That’s not faith, that’s insanity.”

      A few of my religious friends, and yes, I do have some of them, took me to task about this meme. They proceeded at long lengths about how they were able to come to determine the “accuracy” of the Bible. I will paraphrase what they have said to avoid quoting or misquoting each person. Those words in bold are my thoughts.

As a Christian I struggled with this very issue. I took a Biblical History class. I learned our NT was canonized at Council of Nicaea. [A group of mortal men].  [While later another group of mortal men] decided the 4 books of Apocrypha didn’t belong. I was disillusioned. But then I am affirmed that thru history God has and is inspiring these works. Hence one must approach the Bible with faith in God and His divine works. Otherwise it is indeed insanity.

­­­­­­­­­­­­­­­­­­­­­­­­­­­­­I have read the Apocrypha and the Nag Hammadi codex, which includes most excluded books. In my view, it is clear why they were determined to not belong in Scripture when you read them (and your view speaks for all of Christianity?). But as [the previous person] said, at the end of the day, it’s a matter of faith informed by prayer.

I would also add that the interpretation of Scripture over the centuries primarily by white males is the real issue. I take comfort in the fact that when I read Scripture, I do not see the condemning language pushed by evangelicals.

My belief  is rooted in the recognition that the Bible is an evolving set of texts that requires continual study using the best scholarship available from a variety of traditions (to what purpose? To what end?). The King James Version is more interesting for what it says about English Christianity than anything else. If we believe God created our brain, why would he/she expect us to check it at the door of the church? (Exactly, which makes me think that if you are using your brain and you questioned the verity of the messages in the bible, why would your God object because you are disagreeing with his supposed words, since you are using his gift of a brain to come to your questions?)

The King James version is a poor example of Biblical scholarship and should never be held up as the crowning achievement. Although very poetic, it is an abysmal translation of source documents (The meme addressed that, they didn’t even follow the ‘source documents.). Luther's work and use of Hebrew scholars almost a century earlier did a far better job in translating those documents into German (How would you know the translations were better? Does it matter if the source documents are centuries old translations themselves?).

We also have a large amount of knowledge about the original texts by studying the variations between the different texts which came from texts being copied by hand. Having gone to seminary and worked in yeast genetics labs, I find the similarity between the evolution of DNA sequences and hand copied biblical texts fascinating. (What percentage of these ‘knowledge of ‘original texts’ is directly verifiable? Is your proof just a guess as well?)

While I agree that the KJV has its weaknesses when the textus receptus is compared to the Nestle-Aland because of the authority of earlier texts over the byzantine texts and the Latin vulgate. It is an amazing feat of transcription among all of these texts and all that time for them to have such agreement that nothing of theological or historical is in disagreement. No other extant historical document has such consistency. The assertions of this meme are inaccurate in the extreme. (Is it consistency of agreement or is it groupthink and wishful thinking working its magic?)

Note that the defenses that these friends of mine calls up do not challenge the historical facts listed in the meme, instead, the arguments involve mental gyrations that dances around the scholarship of the historians who research the documents and the translations. It doesn’t matter how stellar the scholarship is, if the translations, interpretation, and conclusions have a dodgy foundation. Given the nature of what we have, it is a heroic attempt at historical research to cobble together something that resemble an historical account, but their work sits on quicksand. No amount of congruency or consistency can overcome that because no matter how much work and how exalted their scholarship, they are still leaning their ladders on the wrong wall.

Having said that, my mind asked another question: why does anyone care? Does it matter to the believers and non-believers that the verity of their faith and beliefs is dependent on the factual historical records of the written word?

The question of the historical basis for translations, interpretation, and conclusions from incomplete documents is an old and very common issue in history and in philosophy.

Socrates was adamantly opposed to writing down orations. He prided himself as an orator, he felt that memorizing what he had to say and giving his speeches to his pupil was the sole method of communication that mattered. He railed against the act of writing because he felt that writing distorts meanings since humans are forgetful and are susceptible to error. What was left unsaid is that the scribes’ biases and fallacies may, more than likely, skew and distort the spoken words, deliberately or not. His students did write down his speeches, which is how we came to have books of his thoughts. How authentic are those speeches? No one is sure because those words went through the same process as the King James bible. Does anyone care? We have debates and discussions about the substance of those lessons as we explore the ideas in the context of Socrates times, as well as in the context of our time. The richness of his words, whether he said them or not, are still feeding our curiosity and our resulting explorations.

Epictetus’s Discourse and Echeiridion are some of the founding basis of Stoicism, even though oral traditions tell us that Stoicism started out with Greek philosophers, before their ideas, through mostly oral means, were passed to the Romans. Indeed, any writing that is attributed to Epictetus were written by his student Arrian. We are, and have been, playing telephone with the Stoics as well. How much of Discourse and Echeiridion is Epictetus? How much of it is Arrian? Who knows?  There is a recent revival in Stoicism, many authors have written volumes based on their interpretation of unverified translations from the ancients. This fact has not deterred Stoicism from becoming popular, after many years of being ignored.

In the Chinese tradition, Tao Te Ching is attributed to Lao-tzu, as he has been identified as the founder of Taoism. Yet, there is no evidence that such a person ever existed. Some believe that the Tao Te Ching is a collection of wisdom from many teachers, aggregated to form the Tao Te Ching. Meanwhile, there is an apocryphal story of Lao-tzu, a contemporary of Confucius, who grew disenchanted with his unsuccessful attempt to teach the way of the Tao, the creative and binding force which runs through the universe. Legend has it that he was making his way to the borders of the known Chinese kingdoms and into seclusion when someone convinced him to dictate his wisdom to the student so that the lessons can be recorded for posterity. Romantic story, but is it historically true? Who knows? Who cares?

Indeed, the basis of western civilization is based on incomplete and impossible to prove hypothesis and interpretations of artifacts and texts from the distant past. These conclusions drawn by contemporary interpreters were skewed by values and morals of the ancient as well as the contemporary viewpoint. A concatenation of skewing.

As we examine our opinions on historical and philosophical truths, as we know them, rarely do we care about the verity of historical records regarding ancient ideas. We accept that the knowledge that we have is flawed and possibly skewed, but it is what we have, so we adjust our interpretations of the past as more scholarship and artifacts become available by hewing to the scientific method to make our determination as to their verity.

We take the philosophies, histories, and other ancient traditions that have been passed on to us, both Eastern and Western, at face value, even as we understand and accept the ambiguities that is introduced into the accuracy of the interpretation of the knowledge because of historical indeterminacy.

I decided to practice omphaloskepsis over the question of: why would someone be so adamant about a practice that is so commonplace, so pedestrian, so accepted? The following discussion is entirely my own conjectures and wonderings; keep that fact in mind. Any lapses in logic or reasoning are entirely due to my own carelessness.

It occurred to me that the tenets of the religion are not what is debated here, the tenets of the religion could be seen as distilled truths that are travelling the same intellectual path as philosophy, indeed they are travelling in parallel paths in most circumstances. Specifically, regarding the ideas on the ideals and morality of our behavior. What makes people accept or reject the teachings of philosophical schools is not blind faith, it is the individual person’s judgement on how those teachings are aligned with our personal experiences in life. Our philosophical ideas are affirmed or disaffirmed by what our experience and what our reasoning tells us, using both our empirical knowledge and rational instincts. The same can be said of religious tenets. All religions have basic beliefs are mostly shared with one another, the commonalities between religions outnumber the differences. Why is that? I believe that these commonalities are the basis of our survival as communities. I believe that should be the basis of where we start: our humanity.

I thought about how to separate the ethereal philosophical part of religion so that I can get at the answer to my question; I arrived at the idea of the church, or the religious organization, that man-made institution employed to organize the great unwashed constituency of the believers as the difference maker.

We know that we humans have a need to form communities based on our historical need to survive, there is strength in numbers. Once we have been socialized to the idea of a community, we sought to form more communities based on our commonalities. This is not an earth-shaking statement. As we form these communities and organizations, we make a social compact to create constraints that are placed on those who are in those organizations; these constraints are necessary for the peaceful coexistence within the community. One constraint is a means of survival, in more crass and contemporary terms, they need to have a business model, after all they have to ensure the survival of the organization in the future, ethereal things live on in the minds of the believer, but brick and mortar houses of worship needs gold to survive, as with all other organizations, although the monetary needs are a function of the size and ambitions of the organization.

The business model also needs an identity, to define to the members of the organization their official identifiable points of commonalities. Unofficially the statement of identity also separates those who are members of the community from those who do not belong to the community, this can be both an intended and an unintended consequence. This need for identity eventually morphs into a tangible list of beliefs, something that clearly differentiates ‘us from them’; it also morphs into  something to defend, something to identify the non-believers; this is human nature, we want to identify strongly with our preferred community, and we want to know whether others are in the same community. This is an organic practice; the problem usually appears when the members of the community seek to clearly and forcefully differentiate themselves from those who are not in agreement with the community identity and they seek to treat those non-conformists as their enemies.

Why would anyone treat those who are not in their community as enemies?

Which brings us to the third constraint: the organization to expand, to grow, to attract new believers, something more expedient than the natural way: have lots of babies. This of course, applies across all cultures and religious organizations, All major religious organizations, Christians, Buddhist, Muslims, Hindus, Jews, et. al. has the priority mission to proselytize and expand their membership of the religious organization to expedite the first constraint: maximize the income from the business model to survive. To get more members and more importantly, to prevent people from joining other religious organizations that they perceive to be in conflict with their organization: there is strength in numbers. There is a perceived need to market their beliefs as being superior to other religious organizations, to ignore the universal humanity of their common beliefs and to exaggerate the advantages of their organization. If, we assumed that there is such a thing as universal humanity, then it is difficult to argue about the differences between organizations unless one dove into the granularities such as the historical authenticity of their book of beliefs.

Why do people care about the King. James bible being authentic and is the real transcribed words of the Christian god? Two reasons, first, it goes to show those who are about to be converted that they are joining an organization that have history on their side, no matter how dodgy that history is; second, to continue to impress on the already converted that they are on the correct side of their god, because you cannot trust anyone to not use their brains and question their faith.

I had stated previously that belief in philosophy does not depend on the historical authenticity of the source of the message, that adherents to most philosophies believe because they have found that the lesson from philosophy generally agrees with their experiences. This alignment with human experience is the most powerful way to ingrain beliefs in humans; if it is true in my experience then I will doubt the veracity of the amorphous and ambiguous much less. We are practicing our own ability to judge, to be critical, to question, and most importantly, to verify to the best of our abilities. If we happen to experience the contrary to what we believe, we can and will change. It is all in the context of the situation. Some religious friend refer to it derisively as moral relativism, I think of it as a natural part of being human and living on a world where context is changing continually, our philosophical foundations must keep pace as well.  The counter argument usually involves changing our beliefs as the wind changes direction. It is a valid counterargument, but we still have our brain and our perception to question the veracity of the changes brought forth by context. I think the human’s natural inclination to fear of change makes that option less likely.

Another question that appeared from my bout of omphaloskepsis is: why do the non-believers derive such joy in poking the believers with memes such as this one?

I see it as a reaction to the absolutely determinacy that the believers act and react with when dealing with those that they find are opposed to their beliefs. One problem with the staking an unchanging identity is that there is no fluidity to your espoused belief even as the context of the problem changes, those who are a part of the organization are forever chained to tenets set forth by, in this case 8 members of the Church of England in 1611. This is how heresy and persecution of heretics happen. Therefore Copernicus, Spinoza, et.al were branded as heretics. I see the same problem with the original intent argument from the Scalia cabal of conservative thought.

The more humans adhere to the beliefs that are not defensible under their contemporary context, the more their fellow humans will delight in pointing out the contradictions and hypocrisy inherent in their arguments. Because we do use our brains, whether it was God who endowed us with brains, or not.