The other day I came across a very intriguing research paper, bearing a very provocative title: “Why Most Published Research Findings are False”. Published in 2005, this paper was written by John P.A. Ioannidis, a medical professor specializing in epidemiology. His claim is simple (I quote from the abstract of the paper):
The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.
Now, as you might imagine, a scientific paper with such a subject matter would be sure to attract a lot of attention, both positive and negative, and that was indeed the case. But I do not intend to participate in the debate, and this is not the reason I am bringing up this paper here.
I am more interested in the concept of truth, especially in the way it is employed in papers such as Ioannidis’, i.e. in current scientific research.
The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.
Now, as you might imagine, a scientific paper with such a subject matter would be sure to attract a lot of attention, both positive and negative, and that was indeed the case. But I do not intend to participate in the debate, and this is not the reason I am bringing up this paper here.
I am more interested in the concept of truth, especially in the way it is employed in papers such as Ioannidis’, i.e. in current scientific research.
In the excerpt of the abstract quoted above, Ioannidis uses the term “true” in three different occasions.
In the first he speaks about the probability that a research claim is true.
In the second he speaks about true relationships and their ratio to no relationships.
And in the third he speaks about the likelihood of a research finding being true.
So, we have a) a claim that is true; b) a relationship that is true; and c) a research finding that is true.
Now, let’s pause here for a moment. When the words “is true” are used, are they referring to the same thing? What do they mean? What is truth?
Usually people take statements to be true when they correspond to reality.
When I say “I have some money to spend today”, this is true if (and only if) I really have some money to spend on the day that I utter these words.
Or should I announce that “the cat is on the mat”, this statement would be true if and only if at the point of my saying so there was a (real) cat on a (real) mat.
So far, so good.
What about a statement such as “this house is beautiful”? What are the conditions that such a statement be true? It would be true, you would think, if and only if the house in question is beautiful indeed.
But how do we establish this? While it’s relatively easy to establish the reality of an observable state of affairs (such as the cat’s being on the mat), things become a bit more unclear when a statement refers to concepts (or qualities) such as “beautiful”. People usually brush this kind of statements off, explaining that they are subjective. Of course we need to clarify what subjective means, but let’s say that this is so. Still, we can see that there is a tiny little problem with our understanding of truth. It seems, in this case, that we employ the term differently, depending on whether the issue in question is of a subjective or objective matter. Objective truths are somehow more real than subjective ones.
Keeping that in mind, let’s get back to our scientific paper.
As we saw, it speaks about a) a claim that is true; b) a relationship that is true; and c) a research finding that is true.
We can see that when a claim is said to be true this can only be so if said claim corresponds to some specific and observable state of affairs. I claim that “the cat is on the mat”, and on hearing my words, you look, and you confirm that this indeed is the case. Therefore my claim is true. For the sake of argument, let’s accept that the same applies to research claims.
What about a relationship? When can it be true? What does it mean “true relationship”? The paper talks about scientific research, so it is safe to assume that when it mentions the ratio of true relationships to no relationships it actually means statistical relationships (correlations and dependences) between variables or sets of data. Let’s accept then that the same reasoning applies. A statistical relationship will be true if and only if it corresponds to a specific and observable state of affairs, i.e. if and only if the variables or datasets in question do indeed demonstrate said relationship.
Finally, what can be said about a research finding? It is clear that a research finding cannot be true or not true on its own with no reference to anything else. From the context we can presume that when the paper talks about true findings it means findings that confirm a research hypothesis. So, a finding can be said to be true if and only if it confirms a research hypothesis.
So, the words “is true” are used in three different ways. In the first they are taken to represent a correspondence between a claim and a state of affairs; in the second they are taken to mean the demonstrable existence of a correlation between variables or datasets. And in the third they refer to the confirmation of a hypothesis.
It’s easy to see that none of those conceptions of truth refers to reality as such. The cat might really be on the mat, independently of the observer, but a research claim does not exist outside the framework in which it can have a meaning. It is real, ok, but not in the same way that the cat is real.
As a work hypothesis, then, let us differentiate between “real truth” and “scientific truth”.
Scientific truth only refers to the internal consistency of theories, hypotheses, claims and statements within a field of research and their verifiability by data collected according to other theories, hypotheses, claims and statements within the same field of research. It never refers to real truth.
That is a very important point which many times goes unnoticed.
Let me spell it out. Scientific research of the highest standard strives for scientific truth, i.e. consistency within a field of research. Inconsistencies make researchers re-evaluate their concepts and theories, sometimes abandon them, sometimes change direction and sometimes replace them. Scientific research never aims at real truth as such. It can never do so. Real truth is irrelevant, as far as science is concerned.
To put it more boldly: Science does not have anything to do with Reality.
That was perhaps too bold. I should clarify.
I shall return.
(added 27/7/11: Please click here.)
In the first he speaks about the probability that a research claim is true.
In the second he speaks about true relationships and their ratio to no relationships.
And in the third he speaks about the likelihood of a research finding being true.
So, we have a) a claim that is true; b) a relationship that is true; and c) a research finding that is true.
Now, let’s pause here for a moment. When the words “is true” are used, are they referring to the same thing? What do they mean? What is truth?
Usually people take statements to be true when they correspond to reality.
When I say “I have some money to spend today”, this is true if (and only if) I really have some money to spend on the day that I utter these words.
Or should I announce that “the cat is on the mat”, this statement would be true if and only if at the point of my saying so there was a (real) cat on a (real) mat.
So far, so good.
What about a statement such as “this house is beautiful”? What are the conditions that such a statement be true? It would be true, you would think, if and only if the house in question is beautiful indeed.
But how do we establish this? While it’s relatively easy to establish the reality of an observable state of affairs (such as the cat’s being on the mat), things become a bit more unclear when a statement refers to concepts (or qualities) such as “beautiful”. People usually brush this kind of statements off, explaining that they are subjective. Of course we need to clarify what subjective means, but let’s say that this is so. Still, we can see that there is a tiny little problem with our understanding of truth. It seems, in this case, that we employ the term differently, depending on whether the issue in question is of a subjective or objective matter. Objective truths are somehow more real than subjective ones.
Keeping that in mind, let’s get back to our scientific paper.
As we saw, it speaks about a) a claim that is true; b) a relationship that is true; and c) a research finding that is true.
We can see that when a claim is said to be true this can only be so if said claim corresponds to some specific and observable state of affairs. I claim that “the cat is on the mat”, and on hearing my words, you look, and you confirm that this indeed is the case. Therefore my claim is true. For the sake of argument, let’s accept that the same applies to research claims.
What about a relationship? When can it be true? What does it mean “true relationship”? The paper talks about scientific research, so it is safe to assume that when it mentions the ratio of true relationships to no relationships it actually means statistical relationships (correlations and dependences) between variables or sets of data. Let’s accept then that the same reasoning applies. A statistical relationship will be true if and only if it corresponds to a specific and observable state of affairs, i.e. if and only if the variables or datasets in question do indeed demonstrate said relationship.
Finally, what can be said about a research finding? It is clear that a research finding cannot be true or not true on its own with no reference to anything else. From the context we can presume that when the paper talks about true findings it means findings that confirm a research hypothesis. So, a finding can be said to be true if and only if it confirms a research hypothesis.
So, the words “is true” are used in three different ways. In the first they are taken to represent a correspondence between a claim and a state of affairs; in the second they are taken to mean the demonstrable existence of a correlation between variables or datasets. And in the third they refer to the confirmation of a hypothesis.
It’s easy to see that none of those conceptions of truth refers to reality as such. The cat might really be on the mat, independently of the observer, but a research claim does not exist outside the framework in which it can have a meaning. It is real, ok, but not in the same way that the cat is real.
As a work hypothesis, then, let us differentiate between “real truth” and “scientific truth”.
Scientific truth only refers to the internal consistency of theories, hypotheses, claims and statements within a field of research and their verifiability by data collected according to other theories, hypotheses, claims and statements within the same field of research. It never refers to real truth.
That is a very important point which many times goes unnoticed.
Let me spell it out. Scientific research of the highest standard strives for scientific truth, i.e. consistency within a field of research. Inconsistencies make researchers re-evaluate their concepts and theories, sometimes abandon them, sometimes change direction and sometimes replace them. Scientific research never aims at real truth as such. It can never do so. Real truth is irrelevant, as far as science is concerned.
To put it more boldly: Science does not have anything to do with Reality.
That was perhaps too bold. I should clarify.
I shall return.
(added 27/7/11: Please click here.)
RSS Feed