[mesZibeds]

http://bps-research-digest.blogspot....ortion-of.html

Made it! An uncanny number of psychology findings manage to scrape into statistical significance

Like a tired boxer at the Olympic Games, the reputation of psychological science has just taken another punch to the gut. After a series of fraud scandals in social psychology and a US survey that revealed the widespread use of questionable research practices, a paper published this month finds that an unusually large number of psychology findings are reported as "just significant" in statistical terms. In psychology, a common practice is to determine how probable (p) it is that the observed results in a study could have been obtained if the null hypothesis were true (the null hypothesis usually being that the treatment or intervention has no effect). The convention is to consider a probability of less than five per cent (p < .05) as an indication that the treatment or intervention really did have an influence; the null hypothesis can be rejected (this procedure is known as null hypothesis significance testing).

From the 36 journal issues Masicampo and Lalande identified 3,627 reported p values between .01 to .10 and their method was to see how evenly the p values were spread across that range (only studies that reported a precise figure were included). To avoid a bias in their approach, they counted the number of p values falling into "buckets" of different size, either .01, .005, .0025 or .00125 across the range.

The spread of p values between .01 and .10 followed an exponential curve - from .10 to .01 the number of p values increased gradually. But here's the key finding - there was a glaring bump in the distribution between .045 and .050. The number of p values falling in this range was "much greater" than you'd expect based on the frequency of p values falling elsewhere in the distribution. In other words, an uncanny abundance of reported results just sneaked into the region of statistical significance.

"Biases linked to achieving statistical significance appear to have a measurable impact on the research publication process," the researchers said.

[Coollabioto]

.045 and .050

pfft Fail

[trowUrillioth]

> But here's the key finding - there was a glaring bump in the distribution between .045 and .050.

You'd probably find the same for subatomic physics. Think of it this way, the probability of getting a result with high significance drops off rapidly as the p value decreases, and if the p value is greater than .05 then there's no point in publishing it, so publications are obviously going to have a peak at a p value of .045 to .05.

My objection to psychology findings is quite different.
1) Psychology findings on humans are based on surveys, and that's intrinsically unreliable for several reasons.
2) The subjects are never given the option to provide feedback on whether the survey questions are appropriate.
3) The subjects are lied to by the experimenter, but the subjects are expected to be truthful.
4) There's a bias towards choosing subjects who are university students and therefore in no way representative of the population as a whole.

[Yartonbler]

> But here's the key finding - there was a glaring bump in the distribution between .045 and .050.

You'd probably find the same for subatomic physics. Think of it this way, the probability of getting a result with high significance drops off rapidly as the p value decreases, and if the p value is greater than .05 then there's no point in publishing it, so publications are obviously going to have a peak at a p value of .045 to .05.

They found the probability of p values declined as expected as p went to zero except there was a spike of results just on the significant side of 0.05.

[sanddrareyk]

> My objection to psychology findings is quite different.
1) Psychology findings on humans are based on surveys, and that's intrinsically unreliable for several reasons.
2) The subjects are never given the option to provide feedback on whether the survey questions are appropriate.
3) The subjects are lied to by the experimenter, but the subjects are expected to be truthful.
4) There's a bias towards choosing subjects who are university students and therefore in no way representative of the population as a whole.

1) Any kind of data gathering is a "survey" of something. What alternative methods would you propose regarding a scientific methodology to psychology?

2) Such feedback would introduce variables to the study which are unrelated to the initial hypothesis. People often complain that "none of the options are exactly what I think", but they miss the point that phschological studies are not generally interested in 'what' they think, but rather 'how' they think ***. Psychology is not interested in Neilsen-poll-style opinions. In fact, psychological experiments are specifically designed to exclude confounding variables (of which there is many).

3) Is this an empirical objection, or a moral one? The "lies" are intended to control for variables which the study is not investigating. I don't really see the problem here.

4) A valid point, unfortunately somewhat unavoidable.

[Inettypofonee]

> 3) ...but the subjects are expected to be truthful.

Also, I think this is not the case at all. Psychological studies colate the answers of subjects. Only very specific studies seek to determine the "truthfulness" of those answers. Generally, psychologists expect that their subjects are engaged in some form of lie when they answer questions. The trick is to try to penetrate those lies by careful design of the survey - the very essence of experiment even in the harder sciences, IMO.