Author Topic: Ouchie for Social Sciences  (Read 3512 times)

0 Members and 1 Guest are viewing this topic.

Offline Mika

  • 28
Ouchie for Social Sciences
Did I mention I was last summer in a Social Sciences conference? Me, the hardcore physicist? Yes, I indeed was there. There were some good or bad reasons to attend that conference. I wouldn't call it complete waste of time but still.

My experience there was that it would help tremendously in Social Science studies if physicists and engineers would actually be involved in reviewing the actual measurement equipment before results are claimed and published. I alone pointed out several errors (and generally being the prick from the back row who always throws nasty questions at you) during the short presentations there. Some of them very fundamental to the systems they were using, such as not knowing measurement system limitations, and even more so on what it comes to sampling rates and Nyqvist frequencies. There were also even more fundamental problems not related to Physics, such as assuming that the people playing organs in the cathedrals would not know Bach's melodies from the notes alone(!).

Well, I'm not apparently alone with those experiences. Recently, couple of studies have been brought to my attention - first that a significant number of MRI discoveries have a significant probability of being false due to software problems. All results from the last 10-15 years are under question at this point.

Then we get a study where less than half of the experiments done in Psychology are not repeatable. That's worse than monkeys throwing coins, and one can only hope that these studies push the Social Science for the better and self-improvement. This indicates there has to be also a far more deeper underlying problem with insufficient reviewing (that's true for Physics too but not at this extent).

But seriously, I think this is a wake up call for all Social Sciences. Some of those results have pretty likely already been used in the decision making, so you'll have to improve if you would like to get avoided being labelled as the shills who are used to justify anything the decision makers do.

Feynmann has warned about this in the 1970s by his writing of the Cargo Cult Science, which describes a very special rat experiment at the end. This is worth reading for any scientist regardless of the field. He also mentions Social Sciences have already had this problem for 40 years now, and the culture does not seem to be improving.
Relaxed movement is always more effective than forced movement.

 

Offline Goober5000

  • HLP Loremaster
  • 214
    • Goober5000 Productions
Re: Ouchie for Social Sciences
These are excellent points, and the story of the rat experiment is both interesting and telling.  There is so much pressure these days toward finding unusual results, or to getting papers published, or to obtaining the next grant, that it creates a perverse incentive away from actually finding the truth.

 

Offline Mika

  • 28
Re: Ouchie for Social Sciences
That's the thing.

One would be well advised to remember this every time a new discovery is made in Social Sciences. The first question to ask is, has it been repeatable?

Natural Sciences and Engineering are sort of better protected from this, although they also suffer from the funding guidance by the results. But there's only that long you can hide the errors with the nature, but once humans get included, it becomes a statistical process instead of deterministic one, and the Science's self-correcting seems to get postponed way longer than in other fields of research.

Just to clear one thing though: I'm not saying that Social Sciences are not Sciences. I'm actually curious to hear about their results too. It's just that once politicians are using the results of a flawed scientific process - which affects Social Sciences the most currently - to give reasons why nations should do what they suggest (giving grounds to their policies via science), the scientific people involved should be very vocal about it.

From what I have seen, maintaining scientific integrity is very hard. I'm starting to think you need to be economically independent to do it.
Relaxed movement is always more effective than forced movement.

 

Offline CP5670

  • Dr. Evil
  • Global Moderator
  • 212
Re: Ouchie for Social Sciences
I've been to some economics and finance conferences (and come from a math and EE background originally). The papers and talks tend to be of two types, either model-based or empirical.

The modeling papers usually build up a framework from economic first principles, and can actually get very mathematically sophisticated and rigorous (sometimes more so than physics). However, they are ultimately designed to tell a story of some sort and the models are not verifiable in any way, so they feel disconnected from reality. Some of this work is actually done by former math and physics people, especially in finance, but that doesn't make it any better.

The empirical papers try to find some sort of causal relationship between things (say, education and wages) using simple regression-type analysis on some data, with conclusions ranging anywhere between obvious to ridiculous. There is a lot of misuse of basic statistics in these papers, especially because only the statistically significant results can get published, which means people tweak the data and try a lot of different things until they get such a result. There is also a trend today of papers using confidential data that nobody else has access to, such as wage data from the IRS. Many of the papers in the top journals like AER are like this, and it's impossible to verify the conclusions. To be fair though, this is a problem in many other fields too, and there is a movement among some journals to require that any data or code that produced the plots in the paper should be publicly available.

 

Offline Herra Tohtori

  • The Academic
  • 211
  • Bad command or file name
Re: Ouchie for Social Sciences
The modeling papers usually build up a framework from economic first principles, and can actually get very mathematically sophisticated and rigorous (sometimes more so than physics). However, they are ultimately designed to tell a story of some sort and the models are not verifiable in any way, so they feel disconnected from reality. Some of this work is actually done by former math and physics people, especially in finance, but that doesn't make it any better.

First, let us assume that horses are elliptical...

My impression is that these models are basically abstract, formal systems based on some premises (or axioms) constructed to model a real thing. However, if these models are consistent they can't be complete, and if they are complete they can't be consistent, so there you go - even the BEST of such models are not necessarily reliable or accurate.


Quote
The empirical papers try to find some sort of causal relationship between things (say, education and wages) using simple regression-type analysis on some data, with conclusions ranging anywhere between obvious to ridiculous. There is a lot of misuse of basic statistics in these papers, especially because only the statistically significant results can get published, which means people tweak the data and try a lot of different things until they get such a result. There is also a trend today of papers using confidential data that nobody else has access to, such as wage data from the IRS. Many of the papers in the top journals like AER are like this, and it's impossible to verify the conclusions. To be fair though, this is a problem in many other fields too, and there is a movement among some journals to require that any data or code that produced the plots in the paper should be publicly available.

An obvious problem with this approach is how to separate correlation from causation.

The second problem is eliminating bias, since setting up proper double blind trial with truly separate test group and control group in this field of study would be extremely difficult, not to mention ethically dubious.
There are three things that last forever: Abort, Retry, Fail - and the greatest of these is Fail.

 

Offline Bobboau

  • Just a MODern kinda guy
    Just MODerately cool
    And MODest too
  • 213
Re: Ouchie for Social Sciences
I love how so many of the most influential and foundational experiments ever performed are in modern contexts considered ethically indefensible.
Pavlov, Milgram, Stanford Prison. None of these would be allowed today, yet they were some of the things we learned the most from, or at the very least they had a huge impact, and we can't disprove them or further our understanding like we could then. and yes, I understand why.
Bobboau, bringing you products that work... in theory
learn to use PCS
creator of the ProXimus Procedural Texture and Effect Generator
My latest build of PCS2, get it while it's hot!
PCS 2.0.3


DEUTERONOMY 22:11
Thou shalt not wear a garment of diverse sorts, [as] of woollen and linen together

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Ouchie for Social Sciences
The Stanford Prison experiment was actually quite flawed. Funnily enough people willing to come in off the streets and work in a simulated prison aren't a representative sample.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline T-Man

  • 210
  • I came... I saw... I had a cuppa!
Re: Ouchie for Social Sciences
Thanks for sharing Mika great finds :yes:, particularly Feynman's scientific integrity way of putting it; it's a problem you could argue in a lot of science. People want 'research' but they don't want research that fails (I suppose understandably when it's their money getting spent and they need a return on it), yet that failure is vital to the research process. The funders need to be prepared for it to fail a lot and few are.

It reminds me of a Philosopher I came across and liked called Karl Popper who made a quite similar point (he called it Critical Rationalism); 'In science you shouldn't prove something directly as it's too easy to (willingly or unwillingly) misunderstand or fudge your research, you should prove it through disproving all other possibilities'. Such science brings integrity to the results, but requires a lot of money being spent for little visible progress, which investors hate, and without investors you can't do more research, so your forced to give what sells in the name of survival.
Also goes by 'Murasaki-Tatsu' outside of Hard-Light

UEF fanboy. Rabid Imagination.

 

Offline Bobboau

  • Just a MODern kinda guy
    Just MODerately cool
    And MODest too
  • 213
Re: Ouchie for Social Sciences
The Stanford Prison experiment was actually quite flawed. Funnily enough people willing to come in off the streets and work in a simulated prison aren't a representative sample.
and part of what I was talking about was that we cannot so easily prove that due to changes in what is considered ethical in more recent times. You don't dispute it was very influential, right?

BTW, I have no point I'm getting at here, just sort of an observation.
Bobboau, bringing you products that work... in theory
learn to use PCS
creator of the ProXimus Procedural Texture and Effect Generator
My latest build of PCS2, get it while it's hot!
PCS 2.0.3


DEUTERONOMY 22:11
Thou shalt not wear a garment of diverse sorts, [as] of woollen and linen together

 
Re: Ouchie for Social Sciences
The Stanford Prison experiment was actually quite flawed. Funnily enough people willing to come in off the streets and work in a simulated prison aren't a representative sample.
and part of what I was talking about was that we cannot so easily prove that due to changes in what is considered ethical in more recent times. You don't dispute it was very influential, right?

BTW, I have no point I'm getting at here, just sort of an observation.

The stanford experiment was considered unethical at the time it was done (it was stopped for that reason), and (also at the time) critique was made because it was unrepeatable, suffering heavily from selection bias and outside interference. It's very influential not because of it's results but because it's many issues inspired other researchers to take action preventing exactly that.

You can't do the stanford experiment anymore, but you can't because people at the time realized that you shouldn't have been able to do it then, and that the measures they had at the time did not prevent it from happening anyway.

 

Offline Luis Dias

  • 211
Re: Ouchie for Social Sciences
The notion that the stanford experiment was only influential viz a viz the methodologies used and not its conclusions is ridiculous.

 
Re: Ouchie for Social Sciences
The notion that the stanford experiment was only influential viz a viz the methodologies used and not its conclusions is ridiculous.

There is a difference between conclusions and results, which is the point I am trying to make here: The conclusions drawn from the Stanford experiment are drawn from why it had to be aborted long before it had even ended, becuase even the researchers themselves got too absorbed into their roles (which is a result of how the experiment was set up). As an experiment enacted according to the scientific method, it produced no viable results due to the massive biases involved and the researcher himself directly influencing the experiment beyond the initial setup.

This is important in both the context of the thread placing an heavy emphasis on the scientific method in itself and Bobbeau's claims that a repeat of the experiment is not possible due to more recent ethical changes.

It should be noted that an experiment that is somewhat similar to the Stanford experiment (known as the BBC Prison experiment) was conducted in 2002. In a similar sense, variants of the Milgram experiment have been performed: Here's an example in nature. I also distinctly remember that an example that I heard on dutch radio, a milgram style experiment where the particpants were asked to validate an unethical scientific study. Unfortunately I am having difficulty finding it as faking faking test results is too meta for google.
« Last Edit: December 19, 2016, 06:00:02 am by -Joshua- »

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Ouchie for Social Sciences
I'll assume BBC stands for the British Broadcasting Corporation and not as also might be relevant Big Black Cock. :p

Although if it did mean the latter it would completely blow Bobboau's theory out of the water.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 
Re: Ouchie for Social Sciences
I'll assume BBC stands for the British Broadcasting Corporation and not as also might be relevant Big Black Cock. :p

Although if it did mean the latter it would completely blow Bobboau's theory out of the water.

But that's the problem I have with Bobbeau's statements: In part because it attributes something which hasn't actually happened to something which hasn't actually changed, but mostly becuase it underestimates how complex these situations are. If you'd want to do a prison experiment with big black cocks we first need to objectively define when a cock is big, black, and a cock. When we have found our respective big black cocks we also need a control group (and what exactly is a normal cock?), we need to define the prisons... All that stuff is a rather complex situation where you have to define the one variable that is different and have to exclude all other variables. You'd have to look at the cocks themselves and try to figure out if they are not the result of overextensive breeding practices that happens so much in the industry... And there's probably a lot of things I am forgetting here. All that stuff is important and it was important even before the PETA and variants thereoff was something scientists needed to worry about.

(ALSO IT"S NOT A THEORY ITS A HYPOTHESIS)
« Last Edit: December 19, 2016, 06:30:44 am by -Joshua- »

 

Offline Luis Dias

  • 211
Re: Ouchie for Social Sciences
The notion that the stanford experiment was only influential viz a viz the methodologies used and not its conclusions is ridiculous.

There is a difference between conclusions and results, which is the point I am trying to make here: The conclusions drawn from the Stanford experiment are drawn from why it had to be aborted long before it had even ended, becuase even the researchers themselves got too absorbed into their roles (which is a result of how the experiment was set up). As an experiment enacted according to the scientific method, it produced no viable results due to the massive biases involved and the researcher himself directly influencing the experiment beyond the initial setup.

This famous experiment has influenced not only academia but the wider culture all around the world regarding the relations between individuals and power dynamics. All the scientific criticisms we can list towards the experiment are utterly irrelevant towards that first sentence.

That social science studies constantly fail at their methodologies is also a good reliable and repeatable finding in and of itself, but hey, that's social sciences for you.

 

Offline The E

  • He's Ebeneezer Goode
  • 213
  • Nothing personal, just tech support.
    • Steam
    • Twitter
Re: Ouchie for Social Sciences
This famous experiment has influenced not only academia but the wider culture all around the world regarding the relations between individuals and power dynamics. All the scientific criticisms we can list towards the experiment are utterly irrelevant towards that first sentence.

See also: the myth of the Alpha wolf.
If I'm just aching this can't go on
I came from chasing dreams to feel alone
There must be changes, miss to feel strong
I really need lifе to touch me
--Evergrey, Where August Mourns

 
Re: Ouchie for Social Sciences
Also also, consider Andrew Wakefield's experiments (the vaccines cause autism one) as an example where the impact of the test itself was completely askew compared to it's actual scientific value. It's results did not support the conclusion Wakefield eventually publicized.

And that thing got trough peer-review!

 

Offline CP5670

  • Dr. Evil
  • Global Moderator
  • 212
Re: Ouchie for Social Sciences
First, let us assume that horses are elliptical...

My impression is that these models are basically abstract, formal systems based on some premises (or axioms) constructed to model a real thing. However, if these models are consistent they can't be complete, and if they are complete they can't be consistent, so there you go - even the BEST of such models are not necessarily reliable or accurate.

Check out the work on DSGE models. They are stylized descriptions of various parts of the economy and how they interact, but become very complicated and elaborate. They cannot be solved exactly, so people make a lot of approximations (like linearizing around fixed points) to get exact answers, but that completely changes the properties of the model. It's like a wrong approximation to something that was wrong to begin with. :D These kinds of models are actually used at the Federal Reserve and ECB.

Quote
An obvious problem with this approach is how to separate correlation from causation.

The second problem is eliminating bias, since setting up proper double blind trial with truly separate test group and control group in this field of study would be extremely difficult, not to mention ethically dubious.

They try to control for various other quantities and include them in the regression, but you can only account for a few things. The main problem comes down to the fact that you can only publish the positive results and not the negative ones. There is a very high chance of getting false statistical significance if you do the same test many times on slight variations of the data, which is what everyone does.

That being said, I wouldn't beat up the social scientists too much because other fields have their problems too. It happens even in math to some extent, which is more objective and precise than any other field. I've occasionally gotten math papers to review that had 50+ pages of dense proofs, and although I could in principle go through it in detail and verify that everything is correct, there is no way I'm spending the time to do that (and do it for free). The ABC conjecture is an extreme example of this, where entire conferences have been devoted to just understand what one person did in his 500 page paper. :p

 
Re: Ouchie for Social Sciences
There's lots of lovely accounts in the lovely book "A short history of nearly everything" by Bill Bryson of geology and continental drift, and their (our?) difficulty with accepting that, which eventually involved drawing land bridges everywhere to explain why african and american plants and animals have remarkable similarities. I have a geology book from the eighties that barely acknowledges the theory, yet here we are: Continental drift is now the foundation on which geology is based, in the same way that the evolution theory is the foundation on which biology is based.

 

Offline Mika

  • 28
Re: Ouchie for Social Sciences
Quote
That being said, I wouldn't beat up the social scientists too much because other fields have their problems too. It happens even in math to some extent, which is more objective and precise than any other field. I've occasionally gotten math papers to review that had 50+ pages of dense proofs, and although I could in principle go through it in detail and verify that everything is correct, there is no way I'm spending the time to do that (and do it for free). The ABC conjecture is an extreme example of this, where entire conferences have been devoted to just understand what one person did in his 500 page paper.

Been there done that, amen! I once had to review a paper on my field of Physics coming from Italy. What was said in the abstract did not reflect at all what the actual content was. Their original manuscript was like 6 pages long. My review notes were 8 pages long, given the numerous errors, unsubstantiated claims and that the paper did not provide enough information for a repetition of the experiment. I believe I made a mistake in giving them a chance to improve it, in the hindsight I should have failed the paper immediately. It took a bloody weekend to go through that. My biggest gripe here is that I don't get paid anything for reviewing papers, but it is OK for the publishers to charge like $ 40 from a single download! :hopping: Lesson learned, I don't review publications any more - it was that one contribution of mine for betterment of human kind.

There's lots of lovely accounts in the lovely book "A short history of nearly everything" by Bill Bryson of geology and continental drift, and their (our?) difficulty with accepting that, which eventually involved drawing land bridges everywhere to explain why african and american plants and animals have remarkable similarities. I have a geology book from the eighties that barely acknowledges the theory, yet here we are: Continental drift is now the foundation on which geology is based, in the same way that the evolution theory is the foundation on which biology is based.

Incidentally, one of my colleagues lost the sense of touch from the other half of his face due to an acute nerve infection. Today he mentioned that doctors are saying that nerves do grow back, but it takes a lot of time for that to happen. So he can look forward of getting that sense back to his face in about half a year. But in the 1980s books when I attended primary school there was a clear distinction between the damage to the nerves and to pretty much everything else in the body. Nerve damage does not recuperate was the statement, and everyone of us listening recalled that. Apparently things have changed. Then again, the evidence has been there for some time. Some quadriplegics and paralyzed people have recovered, and that's not possible without some kind of recuperation process in the nerve system.

The thing with Maths and String Theory is that there are very few people who can understand what's being told. However, these studies are not affecting currently a lot of things. The same cannot unfortunately be said about the research of Social Sciences. I've said that I accept the now prevailing explanation why galaxies spin faster than they should, but I note as well that nobody has sent a satellite to another galaxy as of today. That single experiment alone, if possible with nowadays technology, could alone invalidate a bunch of work related to dynamics of galaxies. So I sort of remain sceptical of the astronomy outside our solar system. But they do get nice pictures of the faint galaxies, I'll have to give them that at least. The planetary discoveries are also racking up, and I tend to believe these are actual planets.

What it comes to the ethics of Science, guess what's the best driver for development of Medical Sciences?
Relaxed movement is always more effective than forced movement.