arXiv

[eirias]

Question for physicists and astronomers, and maybe mathematicians!

My super hip postdoc friend and I were talking about research ethics the other day, and lamenting the lack of a good way for psychologists to deal with null results -- we can't publish them, so we're stuck either making revisions to the study until it "works" (which is kind of sketchy), or abandoning the research question, presumably leaving that juicy plum to whichever lab is lucky enough to get a false positive result. ;P We thought it'd be nice to have a database where people could post Stuff that Didn't Work. But littlepurple counters that nobody would contribute to it, because nobody wants to "tip their hand" on the questions they're working on and the methods they're using to get there, not before stuff is published. Hmm.

Now, I know that astrophysicists and such post their work in arXiv, sometimes before formal publication elsewhere (right?). How does intellectual property work here? -- that's what we're really talking about, after all: being able to prove that the cool idea was yours first. Do people ever publish stuff with holes on arXiv, and then get scooped in print by someone who figured out how to fill the holes? Or is that not a sensible question in your field? I'm very curious about this -- a good chunk of my irritation with my current career track has to do with this problem and I'd love to figure out a good fix.

Comments

[cognative.livejournal.com]

re: null results.

I think the problem of null results is an extension of the main issue with behavioural psychology.
We don't know what the hell we're doing.

Given that, null results really are difficult to interpret. If we really *knew* that the task we designed actually did measure what we thought it did then null results would be interpretable.

Also there's the issue of sexy results being more important. Everyone wants to do sexy research and null results aren't usually sexy.

[ex-miang438.livejournal.com]

Well, there's this (http://www.jasnh.com/)... :D

The problem with publishing null results is that you have to be exceptionally certain that you were well-powered (which, I'm sorry, I don't think any study run in our building in the last few decades has been) and that there were no other factors (so-called third variables, imbalance, non-MCAR missing data) that would have biased your estimate downward. That's really, *really* hard to do.

[eirias.livejournal.com]

Oh, I know. It's not publishing practice that's the problem, per se, it's the entire epistemology of the field. Somehow that doesn't make me feel better, though...

[jpfed.livejournal.com]

Yes, it's irritating, but it also represents a tremendous opportunity for progress. That is, if the field isn't irretrievably mired in its own inertia.

It's not like we're the only people thinking these thoughts. Something's got to give, and when it does, things will get interesting.

As someone with only 2 years in a lab, I am still naive enough to think that psychology could be revolutionized if enough of its researchers explicitly rejected its epistemelogical traditions.

[eirias.livejournal.com]

As someone with only 2 years in a lab, I am still naive enough to think that psychology could be revolutionized if enough of its researchers explicitly rejected its epistemelogical traditions.

Who can afford to do that before tenure? Or, more realistically, before making full professor (http://www.bbsonline.org/Preprints/Ceci-04182005/Referees/Ceci-04182005_preprint.pdf)? And after that point, who is really going to bother? "By the way, the last twenty years of work I have done has been based on crappy assumptions, so please ignore it."

Point taken

[jpfed.livejournal.com]

Then progress will have to come from other areas beating psychology at its own game.

[jpfed.livejournal.com]

This was a big reason why I left AG's lab. The methods and theories were both so... shall we say, "flexible", that null results effectively "didn't count." There was no experiment airtight enough that a null result could strike a decisive blow.

[eirias.livejournal.com]

I took a look at the journal you linked to. It doesn't seem terribly popular, which may be support for littlepurple's conjecture that people do not actually want to publish things that didn't work.

The journal format is a little different from what I had in mind. I think there's a good reason for not wanting to go to the bother of writing up a full manuscript when a study didn't work: it takes a lot of time, and people are probably not actually going to read and cite you all that much if you didn't find a sexy result. I think it might be more useful to have something more informal, like a blog, with really brief summaries of what you did. I think allowing comments would be key -- part of the purpose would be to foster discussion of why a particular design might not have worked, and to allow people to chime in with things like "yeah, I totally couldn't replicate that either."

Power issues are important for sure, but ignored in cognitive stuff as far as I can tell. I confess I've been a little afraid of it, because it seems so arbitrary if you don't already have good data on what effect size you're looking for. In neuro studies, it seems that people worry about studies being *over*-powered more than anything else.

I did run a power analysis for my dissertation and it looks like I should have a power of 0.95 with N=75. That seems like a ridiculously small N to get that level of power, but I couldn't find an error anywhere. Like I said, nobody does prospective power analyses in my field, so I felt kind of at sea when DK asked for it, but he was right to do so.

[ursula]

People certainly post proofs with holes on the arxiv. They either patch them or withdraw the paper when the holes are pointed out. If it really is an awesome question, you're also allowed to publish conjectures.

I don't think a tenure committee would agree with me, but practically speaking an arxiv posting is publication-- I'm much more likely to read an article in its arxiv form than track down the officially published version.

[eirias.livejournal.com]

Hmm. I guess I was thinking less of publishing things with unknown holes, and more about publishing works in progress -- for instance "I tried this technique to prove Interesting Conjecture X, but it fell down here." I think there's probably no field that's less analogous to psychology than mathematics (even some of the humanities, like philosophy, are probably closer) so what your field does may not really be a good test case. After all, you don't have to worry about finding a good result by chance, because you're never in the situation of having to say "well, it works on fifty randomly selected integers, so I am going to argue that it works for all of them!"

[ursula]

It depends-- "This was true for the first N examples, so I'm going to conjecture it's always true" would be a valid result (assuming that proving it's always true is hard).

[ksledgemoore.livejournal.com]

I disagree with littlepurple. I think that people would publish stuff there. If you work on something, and it is null, and you mess around a bit, and it's still null, a lot of the time you just try to do something else altogether. You might even try (in vain) to publish these null results because they're actually interesting for whatever reason, but people reject your paper. Once you've abandoned that project, you're willing to let people know about it so they don't get dragged down the same path. I think it would also be a good place to publish failures to replicate, thereby more quickly exposing the many published false positives that are out there today.

[eirias.livejournal.com]

Well, cool :). It's nice to hear another psych person might find it useful.

I think I might add this project to the back burner, and keep my eye out for people to collaborate with -- or more realistically, other, more senior people who've had the idea and are looking for human capital to support it. It's the sort of project that might not take much time or money to maintain, but the social capital needed to make it work is kind of enormous. (On a sort of similar note, I wonder how Jonathan Baron is doing with that electronic journal he was trying to promote a couple years ago... he seems to have stopped updating his blog. Wasn't my field anyway but I thought it was a cool attempt.)

[rms10.livejournal.com]

The way arXiv has evolved in astrophysics -- I can't speak to any other subfields -- is that people post papers there that they intend to publish. It varies by sub-subfield if people post the paper when they submit it to a journal, or after it's been accepted. (It's generally a theory-observer breakdown, but some of the X-ray observers also post when they submit it.)

The argument for posting when you submit the paper is that then you might also get useful comments from others in your subfield, in addition to the referee report. The argument for waiting is that someone might read your paper in its preliminary form, and cite it without going back to check the final refereed version.

Either way, most papers are posted to arXiv before they're actually published, and so the status quo nowadays is to read astro-ph every day to keep up on the latest research. No one goes to ApJ on the first of every month (or whenever) to read the table of contents.

Because of that, people don't really use arXiv for brainstorming, or unfinished work. However, in astronomy null results do get published (I know someone whose thesis was a null result!) -- and if you can convince people that your work was careful, and it's a real null result, it can be a good thing for your career.

arXiv

[drspiff.livejournal.com]

As another career star-gazer I'll chime in on this thread.

I'll admit that my knee-jerk reaction to the question "how does arXiv work" is: not at all. But that's the cynical jaded side of me. The great thing about arXiv is that it is open and unmoderated. The bad thing about arXiv is that it is open and unmoderated.

Crack pots do post the intellectual equivalent of spam. It can make it difficult for someone who is less experienced to sort the wheat from the chaff.

The volume of posts are way too high. Keeping up with the daily email digest is like drinking from a fire hose. Of course, unfortunately, keeping up with the peer reviewed literature can be a similar experience.

Occasionally people post stuff on arXiv just to piss other people off. I've seen it get personal on occasion. But sometimes it's a useful political discussion. There are dueling political white papers on astro-ph as well as real science.

A good aspect of arXiv is that if, like me, your institution is too cheap to spring for a subscription to a journal then you can pull a copy of most papers off arXiv. This is particularly useful when it comes to meeting proceedings, which are not as widely available to people at Somesmallstate U.

People sometimes post to assert primacy on an idea. But if it comes down to that it gets ugly, fast. However we have to realize this has been happening for years. People will say something off hand at a meeting that ends up being the next big idea ten years down the road. Even though they're right, they sound like they're whining as they mumble over and over, "but I said it first!" Good ideas get a life of their own. People don't get credit for good ideas. They get credit for what they do with good ideas. That may not sound ethical or just but it is the way it is. This is the current of reality that I swim against every day. It's also why I tell my students that how you communicate and sell your ideas are just as important as the idea.

At one point in the past there were some April Fools pranksters who would put great effort into some first class crank papers every year on arXiv. Unfortunately those people grew up and stopped. But the potential is there to do it. I lament this in the way that I lament the American Astronomical Society cracking down on the crank-posters at the meetings. My favorite ever: the "Super Huge Interferometric Telescope (SHIT)"

For the record I don't post to arXiv until my paper has passed a referee. If I got a paper to referee that had already appeared on astro-ph, I'd be unhappy because posting to arXiv short-circuits the referee process. Unhappy referees don't need much of an excuse to hold up a paper. If you want to publish unrefereed then fine, do it. But you can't publish the same paper both ways. You're either in or out. In the middle go squash like a grape.

Re: arXiv

[rms10.livejournal.com]

Alas, they cracked down on all pranksters several years ago -- we submitted (http://www.astro.lsa.umich.edu/~grad/astro-ph/) something before the Michigan-Ohio State game a few years ago, and it would have been first on the mailing that went out on the Thursday night before the game. But alas, it was removed before the mailing went out, and our submission email address was banned from astro-ph.

A few null result journals

[(Anonymous)]

Journal of Negative Results in Biomedicine
(also for clinical trials)

The Journal of Spurious Correlations
(more social sciency)

Even though these are "well read" journals, it means your research would appear in literature searches. And, yes, it is something to add to the "peer-reviewed" section of your CV.

-janel