Tuesday, November 24, 2009

Advice to the AGU regarding their journals

I intend to write a proper letter (well email) based on the following thoughts. Before I send it, I'd be grateful for any suggestions.

Dear AGU,

Your previously-respected journals are starting to build a reputation for publishing rubbish papers. Here are a few suggestions for how to remedy the situation.

1. Please advise your journal editors to get off their backsides and try to find at least one reviewer outside of the list of suggestions that you insist all authors provide. I'm sure some conscientious editors already do this, but not all. Under the current system all one has to do in order to publish complete crap is nominate a couple of friends who will wave things through the review process.

2. Please fix your Comment and Reply system. You already have a published policy, please ask your editors to adhere to it. Currently, it seems common for editors to impose an additional "pre-review" stage before even starting the process of asking for a Reply. This means that while any old crap can get published in a few weeks on the say-so of a couple of friends (see 1), even in the best case scenario it takes up to a year, and about 6 reviews (including those nominated by the authors of the original work), for anyone to have any chance of pointing out the problems however glaring they are. Comments are generally relatively urgent in nature and are required to be short by your policies, so a time scale rather closer to that of GRL would seem more appropriate. From what I have experienced and heard from others, editors not infrequently decide that they simply can't be bothered with dealing with comments regardless of their content or validity. If you don't want to deal with comments pointing out errors in misleading and shoddy work, you should work harder to prevent the publication of the erroneous work in the first place rather than blocking any criticism!

3. Please let us know which editor is responsible for each paper. There is no reason for this information to be secret, and EGU journals routinely publish it. I'm sure many hard-working and conscientious editors are upset that the reputation of the journals is being tarred by the actions of a few. Being held to account even in this small way may encourage people to be a little more careful.

4. Please consider introducing a meaningful open review system, like the one in place at many EGU journals. I'm aware of your plans to take baby steps in this direction, but it seems that you are trying to make it as ineffective as possible. Rather than creating some half-hearted process, why not just simply copy one that already works pretty well? (If you feel the need to differentiate yourselves from the EGU, an obvious improvement on their process would be to post all the invited reviews together, rather than letting the later reviewers read the earliest ones.) I'm not aware of any obviously crap papers being published in the EGU journals that practice open review, and their reputation and status is rising as rapidly as yours is falling. Although I'm personally a fan of the EGU open publication system, I would much rather see the AGU journals as a credible and authoritative alternative, rather than sinking into oblivion.

7 comments:

Dan Hughes said...

June 2, 2005 Dan Hughes 324 Plank Road Porter Corners, NY 12859


Journals Editor American Geophysical Union (AGU) 2000 Florida Avenue
N.W. Washington, DC 20009-1277 USA

Subject: Verification and Validation of Computer Codes used as
Basis for Paper Submittals

Dear Editor:

Paper submittals the basis of which are calculations with computer
software continue to be important sources of publications in your
AGU journals. Some of these papers might at some time be used as
audit/checkpoints/benchmarks for other calculations, and other
papers might become part of the basis for public policy decisions.
The software used for these papers needs to be verified and
validated for the applications to which they are applied. Papers
for which the software has not been verified should not be accepted
for publication in archival journals such as those published by the
AGU.

The first crucial aspect of such papers should be the status of the
verification of the software. Verification is the process of
determining that the coded equations are correctly solved.
Editorial boards of several
engineering society journal have
recently put into place technical requirements on the verification
of the software before the paper can be considered for publication.
If the requirements have not been met the paper will not be
published; in some cases the paper will be rejected out-of-hand and
not be sent out for review.

Some of these professional organizations and associated journals
include: The American Society of Mechanical Engineers (ASME) Journal
of Heat Transfer and Journal of Fluids Engineering; The American
Institute of Aerospace and Astronautics (AIAA) Journal of Spacecraft
and Rockets; and the International Journal of Numerical Methods for
Fluid Flow.

References for the editorial polices for these journals are as
follows.

The ASME Journal of Heat Transfer: Editorial Board, “Journal of Heat
Transfer Editorial Policy Statement on Numerical Accuracy,” ASME
Journal of Heat Transfer, Vol. 116, pp. 797-798, 1994.

The ASME Journal of Fluids Engineering: C. J. Freitas, “Editorial
Policy Statement on the Control of Numerical Accuracy,” ASME Journal
of Fluids Engineering, Vol. 117, No. 1, p. 9, 1995
http://www.asme.org/pubs/journals/fluideng/JFENumAccuracy.pdf.

The AIAA Journal of Spacecraft and Rockets: AIAA, Editorial Policy
Statement on Numerical Accuracy and Experimental Uncertainty, AIAA
Journal, Vol. 32, No. 1, p. 3, 1994.

The International Journal of Numerical Methods in Fluids: P. M.
Gresho and C. Taylor, “Editorial,” International Journal of
Numerical Methods in Fluids, Vol. 19, p. iii, 1994.

I have been unsuccessful in locating an editorial policy for AGU
journals on this important issue. If one exists will you kindly let
me know how I can access it. If a policy does not exist do you have
plans to address this issue? Complete and detailed discussion of
all aspects of verification, validation and quality assurance for
scientific and engineering software are given by Roache in the book,
“Verification and Validation in Computational Science and
Engineering,” published by Hermosa Press and a paper by Oberkampf,
et al., “Verification, Validation, and Predictive Capability in
Computational Engineering and Physics,” Sandia National Laboratories
Report SAND 2003-3769, http://www.prod.sandia.
gov/cgi-bin/techlib/access control.pl/2002/023769.pdf, 2003. These
publications contain extensive reference citations to the published
literature on these subjects.

I suggest that the editorial board initiate discussions of the issue and begin planning for
implementation of a sound basis for review and acceptance of papers
the basis of which are numerical calculations with computer
software.

Thank you for your attention to this matter.


Dan Hughes

Deep Climate said...

I'd fix the title (Typo - "regading")

On the letter, I would tone down the rhetoric, e.g. "woefully sub-standard" instead of "rubbish"

Item 1: This should be an ironclad rule, not an exhortation of best effort. And at least one editor-nominated reviewer should be someone who has published an oft-cited work on the same subject. Example: Tenbreth (I think) with his work on ENSO-temperature relationship would have been appropriate for reviewing McLean et al in JGR.

Item 2: An example might be helpful. Whatever happened to Foster et al comment on McLean et al, for example?

Item 3 and 4: Absolutely!!

I also think there should be an Item 5, specifying a code of conduct for authors post-publication. All PR professionals working with the authors should be members of accredited public relations organizations and adhere to a professional code of conduct. The authors should also engage not to issue or approve misleading statements concerning their work. It might even be appropriate that all official press releases approved by the authors also be reviewed by the editor and be posted on the journal website, together with comment by the editor, if any.

Authors should also engage not to release private emails concerning their work, unless they have explicit permission to do so.

I realize this shouldn't be necessary, but the behaviour of authors like McLean et al and Roger Pielke jr needs to be reined in.

William M. Connolley said...

I advise you to learn how to spell "regarding" :-))) and, with some reluctance, I advise you to drop the rude words like "crap", because that is just a convenient excuse for them to bin it.

If you felt like a jolly jape, you could try getting someone like RP Jr to co-sign it.

It would be better with some examples of poor quality papers though.

Otherwise, it looks great.

James Annan said...

Regarding the tone and wording, I wasn't really going to write it like that :-)

Tom C said...

Telling, isn't it, that your co-religionists Belette and Deep Climate actually thought you were going to send it in with those flaming words. That is plausible climate science professional communication, apparently.

I'd really like you to answer the questions I posed on the other thread. Do you understand the upside-down issue regarding the Finnish lake sediments and what do you think of Mann's refusal to address it?

David B. Benson said...

Point three is probably the most important; nothing like having one's name following the words "communicated by..." to produce higher standards.

Point one is, of course, crucial. Either the handling editor knows of an appropriate "outside" reviewer or else should pass the paper to another editor, IMO. That, obviously, has to be made completely clear to the editorial board, but implementing point three will certainly tend to inforce it!

Hank Roberts said...

http://scienceblogs.com/drugmonkey/2009/11/a_videographic_primer_on_how_t.php
A videographic primer on how to respond to reviewer comments