Peer-Review Under Review
in [Relativity]
|
From: "Juan R." González-Álvarez on 10 Apr 2010 07:50 paparios(a)gmail.com wrote on Fri, 09 Apr 2010 11:12:39 -0700: > On 9 abr, 12:04, "Juan R." González-Álvarez > <nowh...(a)canonicalscience.com> wrote: >> papar...(a)gmail.com wrote on Fri, 09 Apr 2010 05:39:27 -0700: > > >> > * 61% of reviewers have rejected an invitation to >> > review an >> > article in the last year, citing lack of expertise as the main reason >> > - this suggests that journals could better identify suitable >> > reviewers. >> >> How? Moreover, I fail to see a problem here. >> >> > Well, the problem is that in certain areas of knowledge, the degree of > specialization is such that only those reviewers working in the same > and, sometimes, quite narrow subject can even understand the used > terminology. This, of course, may not be the area of expertise of the > editor assigning the papers to reviewers and, so, the paper may be sent > to improper reviewers. I understand you. It is a problem for editors in the sense that they may improve their reviewer's databases or ask for advice (sometimes authors are invited to suggest potential reviewers). But what I mean is that I fail to see a serious problem here. If a reviewer reject to review a paper, editor only need to select other. I think it is more work for the editor but not a serious problem for peer-review. I think that other problems are much more important. >> > 3. High expectations: >> > * 79% or more of researchers think that peer review >> > should >> > identify the best papers, determine their originality and importance, >> > improve those papers and, though lower scoring, also determine >> > whether research is plagiarised or fraudulent. >> >> But objective data proves just the contrary all the best papers have >> been sistematically underevaluated by peer-review. By best papers I >> mean revolutionary papers that were rejected but finally awarded by a >> Nobel Prize. I gave a list of 34 of those, with authors explaining how >> their nobel awarded work was rejected by the 'experts'. >> >> > I agree that doing that is quite difficult. Doing so would require the > reviewers to be of the same, or similar level as the authors of those > best papers. For example, Albert Einstein's 1905 papers in Annalen der > Physik were reviewed by the journal's editor in chief, Max Planck (the > father of quantum theory), and its co-editor, Wilhelm Wien, which were > clearly peers (both won Nobel prizes in physics) and quite able to > understand the relevance of Einstein's writings. Hum. I think that example is not so good. Annalen der Physik had not a formal peer-review system, but editors accepting/rejecting papers in terms of aceptability to journal scope. The first time that Einstein submitted a paper to a peer-reviewed journal (Physical Review) was in 1936. Einstein was so perplexed by the process that he subbmited the next letter to the editor: "Dear Sir, We (Mr. Rosen and I) had sent you our manuscript for publication and had not authorized you to show it to specialists before it is printed. I see no reason to address the in any case erroneous comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere. Respectfully, P.S. Mr. Rosen, who has left for the Soviet Union, has authorized me to represent him in this matter." It is funny. Moreover, it is very relevant that the anonymous expert was *right* and Einstein *wrong* when claimed that gravitational waves do not exist. Now his wrong paper with Rosen is considered one of the greatest mistakes of Einstein. According to Silvan Schweber: "By 1930, every European scientific journal would automatically accept and publish any paper that Einstein had submitted." In a recent editorial was written how in 1905 almost any paper was accepted: http://www.nature.com/nature/journal/v425/n6959/full/425645a.html "This is strikingly reminiscent of perhaps the most celebrated editorial judgements of all, in Annalen der Physik in1 905. That was the year in which Einstein published five extraordinary papers in that journal, including special relativity and the photoelectric effect. The journal had a great editor in Max Planck. He recognized the virtue of publishing such outlandish ideas, but there was also a policy that allowed authors much latitude after their first publication. Indeed, in journals in those days, the burden of proof was generally on the opponents rather than the proponents of new ideas. >> > * While 43% of respondents thought peer review was too >> > slow, >> > 65% of authors (a further sub-group) reported that they had received >> > a decision on their most recent paper within 3 months. >> >> I have absolutely no idea about how improve this point. > > I agree it is difficult. One additional difficulty has to do with the > ever increasing production of papers, which overload most of journals. A naive solution could be to increase number of journals, but then this would give serious economic problems for libraries if they want to subscribe to all of them... > Some of them have to, from time to time, publish special issues in order > to limit the delay. I have in my shelves one of these issues with over > 1800 pages published in only one month. That journal only accepts on the > order of 10% of the submissions. > > >> > 4. Reviewers want anonymity: 58% would be less likely to review >> > if >> > their signed report was published. 76% favour the double blind system >> > where just the editor knows who the reviewers are. >> >> Could this imply that 58% of reviewers are abusing from anonimity and >> want to continue to do it? >> >> > I do not see how this could be possible. As the survey reports, the > number of papers per reviewer is quite low (I may review 2 or three > papers per year per journal). So the reviewer impact in this sense is > quite small. > >> Could this imply that 58% of reviewers do not want their names to be >> associated to the papers they accept (maybe because wannot be >> associated to frauds as Schön scandal)? >> >> Could this imply that 58% of reviewers like anonimity to rejected the >> papers of rival theories or rival funding projects? >> >> > Again this is quite impossible in practical terms. >> >> Could this imply that 58% of reviewers like anonimity to accept the >> papers of like theories or friends? >> >> > Again quite improbable. You could instead argue that some editor of a > journal, who you know, as a personal favor could give you a little help > by assigning your paper to more forgiving reviewers. Again I doubt that > has any real impact. > >> I don't know. >> >> However, BMJ reports that when they took the decision to apply an >> improved peer-review system where reviewers are not anonymous, almost >> all the reviewers accepted the new system and continued to review and >> only some few of them resigned. >> >> > My position is neutral with respect to this issue. I do not care if my > reviews are known to the authors. One of the journals I review has a > kind of mixed system, where I can write some specific recommendations to > the authors and/or write some comments only to the editor of the > journal. > >> > 5. Understanding of peer review: Researchers agree that peer >> > review >> > is well understood by the scientific community but just 30% believe >> > the public understands the term. >> >> > 6. Papers aren't recognising previous work: 81% think peer >> > review >> > should ensure previous research is acknowledged; >> >> Do you mean priority issues? >> >> > I think they refer to authors not properly acknowledging the work of > previous authors, on the same subject, which in many cases could lead to > repeat the same research. > >> > 54% think it currently >> > does. This reflects current concerns in the research community. >> > 7. Detecting plagiarism and fraud might be a noble aim but is >> > not >> > practical: A majority think peer review should detect plagiarism >> > (81%) or fraud (79%) but fewer (38% and 33%) think it is capable of >> > this. >> >> I think that the problem with most of current peer-review is that it is >> merely a review of the paper from a print-point of view. >> >> In a review from a datument point of view, referees could ask for extra >> info as raw data and perform extra checks that would detect fraud as >> that of the Schön scandal. >> >> > I doubt most of reviewers are willing to waste more valuable time on > that line. Asking for raw data would detect the fraud in the case of the Schön scandal because he had not the data, he had just invented the figures. > Miguel Rios -- http://www.canonicalscience.org/ BLOG: http://www.canonicalscience.org/publications/canonicalsciencetoday/canonicalsciencetoday.html
From: maxwell on 11 Apr 2010 13:44 On Apr 9, 11:12 am, "papar...(a)gmail.com" <papar...(a)gmail.com> wrote: > On 9 abr, 12:04, "Juan R." González-Álvarez > > <nowh...(a)canonicalscience.com> wrote: > > papar...(a)gmail.com wrote on Fri, 09 Apr 2010 05:39:27 -0700: > > > * 61% of reviewers have rejected an invitation to review an > > > article in the last year, citing lack of expertise as the main reason - > > > this suggests that journals could better identify suitable reviewers. > > > How? Moreover, I fail to see a problem here. > > Well, the problem is that in certain areas of knowledge, the degree of > specialization is such that only those reviewers working in the same > and, sometimes, quite narrow subject can even understand the used > terminology. This, of course, may not be the area of expertise of the > editor assigning the papers to reviewers and, so, the paper may be > sent to improper reviewers. > > > > 3. High expectations: > > > * 79% or more of researchers think that peer review should > > > identify the best papers, determine their originality and importance, > > > improve those papers and, though lower scoring, also determine whether > > > research is plagiarised or fraudulent. > > > But objective data proves just the contrary all the best papers have been > > sistematically underevaluated by peer-review. By best papers I mean > > revolutionary papers that were rejected but finally awarded by a Nobel Prize. > > I gave a list of 34 of those, with authors explaining how their nobel awarded > > work was rejected by the 'experts'. > > I agree that doing that is quite difficult. Doing so would require the > reviewers to be of the same, or similar level as the authors of those > best papers. For example, Albert Einstein's 1905 papers in Annalen der > Physik were reviewed by the journal's editor in chief, Max Planck (the > father of quantum theory), and its co-editor, Wilhelm Wien, which were > clearly peers (both won Nobel prizes in physics) and quite able to > understand the relevance of Einstein's writings. > > > > * While 43% of respondents thought peer review was too slow, > > > 65% of authors (a further sub-group) reported that they had received a > > > decision on their most recent paper within 3 months. > > > I have absolutely no idea about how improve this point. > > I agree it is difficult. One additional difficulty has to do with the > ever increasing production of papers, which overload most of journals. > Some of them have to, from time to time, publish special issues in > order to limit the delay. I have in my shelves one of these issues > with over 1800 pages published in only one month. > That journal only accepts on the order of 10% of the submissions. > > > > > > 4. Reviewers want anonymity: 58% would be less likely to review if > > > their signed report was published. 76% favour the double blind system > > > where just the editor knows who the reviewers are. > > > Could this imply that 58% of reviewers are abusing from anonimity and want > > to continue to do it? > > I do not see how this could be possible. As the survey reports, the > number of papers per reviewer is quite low (I may review 2 or three > papers per year per journal). So the reviewer impact in this sense is > quite small. > > > Could this imply that 58% of reviewers do not want their names to be > > associated to the papers they accept (maybe because wannot be associated > > to frauds as Schön scandal)? > > > Could this imply that 58% of reviewers like anonimity to rejected the papers > > of rival theories or rival funding projects? > > Again this is quite impossible in practical terms. > > > Could this imply that 58% of reviewers like anonimity to accept the papers > > of like theories or friends? > > Again quite improbable. You could instead argue that some editor of a > journal, who you know, as a personal favor could give you a little > help by assigning your paper to more forgiving reviewers. Again I > doubt that has any real impact. > > > I don't know. > > > However, BMJ reports that when they took the decision to apply an > > improved peer-review system where reviewers are not anonymous, almost > > all the reviewers accepted the new system and continued to review and only > > some few of them resigned. > > My position is neutral with respect to this issue. I do not care if my > reviews are known to the authors. One of the journals I review has a > kind of mixed system, where I can write some specific recommendations > to the authors and/or write some comments only to the editor of the > journal. > > > > 5. Understanding of peer review: Researchers agree that peer review > > > is well understood by the scientific community but just 30% believe the > > > public understands the term. > > > > 6. Papers aren't recognising previous work: 81% think peer review > > > should ensure previous research is acknowledged; > > > Do you mean priority issues? > > I think they refer to authors not properly acknowledging the work of > previous authors, on the same subject, which in many cases could lead > to repeat the same research. > > > > 54% think it currently > > > does. This reflects current concerns in the research community. > > > 7. Detecting plagiarism and fraud might be a noble aim but is not > > > practical: A majority think peer review should detect plagiarism (81%) > > > or fraud (79%) but fewer (38% and 33%) think it is capable of this. > > > I think that the problem with most of current peer-review is that it is > > merely a review of the paper from a print-point of view. > > > In a review from a datument point of view, referees could ask for extra > > info as raw data and perform extra checks that would detect fraud as that > > of the Schön scandal. > > I doubt most of reviewers are willing to waste more valuable time on > that line. > > Miguel Rios Einstein's famous 1905 papers were NOT peer reviewed by Planck or Wien: they were not reviewed at all! The procedure in German physics at that time was that an unknown author's FIRST paper only was reviewed (to ensure his competence). After he had jumped that hurdle, an honor system was used, trusting gentlemen to only submit professional quality material. Quality was the professional measure - not quantity of papers published. It is the greed, search for fame, professional jealousies etc that have arisen in the explosion of 20th century science that has led to the need for peer-review.
From: "Juan R." González-Álvarez on 12 Apr 2010 05:06 maxwell wrote on Sun, 11 Apr 2010 10:44:59 -0700: > On Apr 9, 11:12 am, "papar...(a)gmail.com" <papar...(a)gmail.com> wrote: >> On 9 abr, 12:04, "Juan R." González-Álvarez >> >> <nowh...(a)canonicalscience.com> wrote: >> > papar...(a)gmail.com wrote on Fri, 09 Apr 2010 05:39:27 -0700: >> > > * 61% of reviewers have rejected an invitation to >> > > review an >> > > article in the last year, citing lack of expertise as the main >> > > reason - this suggests that journals could better identify suitable >> > > reviewers. >> >> > How? Moreover, I fail to see a problem here. >> >> Well, the problem is that in certain areas of knowledge, the degree of >> specialization is such that only those reviewers working in the same >> and, sometimes, quite narrow subject can even understand the used >> terminology. This, of course, may not be the area of expertise of the >> editor assigning the papers to reviewers and, so, the paper may be sent >> to improper reviewers. >> >> > > 3. High expectations: >> > > * 79% or more of researchers think that peer review >> > > should >> > > identify the best papers, determine their originality and >> > > importance, improve those papers and, though lower scoring, also >> > > determine whether research is plagiarised or fraudulent. >> >> > But objective data proves just the contrary all the best papers have >> > been sistematically underevaluated by peer-review. By best papers I >> > mean revolutionary papers that were rejected but finally awarded by a >> > Nobel Prize. I gave a list of 34 of those, with authors explaining >> > how their nobel awarded work was rejected by the 'experts'. >> >> I agree that doing that is quite difficult. Doing so would require the >> reviewers to be of the same, or similar level as the authors of those >> best papers. For example, Albert Einstein's 1905 papers in Annalen der >> Physik were reviewed by the journal's editor in chief, Max Planck (the >> father of quantum theory), and its co-editor, Wilhelm Wien, which were >> clearly peers (both won Nobel prizes in physics) and quite able to >> understand the relevance of Einstein's writings. >> >> > > * While 43% of respondents thought peer review was >> > > too slow, >> > > 65% of authors (a further sub-group) reported that they had >> > > received a decision on their most recent paper within 3 months. >> >> > I have absolutely no idea about how improve this point. >> >> I agree it is difficult. One additional difficulty has to do with the >> ever increasing production of papers, which overload most of journals. >> Some of them have to, from time to time, publish special issues in >> order to limit the delay. I have in my shelves one of these issues with >> over 1800 pages published in only one month. That journal only accepts >> on the order of 10% of the submissions. >> >> >> >> > > 4. Reviewers want anonymity: 58% would be less likely to >> > > review if >> > > their signed report was published. 76% favour the double blind >> > > system where just the editor knows who the reviewers are. >> >> > Could this imply that 58% of reviewers are abusing from anonimity and >> > want to continue to do it? >> >> I do not see how this could be possible. As the survey reports, the >> number of papers per reviewer is quite low (I may review 2 or three >> papers per year per journal). So the reviewer impact in this sense is >> quite small. >> >> > Could this imply that 58% of reviewers do not want their names to be >> > associated to the papers they accept (maybe because wannot be >> > associated to frauds as Schön scandal)? >> >> > Could this imply that 58% of reviewers like anonimity to rejected the >> > papers of rival theories or rival funding projects? >> >> Again this is quite impossible in practical terms. >> >> > Could this imply that 58% of reviewers like anonimity to accept the >> > papers of like theories or friends? >> >> Again quite improbable. You could instead argue that some editor of a >> journal, who you know, as a personal favor could give you a little help >> by assigning your paper to more forgiving reviewers. Again I doubt that >> has any real impact. >> >> > I don't know. >> >> > However, BMJ reports that when they took the decision to apply an >> > improved peer-review system where reviewers are not anonymous, almost >> > all the reviewers accepted the new system and continued to review and >> > only some few of them resigned. >> >> My position is neutral with respect to this issue. I do not care if my >> reviews are known to the authors. One of the journals I review has a >> kind of mixed system, where I can write some specific recommendations >> to the authors and/or write some comments only to the editor of the >> journal. >> >> > > 5. Understanding of peer review: Researchers agree that peer >> > > review >> > > is well understood by the scientific community but just 30% believe >> > > the public understands the term. >> >> > > 6. Papers aren't recognising previous work: 81% think peer >> > > review >> > > should ensure previous research is acknowledged; >> >> > Do you mean priority issues? >> >> I think they refer to authors not properly acknowledging the work of >> previous authors, on the same subject, which in many cases could lead >> to repeat the same research. >> >> > > 54% think it currently >> > > does. This reflects current concerns in the research community. >> > > 7. Detecting plagiarism and fraud might be a noble aim but is >> > > not >> > > practical: A majority think peer review should detect plagiarism >> > > (81%) or fraud (79%) but fewer (38% and 33%) think it is capable of >> > > this. >> >> > I think that the problem with most of current peer-review is that it >> > is merely a review of the paper from a print-point of view. >> >> > In a review from a datument point of view, referees could ask for >> > extra info as raw data and perform extra checks that would detect >> > fraud as that of the Schön scandal. >> >> I doubt most of reviewers are willing to waste more valuable time on >> that line. >> >> Miguel Rios > > Einstein's famous 1905 papers were NOT peer reviewed by Planck or Wien: > they were not reviewed at all! The procedure in German physics at that > time was that an unknown author's FIRST paper only was reviewed (to > ensure his competence). After he had jumped that hurdle, an honor system > was used, trusting gentlemen to only submit professional quality > material. Quality was the professional measure - not quantity of papers > published. Thanks by confirming this point! "The journal had a great editor in Max Planck. He recognized the virtue of publishing such outlandish ideas, but there was also a policy that allowed authors much latitude AFTER their FIRST publication. Indeed, in journals in those days, the burden of proof was generally on the opponents rather than the proponents of new ideas. > It is the greed, search for fame, professional jealousies etc that have > arisen in the explosion of 20th century science that has led to the need > for peer-review. I think that peer-review has an important role when *properly done*. But I agree, as pointed in [6], that peer-review is commonly being (mis)used today as a dishonest tool to force ortodoxy and to protect careers and fame. [6] http://canonicalscience.org/publications/canonicalsciencereports/20082.html -- http://www.canonicalscience.org/ BLOG: http://www.canonicalscience.org/publications/canonicalsciencetoday/canonicalsciencetoday.html
From: harald on 12 Apr 2010 07:23 On Apr 4, 1:37 pm, "Juan R." González-Álvarez <nowh...(a)canonicalscience.com> wrote: > harald wrote on Sat, 03 Apr 2010 15:40:11 -0700: > > > > > On Apr 2, 5:27 pm, "Juan R." González-Álvarez > > <nowh...(a)canonicalscience.com> wrote: > >> Wendy Warr, an associate editor for ACS, gave a bleak and blistering > >> critique on the current state of peer-review at the recent ACS > >> National. > > >> Problems with Peer-Review: > > >> * It can delay publications for months. * An editor can make or > >> break a paper by sending it to the author's > >> friends or competitors. > >> * Historically biased against women, single authors, etc... * It > >> costs reviewers time (she gave a statistic that 41% of > >> reviewers would like to be paid). > >> * Reviewers tend to favor conservative science and not far-out new > >> ideas. > >> * Difficult finding qualified reviewers for multidisciplinary work.. > >> * Basing the quality of a paper on 2 reviewers, basically just > >> 2-data points, is statistically insignificant. > >> * As more papers are being submitted the burden for reviewers is > >> increasing. > > >> She forgot an important problem: peer-review is useless at detecting > >> fraud. For instance all the papers involved in the Schön scandal > >> (considered the "Biggest Fraud in Physics") were peer-reviewed. > > >> Warr did not give many solutions to these problems. A set of solutions > >> to the problems of peer-review is given in the report "Science in the > >> 21st century: social, political, and economic issues". > > >> Additional info and references are given in Mitch blog entry: "ACS > >> Day 4: Peer-Review Reviewed" > > >> You can state your opinion, share your experience, submit your own > >> proposal for solving the problems, etc. > > > [..] > > > Here's another one that says it all - (and yes this is the 21st > > century!): > > > "Alternative Peer Review: > > Quality Management for 21st Century Scholarship" > > >www.public.iastate.edu/~gerrymck/APR-1.ppt > > > Harald > > Very informative! I am glad that confirms manyo of the fingings in > my report (the author also chose the Schön scandal as illustration > of fraud in physics!). My own proposal to improve Peer-Review is a kind > of mixture of (Slide 43) "Open Peer Review" plus "Commentary-based" Especially "commentary-based" seems to be a good option (slide 53). Note: if you decide to submit your own papers to such a type of review, you should be prepared for a more public exposure to negative criticism... > Please could you post your message in the blog > > http://www.canonicalscience.org/publications/canonicalsciencetoday/ca... > > for that your useful reference can remain there archived? > > Thanks. "comments closed" ... Harald > --http://www.canonicalscience.org/ > > BLOG:http://www.canonicalscience.org/publications/canonicalsciencetoday/ca...
From: harald on 12 Apr 2010 07:26
On Apr 4, 8:39 pm, "Ken S. Tucker" <dynam...(a)vianet.on.ca> wrote: > On Apr 3, 3:40 pm, harald <h...(a)swissonline.ch> wrote: > > > > > On Apr 2, 5:27 pm, "Juan R." González-Álvarez > > > <nowh...(a)canonicalscience.com> wrote: > > > Wendy Warr, an associate editor for ACS, gave a bleak and blistering > > > critique on the current state of peer-review at the recent ACS National. > > > > Problems with Peer-Review: > > > > * It can delay publications for months. * An editor can make or break a > > > paper by sending it to the author's > > > friends or competitors. > > > * Historically biased against women, single authors, etc... * It costs > > > reviewers time (she gave a statistic that 41% of > > > reviewers would like to be paid). > > > * Reviewers tend to favor conservative science and not far-out new > > > ideas. > > > * Difficult finding qualified reviewers for multidisciplinary work. * > > > Basing the quality of a paper on 2 reviewers, basically just > > > 2-data points, is statistically insignificant. > > > * As more papers are being submitted the burden for reviewers is > > > increasing. > > > > She forgot an important problem: peer-review is useless at detecting > > > fraud. For instance all the papers involved in the Schön scandal > > > (considered the "Biggest Fraud in Physics") were peer-reviewed. > > > > Warr did not give many solutions to these problems. A set of solutions to > > > the problems of peer-review is given in the report "Science in the 21st > > > century: social, political, and economic issues". > > > > Additional info and references are given in Mitch blog entry: "ACS Day > > > 4: Peer-Review Reviewed" > > > > You can state your opinion, share your experience, submit your own > > > proposal for solving the problems, etc. > > > [..] > > > Here's another one that says it all - (and yes this is the 21st > > century!): > > > "Alternative Peer Review: > > Quality Management for 21st Century Scholarship" > > >www.public.iastate.edu/~gerrymck/APR-1.ppt > > > Harald > > Hi Harald et al. > I had a problem accessing that link. That's quite possible if either your computer or your internet connection is a bit poor: the file is > 6 Mb. [...] Regards, Harald |