7 min read

Suggestions: introduce the academic peer-review system into utopian.io

Links: steemit.com, cnsteem.com, busy.org, steemdb.com, steemd.com

Problems

Utopian, one of the most spirited flowers blooming from the soil of Steem. On utopian.io, Steemians are writing excitedly, moderators are reviewing the submissions diligently, and supervisors are providing guides to them helpfully. This new system is developing rapidly like a running race car, which is why I like utopian.io, besides the attractive upvotes.

However, there is unpleasant noise from the wheels.

Some new steemians submit anything to utopian without reading the rules. Submitted posts are piling up, waiting for the right moderator’s review. Some moderators are inactive, and some others complain about the low salary. Supervisors are annoyed by the complaints from authors, but have to give response in patience.

Will these problems destroy this race car, like worms destroy this flower some day? I am a little worried.

As a scientific researcher, I often submit research articles to academic journals. The articles must be peer-reviewed before being published, similar to the utopian review system. Editors distribute the submitted manuscripts to reviewers, and summarize their opinions and guide the authors to revise and resubmit their articles. Utopian-io is new, but the academic peer-review system is old and mature.

What if we use the academic peer-review system on utopian.io?

The academic peer-review system

A typical academic peer-review system works in a way shown in the following flow chart as an example.

Image credit: advmodoncolres.com

The procedure is described as follows:

Step 1. An author submits a manuscript to a journal according to the journal rules (topic, format, etc).

Step 2. The journal editor gives a preliminary view of the manuscript ans screens it. If it does not fit the rules, it is rejected and this submission is done. Otherwise, go to Step 3.

Step 3. The manuscript goes two or three reviewers. The reviewers give the feedback to the editor-in-chief before the deadline.

Step 4. if the reviewers have similar opinions (acceptance, minor revision, major revision, rejection, etc), the editor-in-chief summarizes the opinions and give the feedback to the author. If the feedback is revision, then the authors have to follow the review comments and revise the manuscript and go to Step 3. If the feedback is acceptance or rejection, the manuscript is published or rejected, respectively, and this submission is done.

It is not a perfect system, but it has been working for many years.

Apply the reviewer system to Utopian

If you compare the academic review procedure with the review system on utopian.io, you may find that

  • the author (academic) = the author (utopian)
  • the editor (academic) = the moderator (utopian)
  • the reviewers (academic) = the moderator (utopian)
  • the editor-in-chief (academic) = the supervisor (utopian)

Here you see why the submitted utopian posts are piling and the moderators are busy. The moderators act as both the academic editor and the reviewers in the flow chart!

Furthermore, an academic journal has more reviewers than the utopian moderators.

How many reviewers do they have and who are they?

They are all the authors who have ever published an article on this journal!

In Step 3, the editor asks two or three authors if they are able and would like to review the manuscript before a deadline.

But on utopian, the number of moderators is limited. What if all the utopian authors can review new submitted post? Or, to be more strict, some experienced authors? Like this:

  • the author (academic) = the author (utopian)
  • the editor (academic) = the moderator (utopian)
  • the reviewers (academic) = the experience authors (utopian)
  • the editor-in-chief (academic) = the moderator + supervisor (utopian)

The flow chart for utopian would look like this:

Slide1.JPG

I added the author revision into Step 1 because there is an utopian rule of ‘never reject a post at the first time. Always give a change to revise.’ I really appreciate it.

Now let’s define ‘the experience authors’ as the authors who have successfully published, say, 10 posts on utopian.io If all the experience authors can review, I guess the review process will speed up.

There are two things missing:

  1. Time. The seven-day limitation for upvote is a pain. What if the assigned reviewers cannot finish the review on time?

One solution is: display all the submitted posts to all the experienced authors, and add a small reviewer indicator beside the upvote and comment indicator, like this:

Slide2.JPG

Any experienced author can click the indicator to claim the review task. 12 means this post needs two reviewers and one is already claimed. If nobody claims the task within 24 hours, a moderator takes it.

  1. Rewards and Punishment.

In academia, the review work is unpaid. On utopian,io, if the review work is unpaid, would any experienced author volunteer? Maybe utopian can test it. Or they can get paid like the current moderators. There maybe a ranking list of the experienced authors, where all the authors can rate them with five stars.

What if a reviewer claims a task but does not finish it on time? Down vote.

Preferred and non-preferred

So far we have talked about the academic review system, but *peer*s?

Peer-review means a manuscript is reviewed by someone who understands the topic. I don’t exactly know how utopian distributes the submitted posts to the moderators. In my case I would submit a python development post to @yuxi, and a translation post to @wangwenjing, because I know they are experts in these topics, respectively. They are capable to review the related posts fast. But if my post is randomly picked by someone who knows only a little of my topic, it could be reviewed slowly and could get an unsatisfactory comment.

That is why the preferred-reviewer rule exists in the academia. The author is allowed to suggest two or three preferred reviewers, and the editor decides whether to accept them or not. I think this is also possible on utopian. Furthermore, maybe the author could suggest some experts who have not registered on utopian yet, which will bring more visits.

What if a preferred reviewer is a friend of the author and accepts anything the author submits? Maybe the five-star rating could help, or other authors can complain to the moderator or supervisor.

Besides the preferred-reviewer rule, there is the non-preferred reviewer rule. The author is allowed to suggest two or three non-preferred reviewers that the post had better not go to, and the editor makes a decision. This to avoid personal bias of a reviewer on an author. The personal bias could come from anything. Maybe the reviewer fought with the author when they were kids, or maybe the author killed the reviewer’s best friend in a car accident. Who knows.

On utopian it could happen as well. We have seen many fights on steem. Hatred posts are forbidden on steemit, but who can tell whether there is any unfair reviews? I heard that many authors don’t even read the rejection reasons and just accept it and move on, because it takes much more time to complain than to write a new post. A friend told me that he spent 5 hours in writing a post, which was rejected in 20 minutes by a moderator who always rejects his posts, and he knew that it would take him another 5 hours to explain this issue to the supervisor because it was too specific. He knew he would win if he complained, but he gave up.

Summary

In this post I would suggest that utopian.io could use the experience of the academic peer-review system in the utopian review system. Here is the take-home messages:

  1. The experience authors review the submitted posts, and the moderator as well as the supervisor make the decision of revision or rejection.
  2. The preered-reviewer rule speeds up the review process and improve the review efficiency and quality.
  3. The non-preferred-reviewer rule avoids personal bias or interests conflict.

Other academic review experiences not mentioned here might be helpful as well. As the academic peer-review system is a mature system, I hope it could help utopian grow stronger and stronger.



Posted on Utopian.io - Rewarding Open Source Contributors