Proposal for a formal review process

2 replies [Last post]
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019

The revised recommendation field system with four sub-fields appears to be working well. It allows for a flexible definition of the various ways that an AV can be used, and so far I have not come across any situations where none of the four use categories apply.

We also finally have in place a mechanism to sort search results based on various indicators, including but not limited to the "editorial" review fields.

The one thing that has not worked out well is getting reviews from a broad base of users. While some of us have solicited reviews from students as part of various courses, few people submit AV reviews voluntarily. As a result, much of rating reflects my personal bias, since I end up assigning most of the recommendation values. We had hoped to get more users involved in voting through the "AlgoViz Awards" process, but that has two problems. First, we don’t get a lot of people casting votes, and second, even if we did, that only makes small separations among a group of AVs that are already considered of good quality (though the fact that someone nominates an AV for the award means that it gets a more careful re-review, which is significant).

A few of us have kicked around the idea of requesting reviews from reviewers much like a journal or conference paper review process. Archie Korhonen has raised this issue most directly with me recently. Especially in light of the proposed OpenDSA project, which will seek to develop many, many new AVs, there becomes a real need for some sort of formal review process in order to keep the quality high for the project.

One way that a review process could work is to adopt something as close to a journal editor/reviewer mechanism as possible. We would need to establish an editorial board.  Then, in some way certain AVs are selected for review. This might happen when a new AV is added to the catalog, or developers might request a formal review, or the editorial board members might select AVs from the catalog to send for review. At this point, reviewers are selected/recruited, who are asked explicitly to agree to provide a review on some deadline, say in one month. They are given a reviewer form to fill out, and are given any current information about the AV such as what is in the catalog. The editors would then collect the reviews, and integrate the information as appropriate into catalog entries, as well as give copies of the reviews (without reviewer names) to the developers for possible improvement.

What do people think of this idea? We would need to get a small number of people to agree to be editors, and a large pool of people willing to be reviewers. Probably we should limit requests to reviewers to no more than once in a given time period, perhaps four months, so that nobody is asked to review more than two or three AVs in a year.

 

Keith Moss
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2013-06-15
Posts:
Points: 99
Re: Proposal for a formal review process
Ok the ideas expressed seem to point to a process that should give developers and users a better idea about AVs of interest. If you need reviewers I could help if you think that my opinions would be useful
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: Proposal for a formal review process

 Keith — I am sorry, you have mentioned that you are willing to review a couple of times and I have not gotten back to you on that yet! Is it OK if I send you a short list of some AVs in the catalog that could use a review? Thank you for volunteering!