The AlgoViz.org AV Awards: Nomination and Voting Process

16 replies [Last post]
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Here is my initial draft proposal, open for discussion, for how to conduct the process for nominating and voting on awards. While I hope that this is complete and plausible, the purpose of posting this is not to say this is how things will be. It is to put up something that people can react to and suggest changes if needed. PURPOSE AND IDENTIFICATION The goal of this effort is to involve as many people as possible in identifying a small collection of AVs that will be designated as “award winners”. We hope to have some sort of event at SIGCSE 2010 to announce the winners.We need some sort of name for this endeavor. I propose that we call this the “AlgoViz.org Awards”. Deeper goals are to advertise this site, get people participating, and generally raise consciousness on the availability of quality AVs. TIMETABLE I am hoping that we can get the process decided and the technology in place so that we can go public with this on June 15. The nominations and voting process would be open for some months. It is pretty flexible when we close this, so long as we are done a few months prior to SIGCSE 2010. So it could run as late as, say, January 15 before the final voting is closed. NOMINATIONS We have tentatively identified three categories to which something can be nominated: * Standalone AV * Tutorial or Hypertextbook * AV system Within each category, we will seed a few initial nominees. (Note that there are not a lot of options available for the “Tutorial or Hypertextbook” category, but I think we do want to highlight this for the future). Any person who is registered at the Portal will be able to fill out a form to add another nominee. The form will require the following information. * Name of AV. * Link to the AV (ideally, a link to the AV’s entry in the Wiki catalog if one exists). * Link to a screenshot of the AV (we could make this optional, or require it to impose some level of screening). * Brief (couple of sentences) description of the AV and why it deserves recognition. Once the nomination form is submitted, our site staff will (by hand) add it to the voting page if the nomination is acceptable (not a repeat, not a prank). VOTING PAGE The voting page will list all current nominees, by category. Nominees are added to the voting page as we process them. For each nominee, we will show the brief description given by the nominator (possibly edited by our staff), along with voting information. Voting is done on a 5-point scale. For each nominee, we will show the current average vote and the number of votes, and we will have a star bar where the viewer can vote. VOTING PROCESS Voters must be registered (and validated) at the algoviz.org site. Voters can vote for a given nominee only once. Voters can change their vote at any time. Voters can vote for some or all nominees. They can come back later and add more votes. Votes are integer values in the range 1-5. The winner in a category is the one that (a) reaches some minimum threshold of votes, and (b) has the highest average vote in that category. We might give multiple awards in a category. In the end, I think we would like to recognize about six award winners. So far, I have thought of two ways that people can abuse the voting process. (1) Registering with multiple accounts. We can make some effort to purge bogus accounts. We need to think a bit about the process for registering. We don’t want to make that too hard, though, because we want to encourage people to register. (2) Voting for their favorite AV, and then assigning “1” star values to all the competitors. Not sure what if anything we can do about this. We could ask people not to do it. We could purge blatant cases of this. Or we could just accept it and not worry about it.

guido
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-06-03
Posts:
Points: 679
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
Regarding the abuse options, I would not point them out too strongly and hope for (mostly) ethical behavior. What would more concern me is the unfortunate bias we can get if additional nominations arrive after people have voted. Maybe I would have also given a good rating for a system / tool / hypertextbook that appears two days after I have cast my vote. There is no reason why people could not come back and update their vote, as they can give as many “5 stars” as they like, if I understand you right. However, once I have cast my vote, chances are that people will not come back to check for additional nominations. How about making it a two-step approach? Nominations have to be “in” by (say) November 15, and voting on all of them is then possible until January 15. That way, each voting person sees the same choices, you get less corrections, and “late comers” who might be of the same (or more) interest than early entries get the same fair chance of getting a good vote.
ajalon
ajalon's picture
Offline
White BeltYellow BeltGreen BeltRed Belt
Joined: 2009-05-26
Posts:
Points: 60
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
guido wrote:
What would more concern me is the unfortunate bias we can get if additional nominations arrive after people have voted. Maybe I would have also given a good rating for a system / tool / hypertextbook that appears two days after I have cast my vote. There is no reason why people could not come back and update their vote, as they can give as many “5 stars” as they like, if I understand you right. However, once I have cast my vote, chances are that people will not come back to check for additional nominations.
Definitely an issue to be aware of. The simplest response would be to notify voters of new nominations, say via an automated email or RSS feed. I’m not sure if this is an actual solution, and it might just annoy users.
guido wrote:
How about making it a two-step approach? Nominations have to be “in” by (say) November 15, and voting on all of them is then possible until January 15. That way, each voting person sees the same choices, you get less corrections, and “late comers” who might be of the same (or more) interest than early entries get the same fair chance of getting a good vote.
I like this approach.

Cheers! -AJ

shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
Thinking more about the categories: I think the “systems” category has some problems. First, what does this refer to? The ease of using a system to build AVs? That doesn’t seem practical that many people can make a judgement on that. More likely, it would be which “collection” has best overall quality. That permits considering collections related to AV-building systems (such as Jhave) and collections that are just integrated (such as DSV). However, even so this seems impractical that many people can look at the roughly 20 AVs in each system and give any meaningful comparison. People just won’t have the stamina for that. So I think we need to give up on the idea of a systems category. However, I would like to have multiple categories. More if we can get them. One idea is to split “standalone AVs” into “basic” and “advanced” (or terms to that effect). There will be some difficulty in classifying these. But “Basic” would typically be material taught at the lower division (CS1, CS2, sophomore level data structures) and “Advanced” would be topics more typically taught in an upper-division algorithms course. So, I would classify all AVs on linear structures as basic, all sorting as basic (except some exotics like sorting networks), and BSTs at basic. On the other hand anything related to Splay Trees or other (non-BST) search trees as advanced, B-trees, NP-completeness, computational geometry, compression, strings. Things that I am not sure which way they should go yet: Hashing. Graphs.
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
Guido — That is a concern, that someone might not come back and so could miss a chance to vote on some late entries. On the other hand, I am even more concerned that someone will come look at the nominations, maybe make a nomination, and never come back to vote at all. Especially if the nominations process goes on for a long time. What I am imagining is that we are going to make an advertising push at the beginning of this process. Hopefully, a lot of people will come look at us then. If we can’t get them right into voting, I’m afraid we might never see them again. And we can only make so many advertising pushes and expect to attract interest. We will be able to send email to everyone later, once the nominations seem to have reached closure. All voters have to register, so we will have an email address. The other mitigation is to make sure that we have a pretty good slate of nominees from the beginning. That is where the Steering Committee has an important role to play. By Friday, we should have a page up with some initial nominees. I am hoping that you folks will add to that list. Then when we go “live” in a couple of weeks, we can hope that there won’t be a huge number of new nominees. And those will hopefully get on earlier rather than later.
clancy
Offline
White BeltYellow BeltGreen Belt
Joined: 2009-06-05
Posts:
Points: 15
categories
If “basic” means “CS 1+CS 2”, it should be labeled “CS 1+CS 2”. I wouldn’t necessarily categorize a visualization of Quicksort or merge sort as “basic”.
rodger
rodger's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-06-05
Posts:
Points: 196
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
> Any person who is registered at the Portal will be able to fill out a form to add another > nominee. The form will require the following information. > * Name of AV. > * Link to the AV (ideally, a link to the AV’s entry in the Wiki catalog if one exists). I think it should be required to submit an AV to the Wiki Catalog before it can be considered for a nomination. I also agree you need to have a cutoff date for submissions and a period for voting. The dates proposed (Nov 15, Jan 15) sound good. Susan
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
I don’t mind having a cutoff date for new nominations. But is there a reason to not allow voting to overlap with new nominations once we get an initial slate together? I am worried about getting people to come back to vote in a couple of months. I guess the real question (to me) is what approach is going to get the most participation. Will a two-phase process (nominations for a couple of months, then a separate voting phase) increase or decrease participation? Note that the process of voting could take a person some time, depending on how many AVs they choose to review. I suppose that if the list is up and people have time to look at them before voting opens, they might do that.
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
Steve and I talked this over a bit. Here is what I would like to do regarding deadlines for things. Beginning on June 22, we will reset the votes to zero, and start advertising that the Awards process is open. At that time, anyone registered at the site can (1) propose a nomination, and (2) vote, and (3) see the current vote status. Deadline for adding new nominees will be November 15. This will give us opportunities/excuses for new announcements. Final deadline for voting will be January 15. Votes can be changed at any time up until then. Note that voting is not envisioned as something that happens in a few minutes during one session (though it might be that way for many people). It takes time to look at several AVs. So people might do a couple, and then come back later to do a couple more. Many people won’t dedicate that much effort. But we want to encourage people to process the nominees over time. Sort of like voting on movies. OK?
scottgrissom
Offline
White Belt
Joined: 2009-06-02
Posts:
Points: 2
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
I agree with the two step approach as well. It just seems must “cleaner” to the outside world to vote on a stable collection of submissions. This would also be good PR to have two specific deadlines. I agree that a Systems category will be awkward. I would advocate a single category for this first attempt to keep things simple. You could give awards to the top five or some other number to provide wider exposure.
scottgrissom
Offline
White Belt
Joined: 2009-06-02
Posts:
Points: 2
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
Regarding the voting process: What happened to providing a rubric for the voter to use? Didn’t we have more than one category of evaluation? Provide clearer instructions that the viewer is supposed to select one of the five stars. I did not pick up on that. Instead, I followed the link to the resource and tried to submit a comment thinking I was voting. The comment section was odd. There are eight radio buttons to choose from with smiley faces I do not understand. Consider combining the “view nominees” and “cast your vote” into a single page. Why separate them? It seems more straight forward to provide the summary and the voting mechanism together.
shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
Scott — Which page did you look at? The main page that people should be reviewing is the one that you reach under the menu tab “AlgoViz.org Awards” and then click on “Cast Your Vote!”. I see that this setup is more complicated than it should be. I’ll talk to A.J., and we’ll boil this down to just the one item for most people. Right now, there are four menu sub-items: Make a Nomination: I agree, this should be better related to the voting page. Perhaps the form should be linked from the voting page. Certainly it is distracting to have this be the first menu item. View Nominees: This is now obsolete and should be removed. Cast Your Vote!: This should be the main thing that people focus on. And for most people, all that they see. COMMITTEE: Review Nominees: Only people with the right access should even see this menu item. Regarding the Rubric— Yes, we need to get to work on that again. I posted a separate thread to discuss this. I posted Eileen’s original rubric with some editing, and then I also posted a discussion of a related topic: The “editorial” rating recommended/has potential/not recommended in the catalog. We need to have you folks discussing the rubric. And we need to add it somehow to the voting page.
ajalon
ajalon's picture
Offline
White BeltYellow BeltGreen BeltRed Belt
Joined: 2009-05-26
Posts:
Points: 60
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
Scott,
scottgrissom wrote:
What happened to providing a rubric for the voter to use?
The voting slate used to be completely pre-generated server-side. Really easy to configure and setup. However, I recently found a bug in the software we are using to enable the whole voting process; these bugs caused the rubric to provide misleading visual feedback about whether users had voted. Net result is that while I wait for a bugfix from the software devs, I’ve had to hack the voting slate together manually. I’m currently working on the stylesheets that’ll make it look like a proper rubric.
Quote:
I did not pick up on that.
Agreed, the instructions are too vague. Fix that shortly.
Quote:
Consider combining the “view nominees” and “cast your vote” into a single page. Why separate them? It seems more straight forward to provide the summary and the voting mechanism together.
I’ll get rid of the “view nominees” page.
ajalon
ajalon's picture
Offline
White BeltYellow BeltGreen BeltRed Belt
Joined: 2009-05-26
Posts:
Points: 60
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
rodger wrote:
>I think it should be required to submit an AV to the Wiki Catalog before it can be considered for a nomination.
I agree. I know I’ve discussed this with Dr. Shaffer before. I’ve changed the help text on the nomination form slightly to reflect this.
naps
Offline
White BeltYellow BeltGreen BeltRed Belt
Joined: 2009-06-11
Posts:
Points: 65
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
I agree with the other comments that have been posted regarding the requirement that something can be nominated only if there is a wiki entry for it. Ideally the links that trigger the visualization from the voting page should all be links to the wiki entry for that visualization. Right now it seems like some links from the voting page lead me to the wiki, whence I can see the wiki summary of the AV and from there launch it. Others take me straight to the developer’s site. I think a consistency that always has me see the context the wiki entry gives for the AV before I launch it is good, and we can enforce that if all AV’s that are nominated are in the wiki.

shaffer
shaffer's picture
Offline
White BeltYellow BeltGreen BeltRed BeltBlack Belt
Joined: 2009-05-28
Posts:
Points: 2019
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
All current nominees have Wiki entries. A.J., can you please check that all the links from the nomination page go to the Wiki entries? Thanks!
ajalon
ajalon's picture
Offline
White BeltYellow BeltGreen BeltRed Belt
Joined: 2009-05-26
Posts:
Points: 60
Re: The AlgoViz.org AV Awards: Nomination and Voting Process
shaffer wrote:
All current nominees have Wiki entries. A.J., can you please check that all the links from the nomination page go to the Wiki entries? Thanks!
Already did this morning. :)