||Hello CAB members,
Based on the discussion in Slack, you should talk with TC admins and define re-appeal policy.
TC admins confirmed following things:
- if you re-appeal reviewer X, the whole scorecard is reset (or only 1 section?)
- all existing issues are reconsidered
- additionally, copilot/PM can add extra issues that were missed by the reviewer. Multiple TC admins confirmed that (mess, jcori, mtwomey)
My few observations:
- If we reset scorecard from one reviewer, we should also reset another reviewer, because review must be consistent.
- If we fix scorecard from one submission, we should also fix scorecards for other submissions, because of the same reason.
What does it mean in practice?
If any member re-appeals, copilot/PM has a huge job to reset everything and assign scores based on his discretion.
TopCoder is the main source of income for some members. We should have clear and transparent rules.
It seems like all my re-appeals were resolved incorrectly in the past few years.
We just had our last CAB meeting so I am not sure when the next one will be.
However, considering the impact this topic will have, I will check if I can get further clarity on the same asap.
||I'm not a CAB member, but I can talk about this from a copilot perspective:
I think you're overstating the impact and flow of the reappeals, but I agree that it's hard to understand or actually succeed at a reappeal because it is not a formal process and it is not well defined on the site. Copilots and managers have discretion on making changes to the scorecards, but that's not a responsibility taken lightly, and very rarely will I throw out an entire review (although I will admit it has happened more than once).
In practice, I look at the reappeals as follows:
1. The time limit for a reappeal is 24 hours after appeals response ends. Anything past that and the contest is already too far along or closed out.
2. A reappeal must be against a scorecard question that was originally appealed. If you didn't use the formal appeals process, there's no point in entertaining an informal reappeal.
3. A reappeal must be a case where a reviewer is technically wrong with regards to either technology usage or the requirements. There are plenty of cases where reviewers may have understood a requirement differently or would like to see technology used in a specific way. If you can show that the reviewer was incorrect or is asking you to use technology in a way that is not in accordance with best practices, then that's definitely a case where a reappeal is proper.
4. If a reappeal is successful, I will go to all other scorecards from the reviewer in question and ensure they are updated for consistency against the reappealed item. A lot of times the reviewer will make the same mistake in all scorecards, and we need to do that for consistency.
Reappeals are generally successful:
1. If it's clear the reviewer was technically incorrect or if they were adding a requirement that was completely outside the scope of the contest or adding a requirement that actually goes *against* the requirements.
2. If the reviewer is super vague and confusing in either their initial comment or appeals response. I've seen plenty of bad or confusing responses on scorecards that show that the reviewer either didn't spend the time necessary to do a good review or didn't understand the requirements.
Reappeals are generally unsuccessful:
1. If the submitter didn't put the time necessary into the original appeal or the submission. It's massively frustrating as a copilot to see a submitter put in 2-3 times the effort into the reappeals than they did into the original appeal or the actual submission. We have a formal appeal process in place and expect it to be used properly. This would also include many cases I've seen where a submitter will greatly expand on what items they are reappealing vs. their original appeal.
2. If the submitter is emotional and accuses reviewers of being dumb, cheating, or accuses the copilots of all manner of things. Trust me, I've seen a lot after running a couple thousand challenges :)
3. If the submitter and the reviewer are at a disagreement but there isn't a clear technical answer. In cases like this, it's tough, but I will generally give the reviewer the benefit of the doubt over the submitter. We expect the reviewers to be experts in whatever they are reviewing, and we need to respect their expertise.
||I agree in 100%. I have the same understanding and other veteran copilots use the same approach.
In the following situation:
- submission A has an issue X
- submission B has an issue X
- reviewer scores down submission A and misses it for submission B
In such situation, the scorecard for submission B should be fixed and the score should be deducted.
How should we handle a situation when reviewer misses an issue for ALL submissions?
TC admins said that the missing issue should be added. It implicates that we should add all missings issues to all scorecards.
||I'm not going to speculate on what others do, but if a reviewer missed an issue for all submissions, I wouldn't do anything at that point, with the assumption that the scores would even out. I would definitely make sure the issue was fixed in the solution used, one way or another, but I would likely not adjust the scores.
If the reviewer just did a terrible job, it might be prudent to either have them redo their review, or just throw out their review altogether. I have seen both of these things in rare cases in the past. This normally accompanies a lot of scolding directed at the reviewer :)
That being said, in cases like this, it would be extremely rare for multiple reviewers to both miss the same issue in all submissions, which is why we have multiple reviewers. We can't expect every reviewer to catch every issue - otherwise there would be no point in having multiple reviewers.
If multiple reviewers missed the same issue across all submissions, that would imply that not only did the reviewers miss it, but the submitters as well. That would point to a requirements problem, more than anything, in my opinion.
Obviously this is not a black and white issue and every situation is different, so I can't say that this would apply at all times.
||Bad reviews are rare. Responses usually are:
- "can't deploy"
- or score 100 for average submissions
A single reviewer usually finds 60-80% of issues. Both reviewers can cover the whole functionality.
In my last contest, two reviewers found together 100% issue.
I couldn't appeal reviewer X, because he missed issues from reviewer Y.
Reviewer Y also missed some issues from reviewer X.
I don't want to talk about any specific case, but this is something that will affect many future contests.