JOIN
Get Time
forums   
Search | Watch Thread  |  My Post History  |  My Watches  |  User Settings
View: Flat (newest first)  | Threaded  | Tree
Previous Thread  |  Next Thread
Complete code for score calculation | Reply
Would it be possible to publish not only the pseudocode, but the actual code for score calculation which is used for online scoring? (I mean, not the entire script with file loading and so, but only the method which implements the same thing as the pseudocode does.)
Re: Complete code for score calculation (response to post by nofto) | Reply
int MAX_LABEL_VALUE = 1000;
int NUM_VOXELS = 102297600;
int SIZE_Y = 480;
int SIZE_X = 640;
int REGION_MIN_X = 0;
int REGION_MAX_X = 383;
int[] SCORABLE_Z_SLICES = {0, 18, 34, 51, 68, 84, 101, 118, 134, 151, 167, 184, 201, 218, 234, 250, 268, 284, 301, 318, 332};
 
short[] answer = new short[NUM_VOXELS];
Set<Short> uniqueAnswer = new HashSet<Short>();
 
short[] groundTruth = new short[NUM_VOXELS];
Set<Short> uniqueGroundTruth = new HashSet<Short>();
 
// ... data loading not included ...
 
int[][] labelCountMatrix = new int[MAX_LABEL_VALUE + 1][MAX_LABEL_VALUE + 1];
int[] groundTruthLabelCounts = new int[MAX_LABEL_VALUE + 1];
int[] answerLabelCounts = new int[MAX_LABEL_VALUE + 1];
 
for (int sliceIndex = 0; sliceIndex < SCORABLE_Z_SLICES.length; sliceIndex++)
{
  int z = SCORABLE_Z_SLICES[sliceIndex];
 
  for (int x = REGION_MIN_X; x <= REGION_MAX_X; x++)
  {
    for (int y = 0; y < SIZE_Y; y++)
    {
      int voxelIndex = z * SIZE_Y * SIZE_X + y * SIZE_X + x;
 
      labelCountMatrix[groundTruth[voxelIndex]][answer[voxelIndex]]++;
      groundTruthLabelCounts[groundTruth[voxelIndex]]++;
      if (groundTruth[voxelIndex] > 2)
      {
         answerLabelCounts[answer[voxelIndex]]++;
      }
    }
  }
}
 
boolean[] groundTruthLabelMatched = new boolean[MAX_LABEL_VALUE + 1];
 
double TP = 0;
double FP = 0;
double FN = 0;
 
for (short answerLabel : uniqueAnswer)
{
  if (answerLabel > 0 && answerLabelCounts[answerLabel] > 0)
  {
    for (short groundTruthLabel : uniqueGroundTruth)
    {
      if (groundTruthLabel > 2 && groundTruthLabelCounts[groundTruthLabel] > 0)
      {
        int intersection = labelCountMatrix[groundTruthLabel][answerLabel];
        int union = groundTruthLabelCounts[groundTruthLabel] + answerLabelCounts[answerLabel] - intersection;
 
        double jaccardIndex = (double) intersection / (double) union;
 
        if (jaccardIndex > 0.5)
        {
          TP += jaccardIndex;
          groundTruthLabelMatched[groundTruthLabel] = true;
        }
        else
        {
          FP += jaccardIndex;
        }
      }
    }
  }
}
 
for (short groundTruthLabel : uniqueGroundTruth)
{
  if (groundTruthLabel > 2 && groundTruthLabelCounts[groundTruthLabel] > 0 && !groundTruthLabelMatched[groundTruthLabel])
  {
    FN++;
  }
}
 
double fScore = 2.0 * TP / (2.0 * TP + FN + FP);
 
double score = 1000000 * fScore;
Re: Complete code for score calculation (response to post by jonathanps) | Reply
I think there is a bug. Instead of:

answerLabelCounts[answer[voxelIndex]]++;


there should be something like:
if(groundTruth[voxelIndex] > 2) answerLabelCounts[answer[voxelIndex]]++;


The reason for this change is, that non-scorable voxels (with ground truth label 2) should not be counted in the union.
Re: Complete code for score calculation (response to post by nofto) | Reply
jonathanps, any update on this and the other question posted by nofto?
I believe that both cases are relevant and require either a clarification or a fix, and there are only few days left in this contest (with a weekend and a possible holiday ahead)...
Re: Complete code for score calculation (response to post by nofto) | Reply
Thanks for pointing this out nofto. This was a mistake, and it has been updated in the test harness.
RSS