JOIN
Get Time
forums  Revision History
Search My Post History  |  My Watches  |  User Settings
Forums Marathon Matches SpaceNet RoadDetector Differences in APLS metric Revision History (1 edit)
Differences in APLS metric
Hi,

I am having trouble reproducing the APLS metric reported by the testing server on sample data. Specifically, I converted an output graph for Sample Vegas 1454 from pixel coordinates to longitude/latitude coordinates in geojson format, and placed the output graph in sample_data/sample_geojson/test0_prop.geojson and the ground truth graph from AOI_2_Vegas_Roads_Sample/geojson/spacenetroads/spacenetroads_AOI_2_Vegas_img1454.geojson in sample_data/sample_geojson/test0_gt.geojson. But I get this from apls.py:

C_gt_onto_prop, C_prop_onto_gt: 0.560342761102 0.575001651265
Total APLS Metric = Mean( 0.56 + 0.58 ) = 0.57
...
APLS Metric = 0.567577572758

But the website reports this result:

Image id:AOI_2_Vegas_img1454
G1->G2: 0.45312575737285765
G2->G1: 0.3884143548897544
Score: 0.41828201928653147

Any idea what the discrepancy could be? I checked the default parameters in apls.py main function, but only linestring_delta is mentioned in the problem statement, and it is already correctly set to 50.

Note that the apls.py that I am using is the latest commit c74da04022c26ae13dc11895c68429c3a5993c0a from https://github.com/CosmiQ/apls/.

Here is my CSV file submitted to website:https://pastebin.com/SRRsBSGs
And here is the same graph converted to geojson: https://pastebin.com/KXNixp92
Differences in APLS metric
Hi,

I am having trouble reproducing the APLS metric reported by the testing server on sample data. Specifically, I converted an output graph for Sample Vegas 1454 from pixel coordinates to longitude/latitude coordinates in geojson format, and placed the output graph in sample_data/sample_geojson/test0_prop.geojson and the ground truth graph from AOI_2_Vegas_Roads_Sample/geojson/spacenetroads/spacenetroads_AOI_2_Vegas_img1454.geojson in sample_data/sample_geojson/test0_gt.geojson. But I get this from apls.py:

C_gt_onto_prop, C_prop_onto_gt: 0.560342761102 0.575001651265
Total APLS Metric = Mean( 0.56 + 0.58 ) = 0.57
...
APLS Metric = 0.567577572758

But the website reports this result:

Image id:AOI_2_Vegas_img1454
G1->G2: 0.45312575737285765
G2->G1: 0.3884143548897544
Score: 0.41828201928653147

Any idea what the discrepancy could be? I checked the default parameters in apls.py main function, but only linestring_delta is mentioned in the problem statement, and it is already correctly set to 50.

Note that the apls.py that I am using is the latest commit c74da04022c26ae13dc11895c68429c3a5993c0a from https://github.com/CosmiQ/apls/.

Here is my CSV file submitted to website:

ImageId,WKT_Pix
AOI_3_Paris_img84,LINESTRING EMPTY
AOI_5_Khartoum_img304,LINESTRING EMPTY
AOI_4_Shanghai_img1700,LINESTRING EMPTY
AOI_2_Vegas_img1539,LINESTRING EMPTY
AOI_4_Shanghai_img1227,LINESTRING EMPTY
AOI_4_Shanghai_img1887,LINESTRING EMPTY
AOI_4_Shanghai_img1841,LINESTRING EMPTY
AOI_4_Shanghai_img1916,LINESTRING EMPTY
AOI_3_Paris_img235,LINESTRING EMPTY
AOI_3_Paris_img28,LINESTRING EMPTY
AOI_3_Paris_img496,LINESTRING EMPTY
AOI_5_Khartoum_img4,LINESTRING EMPTY
AOI_3_Paris_img406,LINESTRING EMPTY
AOI_5_Khartoum_img194,LINESTRING EMPTY
AOI_3_Paris_img38,LINESTRING EMPTY
AOI_5_Khartoum_img468,LINESTRING EMPTY
AOI_5_Khartoum_img157,LINESTRING EMPTY
AOI_4_Shanghai_img1196,LINESTRING EMPTY
AOI_2_Vegas_img734,LINESTRING EMPTY
AOI_5_Khartoum_img220,LINESTRING EMPTY
AOI_5_Khartoum_img333,LINESTRING EMPTY
AOI_2_Vegas_img230,LINESTRING EMPTY
AOI_2_Vegas_img767,LINESTRING EMPTY
AOI_4_Shanghai_img144,LINESTRING EMPTY
AOI_2_Vegas_img408,LINESTRING EMPTY
AOI_3_Paris_img307,LINESTRING EMPTY
AOI_3_Paris_img432,LINESTRING EMPTY
AOI_2_Vegas_img1454,"LINESTRING (713 0, 677 7, 669 15, 578 15)"
AOI_2_Vegas_img1454,"LINESTRING (75 15, 39 7, 31 7, 38 49, 73 47, 75 15, 95 7)"
AOI_2_Vegas_img1454,"LINESTRING (127 3, 119 62, 73 47)"
AOI_2_Vegas_img1454,"LINESTRING (247 3, 247 47, 247 55, 271 71, 279 79, 326 71, 334 71, 350 11)"
AOI_2_Vegas_img1454,"LINESTRING (482 15, 665 55)"
AOI_2_Vegas_img1454,"LINESTRING (422 27, 382 151, 382 159, 374 167, 366 175, 350 166, 342 334, 342 342, 382 414, 382 422, 446 476, 478 354)"
AOI_2_Vegas_img1454,"LINESTRING (38 49, 15 71, 15 79, 79 175, 87 175, 247 524, 231 542, 223 550, 223 582, 223 590, 255 646, 255 653, 406 789, 414 797, 490 829)"
AOI_2_Vegas_img1454,"LINESTRING (119 62, 141 127, 115 127)"
AOI_2_Vegas_img1454,"LINESTRING (745 79, 937 119)"
AOI_2_Vegas_img1454,"LINESTRING (141 127, 231 319, 231 326, 248 326, 255 275)"
AOI_2_Vegas_img1454,"LINESTRING (1292 307, 1292 370)"
AOI_2_Vegas_img1454,"LINESTRING (248 326, 255 366, 255 374, 342 534, 342 542, 390 606, 390 614, 478 685, 478 693, 558 741, 558 749, 653 789, 653 797, 733 821, 741 829, 877 845, 885 853, 1012 853, 1020 853, 1036 869, 1044 877, 1029 906, 1060 885, 1104 890, 1084 853, 1084 845, 1086 661, 1084 630, 1076 630, 974 636, 933 636, 917 583, 901 590, 893 590, 901 638, 861 653, 853 653, 550 638, 542 630, 510 598, 510 590, 497 509, 526 462, 534 462, 554 462)"
AOI_2_Vegas_img1454,"LINESTRING (626 394, 620 438, 741 462, 749 470, 821 470, 829 462, 853 470, 861 470, 885 502, 917 500, 925 486, 925 478, 1052 470, 1060 470, 1084 510, 1092 518, 1092 558, 1092 566, 1076 590, 1068 598, 1048 598)"
AOI_2_Vegas_img1454,"LINESTRING (620 438, 566 578)"
AOI_2_Vegas_img1454,"LINESTRING (446 476, 470 478, 478 478, 497 509, 462 510, 454 502, 446 476)"
AOI_2_Vegas_img1454,"LINESTRING (885 502, 873 602)"
AOI_2_Vegas_img1454,"LINESTRING (917 500, 933 526, 933 534, 961 538)"
AOI_2_Vegas_img1454,"LINESTRING (247 524, 263 526, 271 534, 303 590, 303 598, 382 685, 382 693, 633 871, 614 861)"
AOI_2_Vegas_img1454,"LINESTRING (901 638, 933 636)"
AOI_2_Vegas_img1454,"LINESTRING (974 636, 996 653, 980 653, 1024 653)"
AOI_2_Vegas_img1454,"LINESTRING (1086 661, 1132 663, 1156 669, 1164 669, 1172 693, 1180 701, 1172 733, 1172 741, 1140 757, 1132 663)"
AOI_2_Vegas_img1454,"LINESTRING (1140 757, 1132 845, 1132 853, 1104 890, 1152 881)"
AOI_2_Vegas_img1454,"LINESTRING (1212 809, 1212 853, 1220 861, 1272 853)"
AOI_2_Vegas_img1454,"LINESTRING (633 871, 653 869, 661 869, 677 891, 653 901, 646 901, 633 871, 610 893)"
AOI_2_Vegas_img1454,"LINESTRING (677 891, 925 911, 1029 906, 1004 933, 1004 941, 1029 906, 1036 933, 1082 941, 1104 890, 1124 933, 1132 941, 1138 1002, 1176 984)"
AOI_2_Vegas_img1454,"LINESTRING (925 911, 925 937)"
AOI_2_Vegas_img1454,"LINESTRING (1082 941, 1068 1128)"
AOI_2_Vegas_img1454,"LINESTRING (1138 1002, 1132 1128)"
AOI_3_Paris_img340,LINESTRING EMPTY
AOI_2_Vegas_img699,LINESTRING EMPTY
AOI_2_Vegas_img517,LINESTRING EMPTY
AOI_3_Paris_img489,LINESTRING EMPTY
AOI_2_Vegas_img1521,LINESTRING EMPTY
AOI_4_Shanghai_img1920,LINESTRING EMPTY
AOI_4_Shanghai_img1890,LINESTRING EMPTY
AOI_5_Khartoum_img66,LINESTRING EMPTY
AOI_5_Khartoum_img74,LINESTRING EMPTY
AOI_4_Shanghai_img308,LINESTRING EMPTY
AOI_2_Vegas_img794,LINESTRING EMPTY
AOI_5_Khartoum_img410,LINESTRING EMPTY


And here is the same graph converted to geojson: https://pastebin.com/KXNixp92