JOIN
Get Time
forums   
Search | Watch Thread  |  My Post History  |  My Watches  |  User Settings
View: Flat (newest first)  | Threaded  | Tree
Previous Thread  |  Next Thread
Lots of Time Limit Exceeded in final results | Reply
It seems the final results are already available: https://community.topcoder.com/longcontest/?module=ViewStandings&rd=16981

But many competitors (e.g. nika, CatalinT, kurenai3110, koyumeishi, atsT5515, hoshi524, hakomo, me) have TLEs on many system tests. What we all seem to have in common is that our timing function assumes a CPU frequency of approx. 2.8 GHz, which was true in the environment used for example and provisional tests (this is consistent with the spec data of AWS c3 instances).

t-mac, can you please check if the same environment was used for the system tests? And if not, was that intended? I always assumed that the same conditions for example and provisional tests will also be used during system tests. After all, that's the point of the example tests - to check how my submision behaves under system testing conditions, on a very small set of example test cases.
Re: Lots of Time Limit Exceeded in final results (response to post by mugurelionut) | Reply
I saw the system test results, it seems that the old instance and the new instance were mixed. The mixing ratio of instances is roughly the same for all competitors, but which seeds and which instances are assigned seem to be disjointed by competitors.

e.g.
test_case_id = 33756916
hakomo = TLE (seems old low spec instance)
mugurelionut = not TLE (seems new high spec instance)

test_case_id = 33756922
hakomo = not TLE (seems new high spec instance)
mugurelionut = TLE (seems old low spec instance)

I think that this is unfair.

I also agree with the opinion of mugurelionut and I would like to rejudge only with new instances.
Re: Lots of Time Limit Exceeded in final results (response to post by hakomo) | Reply
After not so nice experience in MM 94 I used gettineofday this match. I am going to test the why std::chrono does not work in TC Amazon environment as well.
I also proposed running the entire seed on the same VM for everyone, but that is hard to implement, according to t-mac.
Re: Lots of Time Limit Exceeded in final results (response to post by mugurelionut) | Reply
Just a formal, small point :)

In a previous thread, https://apps.topcoder.com/forums/?module=Thread&threadID=905034&start=0 this was stated:

`About our testing environment, for this contest, for the first time in a fun marathon actually, I switched from using m3 instances to using c3 instances. Timing had been brought up as a potential issue previously, and it could be, at least in part, due to m3 instances having the potential of being run on either Xeon E5-2670 (Sandy Bridge) or Xeon E5-2670v2 (Ivy Bridge) processors. For timing-important computations, that could make a difference. c3 instances, BTW, are all E5-2680v2 CPUs.`

The fact that:
a) c3 instances are going to be used for all fun rounds from now on (referenced in another forum post)
b) 'c3 instances, BTW, are all E5-2680v2 CPUs' (at the end of the above)

leads to the explicit implication that our solutions for fun MM's are always going to be run on E5-2680v2 CPU, which via https://ark.intel.com/products/75277 always have a 2.8GHZ CPU frequency.

This means that always assuming a 2.8GHZ CPU frequency in our code is not only reasonable, is the only official way of looking at things :)
Re: Lots of Time Limit Exceeded in final results (response to post by gorbunov) | Reply
@gorbunov: In MM94 the situation was reversed. Older (slower) instances were used for examples (and I guess also provisional) tests, but newer (faster) instances were used for system tests. This did not cause additional TLEs. Instead, it made solutions relying on knowing the CPU frequency not use the full available time (which is bad, but not as bad as getting a TLE).

@hakomo: Thanks for checking. So it seems there are, in fact, two issues:

1) The example+provisional tests all used newer (faster) AWS instances, while system tests used a mix of older and newer AWS instances.
This caused many competitors to get TLE when their submission was run on the older instances, because of the assumption that the CPU frequency is 2.8 GHz (which should be the only reasonable assumption, as CatalinT mentions in his post above).

2) For the same test, some submissions were run on newer (faster) AWS instances, while other submissions were run on older (slower) ones.
This means that a higher quality solution running on a slow instance could get a lower score than a lower quality solution running on a fast instance (due to the latter one having a faster CPU and, thus, being able to run for more iterations). Given how close the absolute scores probably are, this has the potential of making any final results meaningless (even in the absence of issue 1).
Re: Lots of Time Limit Exceeded in final results (response to post by mugurelionut) | Reply
If old rounds are mentioned, I really wonder how the variancle of "old" m3 instances influenced timeouts (and effective calculation time as well!) in earlier rounds, especially TCO2017/round3...
Re: Lots of Time Limit Exceeded in final results (response to post by kovi) | Reply
So, for R3, all system tests were run on c3.

For the lightning round, I spun up additional testers to make sure tests were done in time... and mistakenly some of them (which hadn't been used for a while) had not yet been converted from m3->c3.

I'm rerunning them now. Entirely my fault on this one, please accept my apologies.
Re: Lots of Time Limit Exceeded in final results (response to post by mugurelionut) | Reply
(I changed my post after I see t-mac post).

Thank t-mac for rerunning the tests.
Re: Lots of Time Limit Exceeded in final results (response to post by t-mac) | Reply
I don't want to hijack this forum/post instead I continue in round3 forum (rerun of this lightning round system tests seems pretty straightforward)
Re: Lots of Time Limit Exceeded in final results (response to post by t-mac) | Reply
Thank you t-mac for rerunning the tests! :)
Re: Lots of Time Limit Exceeded in final results (response to post by CatalinT) | Reply
Updated results are loaded, ratings recalculated, etc. Things look a lot more reasonable now.

Let me once again say how sorry I am that it was mucked up the first time.
Re: Lots of Time Limit Exceeded in final results (response to post by t-mac) | Reply
Hi t-mac. Thank you for rerunning the system tests. The results do look OK now. However, the rating updates seem to have considered the initial results (with lots of TLEs). See, for instance, how nika's rating decreased a lot, despite him winning the match (you can see the same pattern for everyone else who was affected by TLEs in the initial results). However, if you look at marek.cygan's rating - this decreased only a bit (he was not affected by the TLE issue). Could you update the ratings based on the new (final) results?
Re: Lots of Time Limit Exceeded in final results (response to post by mugurelionut) | Reply
Ratings seem to be fixed as well. Thanks!
RSS