If you are here and reading this, you are probably a sport programmer, and we already have a lot in common. And, it is from that I know how important it is for a sport programmer to have an interruption-free experience during programming contests.
During the recent on-site contests at Islamic University of Technology (IUT) and Shahjalal University of Science and Technology (SUST), Toph encountered some issues that severely hampered the experience of the participants. First and foremost, we want to apologize for the inconvenience this has caused.
Contest participants usually have very restricted interaction with Toph, limited mostly to the contest arena, barred from most of the features Toph has to offer. And, so regardless of the cause, the way the participants experience them are similar: slow loading of the arena or queued submissions.
The root cause of the imperfect experience at these two contests was quite different.
We monitored our infrastructure closely during the IUT contest. Based on our findings we were able to fix some of the issues that caused the queue to be unstable. Unfortunately, we discovered the remaining issue only after the contest at SUST had begun.
This remaining issue has always been very elusive as it affected our load balancers (which in turn affected our entire infrastructure). But, the severity of this issue depended directly on the geographic distribution of the users. As a result, it rarely affected online contests. But, it hampered on-site events with a large number of participants.
While addressing the issues discovered during the IUT contest, we also implemented more rigorous logging and monitoring for our infrastructure. And, it helped us identify the remaining problem during the event at SUST rather quickly. It took us about 40 minutes from the start of the contest to implement a fix and deploy the changes to our infrastructure. As reported by many participants, it was right about that time when the stability of the platform improved.
The other bug that we encountered during the contest caused truncated I/O files for problems that used the line-ending correction feature in Toph Drafts. We fixed it before the rectified the balancer configurations. We synchronized everything and checked and double checked to make sure no other problems were affected.
It is unfortunate that the Internet issue prevented from completing the contest. We were still glad to see how hard the organizers of the event at SUST tried to recover from the situation.
So what is next for Toph? Improve.
It is important that we do not stop at just addressing the bugs and misconfigurations we have identified in these two events, but that we continue to improve consistently.
We believe that a stitch in time saves nine. And so, with these major issues addressed, we have already begun working on fixing other quirks and issues. We will also make improvements throughout Toph and Toph Drafts over the coming days enhancing your experience on the platform.
We also want to welcome everyone to Toph Community, and we want to encourage everyone to share their constructive (or even destructive) criticism here. It is worth stressing, that these recent events and every feedback that you have shared have helped us improve Toph. And, You can help us more by communicating your feedback directly to us.
Please feel free to share your thoughts and feedback here in Toph Community and for anything that you think deserves immediate attention, you can always @-mention me directly.