After the run posted by BrawlerJesse recently who had some insane loading times, much faster than any we had ever seen before, it prompted me to see just how much loading times vary between digital runs and unfortunately it varies a lot.
Below is a spreadsheet comparing the loading times of 9 of the top 200cc Mushroom Cup (No Items) times.
https://docs.google.com/spreadsheets/d/1N3cJUu65xIRVwweUeHPfg2TXGPKgWy10eMupuLs66jY/edit#gid=0
As we can see, BrawlerJesse's time saves around 11 seconds just in loading time over the average of the other runners. Even among the other runners, we can see that if we were sorting by loadless time then the leaderboard positions would change a lot. It's pretty unfair all around and people get a serious advantage by just having a faster switch.
This problem is harder to see on individual cup runs since the best times will more likely have good loads, so I also compared some digital runs from the Retro Tracks portion of some 32 and 48 Track 150cc (No Items) runs.
https://docs.google.com/spreadsheets/d/1LpRnO3tSwP01OfBW81d7oYCqvUtYzIwM7dejTNI60Mo/edit#gid=0
From this we can see that among 6 digital runs, the loading time varies significantly from an average of 6.967s and 7.779s, over a 48 track run this can have up to a 39 second difference. I assume that there will be people with slower or faster switches as well so the difference could be larger.
As a result, I believe we really do need load removals on the leaderboard, and I would be willing to invest time into making something that makes it easy to do this. For it to be acceptable I would say the minimum requirements are:
- We need to determine what part of a run counts as loading time while at the same time remembering that it needs to be easy for software to detect otherwise it will have too many false positives.
- It should be easy for runners to get the loadless time for their run, assuming they have a capture-card video. Stable and clear video/phone recordings may also work but we will have to test.
- It should be easy for moderators to verify that the loadless times people are submitting are accurate.
- The accuracy of the load remover should be verified against manual load removals to be sure that it is accurate.
A secondary goal:
- There should be a program that runners can install on their computer to do live load removals, but it would need to be re-checked afterwards. Perhaps making use of https://github.com/ROMaster2/LiveSplit.VideoAutoSplit
As I mentioned, I'd be willing to invest time into making something like this, but only if the community wishes to use it and adopt it as it would take some time to get right.
My questions for the moderators:
- Would you be willing to adopt something like this if it hit the above criteria?
- Are there any other requirements you would look for to make this more accurate, accessible, or verifiable?
My questions for everyone:
- Would you be willing to put your runs through a load timer tool before submitting if it was a top time?
- Do you even care about load removals? Or would you be fine accepting that you may have a faster/slower switch than another runner.
- Should it be enforced for the top 5 or so runners for each category?
I believe that by doing this and getting it right it will bring more runners into MK8DX. Especially for those who have a cartridge version of the game.
Seeing this come to fruition would be a dream come true. A fast, automated, and reliable way to remove load times seems more attainable than ever, and based on your collected data a resource like that would level the playing field significantly!
If this is implemented, it might be worth requiring the area of the screen examined for load remover to be clear of splits/other additions. I don’t know how necessary that would be, though, as I can’t recall any player covering up the center part of the screen. Just a thought.
I also have a couple of questions—first of all, (maybe a mod who knows the site can answer this), how would the leaderboard logistically handle such a change? Is it possible to include both the IGT and RTA timings, or will only one be displayable at a time? If only one, which should it be, and how will it be clearly labeled?
Second, “stable and clear video/phone recordings may also work” worries me a little. Not having access to a monitor or capture card, I use my phone to record my runs. How likely is it to work for that? My recordings are pretty high-quality for a phone, but will there be a manual backup if the load remover doesn’t work for me or others? Even if there is, I hate to put someone through all that work for my sake, especially for a 48-track run.
Regarding how the leaderboards can handle times with and without loads, it could be done like how the Skyrim boards work: https://www.speedrun.com/skyrim
With regards to how well it would go on phone recordings. I have honestly no clue, it may work well if the recording is high quality and with good lighting, but if the camera moves around or if there is too much light then it would probably not work. I will have to see when I get it working.
I could see this being applied like the Skyrim example, except maybe the other way around for the time being: RTA first, loadless collumn next. That way I think it wouldn't alienate newer players so much when submitting their first runs, and if people still want to see the boards ordered by loadless times, the Filter options should work quite nicely as an alternative.
Also, top 5 does sound about right to me.
Personally, so as long as the programs/tools are quick and easy to use, I wouldn't have many quarrels switching to this system. Time is definitely something I've been lacking as of late, so I feel like the aide of programs would be very much welcome to help accomplish this. Well, as long as the loads are removed in a consistent manner among multiple players, that is!
IIRC, Obsidiaus, you previously suggested separating the leaderboard based on digital/cartridge. I still believe that to be unnecessary, especially because we may now be able to provide loadless times that will get rid of the loading difference between versions.
The reason this topic is coming up again is that we now know that the loading times are more inconsistent than we realized even within the digital version. We also didn’t have the tools available then to make loadless retimes feasible; Vike has done a lot of research and work on that in the past few months. Only now does it seem like removing loads for all top runs is realistic.
If I’m wrong or misremembering any of that, Obsidiaus, I’m fine with being corrected politely. You’re opinion is valid, and I want to hear it. But please don’t accuse us of hypocrisy so blatantly. I’m doing my human best to make this community and leaderboard run smoothly, and most others on this site are too.
I've got a generic speedrun timer website done which you can use https://mk8dx-speedrun-timer.azurewebsites.net/
I'm still working on the load detection part though which is the hard part. I don't think I will have it done for another month.
How is that even possible? Some sort of code that looks at the picture and realizes it’s a loading screen?
Yes, it works by looking at the screen and determining if it is a loading screen.
And an update is that I still plan on finishing it. it's just I recently moved country and so I haven't had access to my computer to continue writing the code. I should have it back sometime over the next week though so I can finish it off. Probably another month sorry :/
Over the past couple of days, I analyzed the loads and loadless times for the top 3 times for Nitro 150cc No Items (and I have to give Vike credit for most of the formulas/timing/layout I used; I have very little experience with Excel/Google Sheets, so for the most part I copied and modified his sheet comparing Doodle and John's 200cc 48 Track runs).
Like Vike, I started the load timing when the menu buttons had completely faded away and ended the load timing on the first frame the track name appeared (I think this is sometimes inconsistent by 1-2 frames, but [spoiler] as you'll see the differences were large enough that that inconsistency is irrelevant in this case).
BrawlerJesse has the RTA record for this category, but because of his infamous fast loads I wanted to see if his time held up compared to both Power and Miso_ra_never, who are less than 0.1 seconds apart from each other RTA. This also provided an opportunity to compare two "normal" Switches' load times.
Sheet here: https://docs.google.com/spreadsheets/d/1q1PklO7D_ey833m7jo7BFKj05lx320hKYT3IE0x_VSE/edit?usp=sharing
Highlights:
RTA timing:
- BrawlerJesse - 37:59.670
- Miso_ra_never - 38:04.422 (actually 2nd, but listed as 3rd because Power didn't include ms)
- Power - 38:04.486
Average load time (w/o MKS):
- BrawlerJesse - 6.980
- Miso_ra_never - 7.523
- Power - 7.803
Loadless times/Total Track Time:
- Power - 35:57.269
- Miso - 36:01.872
- BrawlerJesse - 36:06.143
So, even though BrawlerJesse's RTA time is the fastest of the three players, he saved 14 seconds by faster loads alone (compared to Power), and Power holds what I consider to be the true record - he achieved an incredible sub-36 loadless time. He saved time over Jesse on almost every track, with the notable exception of SGF (he didn't go for the NISC, while Jesse made it 3/3). Even among "normal" Switches, though, there is quite a bit of variation in load times, and Power lost 4.7 seconds compared to Miso_ra_never as well.
In conclusion, Vike's load remover will fix all this and I won't ever have to go through this manually again. :P
But really, this reaffirmed to me that this is indeed an issue that needs to be addressed, so huge thanks again to Vike for developing that, even with everything that's been going on for you recently!
I've completed the backend algorithm and am confident it can time loads correctly, just working on the frontend. I've had a lot less free time than I expected lately.
So, unfortunately after testing a bunch on different videos, I just don't think the load timer is consistent enough or usable enough for us to use, and it is also going to cost me way too much money to host because of how much CPU the video analysis uses. So I'm going to have to park it :(
I just don't have the time as of late to fix it up/redo it to properly work. I'm sorry for everyone who was waiting for it to be complete. I might pick it up again someday or try and see if I can find some other people who want to work on it together.
For context, I would estimate that server hosting would cost me about $60-80 per month