Honestly you're probably better off talking in the game's forums itself, asking in their discord if they have one, or talk with the moderator that rejected the run personally. I mainly replied because you claimed the run was "getting rejected without any clear reason".
Using an Xbox controller is just fine, plenty of emulator runners using XBox or PS controller. In fact it's highly recommended to start with one of those if you already own one of these controllers. N64 controller is mainly a gold standard because of the C-buttons and notches, which both are fundamental to the run. Thing is if you go for the N64 controller, you want to get an OEM (original) controller because the 3rd party USB ones have sticks and buttons that wear down incredibly fast. Other factor to keep in mind with N64 controller is the stick wear in general from doing Bowser throws and even if you get an OEM N64 controller, it's recommended to have a second one to swap to just for throws, whether that's a Hori, a third party like the Brawler, or a not so good OEM.
As long as the video and audio come in clear, should be fine for the most part. Would probably want to double check with the moderators of the games you are running first since some may not allow it or would require a capture card when you reach certain time goals. If you're running on emulators though, you can just get OBS to record for free.
You should also try to contact the moderators outside of the site if you haven't already done so. Most games should have their own discord at this point but even if not, moderators will usually have a few socials to try to reach out to them on as well.
Wouldn't even call it a new idea. There was an any% tournament a few years back and at least one of the sets of races, the runners did 8 star as described instead of trying for 0/1 star. Definitely wouldn't be worth adding a new category for but nothing is stopping you from doing it if you want.
You need to talk with the specific game's community either through their forums or their discord provided they have one about adding any categories to any game.
Another point with 2 would be streaming to a service directly like Twitch or YT to get around the 1 hour limit for capturing. Only thing would be if it's on Twitch making sure the VoDs are saved for the 7-14 days (whichever you have).
Seterra runs definitely exist https://www.speedrun.com/series/seterra_series
If it's just an issue of parents not wanting you to have a YouTube account or something at this point, you can always talk to them about the situation with your hobby needing video submissions and try to make a compromise where you can have a YouTube where the speedrun videos are put as 'Unlisted', this way they're still able to be submitted and viewable to the mods and anyone looking at the leaderboards without being publicly available to find on YouTube itself.
Mods and verifiers have real lives and considering it's a roblox game, possibly other games with much larger queues they could be helping with. If it's still not approved/rejected in 21 days which is what SRC says should be the max wait, try to DM the mods or join their discord to ask about the run.