Me and my friend want to do a co-op speedrun of a game that doesn't already have a co-op category. He says that since it isn't defined in the rules for that game that co-op should be allowed in any%, but i'm pretty sure that it being 1 player is general rule in any% and we'd have to create a category for co-op. What would we have to do?

Generally speaking, unless explicitly a two player category, all runs are expected to be performed by one person. Usually, co-op runs are defined by cases when you can have multiple players in the game at once (like local split screen); you don’t usually see categories were people trade a controller back and forth during the run. If you’d still like to try to get the category added, do a run and then bring it to the attention of the mods/community for the game and see if they’ll be open to adding a category for it.