Sorry, my English is terrible

Is it compliant for me to use OBS Studio for screen recording without using any timers? Because it's really difficult for me to learn how to set a timer

If I perform a reset midway through the process, there will be many changes after I start the GE route for the second time, such as Flowey no longer appearing in the previous section. Will this affect GE SPEEDRUN?

How to skip conversations faster? I tried to use both buttons on the keyboard at the same time, but it didn't have much effect

Are there any useful techniques for beginners with SPEEDRUN？

I don't think a timer is necessary here, so you'll probably be fine.

Resetting during the run is technically not against the rules, but it's always slower to reset than not to. If you're asking about if you can run the game again on a file that has been reset, then no. Before starting any runs, you have to delete all files in the %LOCALAPPDATA%/UNDERTALE folder. The in-game reset feature doesn't suffice for the reasons you mentioned.

To mash faster, pressing 'Z' and 'X' on the same frame doesn't help, as doing that will either skip the text or advance the dialogue, not both. The usual way to mash is to alternate the four keys in the order 'Z', 'X', 'ENTER', 'SHIFT' or 'X', 'Z', 'SHIFT', 'ENTER', whatever you prefer. It's also better not press any keys on the same frame, as you're just wasting key presses. If you're running on versions 1.02+, make sure to release each key before you press another key of the same type, as for example pressing 'ENTER' while 'Z' is pressed down will cancel that 'ENTER' input, same thing for 'X' and 'SHIFT'. If you're running on Linux 1.001 that isn't be an issue. Linux 1.001 is often crowned as the best version of the game to run on, as it has a superior input handling that make mashing a lot easier and glitches easier/possible. (Guide on how to get Linux 1.001 on Windows, as well as how to RESET before starting a run)

I assume you're running Geno, and there are a couple of major strats such as Seagrass Skip and Early Burnt Pan, but besides from those, it's mostly about doing things fast and RNG. If you're running other categories, there can be MANY more strats. You should check out the Undertale General Guide and the Geno/Neutral Glitchless guide. These guides are missing a few strats, but overall they are pretty good for beginners.

(Other main categories with guides: Nooch/TPE, TPEG, category extensions: EPC)

For extra help, you can join the Undertale Speedrunning Discord Server.