I've noticed that after I reach the flagpole, it takes me about 1-2 seconds longer to walk to the castle than other runners. Does anybody know how I can fix this issue?

Extract from Warpless No OoB Complete Guide:

The speed of walking after touching a flagpole depends on the acceleration of the cat when it touches the flagpole. As a result, we want the cat to be accelerating to the right (i.e. moving to the right while speeding up or moving to the left while slowing down) because this will make the cat walk towards the castle faster.

The easiest case for Fast Flag is in L1, where after jumping over the flagpole, you jump towards the flagpole then hold right while touching it. In L2-L5, it is done by jumping to the right into the flagpole, holding left briefly, then turning right again just before touching the flagpole. This saves 2 seconds in L1 and 0.75 seconds for each of L2-L5, meaning a total of 5 seconds can be saved with Fast Flag.