There are 4 difficulty levels: Easy, Medium, Hard and Heroic (You have to unlock Hard and Heroic).

When the difficulty is high, you will have more things to do in the challenges so you will have to be faster, and you will have less indications.

I was wondering if there's a difference between the difficulty we can choose in the Options and the one we have to choose when we press Adventure (or Boyard Party), so I asked Microids which modes are affected by the difficulty chosen in the Options and they replied:

"La difficulté prend généralement effet sur totalité des modes de jeu, sauf si un mode de jeu vous propose de la modifier avant de démarrer." (The difficulty usually takes effect on all game modes, unless a game mode offers you to change before starting.)

So the difficulty in the Options is only for Training mode because you have to choose the difficulty when you go to Adventure or Boyard Party.