On hardest, you should get the enemy to set diplomatic stance to enemy earlier, to lure the catapults quicker. This is why I used my villager to attack the wall, since it was conveniently right next to him en route to the government center.