Every time I try to do it, Swank and the other chairmen go hostile instead of just Benny and his personal bodyguards, am I dumb and just doing something wrong?

Make sure you’re only hitting him and his guards with the explosion, and put the grenade launcher away to de aggro everyone else.

Thanks