Both families owned slaves but the way I see it, atleast the Grays didn't resort to selling moonshine to the Lemoyne Raiders for the local mafia, kidnapping a child, or locking their children in outhouses to hide them.
Sean burned their crops and murdered a bunch of the guards at Caliga Hall, the slave thing is obviously inexcusable but they were justified in executing him like they did