I definitely don’t think the target minions’ mods apply; yes, their dice are being rolled, but they’re not taking any action — you are — and it’s your turn / reaction, and your ability that you’re using.
As for whether your mods apply . . . I dunno.
Wait, I think that we might’ve already discussed this in my Iron Five thread, starting here. I think that your conclusion then was that any die or dice that are used to determine the value of an action are effect dice, and thus can have mods applied to them.