(4.2.5) PvP alerts triggering despite being off

[Cyrup]

It looks like this should be fixed for 4.2.5 but we need to test and confirm (at a later point).