Gems of War (probably any other game) can’t tell if you’re trying to use a controller for the game or not. Unless there’s an option to pick one over the other in the games options.
So the only solution. (in my opinion)
Is to unplug the controller.
But if you want the devs to actually weigh in on this, you have to change this thread to bug report is essentially what I’m trying to help you with.
(Nevermind I see (Kafka is here now.)