There seems to be an intermittent issue with swapping XB1 controllers. I am able to get into a state where the UI does not respond to any input. While it is in this state if I swap the controller again (to a third controller) the game will exit back to the main menu.
It may be that your application needs to be running for an extended period of time to exhibit this bug… because a few days ago it passed all my controller swap tests. Was able to replicate this bug twice today.