it happens all the time and is easy to reproduce, It happens when moving the mouse at an angle. My belief is that with the angle of the movement, if the game sees the angle as a little bit to the side first before you start to move up, it thinks you are going sideways. One way to fix it would have the gem move when you break the edge of the gem, not at the first little movement of the mouse. So you would have 4 walls around each gem, break the wall to move it
And can we disable right clicking moving gems, or at least make it move gems at the same location as the mouse? If the Devs would like me to make a video demonstrating either or both of these issues let me know and I will be happy to. But since it is a year old, I am not sure me making a video would be helpful, or looked at.