1st Person Hands Poking the 3rd Person Character

So I built a UMA based player character in 3rd person combat mode and also 1st person combat mode. Apparently if I am holding a sword or "body", then when I try to look up in 3rd person mode the hands are poking me in the back and making the character move around. The same happens when I am moving between 3rd and 1st person modes. Is there a way to disable the colliders of the 1st person hands while in 3rd person?
 
Are you sure that it is the colliders? The first person item colliders are on the Overlay layer so that won't affect the character movement. The third person item colliders are on the SubCharacter layer so that also won't affect the character movement.
 
Top