Ok, so I think the problem was that the 3rd person animator was somehow controlling my First person objects. I was using the same Avatar and Animator Controller on both. I'm still not sure why that was the case, but when I explicitly created a new Avatar for the first person hands, it fixed things.