There are non-binary people too. Being bigoted towards them is generally called transphobia, though maybe nb-phobia would be more accurate.

Edit: also this is usually trying to claim that sex = gender so is transphobic from that perspective as well.