The first thing I thought about was to remove the validator from your myElement's ValidatorChain. You could get it within the controller with:
$form->getInputFilter()->get( 'myElement' )->getValidatorChain()
It seems like you can't remove from the ValidatorChain, just add. Check this post. Matthew Weier O'Phinney, from Zend, explains why it can't be done and a possible solution for your scenario.
The way I solve this problem, is checking the 'remove condition' when I create the validator in the FormFilter class. If you use annotations I think it doesn't works for you, so Matthew suggestions is the one you should use.
Or you could try the one in this post from @Stoyan Dimov: define two forms, a kind of BasicForm and ExtendedForm. The first one have all the common form elements, the second one is an extended one of the other with the rest of fields. Depending on your condition you could use one or another.