Found out myself. Of course the translation files need to be loaded... Also the Symfony config component needs to be added due to a dependencie to the FileLoader class.
$translator = new Translator('de');
$translator->addLoader('xliff', new XliffFileLoader());
$translator->addResource('xliff', '<path-to-compontnt>/Resources/translations/validators.de.xlf', 'de','validation');
$builder = new ValidatorBuilder();
$validator = $builder
->setTranslator($translator)
->setTranslationDomain('validation')
->getValidator();