Silex ships with a ServiceControllerServiceProvider, which you can use to get controllers as services.
$app->register(new Silex\Provider\ServiceControllerServiceProvider());
$app['my.controller'] = $app->share(function ($app) {
return new MyController($app['my.service']);
});
$app->get('/info', 'my.controller:info');
This should work with your custom loader as well.