try https://drupal.org/project/ctools_automodal
and use hook_form_alter() to edit registration form https://api.drupal.org/api/drupal/modules%21system%21system.api.php/function/hook_form_alter/7
I used it for making popup contact form, worked like a charm Maybe you will need some jQuery customisation for fadeout or other jQuery effects