One way to do this is to apply a clip-path
to the root svg element.
If you want it can be a simple rectangular region, or another more complex shape, as in the example below:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg xmlns="http://www.w3.org/2000/svg"
xmlns:xlink="http://www.w3.org/1999/xlink"
viewBox="-100 -100 300 300"
clip-path="url(#clip)">
<defs>
<clipPath id="clip">
<rect width="100" height="100" rx="10"/>
<path d="M40 99l10 11 10 -11z"/>
</clipPath>
</defs>
<rect id="background" width="120" height="120" fill="slateblue"/>
<image xlink:href="images/man.png" width="100" height="110"
preserveAspectRatio="xMidYMax meet"/>
</svg>
View it online here.
In your case you would probably want just one <rect>
inside the <clipPath>
element that had the same dimensions as the viewBox.