If I understood your question, I would try this:
- find the convex hull of your set of points, which I will call S, using
convhull
find the convex hull of the set S', where
S' := S - points_defining_the_convex_hull(S)
(i.e., S' contains the points of S which do not "enlarge" its convex hull, thus the ones which are inside the convex hull itself.)
make the difference/proportion between the volumes of S and S' (trivial, both are convex).
There is a strong assumption on the topology of the hole considered, i.e.
"the convex hull of the S' is the hole".
If you have a more complex topology of holes you can't avoid making active use of it (my guess, of course).