Since a cell can disappear during insertion of a new point, the handle you have saved are not guarantee to point on what you expect.
You have a crash because you use the triangulation hierarchy that internally creates and remove cells in the internal container. If you use CGAL::Delaunay_triangulation_3, you will not have the crash.
For your problem, you should store a quadruplet of Vertex_handleS and use the is_cell function (documented here).