I read somewhere that a good optimization could be to first check if the bounding spheres of two vessels intersect, followed by bounding boxes (not axis aligned), followed finally by the collision mesh, which would either use multiple convex hulls or tri -> tri intersection tests.
You can throw out the sphere. You check the bounding box, then you check the collision mesh. Checking for sphere means calculating distance, which is more expensive than checking the bounding box, for which a check merely involves comparing a maximum of 6 variable pairs at worst, and potentially as little as two if the result is negative.
However, the issue is with what you compare exactly. Iterating through bounding boxes and comparing each to each is inefficient, so you usually build a quad tree with point values that allows to find nearest neighbours in a 3-d space a
lot faster. But that isn't the real problem either. Somebody like artlav can probably do all that during his lunch break.
The real work starts once the collision is detected. Calculating impact vectors, applying energies, and then most of all what on earth now happens. There's multiple levels of details that one can address here. You can just treat both objects as indestructible spheres, pool style, in which case the result becomes fairly easy to calculate, but also oftentimes hilarious, especially with time accelleration involved (accidentally being ejected from the solar system at light speed is one of the classical ones...).
Next you can start consider mesh geometry to actually turn part of the energy into rotational velocity based on the impact, then you can start considering things like rigidity and how much of the impact energy is absorbed by each object instead of just redirecting momentum, then you can start wondering about how that energy would
deform the objects, etc. etc.
Really, detecting the collision is the easy part of collision detection. The hard part is all that comes afterwards, which is why libraries like bullet exist.