Not sure why that's CTDing, but why not just create the MGROUP on the stack?
In otherwords, just do
MGROUP_ROTATE payloadrotatex = MGROUP_ROTATE(payload[pns].msh_idh[nm],NULL,NULL,reference,_V(1,0,0),(float)2*PI);
Then if you're passing it to a function that needs a pointer to it, just do it...