As far as I understand it the Xposed installer itself does nothing, it’s just needed to install the modules. So there is no drawback.
Gravity Box has a fix if you have the problem that your location is sometimes calculated inaccurately although you have a GPS fix. It doesn’t help with the TTFF (time to first fix)
For more info see the gps guide