I never used Google, so I’m no expert, but I’d guess you can remove your Google Account in Settings > Accounts.
Whether that is enough to be save I don’t know. Especially if your account was already hacked they could have opened some backdoors in your phone so they don’t need your account anymore to enter.
I can’t tell you if it’s a good time to panic, but if you do then to put your mind at ease you could do a hard reset and/or (re)install FP (Open) OS.