You can do that, but you cannot switch the source from the (Fairphone TWS) earbuds themselves. You need to activate/deactive the connection device ↔ earbuds on the source device.
I have been able to make the earbuds work with all the FP1, FP2, FP3+, a (now almost 14 years old) Mac and a Windows Surface:
I now mainly use my earbuds with my FP3+, but just yesterday used them on the old Mac again (works a bit wobblier though, probably because of the old Bluetooth standard on the Mac).