It is a legitimate concern, but whether you can restore any backup depends mostly on the quality of your backup strategy.
With computer tech, things can go wrong at any time in any way. So have plans B, C etc. ready if you want to be safer.
What can possibly go wrong? And be very sceptical here, don’t assume any given stuff will work!
What would you do then?
What can you perhaps do beforehand to make your life easier in such a case?
For a start:
Don’t just blindly trust any backup software which creates one monolithic data file out of several components like e.g. the OS on a phone and your personal data on a phone.
It’s super convenient if this works, as e.g. a TWRP backup usually should. But as you legitimately are concerned about: What if this somehow fails to restore?
Then you could still save the day if you had a way to reinstall the OS you need as well as a different backup of just all your important data, and the latter in a preferably plain way so you could even restore it to a different phone, if all else fails:
Is your phone the only place you keep your contacts, messages, calendar data on? Bad idea. If possible, regularly sync this to a computer or to the internet. Or try to figure out whether the Apps you use managing this data already sync it or whether they can at least backup this data into files you can then transfer somewhere else.
Important files in Internal Storage such as pictures, videos, documents, downloads etc.? Just copy the whole Internal Storage over to a PC via USB (this can perhaps even be done when TWRP is running, depending on encryption). It’s just a bunch of folders and files, which is good for simplicity.
Keep everything around which would be necessary for you to install the OS on your phone from scratch.
Just upgraded to a major new Android version? Keep the last install file(s) for the older version around … might come in handy.
For being prepared to exchange the phone, should the need arise:
Can you export contacts, messages, calendar data into generic file formats which other Apps could import, or do you have a sync method in place that would work on a different phone?
You have a copy of the whole Internal Storage on your computer? Good. But do you actually know whether your important files are really in it, and where exactly they are? Have a look. When changing phones it will not be the best concept to just dump the whole Internal Storage from the previous phone over, even if there is a chance this may possibly work when staying in the Android realm.