I don’t know if it makes any difference, but I was able to update twrp to version 3.1.1-0 from within twrp. I went into recovery, copied the twrp .img file to the sd-card via adb, then installed it with the “Install” button. What you have to be careful about: After install is finished, you need to restart directly into recovery again; in case you do a normal (system) restart, twrp gets set back to the previous version.
Anyway, my os downgrade didn’t work (no luck with @maba007’s suggestion either since I couldn’t get into fastboot anymore), but I managed to get 17.06.04 running by following @Roboe’s instructions, only with the 17.06.04 (OTA Build switcher version) instead of the 17.04 file.
If you want to downgrade, it looks like your only option might be a factory reset. Unless someone more skilled than me can come up with some other magic.