Good morning!
Crazy enough: I seem to have used a different port - and while it then became visible in fastbootd, it didn’t for fastboot.
Kudos & thanks to hirnsushi!
As documentation for those having the same problem (it seems as if fastbootd took over even low-level USB communication):
Bootloader/Kernel fastboot on the wrong port:
not listed in fastboot devices
not listed in lsusb
dmesg output showing wrong device speed:
[18971.925179] usb 1-1: new full-speed USB device number 109 using xhci_hcd
[18972.075191] usb 1-1: device descriptor read/64, error -71
after switching to Recovery/Fastbootd (side remark: the mentioning of ‘bootloader’ is not helping, here 
) :
device listed in fastboot
device listed in lsusb as
Bus 001 Device 120: ID 18d1:4ee0 Google Inc. Nexus 4 (bootloader)
dmesg:
[19463.145117] usb 1-1: new high-speed USB device number 119 using xhci_hcd
[19463.327071] usb 1-1: New USB device found, idVendor=18d1, idProduct=d001, bcdDevice= 4.19
[19463.327077] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19463.327081] usb 1-1: Product: FP4
[19463.327084] usb 1-1: Manufacturer: Fairphone
[19463.327087] usb 1-1: SerialNumber: XXXXXX
Bootloader/Kernel fastboot on the only holy, just and right port:
[19013.535165] usb 1-2: new high-speed USB device number 113 using xhci_hcd
[19013.715752] usb 1-2: New USB device found, idVendor=18d1, idProduct=d00d, bcdDevice= 1.00
[19013.715754] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[19013.715755] usb 1-2: Product: Android
[19013.715756] usb 1-2: Manufacturer: Google
[19013.715757] usb 1-2: SerialNumber: XXXXXX
Matter resolved, TWRP successfully booted via fastboot 