Thanks Jeroen, that's exactly what I was looking for. Normally Google Pay is managed via the Google Pay app, but the Google Pay app is not available in The Netherlands.
I can indeed remove cards by going to Settings > Google Services & Preferences > Account services > Google Pay (vanilla Android 11 on Pixel device).