All this because of a bug with the ubiquity installer that mismaps (and it is next to impossible to correct) the location of the bootloader.
Reinstalling GRUB doesn't change anything.
Removal of the FAT32 EFI boot partition and installing grub to the main EXT4 partition where the system is installed to, also doesn't work.
Installing to a solo ext4 partition and manually installing grub doesn't work either (despite marking it as "boot" in gparted).
However, debian and other distributions that don't use ubiquity don't have this problem. You are free to use any/all distributions for a NON-UEFI USB installation on older machines.