I was using the defaults in my original scene (which wouldn't denoise).
Then tried your scene which worked but I couldn't understand what was different.
It was just by chance that I put the samples back to the defaults and the denoisng stopped working again.
Setting them at 4 and 8 from now on.