Long answer:
A lot of folks have made a lot of hay about high mass bridges - how they affect sustain. Well, a string has two ends, and adding metal mass to the one with a tiny amount of metal anchoring it makes a much bigger difference i sustain and sound than adding more metal to the one that already has a decent amount of metal.
The amount of fret material affects the end condition of that end of the string - it affects how the string vibrates. A smaller fret not only has metal mass there, it means the vibrational forces into the wood are distributed across less surface area, so the wood appears "softer" to the string. Think of walking on snow in snow shoes or high heels (I've done the first, the second is a thought experiment) - the less the surface area, the more you sink in - the ground appears "softer" as a result. A similar thing happens with frets.
I've done neck swaps where the only variable between necks was fret size - the neck and fretboard materials were the same otherwise, and yes, smaller frets are less "clanky", and more "woody" sounding - it is a real, if somewhat subtle effect. I have a bass I'm rebuilding - it was too bright and clanky. There are several things I'm changing to "unclank" it, but yes, smaller frets are part of the recipe I'm using - I have a neck on order with smaller frets as part of that rebuild.
Short answer: Yes.