Increasing refinement panel MetadataThreshold="1" from 1 to many has no effect

I do not want to show certain refiner categories when there is only one or two options to click in the category, so I have increased the threshold for all the relevant refiners to 3 or 5. Unfortunately this doesn't seem to have any effect at all. All refiners still show even though many only have one option.

I have double checked that the settings have been saved and the page checked in and published. I have worked with refiners many times before, but never seen this problem. Any ideas?

And yes, I have unchecked "Use Default Config"

Best regards

Henning

<Category    Title="RawMaterialCategory"    Description="Process"    Type="Microsoft.Office.Server.Search.WebControls.TaxonomyFilterGenerator"   MetadataThreshold="3"    NumberOfFiltersToDisplay="5"    MaxNumberOfFilters="20"    ShowMoreLink="True"    MappedProperty="ows_MetadataFacetInfo"    MoreLinkText="show more"    LessLinkText="show fewer" ShowCounts="Count" />