Alright, it seems like I undertand how it works 
My default button uses an icon with no overrides. As soon as I switch to a different style, primary, thr override applies and the icon turns white. And when I revert back to the default button style the colour stays white because there is no instruction to revert it back.
The same happens with another style where I make the text and the icon blue (toolbar button). Interesting enough, the toolbar button hover state uses the same default state icon with no overrides. When I set state to hover, the icon stays blue, well, because the blue is the override now and it has a priority over the default fill colour.
I fixed the issue by forcing colour override on my icons. Literally, I choose an icon and set its colour to something else and then set it back to the value I need. So now there is an override and everything works as expected