Leave the size for the toolbar itself set to "default", then change the sizes of the individual buttons that you want to be large. Each button has a "large icon" option (the toolbar setting overrides it, unless the toolbar setting is set to "default").
I had actually tried that before but the one icon I had tried it with wasn't scaling up so thought it wasn't designed for that, but it turned out to be specific to that one icon! After giving it another try all the others scaled up and I'll just replace the troublemaker.