Hi mvdb , I thought this option was enabled by default in earlier versions. I'll make sure to take a look and check the behaviour 🙂 Probably we need to change this back to default and making the option opt-out.
The extra loadtimes depend on the amount of fonts you load on your website. We currently use the FontAwesome 4.7 CDN ( bootstrapcdn.com ) by default. For the FontAwesome 5 icons we use the use.fontawesome.com CDN., so there is a big chance the users have loaded the files before on another website. But the new users indeed need to load them.
I also see your theme is loading in FontAwesome 4 icons, but only for the theme components. But you may be able to manipulate that and use these resources to improve the page speed 🙂