We had this issue, and it turned out that we were simply not using the “correct” font name. In our case, we were using “Menlo” instead of “Menlo-Regular” when instantiating our SKLabelNode, and it was incurring the several-second penalty. Once we used the correct font name, the delay no longer happened.
(Curiously, the SKLabelNode still found Menlo and used it, so it was not immediately obvious that we had the wrong font name. Presumably, the delay is caused by the system having to figure out the appropriate substitute to use. It does a good job, finding the font we intended to use, but it takes a while to do it, hence the delay.)