Firefox has no such problem - so I'm pretty sure my ISP is OK (I have fibre at approx 40Megs)
So, with thanks to http://www.webnots.com/fix-safari-slow-loading-pages-mac-os-x/ I found the following seems to have fixed my problem.
Disable the DNS Prefetching in Safari:
defaults write com.apple.safari WebKitDNSPrefetchingEnabled -boolean false
(You can re-enable it by doing:
Simple as that - don't even need sudo! Just restart Safari and off you go.
(For info, MacBook Pro 15" running Sierra)