No, afaik they are only loaded via the txt files of the addons (there is the ## SavedVariables: tag in it) and if the addon is not enabled their SV file should not get loaded.
So loading times do not decrease by deleting or moving any disabled addon's SV, as far as I know.
Loading times decrease by removing contents from SV files of loaded addons, so their files get smaller. By the addon settings e.g. to strip old data or accounts or by manually checking the files with a text editor like described here, and e.g. delete sub tables of old @accountNames or Character names/IDs that are not there anymore:
https://www.esoui.com/forums/showthread.php?t=9235