Size or access limit for saved variables?
 

Greetings all,

Just wondering if anyone has had experience with any sort of limit to the saved variables table size, or how many saved variables can be created/set/changed in a period of time? This is WITHOUT the feature to instantly save enabled.

I have a large table (maybe a couple MB) that gets built dynamically and stored to a temporary sub-table in the account wide saved variables. Using zgoo I can see that everything is there, and that the target value in question is indeed set as expected.

However when I reload the client this setting and others have reverted back to their default values.

Can't give you exact info but I recall the limit being more about number of keys than a raw size limit

I'm not certain where things stand after the updates awhile back. Pre-improvements, a low information density table at 10k rows could have save problems.

I can only provide you this information:
https://www.esoui.com/forums/showthr...+variable+size

Thanks, some very helpful information in that link discussion.

The bit about number of keys rather than size was helpful as well. I am testing a solution for compression of core data into character separated strings and then dynamically 'unpacking' that on addon load. Which should be relatively quick using zo_strgmatch(tString, "%S+") type of step process.

According to wiki Lua can process text strings in excess of 230,000 characters so this should be more than adequate for any conceivable use scenario.

I believe the ~2k characters per string in the saved vars is still in play.

Oof. Well good to know. Perhaps a slimmed down set of simple k/v pairs so the rebuild routine has minimal work to do while leaving the database containing as few keys as possible would be the best compromise.

EDIT: Although, that thread Bertram linked makes it seem like this should no longer be a factor after the move to 64 bit? Maybe my issue is unrelated, although a cleaner saved variable table is better in general.

Maybe ZOS_DanBatson is able to give an insight here about the 2k character limitation. Perhaps it's still a "bug".

The single string character limit of 2000 is still there. This limitation is there for both performance and memory reasons. It is not a bug.

Talking about performance, would it be possible to allow addons to store raw strings as saved variables? For example instead of specifying "## SavedVariables: MyAddOn_SomeVar1 MyAddOn_SomeVar2" in the manifest an addon could instead use "## RawSavedVariables: MyAddOn_SomeVar1 MyAddOn_SomeVar2" and when the variable is a string, the game just dumps it into a file <addon name>_<varname>.dat as is. MyAddOn_SomeVar1 would for example be stored in MyAddOn_MyAddOn_SomeVar1.dat and on load the game just needs to load it back into a string without having to do any parsing and interpreting.
I imagine that would make loading screens a lot faster for data heavy addons that do not need everything in Lua tables right away.

One thing I have always wanted was to have a switch, that would compress or decompress the file without the mod author having to make an external EXE to do so. Maybe you could add that to the manifest file with ## UseCompression: true for example.

If you choose to do that please allow the mod author to change that value as needed. Changing it would cause ESO to automatically know the difference and compress or decompress the information. Setting the compression to true for normal operation and false so the author could easily read the file. Not being able to see the contents of the file is not helpful.

Currently save games use LZ4 compression for Skyrim SE and Fallout 4 and I do not feel anyone has ever complained about there being a performance issue. LZ4 is intentionally meant to be fast even if it doesn't compress as much as other formats.

Would you please consider doing that? I don't see it being needed at all for any mods except mods with data collection such as Harvest Map, Master Merchant, or Tamriel Trade Center to name a few.

Using an exe from https://github.com/lz4/lz4/releases I was able to compress a file to be significantly smaller.

Once you unzip the attached files, as you can see the uncompressed .lua file is 6610KB and the compressed version is 472KB,

Thanks for confirming this, it likely saved me a bunch of time testing. :)

Since the LuaVM was changed to 64 bit the key/var limit is pretty much nonexistent. However if saved vars loading takes too long the client may crash and can't login. I played a bit with it a while back and found that for me 130MB was about as much as possible with very long load times.