Thread Tools Display Modes
05/19/20, 08:55 AM   #1
Phinix
 
Phinix's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 186
Size or access limit for saved variables?

Greetings all,

Just wondering if anyone has had experience with any sort of limit to the saved variables table size, or how many saved variables can be created/set/changed in a period of time? This is WITHOUT the feature to instantly save enabled.

I have a large table (maybe a couple MB) that gets built dynamically and stored to a temporary sub-table in the account wide saved variables. Using zgoo I can see that everything is there, and that the target value in question is indeed set as expected.

However when I reload the client this setting and others have reverted back to their default values.
  Reply With Quote
05/19/20, 09:28 AM   #2
Kyoma
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 125
Can't give you exact info but I recall the limit being more about number of keys than a raw size limit
  Reply With Quote
05/19/20, 09:28 AM   #3
Rhyono
AddOn Author - Click to view addons
Join Date: Sep 2016
Posts: 659
I'm not certain where things stand after the updates awhile back. Pre-improvements, a low information density table at 10k rows could have save problems.
  Reply With Quote
05/19/20, 10:04 AM   #4
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
I can only provide you this information:
https://www.esoui.com/forums/showthr...+variable+size
  Reply With Quote
05/19/20, 12:54 PM   #5
Phinix
 
Phinix's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 186
Thanks, some very helpful information in that link discussion.

The bit about number of keys rather than size was helpful as well. I am testing a solution for compression of core data into character separated strings and then dynamically 'unpacking' that on addon load. Which should be relatively quick using zo_strgmatch(tString, "%S+") type of step process.

According to wiki Lua can process text strings in excess of 230,000 characters so this should be more than adequate for any conceivable use scenario.
  Reply With Quote
05/19/20, 01:44 PM   #6
Rhyono
AddOn Author - Click to view addons
Join Date: Sep 2016
Posts: 659
I believe the ~2k characters per string in the saved vars is still in play.
  Reply With Quote
05/19/20, 04:04 PM   #7
Phinix
 
Phinix's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 186
Originally Posted by Rhyono View Post
I believe the ~2k characters per string in the saved vars is still in play.
Oof. Well good to know. Perhaps a slimmed down set of simple k/v pairs so the rebuild routine has minimal work to do while leaving the database containing as few keys as possible would be the best compromise.

EDIT: Although, that thread Bertram linked makes it seem like this should no longer be a factor after the move to 64 bit? Maybe my issue is unrelated, although a cleaner saved variable table is better in general.

Last edited by Phinix : 05/19/20 at 04:46 PM.
  Reply With Quote
05/20/20, 01:54 AM   #8
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
Maybe ZOS_DanBatson is able to give an insight here about the 2k character limitation. Perhaps it's still a "bug".
  Reply With Quote
05/20/20, 08:23 AM   #9
ZOS_DanBatson
ZOS Staff!
 
ZOS_DanBatson's Avatar
Yes this person is from ZeniMax!
Join Date: Jul 2015
Posts: 171
The single string character limit of 2000 is still there. This limitation is there for both performance and memory reasons. It is not a bug.
  Reply With Quote
05/20/20, 10:01 AM   #10
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,566
Talking about performance, would it be possible to allow addons to store raw strings as saved variables? For example instead of specifying "## SavedVariables: MyAddOn_SomeVar1 MyAddOn_SomeVar2" in the manifest an addon could instead use "## RawSavedVariables: MyAddOn_SomeVar1 MyAddOn_SomeVar2" and when the variable is a string, the game just dumps it into a file <addon name>_<varname>.dat as is. MyAddOn_SomeVar1 would for example be stored in MyAddOn_MyAddOn_SomeVar1.dat and on load the game just needs to load it back into a string without having to do any parsing and interpreting.
I imagine that would make loading screens a lot faster for data heavy addons that do not need everything in Lua tables right away.
  Reply With Quote
05/20/20, 11:56 AM   #11
Sharlikran
 
Sharlikran's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 626
One thing I have always wanted was to have a switch, that would compress or decompress the file without the mod author having to make an external EXE to do so. Maybe you could add that to the manifest file with ## UseCompression: true for example.

If you choose to do that please allow the mod author to change that value as needed. Changing it would cause ESO to automatically know the difference and compress or decompress the information. Setting the compression to true for normal operation and false so the author could easily read the file. Not being able to see the contents of the file is not helpful.

Currently save games use LZ4 compression for Skyrim SE and Fallout 4 and I do not feel anyone has ever complained about there being a performance issue. LZ4 is intentionally meant to be fast even if it doesn't compress as much as other formats.

Would you please consider doing that? I don't see it being needed at all for any mods except mods with data collection such as Harvest Map, Master Merchant, or Tamriel Trade Center to name a few.

Using an exe from https://github.com/lz4/lz4/releases I was able to compress a file to be significantly smaller.

Once you unzip the attached files, as you can see the uncompressed .lua file is 6610KB and the compressed version is 472KB,
Attached Files
File Type: zip uncompressed.zip (272.7 KB, 461 views)
File Type: zip compressed.zip (292.8 KB, 464 views)
  Reply With Quote
05/20/20, 03:55 PM   #12
Phinix
 
Phinix's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 186
Originally Posted by ZOS_DanBatson View Post
The single string character limit of 2000 is still there. This limitation is there for both performance and memory reasons. It is not a bug.
Thanks for confirming this, it likely saved me a bunch of time testing.
  Reply With Quote
05/20/20, 05:04 PM   #13
Solinur
AddOn Author - Click to view addons
Join Date: Aug 2014
Posts: 78
Since the LuaVM was changed to 64 bit the key/var limit is pretty much nonexistent. However if saved vars loading takes too long the client may crash and can't login. I played a bit with it a while back and found that for me 130MB was about as much as possible with very long load times.
  Reply With Quote

ESOUI » Developer Discussions » Lua/XML Help » Size or access limit for saved variables?

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off