View Single Post
11/19/17, 06:58 PM   #3
Uesp
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 15
I've been testing this too and while I've found results similar to yours I'm left more confused now about exactly what the limit is and how it changes depending on the content. I've made an empty addon that just populates the saved variable file with a structure like:


Code:
TestNoneLogSavedVars =
{
    ["Default"] = 
    {
        ["@Reorx"] = 
        {
            ["$AccountWide"] = 
            {
                ["test1"] = 
                {
                },
                ["version"] = 3,
            },
        },
    },
}
  • The file is saved fine but it breaks on load.
  • File corrupt can result in "random" data being added, missing data, sibling data being added up one level or complete removal of the data and a reset of the saved variable file.
  • If I save "true" values the file breaks at 131066 entries. This is suspiciously close to 2^17=131072.
  • If all inserted data is the same it doesn't seem to matter how long it is. The break size of an array with trues is the same as one with 100 byte strings.
  • The limit seems to only affect the number of entries in a single variable. For example, 2/3/4 different arrays with 131000 entries each save and load fine.
  • The depth of the field seems to have some effect on the max length of the array. For example, adding one level to the saved variable data like:
    Code:
                ["$AccountWide"] = 
                {
                    ["test1"] = 
                    {
                       ["test2"] = 
                       {
                           -- Lots of entries
                       },
                    },
                },
    changes the break size from 131065 to 131064.
  • Variation in the data seem to affect the break size. If we have:
    Code:
     
    savedVars[i] = i + 101713    -- Arbitrary constant here
    Then it breaks at 65533 (close to 2^16). If we have:
    Code:
     
    savedVars[i] = (i % 10) + 101713
    then it breaks at 131066, one more than the constant case.
  • Files that break when loaded into ESO seem to load fine with a command line Lua v5.2. I believe ESO uses 5.1 but I didn't see any bugs/change log that might affect this.

So more data but I'm not sure I'm any closer to understanding the exact issue or exactly when it occurs. I know in our uespLog addon I've limited the logging data array to 60k elements to prevent issues although I do run into random corruption during various data mining operations involving large amounts of data.

It sort of looks like someone is using a fixed buffer with a size around 131072 elements which is somehow overflowed during file loading. I assume the ESO code somewhere is just using a dofile()/loadfile()/dostring() Lua API call and if so the issue would in the Lua library somewhere. Or perhaps the ESO code uses a custom file load which has a bug in it.

Would be nice if a ZOS dev (Chip?) could step through the saved variable loading with a known bad file to see where the issue is, confirm it, and hopefully fix it. For the record, all you need to is create an addon with saved variables that outputs a single large array like:

Code:
function testNone.SetVar1(count)

	if (count == nil) then count = 131072 end
	
	testNone.savedVars.test1 = {}
		
	for i = 1, count do
		testNone.savedVars.test1[i] = true
	end
end