[fixed] There's something terribly broken in saving and/or loading SavedVariables

[merlight]

Here's a sample of saved vars from a combat log add-on, section for character Sindari, i.e. below ["Default"]["@Account"]

["Sindari"] = 
{   
    ["version"] = 2,
    ["combatlog"] = 
    {   
        [20150720] = 
        {   
            [1] = "00:00:00.123 Queued 0 by Sindari to Reine Noir using Light Attack -- ok slot:Light_attack source:Self target:Enemy ptype:Invalid dtype:Flame",
            [2] = "00:00:00.124 Heal 7 by <unknown> to Sindari using <unknown> -- ok slot:Ability source:Mob target:Self ptype:Magicka dtype:Generic",
            [3] = "00:00:00.125 Heal 7 by <unknown> to Sindari using <unknown> -- ok slot:Ability source:Mob target:Self ptype:Magicka dtype:Generic",
            [4] = "00:00:00.415 Hot_tick 594 by <unknown> to Sindari using <unknown> -- ok slot:Ability source:Mob target:Self ptype:Magicka dtype:Generic",
            [5] = "00:00:01.939 Blocked_damage 320 by Sindari to Reine Noir using Crushing Shock -- ok slot:Ability source:Self target:Enemy ptype:Invalid dtype:Flame",
            [6] = "00:00:01.943 Blocked_damage 64 by Sindari to Reine Noir using Befouled Weapon -- ok slot:Ability source:Self target:Enemy ptype:Invalid dtype:Disease",
            [7] = "00:00:01.231 Blocked_damage 311 by Sindari to Reine Noir using Light Attack -- ok slot:Light_attack source:Self target:Enemy ptype:Invalid dtype:Flame",
            [8] = "00:00:01.491 Critical_damage 1062 by Sindari to Reine Noir using Crushing Shock -- ok slot:Ability source:Self target:Enemy ptype:Magicka dtype:Shock",
            [9] = "00:00:01.492 Damage 666 by Sindari to Reine Noir using Crushing Shock -- ok slot:Ability source:Self target:Enemy ptype:Magicka dtype:Cold",

When the log gets really long, more than 10k+ lines, it completely screws the saved vars structure. Here's a sample of the garbage from the same level of the table:

["00:23:24.060 unit 'reticleoverplayer' is dead"] = 
{   
    ["version"] = 2,
    ["combatlog"] = 
    {   
    },  
},  
["00:21:49.585 Damage 237 by Sindari to Aoichan using Caltrops -- ok slot:Ability source:Self target:Enemy ptype:Magicka dtype:Physical"] = 
{   
    ["version"] = 2,
},  
[1385] = 
{   
    ["version"] = 2,
    ["combatlog"] = 
    {   
    },  
},

I don't have a character named 1385, or "00:23:24.060 unit 'reticleoverplayer' is dead". Either the saving or the loading code takes keys and data from deeper parts of the table and places them as keys on the character level. The original structure is not complex, it's a simple array of log messages shown above, nothing else; it's just huge.

[Ayantir]

Yeah, I hitted it with pChat at the begenning of Chat restoration (I was dumping whole chat in saved vars).

You should look at mastermerchant way of managing this. I'm also working on a big project which will meet this limit, so i'll try to find a workaround.

Consider that if lua file is > 20MB , it has huge chance to be corrupted. After 32MB , it's 100% corrupted.

Maybe array indexes which can fail after some limit .. 4k or something.

[sirinsidiator]

The serialization also fails to recognize recursive table structures and continues to write to your disk until the game crashes.

And I also happened upon a strange bug where my saved vars got cleared randomly when I used 2 separate ones in one file. I tried to find the reason for a long time, but gave up after a month or two.

Deleting an old var is also impossible, because setting it to nil leaves the old data unchanged. The best we get is setting it to an empty table and remove the unused var some versions later once we transferred data over to the new one.

[ZOS_ChipHilseberg]

Hmm, that's worrisome. Is there anything I would have to do to repro this other than making a giant array in a saved var?

[QuadroTony]

idk but this can be conencted

when you mouseover guild roster players one by one with Shissu guild tools installed your lua usage growing up very fast = memory leak

and you can reach limit fast(for me limit is 1024mb)

[kerb9729]

Originally Posted by QuadroTony idk but this can be conencted when you mouseover guild roster players one by one with Shissu guild tools installed your lua usage growing up very fast = memory leak and you can reach limit fast(for me limit is 1024mb)

I think this is a separate issue but -
You sure Shissu guild tools isn't just doing something silly?

What are you using to determine what lua's memory usage is? I've wondered how to do that.

[circonian]

Originally Posted by sirinsidiator The serialization also fails to recognize recursive table structures and continues to write to your disk until the game crashes.

I've never had 114k lines, but the only time I've seen this corruption was when I tried to use saved variable data for dataItems in a scroll List.

local entry = ZO_ScrollList_CreateDataEntry(ROW_TYPE_ID, savedVarDataItem, categoryId)

which creates recursion in the saved variable file. That definitely crashes the game & corrupts the saved variable file.

[merlight]

Well, this is embarrassing. Instead of publishing a new geeky widget I have for you, I've spent the whole evening creating huge saved vars with varying layouts, up to 300k lines / 80MB, trying to replicate the corruption I'm seeing with the combat log. Without success. It's a tiny add-on my friend had written, and I modified it a bit... two bits, actually... and months ago. You see where this is heading?

Mod #1 is that I do

charvars = ZO_SavedVars:New("SV", 1, nil, {combatlog = {}})
setmetatable(charvars.combatlog, {__index = autovivify})
-- autovivify is a function that creates nested tables for nonexistent keys

I suspect that when I wrote that, I didn't realize ZO_SavedVars used metatables for the defaults magic; but even taking that into account, I can't see anything wrong with this. I'm simply replacing their metatable with my own, I don't want defaults at that level anyway.

Mod #2 is pruning old logs. Basically I have a flat array of log messages for each day, saved under charvars.combatlog[date]. When the add-on loads, I do

for date, log in pairs(charvars.combatlog) do
  if date < today - 14 then
    charvars.combatlog[date] = nil
  end
end

For a plain table, modifying/nilling existing keys during traversal is allowed. I'm not quite sure whether it's okay with the metatables from ZO_SavedVars.

Anyway, I cleaned the saved file by hand, and added some checks to the add-on, so that next time it gets corrupted, it'll give a hint. I shall let it keep growing naturally and see.

[Randactyl]

Well let's just call it foresight then, merlight!

I noticed nothing was being saved on PTS 2.1 and mentioned it to Ayantir ingame. He figured out that in this patch saved vars switched to ANSI, so files from live would not work as he mentioned in the 2.1 thread.

But I'm having a problem more dire than that - all saved variables seem to be cleared on initial game load.

I tested it with a dummy addon that just saves a few strings to a table.

Steps:

1. Delete all saved variables files
2. Start PTS and load a character
3. Save some data to saved vars
4. Reload UI and notice data was saved
5. Logout
6. Load a character
7. Notice saved variables were deleted

If you take a moment between steps five and six to check the files on disk, you should see that the data is still okay after logout. So the issue must be somewhere in initial game load since /reloadui also doesn't delete the data.