[fixed] There's something terribly broken in saving and/or loading SavedVariables
Here's a sample of saved vars from a combat log add-on, section for character Sindari, i.e. below ["Default"]["@Account"]
Lua Code:
When the log gets really long, more than 10k+ lines, it completely screws the saved vars structure. Here's a sample of the garbage from the same level of the table: Lua Code:
|
Yeah, I hitted it with pChat at the begenning of Chat restoration (I was dumping whole chat in saved vars).
You should look at mastermerchant way of managing this. I'm also working on a big project which will meet this limit, so i'll try to find a workaround. Consider that if lua file is > 20MB , it has huge chance to be corrupted. After 32MB , it's 100% corrupted. Maybe array indexes which can fail after some limit .. 4k or something. |
The serialization also fails to recognize recursive table structures and continues to write to your disk until the game crashes.
And I also happened upon a strange bug where my saved vars got cleared randomly when I used 2 separate ones in one file. I tried to find the reason for a long time, but gave up after a month or two. Deleting an old var is also impossible, because setting it to nil leaves the old data unchanged. The best we get is setting it to an empty table and remove the unused var some versions later once we transferred data over to the new one. |
Hmm, that's worrisome. Is there anything I would have to do to repro this other than making a giant array in a saved var?
|
idk but this can be conencted
when you mouseover guild roster players one by one with Shissu guild tools installed your lua usage growing up very fast = memory leak and you can reach limit fast(for me limit is 1024mb) |
Quote:
You sure Shissu guild tools isn't just doing something silly? What are you using to determine what lua's memory usage is? I've wondered how to do that. |
Raetia info hub, Votans settings menu or CharInfo inspector can show you current lua usage
|
I generated a 9MB Saved Vars file with 60000 strings in an array and did not see any corruption. Maybe there is something more to it?
|
Something in the logged data that the save routine is interpreting as a table structure delineator maybe?
|
Quote:
It may be anything, a combination of array size, number of elements, depth, perhaps Lua memory setting. Or a bug I put in the add-on, that'd be stupid :D A few years ago I had to patch Apache Flood because they were reading XML in fixed size chunks, and if you had a #text node crossing the read buffer boundary, it got split into two nodes, and the original code wasn't able to match such split nodes to keywords. This looks remotely similar, perhaps there's a point where the SavedVars reader/writer forgets the context and starts reading from the wrong offset / writing wrong data. In the sample above I showed how data go "upwards", becoming keys in higher level. Here's a sample of the inverse, where a part of the root SavedVars table is flattened and written as values in the lowest level: Lua Code:
|
Do your samples reflect the indentation of those pieces of code in the file?
|
The 2 in the OP are with 12 leading spaces (3x indent) removed.
The 3rd one is with 24 leading spaces (6x indent) removed, it's the deepest level. |
If you could, would you email that saved variable file to charles.hilseberg at zenimaxonline dot com.
|
Quote:
Lua Code:
|
Well, this is embarrassing. :o Instead of publishing a new geeky widget I have for you, I've spent the whole evening creating huge saved vars with varying layouts, up to 300k lines / 80MB, trying to replicate the corruption I'm seeing with the combat log. Without success. It's a tiny add-on my friend had written, and I modified it a bit... two bits, actually... and months ago. You see where this is heading?
Mod #1 is that I do Lua Code:
I suspect that when I wrote that, I didn't realize ZO_SavedVars used metatables for the defaults magic; but even taking that into account, I can't see anything wrong with this. I'm simply replacing their metatable with my own, I don't want defaults at that level anyway. Mod #2 is pruning old logs. Basically I have a flat array of log messages for each day, saved under charvars.combatlog[date]. When the add-on loads, I do Lua Code:
For a plain table, modifying/nilling existing keys during traversal is allowed. I'm not quite sure whether it's okay with the metatables from ZO_SavedVars. Anyway, I cleaned the saved file by hand, and added some checks to the add-on, so that next time it gets corrupted, it'll give a hint. I shall let it keep growing naturally and see. |
Well let's just call it foresight then, merlight!
I noticed nothing was being saved on PTS 2.1 and mentioned it to Ayantir ingame. He figured out that in this patch saved vars switched to ANSI, so files from live would not work as he mentioned in the 2.1 thread. But I'm having a problem more dire than that - all saved variables seem to be cleared on initial game load. I tested it with a dummy addon that just saves a few strings to a table. Steps: 1. Delete all saved variables files 2. Start PTS and load a character 3. Save some data to saved vars 4. Reload UI and notice data was saved 5. Logout 6. Load a character 7. Notice saved variables were deleted If you take a moment between steps five and six to check the files on disk, you should see that the data is still okay after logout. So the issue must be somewhere in initial game load since /reloadui also doesn't delete the data. |
At character loading, AddonManager create files of Savedvars with 0bytes, like a fopen("w").
For all addons, even disabled ones. On live thoses files are not created at character loading And ZO_ingame.lua don't have problem Thanks :) |
Quote:
|
This has been resolved internally and the fix will be pushed out in the next PTS build. Thanks, everyone!
|
Quote:
Today I caught it not long after my saved vars got corrupted, and eventually isolated a 20MB/113k lines file, wrapped it in an add-on with a ## SavedVariables directive and no code, and the game reliably destroys the saved vars file (for me at least, although I should probably try with different memory limits). I think it could be more useful than the corrupted saved vars file I sent to Chip earlier. |
All times are GMT -6. The time now is 10:13 AM. |
vBulletin © 2024, Jelsoft Enterprises Ltd
© 2014 - 2022 MMOUI