Thread Tools Display Modes
08/10/15, 09:19 AM   #21
merlight
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 671
I took a chunk of data from my saved vars and put it in an add-on that throws several copies of that data into saved vars. Here's the add-on: svFailTest.7z

Here are the steps I used to get corrupted saved vars:
  1. disable all other add-ons, enable svFailTest (no saved vars file yet)
  2. login
  3. /svfail
  4. /reloadui
  5. check SavedVariables/svFailTest.lua: mine has 118715 lines / 21802807 bytes
  6. /reloadui
  7. check SavedVariables/svFailTest.lua:
    • if empty (this is an error, too), go to step 3.
    • otherwise look for the string "svFailTestSV"
      for me the corrupted file has 118715 lines / 21595305 bytes (~200k less), and the corruption starts at line 115950
 
08/10/15, 04:32 PM   #22
ZOS_DanBatson
ZOS Staff!
 
ZOS_DanBatson's Avatar
Yes this person is from ZeniMax!
Join Date: Jul 2015
Posts: 171
Originally Posted by merlight View Post
Do you mean the saved vars truncation issue on the PTS? Because this was originally about mysterious saved vars corruption.

Today I caught it not long after my saved vars got corrupted, and eventually isolated a 20MB/113k lines file, wrapped it in an add-on with a ## SavedVariables directive and no code, and the game reliably destroys the saved vars file (for me at least, although I should probably try with different memory limits). I think it could be more useful than the corrupted saved vars file I sent to Chip earlier.
Sorry, I meant to have that quote Chip's reply to the post by Ayantir, in regards to the 0 bytes file issue. i hit "Post Reply" instead. Feel free to send your new file to Chip and myself (dan.batson at zenimaxonline dot com).
 
08/18/15, 09:57 AM   #23
ZOS_ChipHilseberg
ZOS Staff!
Premium Member
Yes this person is from ZeniMax!
Join Date: Oct 2014
Posts: 551
I made some changes internally to expand the size of the saved variable serialization we can handle from 2^24 characters to 2^32 characters. After doing this the example table you supplied was no longer running out of storage space and I did not see any corruption in the file.
 
08/18/15, 10:24 AM   #24
Ayantir
 
Ayantir's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 1,019
Many thanks
 
08/18/15, 07:12 PM   #25
merlight
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 671
Originally Posted by ZOS_ChipHilseberg View Post
I made some changes internally to expand the size of the saved variable serialization we can handle from 2^24 characters to 2^32 characters. After doing this the example table you supplied was no longer running out of storage space and I did not see any corruption in the file.
Great.

I was curious whether shrinking SavedVars files could speed-up their loading. Currently they're indented with spaces. Typical saved vars from add-ons I have are literally full of spaces. I tried replacing them with tabs (4 spaces -> 1 tab) in files larger than 10k. The results were 23% to 57% smaller (not counting the above-mentioned combat log add-on, which was only 10% because it stores long strings).
 
08/19/15, 03:23 AM   #26
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,567
We could avoid this problem altogether with a sqlite database.
No more loading everything into memory on ui load and (almost) no data is lost on crashes.
Addons like Harvest Map or Master Merchant would really profit from a database.
 
08/19/15, 03:40 AM   #27
QuadroTony
Banned
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 828
i like a database idea!!!
what i must to learn first to create it
 
08/19/15, 04:57 AM   #28
Ayantir
 
Ayantir's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 1,019
Originally Posted by sirinsidiator View Post
We could avoid this problem altogether with a sqlite database.
No more loading everything into memory on ui load and (almost) no data is lost on crashes.
Addons like Harvest Map or Master Merchant would really profit from a database.
In the road for almost more than a year, but it's a very big and very long and very complicated project. and it's also a big huge time crusher when it'll be in prod so it's coming slowly.
 
08/21/15, 10:42 AM   #29
Sasky
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 231
Originally Posted by merlight View Post
Great.

I was curious whether shrinking SavedVars files could speed-up their loading. Currently they're indented with spaces. Typical saved vars from add-ons I have are literally full of spaces. I tried replacing them with tabs (4 spaces -> 1 tab) in files larger than 10k. The results were 23% to 57% smaller (not counting the above-mentioned combat log add-on, which was only 10% because it stores long strings).
Depends on disk access speed, but it'd probably be minor. If disk access is really slow, compressing the SavedVars file before writing could actually speed it up.

Originally Posted by sirinsidiator View Post
We could avoid this problem altogether with a sqlite database.
No more loading everything into memory on ui load and (almost) no data is lost on crashes.
Addons like Harvest Map or Master Merchant would really profit from a database.
Sure, any of the data-oriented addons would benefit from having a partially loaded database. Most addons probably would not benefit really from that, though, and there's also the consideration of structure. Lua tables are extremely unstructured which means the SavedVars are "just use it". A relational database would need to define table structure. Just imagine how many more "SavedVars" threads there'd be with SQL added to it.

Not sure how hard it'd be to integrate Lua drivers with it, but looks like this would be the way to go for ZOS to add SQL support:
http://keplerproject.github.io/luasql/doc/us/
Potentially it could be run under a separate memory pool as well, which would take away a lot of the need to increase the Lua memory limit.
 
08/21/15, 11:08 AM   #30
Ayantir
 
Ayantir's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 1,019
That's also what i'm doing in background

I've chozen a sqlite db because of the hugeness of data to store. MySQL webhostings can't fit it
 
08/23/15, 09:43 AM   #31
Sasky
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 231
Originally Posted by Ayantir View Post
I've chozen a sqlite db because of the hugeness of data to store. MySQL webhostings can't fit it
That's more of a problem with the hosting company than any MySQL (or client/server SQL) limitation. AWS for example limits to 2TB per table for MySQL. I'm having a hard time thinking of what could be stored from addons to even hit the GB range.

Also, the SQLite page recommends against using it for client/server interactions. It's appropriate for local managed storage for an application, like if they added SQL support to addons.
 
08/23/15, 11:12 AM   #32
Arterion
Join Date: Apr 2015
Posts: 10
The only AddOn I use with a problem is PriceTracker. When its SavedVariables grows to more than 60 MiB, it will eventually save an empty table when it writes to disk. Very glad to hear that the serialization will be improved for 2.1. 2^32 is quite a lot of bytes.
 
09/17/15, 09:13 AM   #33
ZOS_ChipHilseberg
ZOS Staff!
Premium Member
Yes this person is from ZeniMax!
Join Date: Oct 2014
Posts: 551
We have official word on the cause of this bug now. The lua virtual machine implementation that we use has a hard limit on the number of string literals that can appear in a single file. The work around for this would be to use multiple saved variable files if one gets too large. It's not optimal, but it's what we have in the short term. There is the potential to auto-split the tables across files in the future, but that won't be for a while and may not be a better solution after we investigate it.
 
09/17/15, 10:16 AM   #34
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,567
I don't know what exactly happens when you save or load the file, but have you tried to save it in a different form (e.g. JSON) instead of generating and parsing lua code? Splitting saved vars over multiple files sounds complicated and error prone.
 
09/17/15, 10:52 AM   #35
merlight
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 671
Originally Posted by ZOS_ChipHilseberg View Post
We have official word on the cause of this bug now. The lua virtual machine implementation that we use has a hard limit on the number of string literals that can appear in a single file. The work around for this would be to use multiple saved variable files if one gets too large. It's not optimal, but it's what we have in the short term. There is the potential to auto-split the tables across files in the future, but that won't be for a while and may not be a better solution after we investigate it.
Wow

I was wondering why my combat log still gets corrupted after ~3 evenings of PvP, and started to think it might be because I regularly force /quit without waiting those 10s, and thought that events might still be appending to the table while it's being written out.

Anyway, auto-splitting tables sounds like a daunting task. The "LuaON" that goes into saved variables is such a simple subset of Lua (no expressions, keys can only be string/number/boolean, values can also be tables, and that's it) that I think it'd be much simpler to write a custom parser instead of using the limited implementation. With re2c or another simple parser generator, you could probably get it working in a single coffee buff's time (provided you already know the generator).
 
09/17/15, 11:56 AM   #36
Fyrakin
 
Fyrakin's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 129
Why not use XML datasets instead?
 

ESOUI » Developer Discussions » Bug Reports » [fixed] There's something terribly broken in saving and/or loading SavedVariables

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off