[open] New Auto Saving Is Not Working - ESOUI
Thread Tools Display Modes
03/16/19, 04:20 PM   #1
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 109
[open] New Auto Saving Is Not Working

I am the author of Notebook 2018 and recently got some comments about the issue of losing saved data.

Specifically, how a game crash will lose saved data, in this case user notes in the addon. I noticed that update 20 API V100025 introduced some game-system enhancements to help resolve this issue.

Sadly, my tests showed that other than doing a /reloadui, no changes were written to file. I update my addon to the latest API version and tried the following, after saving a change in my Notebook:
  • Wait about 10 minutes.

  • Enter a building.

  • Change zone (into my guild house. That means a loading screen.)

Considering the impact this has on my specific addon, me and my users would appreciate a solution. Whether I'm missing something, or the new autosave system needs fixing. Thanks.
  Reply With Quote
03/16/19, 06:57 PM   #2
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 679
I tried testing auto-saving back in January. I don't know whether the system has changed since then, and I also think my script watching for file changes was bugged and may have missed something. I was planning to review it and do another round of tests, but stuff got in my way so that's still pending.

@Phuein: Wall of text incoming, and I'm afraid it won't be very useful to you, it's just something that's been bugging me for some time. The only reliable way to save your data is /reloadui. Even if the auto-save system works (or worked), even if it was changed to work differently / better, it will not save reliably on demand, because that'd be abusable.

Anyway, here's what I found out back in January.

I had 15 addons + 9 libraries enabled. Only one of those addons had ## DisabledSavedVariablesAutoSaving: 1

Test 1: Left a character standing near crafting station in Shornhelm for 110 minutes, mostly AFK just moved the mouse occasionally. Haven't seen a single save for any addon during that time.

Test 2: Left that same character at Rayles keep in Cyrodiil for about two hours. Called RequestAddOnSavedVariablesPrioritySave a couple times for that single "disabled-auto-save" addon manually via /script command.
The client occasinally saved one file. Delays between saves were 3, 6, 12, 15, 21 or 24 minutes. The "disabled-auto-save" addon never got saved. The rest appeared to save in a fixed order -- after all were saved, the first was saved again, and delays seemed to repeat as well.

Given how long it took to make full circle I obviously didn't have enough data to confirm this, but I have a theory. The client makes a circular queue of addons that have auto-save enabled. Then every 3 minutes an "auto-save-event" triggers. The first addon from the queue is picked. If that addon doesn't have saved variables at all (e.g. libraries), the "auto-save-event" is wasted.

Test 3: Last test was to just leave the file-watching script running while actually playing in Cyrodiil. During this time I didn't watch the output on-the-fly, only inspected it afterwards. But it showed that several (1 to 4) files were being saved within 30 seconds after teleports via keep shrines. "disabled-auto-save" addon still didn't get any saves (though I didn't request any manually).

I have no explanation as to why I saw no saves standing in Shornhelm, and regular saves standing in Cyrodiil.

What's wrong with auto-save.

I considered the system flawed from the moment I read about it here on forums. Not speaking about implementation, I still suspect my testing method and logging script may be responsible for those weird observations. But regardless of observations, in my opinion the system is flawed because it's opt-out.

The core of the auto-save system is that the client decides, in some undisclosed manner and indeterminate intervals to prevent abuse by add-ons, that it should save some files. During your play session, there will be a finite number of such "auto-save-events". Thus, they're a scarce resource that all add-ons with auto-save enabled compete for.

Auto-save is on by default. This lowers the utility this system provides to add-ons that can actually benefit from it. Every add-on that doesn't need auto-save but doesn't explicitly disable it (it may be abandoned, or its author doesn't even know about the setting) consumes an "auto-save-event" in vain. For example I have a 3+yr old still working add-on that uses saved vars to remember window positions. This information is not critical, if the saved vars get lost it just falls back to default positions, no big deal. Many add-ons store configuration that very rarely changes -- users install them, tune them to their liking and then don't touch the settings for months. Again, these add-ons consume "auto-save-event" for no benefit.

And don't get me started on the saved-vars-condensed-to-single-line-to-make-auto-save-faster. Maybe that makes it 2% faster and 100% more difficult to use a saved vars file from a crashed session, because some editors go bonkers. 4 years ago I suggested using tabs for indentation, that alone would reduce saved vars size 20-50% depending on table depth, yet the client still indents with 4 spaces on clean exit. If you really want to squeeze more on auto-save, okay don't indent, but please at least keep newlines.

Here's how I would implement auto-save.

Drop the setting from manifest, it's not needed. Default to auto-save disabled, i.e. like before it was a thing.

Add method AddOnManager:MarkSavedVariablesDirty(addOnName) and AreSavedVariablesDirty(addOnName).

MarkSavedVariablesDirty will place the named add-on at the end of a queue (not front, as this post suggested), unless that add-on already is in the queue (in which case it's not added a second time, it stays where it is in the queue).
Every time an "auto-save-event" triggers, you take one or more add-ons from the head of the queue. For each of them, check whether it's eligible for saving (some time elapsed since last save) -- if eligible then save, otherwise put it at the end of the queue again. After the save, AreSavedVariablesDirty will return false, and you can fire an event to let add-ons know their data is safe (I thought EVENT_SAVE_DATA_COMPLETE was meant for this, but never saw it occur).
  Reply With Quote
03/17/19, 03:12 AM   #3
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,416
In my observation the current auto save works fine (on windows). I have more then a hundred addons active and it would save every single one of them regularly, given there are enough loading screens to store the bigger saved vars that could not be saved otherwise due to the rules to minimize performance impact. During a loading screen it would save ~3-8 files depending on their size. The minimum wait time between two saves of the same addon is exactly 15 minutes, so it's quite possible that it saved right before you started to wait for 10 minutes.

I do agree with merlight that the one line no spaces way of saving them is a bad idea. It messes with my external applications that try to read saved vars and I have to log in again after a crash in order to store the files normally. :P
https://www.imperialtradingcompany.eu/ - My Addons - The Vault (Early updates and experimental projects) - My patreon - My blog
>siri.exe MyAddon
Does your addon work? [y/n] n
There is a typo in there.

Last edited by sirinsidiator : 03/17/19 at 03:15 AM.
  Reply With Quote
03/21/19, 09:41 AM   #4
AddOn Author - Click to view addons
Join Date: Jun 2014
Posts: 51
SirI, you can use an LUA parser to import the data, then re-save it formatted better. It's one additional step, but the time savings in writing with minimal spacing is significant.

SavedVariables\IIfA.lua in it's compressed form is less than half it's normal size of 2M for me. Multiply that by dozens of addons, and the cumulative time adds up.
  Reply With Quote
03/24/19, 04:07 AM   #5
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 109
Welp, pinging Zeni on Twitter to check this thread didn't do much. Would be nice to even just get a general Nay from the devs or something.

Nice general conversation growing in here, though.
  Reply With Quote

ESOUI » Developer Discussions » Bug Reports » [open] New Auto Saving Is Not Working

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off