Thread Tools Display Modes
06/29/18, 08:42 AM   #1
ZOS_DanBatson
ZOS Staff!
 
ZOS_DanBatson's Avatar
Yes this person is from ZeniMax!
Join Date: Jul 2015
Posts: 171
Lightbulb Auto Saving SavedVariables

Hi all,

We're looking to come up with a robust solution to the problem many of you have had when it comes to SavedVars not saving when the game closes unpredictably. We want to implement an auto saving system but also include options to opt out and request a save manually. By default, this system will automatically save SavedVar data for each addon, one at a time, with a large (minute(s)) time interval between addon saves. For addons that opt out, there will be a function to manually request that their saved vars be placed at the front of the list to be saved on the next opportunity. This manual request will have a larger cooldown (15min-ish).

We’ll disable auto saving by checking an addon config:
## DisabledSavedVariablesAutoSaving: 1

When a save occurs, we will attempt to do the work to save it out. If the save was not triggered manually and the work takes longer than some reasonable amount of time that we’ll determine, we will abort the save and increment numSaveAttempts. If numSaveAttempts goes higher than some reasonable number of attempts (maybe 3?), we’ll consider the addon to be too intensive to auto save and it won’t try again. Manual saves go through regardless of how long they take.

We would like to get your thoughts on this idea. Do you think this will be helpful to you? Does it give you enough control, and if not, what do you think is it missing? Can you see any serious issues this may cause for you?

Thanks!
  Reply With Quote
06/29/18, 09:12 AM   #2
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,567
I really would love to see this!
My main concern is that right now the table keys in the saved variables are serialized in a random order. This means every time a file is saved, it looks vastly different to the previous revision, which in turn makes it impossible to efficiently create incremental backups. My saved variables folder is close to 250MB right now, which could easily lead to a gigabyte of unnecessary backups every hour.
Ideally this auto save (and also the regular save) would store the keys in alphabetical order and check if the resulting file would be different so it skips writing if there is no change at all.
  Reply With Quote
06/29/18, 01:32 PM   #3
Dolgubon
 
Dolgubon's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2016
Posts: 408
This sounds great! I don't think it's really something that I will use explicitly, but it's a really good QOL improvement, not just for add-on authors, but any players who use add-ons. As far as explicit uses go, I can see my lazy farmer using it, but I don't know if I'd spend the time to update it to use that.
  Reply With Quote
06/29/18, 01:54 PM   #4
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
I think it's a good addition but some ppl mentioned it shouldn't save if you are in a fight, especially not if you are in a dungeon or even trial!

Not sure if the saving will produce any delay.
But I support the idea of Sirinsidiator to compare the savedvars in memory with the file somehow (by help of a hash e.g.) before saving it, if nothing has changed.
  Reply With Quote
06/29/18, 02:11 PM   #5
votan
 
votan's Avatar
AddOn Author - Click to view addons
Join Date: Oct 2014
Posts: 577
Are they saved asynchronous? E.g. via memory snapshot. I can imagine some saved-vars could take a bit longer.
  Reply With Quote
06/29/18, 08:33 PM   #6
SilverWF
 
SilverWF's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 55
Question

Hey Dan, this is great!

Any news on when it would be possible to receive new mails without need to relog or change map?
  Reply With Quote
06/29/18, 09:47 PM   #7
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
I think it would be a better idea to make the option so you have to opt in to have it work. Making it run by default can cause performance issues with addons that store large amounts of cache data to their saved variables until the author gets around to turning the option off.

Something like this:
Code:
## SavedVariablesAutoSave: 1
  Reply With Quote
06/30/18, 02:27 AM   #8
votan
 
votan's Avatar
AddOn Author - Click to view addons
Join Date: Oct 2014
Posts: 577
Maybe the author can mark a savedVar as dirty. The authors know best, if a critical change was made.

Last edited by votan : 06/30/18 at 04:09 AM.
  Reply With Quote
06/30/18, 06:36 AM   #9
Sorondor
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 17
I agree that opt in is much preferred. I use more than a few addons that are no longer updated/supported, but still work. (Some just work and others I've fixed so they work again.) I'm unsure how autosaving might impact them or my client's performance. Sure, I could edit all the old addons with "## DisabledSavedVariablesAutoSaving: 1", but that wouldn't be necessary if the system was opt in.

Using the example Dan posted, implement "## EnabledSavedVariablesAutoSaving: 1" to opt in instead.

I also like the dirty flag suggestion. What would really be nice is if specific keys could be marked dirty and only those keys saved during an autosave cycle. However, I have no idea if it's possible to write to SavedVars in part or if it must be written whole every single time.
  Reply With Quote
06/30/18, 07:27 AM   #10
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,567
I would also love it if that feature could be turned off globally via the user settings and maybe start turned off in the first iteration, until we see how it impacts the game on a larger scale.
  Reply With Quote
06/30/18, 09:36 AM   #11
silvereyes
 
silvereyes's Avatar
AddOn Author - Click to view addons
Join Date: Aug 2015
Posts: 66
imo, any behavior change like this that could adversely affect performance should be opt-in; Edit: or perhaps just disabled for out of date add ons.

Last edited by silvereyes : 06/30/18 at 12:12 PM.
  Reply With Quote
06/30/18, 02:23 PM   #12
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by votan View Post
Maybe the author can mark a savedVar as dirty. The authors know best, if a critical change was made.
In Dan's example, running the manual save function registers the saved var to be picked up later by this process. What part of that isn't marking it as dirty?



Originally Posted by Sorondor View Post
What would really be nice is if specific keys could be marked dirty and only those keys saved during an autosave cycle. However, I have no idea if it's possible to write to SavedVars in part or if it must be written whole every single time.
I highly doubt writing only specific keys would even be possible. Lua memory doesn't look like the code you see in the saved var files. It also doesn't have any stock method of turning data into such code. This is where a serializer comes in. A serializer takes data you give it and encodes it into some sort of string that can be read in by a deserializer to reconstruct the original data later. The serializer for saved vars takes the entire table and writes it into Lua code that can be read in by loadfile() as the deserializer.

If an attempt were to be made, the original file would have to be analyzed with a tokenizer to find where the original data for the key/value pair is in the file to replace it. This would take up a lot more processing than just overwriting the entire thing.

Last edited by SDPhantom : 06/30/18 at 02:26 PM.
  Reply With Quote
06/30/18, 02:58 PM   #13
votan
 
votan's Avatar
AddOn Author - Click to view addons
Join Date: Oct 2014
Posts: 577
Originally Posted by SDPhantom View Post
In Dan's example, running the manual save function registers the saved var to be picked up later by this process. What part of that isn't marking it as dirty?
There are repeatable actions and non-repeatable events.
repeatable action: Checkbox in the settings: Worse case: User has to do it again. => Can be handled like now.
non-repeatable event: Completing a quest: Worse case: Gone forever. => Should be saved soon.

Last edited by votan : 06/30/18 at 03:28 PM.
  Reply With Quote
06/30/18, 04:33 PM   #14
Dolgubon
 
Dolgubon's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2016
Posts: 408
Originally Posted by votan View Post
There are repeatable actions and non-repeatable events.
repeatable action: Checkbox in the settings: Worse case: User has to do it again. => Can be handled like now.
non-repeatable event: Completing a quest: Worse case: Gone forever. => Should be saved soon.
As another example:


Craft 5 legendary items with the set crafter. Worst case: They make it again and lose 300k because that wasn't saved. A similar issue would exist with Writ Worthy, especially since some people do a large amount of them.


As another request, would it be possible to have the Save Request function return a value letting the add-on know if the save request has been approved?


Also, it would be great if an add-on could call it twice a few minutes apart, but then have an even longer cooldown. My use case here is actually with the set crafter/writ crafter. If a player adds an item to the queue (saved vars) then they will probably make it pretty quickly after. If it's just the one save request, then I'd save it for the actual crafting, but if it's possible to have two it would be nice to also have it save after a large amount of items are added to the queue. If that's too abusable for you, I'm still excited even for just the one save request option.




I'm seeing a lot of people worried about performance issues from this. How much of a concern is that actually? Maybe if an add-on's saved vars table is less than x amount the game can do default of auto saving, but if it has more than x amount of values it defaults to the current behaviour. Or even just make the 'reasonable' amount of time very short. Depending on the reasonable amount of time chosen, I don't see why there should be an issue with performance, in PvP, trials, or not.
  Reply With Quote
07/01/18, 01:55 AM   #15
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by votan View Post
There are repeatable actions and non-repeatable events.
repeatable action: Checkbox in the settings: Worse case: User has to do it again. => Can be handled like now.
non-repeatable event: Completing a quest: Worse case: Gone forever. => Should be saved soon.
I still don't see how this changes anything. You either need to save now and call the save function, and if it has to, it waits for the cooldown to finish. There is no way around this as per the design of it. If you call for it to save something not so important (as if marking the save dirty), it does the exact same thing. Which of these two processes do you see as different? A suggestion I would make, if this is the direction we're going, is to have the autosave and manual saves share the same cooldown. Something like 5 minutes seems reasonable to me.



An alternative I would suggest is to add a directive in the manifest that would both have an addon opt-in for autosave and configure how often the author wants it to occur. Something like ## AutoSaveInterval: 5 would set the interval for every 5 minutes. This has to be an integer greater than zero or the directive will be ignored.

In addition, a manual save function will be made available to all addons, regardless of this setting. For the sake of this example, the cooldown for an addon is 1 minute regardless of if processed by autosave or manually done so. A scheduler system should be made to handle the order in which addons save their data. This will act as a queue that iterates through one step every 5 seconds. When processed, addons configured for autosave will be rescheduled according to the interval set in their manifest. Manually calling the save function will modify the schedule to run at the the current timestamp, or 1 minute after the timestamp of the last save, whichever is greater.

To clarify, if an addon that's registered for autosave calls the manual function, it will still reschedule itself the same as if it was triggered by autosave. This supports more urgent saving and preserves the autosave feature at the same time. Duplicate entries for a single addon is not allowed. If there an entry already exists, it should be modified appropriately if the requested timestamp is earlier and the queue re-sorted. This design also allows the queue to fall behind if the step doesn't cycle through fast enough if there are too many addons using it.
  Reply With Quote
07/01/18, 03:30 AM   #16
Scootworks
 
Scootworks's Avatar
AddOn Author - Click to view addons
Join Date: Nov 2014
Posts: 312
in general i like the idea of auto savings. in my opinion, one time each 15 or 30 is enough. in what cases you want to save it every 5‘?
what addon needs it?
- how many times per day you gonna change settings?
- merchant addon can read the history, not really needed

maybe some log addons or chat savings, but how many people are using this?

why not a button to save it manually? (with a function that can be called via lua code to be able to allow autosavings via RegisterForUpdate)

and after writing this text i believe, we don’t need SV autosaving
  Reply With Quote
07/01/18, 04:49 AM   #17
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
Saving any x minutes where x is below 5 won't happen imo as Dan has already stated in his thread above.
Making the opt in would be better imo, as existing addons need to be changed to adopt it.
  Reply With Quote
07/01/18, 05:44 PM   #18
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by Scootworks View Post
in general i like the idea of auto savings. in my opinion, one time each 15 or 30 is enough. in what cases you want to save it every 5‘?
what addon needs it?
- how many times per day you gonna change settings?
- merchant addon can read the history, not really needed

maybe some log addons or chat savings, but how many people are using this?

why not a button to save it manually? (with a function that can be called via lua code to be able to allow autosavings via RegisterForUpdate)

and after writing this text i believe, we don’t need SV autosaving
If this causes the game to stutter every 5, 15, or 30 mins, people are going to complain the same regardless of how long between. I thought 5 mins between saves would be more reasonable than 15. It doesn't matter if I'll have a use for it, but if it's an option that'll fit into more peoples' needs with hardly any impact, why not? Especially with something that'll effect everyone that plays the game in some way. In all honesty, I don't have a "need" for autosave either and I've crashed the game more times than I care to count.



Originally Posted by Baertram View Post
Saving any x minutes where x is below 5 won't happen imo as Dan has already stated in his thread above.
From the sounds of it, Dan's team hasn't fully figured out how autosave is going to work, let alone ironed out an interval between saves. He posted here for suggestions, I made mine.

Last edited by SDPhantom : 07/01/18 at 05:52 PM.
  Reply With Quote
07/01/18, 06:03 PM   #19
Dolgubon
 
Dolgubon's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2016
Posts: 408
Originally Posted by SDPhantom View Post
If this causes the game to stutter every 5, 15, or 30 mins, people are going to complain the same regardless of how long between. I thought 5 mins between saves would be more reasonable than 15. It doesn't matter if I'll have a use for it, but if it's an option that'll fit into more peoples' needs with hardly any impact, why not? Especially with something that'll effect everyone that plays the game in some way. In all honesty, I don't have a "need" for autosave either and I've crashed the game more times than I care to count.
I doubt it will cause the game to stutter. If it's something that would cause the game to stutter, I don't think it would be implemented.
  Reply With Quote
07/02/18, 01:25 AM   #20
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by Dolgubon View Post
I doubt it will cause the game to stutter. If it's something that would cause the game to stutter, I don't think it would be implemented.
Missing the point. It's not likely, but it is the worst that could happen and is the reason Dan suggested having a cooldown for saving in the first place.
  Reply With Quote

ESOUI » Developer Discussions » General Authoring Discussion » Auto Saving SavedVariables

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off