Quantcast support more frequent write (only) of saved variables - ESOUI
Thread Tools Display Modes
08/06/18, 09:56 AM   #1
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 52
support more frequent write (only) of saved variables

Dear ZOS, one of the biggest pain points around addons, for me, is that the model of saving data only at the clean exit of the application leads to quite some pain when I care about the content stored in them.

As a concrete example, I have been using Notebook 2018 to keep a "travel journal" of sorts for my adventures in Tamriel.

On a couple of occasions I have had the game hang, and lost quite a few notes. Enough pain, for me, that I have started to simple /reloadui after writing one, which is rather uncomfortable, and definitely slows down and breaks up my gameplay experience.

To mitigate this I would propose two possible solutions:

Solution one, my preference, would be to add an API mechanism to permit an addon to signal that it has made "significant" changes, and would like to have them saved to disk. By making this an active request, it helps ensure that authors like myself can request a save when the variables are in a "good", consistent state, and ready to write. (Avoiding, for example, saving half the configuration changes the user has made, but dropping the second half, if the game was to crash.)

Solution two, far less optimal, would be to save more frequently: on zone transitions, or loading screens, or simply on a timed basis. This removes direct control from addon authors, but still delivers the primary benefit, which is that addon users are much less likely to lose their data -- or at least as much of the data -- in the event of a crash.

I see that in the past you expressed significant concerns about real-time communication with applications outside the game:

Originally Posted by ZOS_ChipHilseberg
Our concern is with real time communication to programs outside of the game. There are certain design choices made about what information should be available to other players which could be violated by allowing file writing like this. A solution to your problem would be a lot more restricted than allowing arbitrary data to be written to files at any time. Discussion in the direction of allowing safe data (as dictated by the game design) to be shared in real time is fine. It's just not something we've put a lot of though into at this time.
I absolutely recognise this as a valid and important consideration, and think that real-time access to data externally has both good and bad sides. For example, in GW2, real-time access to player position allowed for "GPS-like" external tools to help navigate maps, which I can certainly see being undesirable from a design perspective.

Given the existence of the /reloadui command as a facility to support writing data at an arbitrary time. A number of addons use exactly that method to ensure they "commit" results to disk - to the point that libSaveToDisk exists to automate the process under user control; examples:

Adding an API driven mechanism that allowed these addons to save their data in a less disruptive way to gameplay would not significantly reduce their ability to transmit data to external programs, though it would limit their ability to read it from them. (Which suggests that in the later case, they may retain the reloadui mechanism.)


The API driven mechanism would also, I would hope, encourage using a mechanism that could be appropriately rate-limited by ZOS, so that data is only saved as frequently as you judge appropriate. (For example, taking an appropriate copy of the data on call, but only writing it to disk every minute.)

API-wise, I'd suggest something like this as what I would personally find most valuable:

Lua Code:
  1. -- wrapper on the existing saved var abstraction
  2. function ZO_SavedVars:SaveNow(importance)
  3.  
  4. -- values for importance:
  5. -- the user is likely to be very upset if this is lost
  6. SV_IMPORTANCE_CRITICAL
  7.   -- the user will probably be a bit annoyed if this is lost
  8. SV_IMPORTANCE_NORMAL
  9. -- the user probably won't notice unless you lose a lot of these
  10. SV_IMPORTANCE_LOW
  11.  
  12. -- SaveNow returns:
  13. -- request was accepted, and data saved
  14. SV_RESULT_SUCCESS
  15. -- request was accepted, and data will be saved later
  16. SV_RESULT_DELAYED
  17. -- too many requests, do not try again for <seconds> seconds
  18. SV_RESULT_RATE_LIMITED, seconds

However, that high level abstraction could obviously be built by the community over a much simpler ZOS API, such as:

Lua Code:
  1. -- just ask for everything to save, nil return
  2. function WriteSavedVars()
  3. -- either a string for "addon", or the SV itself, or something
  4. function WriteSavedVars(what)

If you chose to implement rate limiting, then having an appropriate return from the low level function would be hugely valuable, to prevent foolish loops that call this every second or two. Even a boolean to indicate if it worked or not would reduce the likely pain of repeated calls into the runtime.

Alternately, an event to signal that the variables have been saved, and simply ignoring calls to write them during the period between the two -- with an arbitrary delay possible, making for natural rate limiting -- would also meet the needs; then the high level implementation would be more or less:

Lua Code:
  1. local state = { isDirty = false, isWriting = false }
  2.  
  3. -- I'd actually implement the priority system, and basically
  4. --  aggregate lower priority things until we had either
  5. -- (a) sufficient count of them, or
  6. -- (b) enough time passed.
  7. function state:SaveNow(self)
  8.     self.isDirty = true
  9.     if not self.isWriting then
  10.         -- in my library, I'd probably animate a "saving now" icon
  11.         -- that showed we were busy writing stuff out, so users know
  12.         -- when things are safe, because failing when they know
  13.         -- we are trying to behave as desired is less annoying
  14.         -- than failing when the user thinks everything is just fine.
  15.         self.isWriting = true
  16.         WriteSavedVars()
  17.     end
  18. end
  19.  
  20. function state:OnWriteComplete(self)
  21.     self.isWriting = false
  22.     if self.isDirty then
  23.         self.SaveNow()
  24.     end
  25. end
  26.  
  27. EVENT_MANAGER:RegisterForEvent(
  28.     "MyAddOn", EVENT_SAVED_VARIABLES_WRITTEN,
  29.     function() self.OnWriteCompleted() end
  30. )

Finally, the purpose of this is to implement a user friendly "your data has a high probability of being safe" mechanism; making sure that we can effectively communicate the robustness to the user would have huge value.

Thank you for your time and consideration.


PS: also mentioned here on the official forums.

PPS: I am, as yet, unable to publish any code I have written to the ZOS addon API, as I am awaiting some internal processes at my employer to permit that. Forgive me making requests without published code to back them, please.

Last edited by SlippyCheeze : 08/06/18 at 04:51 PM.
  Reply With Quote
08/06/18, 10:04 AM   #2
Kyoma
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 70
If you want some insight in all this you can check out this thread

Originally Posted by SlippyCheeze View Post
PPS: I am, as yet, unable to publish any code I have written to the ZOS addon API, as I am awaiting some internal processes at my employer to permit that. Forgive me making requests without published code to back them, please.
I gotta say, I'm a bit curious how code written for an ESO addon would somehow have *permission* issues with regards to real-world projects.
  Reply With Quote
08/06/18, 11:15 AM   #3
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,131
You are preaching to the choir. Check this thread. They are already working on something.
__________________
>siri.exe MyAddon
Does your addon work? [y/n] n
There is a typo in there.
  Reply With Quote
08/06/18, 12:14 PM   #4
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by Kyoma View Post
I gotta say, I'm a bit curious how code written for an ESO addon would somehow have *permission* issues with regards to real-world projects.
What the addon API does is register C functions to be called by Lua code. If the underlying C code isn't written yet, there's no point in posting what the API function is going to look like.
__________________
ESOUI AddOns | WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)
  Reply With Quote
08/06/18, 02:54 PM   #5
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 52
Originally Posted by sirinsidiator View Post
You are preaching to the choir. Check this thread. They are already working on something.
Ah! thank you very much. I did not locate that in my searching, but will update this thread appropriately to reflect that there is an existing and more accurate discussion.
  Reply With Quote
08/06/18, 03:03 PM   #6
Kyoma
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 70
Originally Posted by SDPhantom View Post
What the addon API does is register C functions to be called by Lua code. If the underlying C code isn't written yet, there's no point in posting what the API function is going to look like.
Yes but SlippyCheeze isn't adding anything "C-related" to the game so doesn't really apply here. I mean, it's his employer he's talking about.
  Reply With Quote
08/06/18, 03:05 PM   #7
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 52
Originally Posted by Kyoma View Post
If you want some insight in all this you can check out this thread

I gotta say, I'm a bit curious how code written for an ESO addon would somehow have *permission* issues with regards to real-world projects.
Simple: as a condition of my employment, I agreed to an IP and copyright assignment. While that was active for work on ESO addons, every addon I touched would have to go through an individual patch release process, or be released as part of my employers standard open source license. There are also a significant number of addons which, by virtue of the license agreement, I would be unable to patch.

I'd also be in the position where copyright would be retained by my employer, requiring a statement in the code to that effect, and I honestly don't want to deal with the oddities that would likely cause out in the wider world.

The delay was getting the approval of a release covering ESO addons; it was hardly a controversial thing, but it was something that required waiting on the request to be looked at by a committee that meets infrequently. Now it is granted, I can touch anything related to any addon for ESO without blinking. At least, as long as it is in lua.

So, it isn't exactly that I couldn't write code, it was that I couldn't release it without going through processes I didn't want to deal with. "Can't release" was a shorthand for this, honestly, over-long explanation.

FWIW, depending where in the world you are, even without a written agreement on the subject, you are probably either subject to a "by default" assignment to your employer, or in a grey area where it isn't clear under law who owns those rights, if you are employed to produce software for a living, or anything like that.

If you are a university student, I'd upgrade that to "are subject, 100 percent", everywhere I'm aware of. Universities are even more grabby than private companies. Government workers are also usually 100 percent subject to something similar, but the processes are fairly different.

This is a horribly unclear area, and IMO subject to vast overreach, but I'd rather understand the problem area and work in ways that don't subject me to potential pain and annoyance later.
  Reply With Quote
08/06/18, 03:07 PM   #8
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 52
FINAL UPDATE: see http://www.esoui.com/forums/showthread.php?t=7901 instead of this thread.
  Reply With Quote
08/06/18, 03:09 PM   #9
Kyoma
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 70
Originally Posted by SlippyCheeze View Post
Simple: as a condition of my employment, I agreed to an IP and copyright assignment. While that was active for work on ESO addons, every addon I touched would have to go through an individual patch release process, or be released as part of my employers standard open source license. There are also a significant number of addons which, by virtue of the license agreement, I would be unable to patch.

I'd also be in the position where copyright would be retained by my employer, requiring a statement in the code to that effect, and I honestly don't want to deal with the oddities that would likely cause out in the wider world.

The delay was getting the approval of a release covering ESO addons; it was hardly a controversial thing, but it was something that required waiting on the request to be looked at by a committee that meets infrequently. Now it is granted, I can touch anything related to any addon for ESO without blinking. At least, as long as it is in lua.

So, it isn't exactly that I couldn't write code, it was that I couldn't release it without going through processes I didn't want to deal with. "Can't release" was a shorthand for this, honestly, over-long explanation.

FWIW, depending where in the world you are, even without a written agreement on the subject, you are probably either subject to a "by default" assignment to your employer, or in a grey area where it isn't clear under law who owns those rights, if you are employed to produce software for a living, or anything like that.

If you are a university student, I'd upgrade that to "are subject, 100 percent", everywhere I'm aware of. Universities are even more grabby than private companies. Government workers are also usually 100 percent subject to something similar, but the processes are fairly different.

This is a horribly unclear area, and IMO subject to vast overreach, but I'd rather understand the problem area and work in ways that don't subject me to potential pain and annoyance later.
I see, still seems weird your work license would apply to something you write in your own time that would have no relation to your work.
  Reply With Quote
08/07/18, 11:45 AM   #10
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
A lot of big companies have a blanket copyright claim to any creative work done by their employees mainly to prevent corporate secrets from leaking out. In such cases, they have legal right to order such work to be taken down without the author's consent.
__________________
ESOUI AddOns | WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)
  Reply With Quote
08/07/18, 01:05 PM   #11
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 52
Originally Posted by SDPhantom View Post
A lot of big companies have a blanket copyright claim to any creative work done by their employees mainly to prevent corporate secrets from leaking out. In such cases, they have legal right to order such work to be taken down without the author's consent.
Yeah. Sometimes, as in my case, that is explicit in the contract, sometimes it is implicit.

As I noted below, it also is a grey area of the law: in many places, it isn't actually clear-cut if you personally own something if, for example, it might have been developed using knowledge that was part of your job, or does something substantially similar to a product of your employer.

Those y'all can go to court and fight about, but there is no certainly if you, or your employer, would prevail.

Most of this also applies to universities, and government agencies, not just large private companies.

Though in the government case, it usually means that the government owns the content, and there are rules on release, but it more or less equates to "the entire country owns" than anything else.

Anyway, point was, even though I'm really not a fan of this sort of thing, and I think it reaches far beyond what is reasonable, I can't just ignore the law because it is a stupid law.
  Reply With Quote
08/08/18, 09:54 AM   #12
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
My experience with this is limited, but I had a friend that worked the phones at a call center, definitely not a software development job, and they had a contract like this. This issue was one of the many things he constantly complained about.
__________________
ESOUI AddOns | WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)
  Reply With Quote
08/08/18, 12:55 PM   #13
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 52
Originally Posted by SDPhantom View Post
My experience with this is limited, but I had a friend that worked the phones at a call center, definitely not a software development job, and they had a contract like this. This issue was one of the many things he constantly complained about.
O_o now, that is definitely overreach, and I doubt a court would hold it up. but ... who has the $$$ to fight that out, eh?
  Reply With Quote
08/09/18, 10:16 AM   #14
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
It's a contract like any other that involves signing over rights to your work. Court would definitely uphold it, but with small cases like these, it wouldn't even get that far. DMCA (Digital Millennium Copyright Act) notices don't require court action to be issued and most content hosting services obey them without any question. Think of these notices as a threat of a lawsuit if action is not taken.
__________________
ESOUI AddOns | WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)
  Reply With Quote
08/09/18, 12:57 PM   #15
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 52
Originally Posted by SDPhantom View Post
It's a contract like any other that involves signing over rights to your work. Court would definitely uphold it, but with small cases like these, it wouldn't even get that far. DMCA (Digital Millennium Copyright Act) notices don't require court action to be issued and most content hosting services obey them without any question. Think of these notices as a threat of a lawsuit if action is not taken.
The "it is a contract" part is true, but there have certainly been cases where the overreach was considered unreasonable, and the courts found in favor of the employee under similar ownership of IP situations.

It also depends on employment type; if you are a contractor or consultant, often the other party doesn't own the rights other than anything explicitly enumerated in the contract. Not applicable to that situation, but ...

The law is complicated. Complicated law is totally not fun.
  Reply With Quote

ESOUI » Developer Discussions » General Authoring Discussion » support more frequent write (only) of saved variables

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off