Quantcast Auto Saving SavedVariables - Page 2 - ESOUI
Thread Tools Display Modes
07/02/18, 02:43 AM   #21
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,127
Originally Posted by SDPhantom View Post
Missing the point. It's not likely, but it is the worst that could happen and is the reason Dan suggested having a cooldown for saving in the first place.
It's not mentioned here, but on Gitter Chip told us that the saved variables are written in a background thread, so there should not be any stutter or performance issue, unless you use a single core CPU or limit ESO to one core.
__________________
>siri.exe MyAddon
Does your addon work? [y/n] n
There is a typo in there.
  Reply With Quote
07/02/18, 01:55 PM   #22
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by sirinsidiator View Post
It's not mentioned here, but on Gitter Chip told us that the saved variables are written in a background thread, so there should not be any stutter or performance issue, unless you use a single core CPU or limit ESO to one core.
Actual writing is a different stage of the process than serialization. In order to do it while the addon system is running, Lua cannot be allowed to execute any code or you'll have the possibility of data corruption. This means it needs to be done with the UI thread or at least lock it meanwhile. Locking the UI thread either way means the UI will be unresponsive for the duration and when that happens, it ties up the rendering engine as well. It's a cascading effect.

Just because something runs on a different thread doesn't mean thread locking doesn't exist, it's a necessity in certain applications.
__________________
ESOUI AddOns | WoWInterface AddOns
"All I want is a pretty girl, a decent meal, and the right to shoot lightning at fools."
-Anders (Dragon Age: Origins - Awakening)

Last edited by SDPhantom : 07/02/18 at 01:59 PM.
  Reply With Quote
07/02/18, 03:13 PM   #23
ZOS_DanBatson
ZOS Staff!
 
ZOS_DanBatson's Avatar
Yes this person is from ZeniMax!
Join Date: Jul 2015
Posts: 31
Hi everyone,

Thank you so much for the feedback! I have another question, then I'll give a bit more info:

Can you provide me with some numbers on how big your addon's saved var files typically are? As many real examples as you can give me. If you know someone else's I'll take that too. I'm trying to get an idea of what's small, what's large, and what's typical.

As for the feedback, I can tell you that performance is a big focus here, and we will be trying a number of different avenues to ensure we aren't causing spikes. You're not the only ones concerned about that, believe me it's on our minds too. We may look into using some of your suggestions for that, as well.

As for auto on vs auto off, assuming performance is under control, a key to this feature is that not every addon developer is going to think or remember to enable this, nor will the many addons that are fully functional but not maintained anymore. We want this to help as many people as possible while not causing issues. We will strike that balance between speed and size to find a way to allow the most number of addons to leverage it. And priority saving will always be available to those whose addon is too beefy to do it automatically but they still want to save.
  Reply With Quote
07/02/18, 04:02 PM   #24
Dolgubon
 
Dolgubon's Avatar
AddOn Author - Click to view addons
Join Date: Jan 2016
Posts: 314
So I have 64 add-ons with saved variables. 25 have a file size less than 10 kilobytes.
10 have a file size between 10 and 50 KB
5 are between 50 and 100, and another 5 between 100 and 200.
There's just two between 200 and 1000
For the next few, I'm ignoring the MM00-MM15 files.
2 are between 1000 and 2000, and then there's one at 7.1 MB.


If you include MM, the lowest is 3.8 MB, largest is 8.5, and the average is 6.2 MB, with a deviation of 1.4 MB.

I do have a spreadsheet with the names of the addons and sizes here: https://docs.google.com/spreadsheets...it?usp=sharing


For one of my addons, the Set Crafter, I tried to see what some reasonable sizes would be.


With no use of it, it's about 6 KB (0 items). What I'd consider normal use (20 items) was about 50 KB Moderately heavy (80) was about 160. Very heavy use (200 items) gave me a size of 410.

The Writ Crafter uses 32 KB for me, and I don't think that the size will vary a lot.

Last edited by Dolgubon : 07/02/18 at 04:08 PM.
  Reply With Quote
07/02/18, 04:21 PM   #25
Kyoma
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 70
With regards to saved vars size, are we talking raw file size? Number of lines? Size of tables? All of the above?
  Reply With Quote
07/02/18, 07:22 PM   #26
SilverWF
 
SilverWF's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 49
Exclamation

My SavedVars folder weight is ~124 MB - yes, I'm addons addict, more than 60 addons overall (if not count all MM clones) Not all of them are enabled on all characters tho...

So, ~124 MegaBytes, 78 files, there is some from uninstalled addons, sure, but their weight aren't big.
Most heavy files are at image. This is only 38 files, so also 40 more with size less than 70 KB
Attached Thumbnails
Click image for larger version

Name:	img-2018-07-03-04-18-09.png
Views:	52
Size:	22.3 KB
ID:	1017  
__________________
ESO PC EU: @SilverWF
Addons: Advanced Autoloot Renewed, BeggingTheGear, Wykkyd Toolbar Updated
  Reply With Quote
07/02/18, 08:36 PM   #27
Scootworks
 
Scootworks's Avatar
AddOn Author - Click to view addons
Join Date: Nov 2014
Posts: 215
36.4 MB
__________________
EU Server - SŁĄuGhTerHĜuSe
  Reply With Quote
07/03/18, 02:58 AM   #28
Shinni
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 122
https://i.imgur.com/nEoxpfd.png
The average is 3.1mb per file.
When looking at my SavedVars folder, I feel like the files with more than 1mb are not important to be auto-saved.

The really large files (MM**.lua and ArkadiusTradeToolsSalesData**.lua) contain sales data from the guild history. If saving data of the current play session fails, the data is simply re-obtained upon the next login when scanning the guild's history. So I don't think these are files that would require auto-saving.

The HarvestMap**.lua files contain locations of crafting resources. If these files are large, then the player typically used some external tool to obtain data, in which case it is also not a big issue, if the data is lost.
However, if the files are small, then they typically only contain data collected by the player themself. In that case the player would be rather sad to lose the collected data from the current play session.
Personally, I think I would use the optional save request for HarvestMap to accommodate for these two different cases.
  Reply With Quote
07/03/18, 05:38 AM   #29
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,127
After trimming some old files my saved variables folder has ~200MB where ~150MB are Master Merchant files. Second place is HarvestMap with 8.15MB and third comes QuickEnchanter with 8MB. Then I have 10 more files with a size between 1 and 6MB, 10 between 100kB and 1MB and the rest (59 files) below 100kB with an average size of 13kB (but most of them are <10kB actually).
__________________
>siri.exe MyAddon
Does your addon work? [y/n] n
There is a typo in there.
  Reply With Quote
07/03/18, 07:00 AM   #30
Baertram
 
Baertram's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 1,768
My addon savedvariables folder:
I'm not using Master Merchant so my savedvars folder is "only" about 60MB in total

HarvestMap with 8.07 mb.
3 other addons with about 7 mb.
2 with 4 MB where 1 only stores set item data and won't be updated often (only manually) -> Would be cool if the savedvars would only save the "changed addon data" and not the "non changed set items database" on autosave!)
Several with 1-2 mb size
Several >250kb and < 1 mb
Several 10kb to 250kb
Lots of 1kb to 10kb

Last edited by Baertram : 07/03/18 at 07:34 AM.
  Reply With Quote
07/03/18, 08:20 AM   #31
AssemblerManiac
AddOn Author - Click to view addons
Join Date: Jun 2014
Posts: 46
MM is largest with 213M
CraftStore is next largest at 7.7M
Furniture Catalogue and Inventory Insight are about 1.7M each
Everything else is 700k or smaller

Saved vars directory:

57 files
Code:
      481,049 AIResearchGrid.lua
      168,801 AUI.lua
       40,553 AwesomeGuildStore.lua
      254,398 CombatLogStatistics.lua
       29,874 CraftingSwit.lua
    7,750,709 CraftStoreFixedAndImproved.lua
        4,621 DefGB.lua
      729,101 Destinations.lua
        4,250 DoItAll.lua
        4,370 DolgubonsLazySetCrafter.lua
       34,033 DolgubonsLazyWritCreator.lua
      221,260 FCOItemSaver.lua
        6,326 FCOLockpicker.lua
    1,799,506 FurnitureCatalogue.lua
        3,353 GameInfo.lua
        1,718 GuildNotificator.lua
      151,794 HarvestMap.lua
      150,356 HarvestMapAD.lua
      205,990 HarvestMapDC.lua
      288,920 HarvestMapDLC.lua
      134,026 HarvestMapEP.lua
      308,569 HarvestMapNF.lua
    1,761,378 IIfA.lua
       12,957 LoreBooks.lua
        8,275 LostTreasure.lua
      129,135 MasterMerchant.lua
        1,165 merTorchbug.lua
   11,105,666 MM00Data.lua
   12,954,578 MM01Data.lua
   12,141,318 MM02Data.lua
   16,503,415 MM03Data.lua
   12,749,717 MM04Data.lua
   13,112,213 MM05Data.lua
   12,821,735 MM06Data.lua
   12,664,635 MM07Data.lua
   12,785,782 MM08Data.lua
   12,425,072 MM09Data.lua
   13,453,748 MM10Data.lua
   14,256,713 MM11Data.lua
   14,031,273 MM12Data.lua
   13,144,873 MM13Data.lua
   16,008,642 MM14Data.lua
   13,416,008 MM15Data.lua
        5,960 NoThankYou.lua
        9,430 PotionMaker.lua
      335,441 RAETIA_InfoHub.lua
       29,485 ResearchAssistant.lua
          610 Roomba.lua
       11,318 SkyShards.lua
      706,425 Srendarr.lua
          354 VotanSearchBox.lua
          842 VotansFishFillet.lua
      233,749 WritWorthy.lua
          175 ZAM_Notebook.lua
        1,109 Zgoo.lua
      142,909 ZO_Ingame.lua
          362 ZO_Pregame.lua
  229,740,044 bytes

Last edited by AssemblerManiac : 07/03/18 at 08:27 AM.
  Reply With Quote
07/03/18, 11:11 AM   #32
Tarlac
 
Tarlac's Avatar
AddOn Author - Click to view addons
Join Date: Jun 2018
Posts: 10
I'm not sure all addons need an autosave. Maybe just being able to program a save within the addon, will save a lot of unnecessary saves. In Cashier for example, it only needs to be saved when a transaction has occurred with the bank. Otherwise it does not care. I would prefer to not have it autosaving every X minutes wasting processing power when it does not do anything. But if I disable the autosave, I would like to be able to program into the code that it causes a save when its data is changed. So no lost data will occur.
  Reply With Quote
07/03/18, 12:06 PM   #33
Marazota
Join Date: Sep 2015
Posts: 77
Originally Posted by sirinsidiator View Post
It's not mentioned here, but on Gitter Chip told us that the saved variables are written in a background thread, so there should not be any stutter or performance issue, unless you use a single core CPU or limit ESO to one core.
if its true, why when MM saving data after a scan of a new guild i just joined, game become a slide show for 30s at 8-core cpu?
  Reply With Quote
07/03/18, 12:18 PM   #34
Shinni
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 122
Originally Posted by Marazota View Post
if its true, why when MM saving data after a scan of a new guild i just joined, game become a slide show for 30s at 8-core cpu?
sirinsidiator was talking about the future auto saving of savedvariables, not about anything that currently exists in the game.
btw the MM lag has nothing to do with saving data. the lag does not come from saving something to the hard drive. in the current game version, saving is only done when you logout or /reloadui.
the lag comes from mm scanning and processing the guild sales history.
  Reply With Quote
07/05/18, 03:22 AM   #35
Letho
AddOn Author - Click to view addons
Join Date: Apr 2016
Posts: 198


Avg: ~ 218kb
  Reply With Quote
07/07/18, 08:46 AM   #36
nightstrike2
Join Date: Sep 2017
Posts: 12
Holy cow, my request got through?!?!?! I will happily do anything required to help test, integrate, and even design this feature!!!
  Reply With Quote
07/07/18, 08:56 AM   #37
nightstrike2
Join Date: Sep 2017
Posts: 12
And, your requested file size information:

Code:
total 273M
248K AlphaGear.lua
 32K AutoCategory.lua
 56K AwesomeGuildStore.lua
1.8M CraftStoreFixedAndImproved.lua
4.0K CrownCrateLogger.lua
4.0K FurniturePreview.lua
4.0K GearSetsSort.lua
1.7M HarvensQuestJournal.lua
180K HarvestMap.lua
140K HarvestMapAD.lua
 48K HarvestMapDC.lua
276K HarvestMapDLC.lua
 40K HarvestMapEP.lua
 36K HarvestMapNF.lua
2.3M kyoAchievenizer.lua
144K MasterMerchant.lua
1.1M MasterRecipeList.lua
 16M MM00Data.lua
 18M MM01Data.lua
 16M MM02Data.lua
 18M MM03Data.lua
 12M MM04Data.lua
 16M MM05Data.lua
 15M MM06Data.lua
 19M MM07Data.lua
 16M MM08Data.lua
 17M MM09Data.lua
 17M MM10Data.lua
 20M MM11Data.lua
 18M MM12Data.lua
 16M MM13Data.lua
 17M MM14Data.lua
 20M MM15Data.lua
236K pChat.lua
8.0K RareFishTracker.lua
1.0K Roomba.lua
4.0K SkyShards.lua
1.0K VotansImprovedOutfit.lua
4.0K Zgoo.lua
 76K ZO_Ingame.lua
4.0K ZO_Pregame.lua
  Reply With Quote
08/07/18, 04:26 PM   #38
decay2
AddOn Author - Click to view addons
Join Date: Aug 2014
Posts: 69
Here you go:

Code:
     Length Name
     ------ ----
   35.68 KB AddonSelector.lua
    1.04 MB AlphaGear.lua
    8.26 KB AssistRapidRiding.lua
     630  B AsylumNotifier.lua
     826  B AuraMastery.lua
   82.74 KB AutoCategory.lua
    1.07 KB AutoInvite.lua
   23.03 KB AutoResearch.lua
   22.91 KB AwesomeGuildStore.lua
   10.50 MB BankManagerRevived.lua
     569  B Chat2Clipboard.lua
   66.83 KB CombatMetrics.lua
    6.88 MB CombatMetricsFightData.lua
   30.04 KB CombatReticle.lua
     458  B Constellations.lua
     734  B CraftBagExtended.lua
   15.00 KB CraftingStations.lua
   10.06 MB CraftStoreFixedAndImproved.lua
    3.93 KB CyrodiilAlert.lua
    1.28 KB DailyAlchemy.lua
     293  B DailyProvisioning.lua
     797  B DarkUI.lua
  802.06 KB Destinations.lua
    4.78 KB DeveloperSuite.lua
   12.53 KB DevTools.lua
   41.47 KB DolgubonsLazyWritCreator.lua
  137.90 KB Dustman.lua
    8.28 KB FCMQT.lua
  318.49 KB FCOItemSaver.lua
    2.10 KB FCOLockpicker.lua
    8.29 KB FCOStarveStop.lua
    7.21 KB filelist.txt
   14.38 KB FoundryTacticalCombat.lua
  134.80 KB FurnitureCatalogue.lua
     328  B FurnitureCatalogue_Export.lua
     739  B FurniturePreview.lua
   15.13 KB GroupDamageShare.lua
    4.23 KB GuildBankTwiddler.lua
    2.94 KB HarvensImprovedSkillsWindow.lua
     453  B HarvensStackSplitSlider.lua
  151.06 KB HarvensTraitAndStyle.lua
  103.55 KB HarvestMap.lua
    1.20 MB HarvestMapAD.lua
    1.46 MB HarvestMapAD-backup.lua
    1.38 MB HarvestMapDC.lua
    1.60 MB HarvestMapDC-backup.lua
    1.35 MB HarvestMapDLC.lua
  818.61 KB HarvestMapDLC-backup.lua
    1.68 MB HarvestMapEP.lua
    1.55 MB HarvestMapEP-backup.lua
    1.83 MB HarvestMapNF.lua
    1.07 MB HarvestMapNF-backup.lua
    1.64 KB HelmetToggle.lua
     353  B HoFNotifier.lua
    1.69 MB IIfA.lua
    1.19 MB ImprovedDeathRecap.lua
    3.03 KB JoGroup.lua
     315  B jovAST.lua
  329.85 KB KillCounter.lua
     789  B LibGroupSocket.lua
     947  B LightAttackHelper.lua
    2.90 KB LootDrop.lua
     363  B LootLog.lua
    7.26 KB LoreBooks.lua
   13.87 KB LostTreasure.lua
   83.07 KB MasterMerchant.lua
    1.17 MB MasterRecipeList.lua
     532  B merlinsRezHelper.lua
    1011  B merTorchbug.lua
   11.43 KB MitigationPercent.lua
    9.17 MB MM00Data.lua
   10.06 MB MM01Data.lua
   10.26 MB MM02Data.lua
   10.68 MB MM03Data.lua
    9.69 MB MM04Data.lua
    8.56 MB MM05Data.lua
    9.03 MB MM06Data.lua
   10.20 MB MM07Data.lua
    9.23 MB MM08Data.lua
    9.71 MB MM09Data.lua
   10.84 MB MM10Data.lua
   11.04 MB MM11Data.lua
   11.24 MB MM12Data.lua
    9.02 MB MM13Data.lua
   11.97 MB MM14Data.lua
   10.42 MB MM15Data.lua
     215  B NoobFilter.lua
    5.31 KB NoThankYou.lua
  390.35 KB pChat.lua
     794  B PenTest.lua
    8.96 KB Photographer.lua
    2.03 KB Postmaster.lua
   12.38 KB PotionMaker.lua
    9.25 KB PvPJ.lua
    9.29 KB RaidNotifier.lua
   29.29 KB Ravalox'QuestTracker.lua
   18.63 KB Recharge.lua
    1.63 KB RipFilter.lua
     522  B Roomba.lua
   33.39 KB SetManager.lua
     188  B sidTools.lua
    9.57 KB Siroria.lua
    7.92 KB SkyShards.lua
    2.98 KB SocialIndicators.lua
    3.33 KB SpentSkillPoints.lua
   20.61 KB SummonAssistant.lua
    7.77 KB SuperStar.lua
    1.26 KB Tank.lua
    2.70 KB TextureIt.lua
  240.95 KB tom.lua
   19.47 KB Untaunted.lua
   55.48 KB VotansAchievementsOvw.lua
    1.35 KB VotansAdaptiveSettings.lua
     543  B VotanSearchBox.lua
    2.09 KB VotansGroupPins.lua
    6.34 KB VotansImprovedLocations.lua
      38  B VotansImprovedOutfit.lua
  158.13 KB VotansKeybinder.lua
   10.92 KB VotansMiniMap.lua
     561  B VotansRuneTooltips.lua
    3.42 KB VotansSettingsMenu.lua
  424.05 KB vSAForDummies.lua
   54.65 KB WritWorthy.lua
   17.26 KB wykkydsToolbar.lua
     340  B Zgoo.lua
  209.25 KB ZO_Ingame.lua
     354  B ZO_Pregame.lua
    4.25 KB Zolan_AutoRepair.lua
   47.68 KB Zolan_ChatNotifications.lua
  Reply With Quote
08/07/18, 10:04 PM   #39
Phuein
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 47
I'm the developer for Notebook 2018. A fan of the addon (yes, a fan! lol) directed my attention to this post.

At this point, ANY solution is welcome. Opt-in only for "large" savedvars files strongly preferable. My addons savedvars are generally "small", but the content is only user hand-made, so losses are terrible.

Also, when it is added, I wouldn't mind getting some notification with clear instructions / explanation. Be it a message here or an email as an addon developer. It'll be a celebration!
  Reply With Quote

ESOUI » Developer Discussions » General Authoring Discussion » Auto Saving SavedVariables

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off