Thread Tools Display Modes
09/08/15, 05:51 PM   #1
Sp00sty
Join Date: Apr 2014
Posts: 67
1.7 FPS and loading screen death - ZOS says all addons fault?

ZOS seems to want to just blame addons and force me to just play without any addons to get the game to run right.

I've been gathering all the updates but even with swapping addons in and out I am still getting 999+ char load pings and freezes after reloadui.

Has anyone gotten anything from ZOS about what they have done that is making addon use so difficult?

I'm happy to keep trying diff combos of addons or no addons to resolve but if ZOS has spilled the beans at all it might help me decide which I can try to live with or not.
  Reply With Quote
09/08/15, 10:41 PM   #2
Fyrakin
 
Fyrakin's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 129
Originally Posted by Sp00sty View Post
ZOS seems to want to just blame addons and force me to just play without any addons to get the game to run right.

I've been gathering all the updates but even with swapping addons in and out I am still getting 999+ char load pings and freezes after reloadui.

Has anyone gotten anything from ZOS about what they have done that is making addon use so difficult?

I'm happy to keep trying diff combos of addons or no addons to resolve but if ZOS has spilled the beans at all it might help me decide which I can try to live with or not.
There were undocumented changes in API as well as new additions that made some add-ons significantly slow down UI. There are still performance issues with ESO client itself. Add-on developers have a limited ability to verify/profile their add-ons on a wide range of systems/platforms as well as combinations with other add-ons. I would suggest you to start from identifying the problematic add-on and asking for help.
  Reply With Quote
09/09/15, 04:44 AM   #3
Sp00sty
Join Date: Apr 2014
Posts: 67
Thanks Fyrakin,

Ive been following your updates and comments and have tried to group addons that might be related in issues from the API, pin calls for maps for example, and quest/progress calls or ones that query the client continuously, and loading/pulling them in blocks.

It looks like I'll have to go back to load/pull each one individually until I find either a single major culprit or the specific grouping that mitigates the issues as best can be done.

And continued thanks to you and the other authors that still have the desire and patience to try to keep up with ZOS known and unknown changes.
  Reply With Quote
09/09/15, 08:01 AM   #4
ZOS_ChipHilseberg
ZOS Staff!
Premium Member
Yes this person is from ZeniMax!
Join Date: Oct 2014
Posts: 551
Originally Posted by Sp00sty View Post
Thanks Fyrakin,

Ive been following your updates and comments and have tried to group addons that might be related in issues from the API, pin calls for maps for example, and quest/progress calls or ones that query the client continuously, and loading/pulling them in blocks.

It looks like I'll have to go back to load/pull each one individually until I find either a single major culprit or the specific grouping that mitigates the issues as best can be done.

And continued thanks to you and the other authors that still have the desire and patience to try to keep up with ZOS known and unknown changes.
Can you tell me which addons you are using and the size your saved variable files? The saved variables will be in Documents\Elder Scrolls Online\live\SavedVariables.
  Reply With Quote
09/09/15, 10:37 AM   #5
Sp00sty
Join Date: Apr 2014
Posts: 67
Do you need the saved variable size for each individual addon or just the total?
  Reply With Quote
09/09/15, 11:02 AM   #6
ZOS_ChipHilseberg
ZOS Staff!
Premium Member
Yes this person is from ZeniMax!
Join Date: Oct 2014
Posts: 551
The total would be fine. Also what framerates are you seeing and how long is a reload ui taking?
  Reply With Quote
09/09/15, 01:11 PM   #7
Sp00sty
Join Date: Apr 2014
Posts: 67
Well it looks like they may have done some server side tweaking today. No kickouts or server disconnects. Last few days were terrible.

As seen below today's experience is fine as long as server performance stays like this. There is still something that is churning Ping and FPS upon load though and locking initial char load for several secs every time.

Ran several logins/outs reloadui and fast travels:

Character starts in Belkarth Bank.

ESO login no issues. Char page: 8.625s

Char Load: 59s and immediate 999+ ping, 0-50FPS

Screen then locks with CLS window initialize 8s (CLS set to OFF, both windows, and output to chat tab. Don't know why it needs to display at char load.) Also Fyrakin Minimap is blank and takes a sec to load pins (I haven't tweaked any performance settings).

Then Screen is back with another 999+ ping and 0-25 FPS for 4-5s

Next did Reloadui 9.34s

Loads again with 999+ ping 0-50 FPS for about 4-5s. But no screen lock CLS.

Fast Travel Belkarth-Daggerfall 46s and follows 999+ ping sequence. But no Screen lock CLS.

After static Belkarth bank - 28-99 FPS 78-184 Ping Some immediate stutter inside bank but cleared quickly

Here's what I have been using prior to patch 1.7 with no issues. And today with details above. *LUA memory set at 768*

Warning: Spoiler

Last edited by Sp00sty : 09/10/15 at 05:13 AM.
  Reply With Quote
09/09/15, 01:57 PM   #8
@AlphaLemming
 
@AlphaLemming's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 122
Autsch. You should definitely clear out that list. Some addons are outdated and i know at least four addons that can replaced by a single one .
  Reply With Quote
09/09/15, 03:02 PM   #9
Sp00sty
Join Date: Apr 2014
Posts: 67
I understand its a long list and some may look to be duplicates but perform distinct things that aren't in an all-in-one addon.

In major blocks of categories I can try to consolidate but the issue is that they have been functioning and all are functioning now without throwing any errors into bugeater but now the ping/lockups.

Some of the functions/addons are prob never going to be updated but continue to work w/o error.

Is the consensus now that there is just a hard limit that ESO simply won't handle anymore after this patch?

Or is there just some function(s) in the new API that are just going to cause these Ping/FPS/load issues with some addons even if they aren't throwing actual errors?

Like I said I'll go into them and see what I can consolidate, but still hoping there is some common thread to what is demonstrating that max ping and lockup at char load. It is when that ping/lockup finally breaks sync with the server that I appeared to get the Server disconnects or "there is an unknown problem" kickouts to the login screen.

Thanks again all for taking a look.
  Reply With Quote
09/09/15, 03:57 PM   #10
Ravalox Darkshire
Ravalox of Lake Superior
 
Ravalox Darkshire's Avatar
Premium Member
AddOn Author - Click to view addons
Join Date: May 2014
Posts: 22
I know this is coming from another direction but still worth asking ...

What's your rig config? (CPU, RAM, Graphics)
What's your internet connection? (DSL, Broadband, Satellite, etc)
What kind of Hard Drive do you have? (SSD, HDD)

If you have an HDD, what is your fragmentation %?
For SSD, do you manually trim it often?
- also if SSD do you defrag?

While experiencing the issue, try running (from CMD window) netstat -s
This is a snapshot of the NIC counters. You can repeat the command a number of times to see if it is actively counting.

To easily save the output enter this command netstat -a >>nic_stats.txt
Each time you execute this it will append the new output to the same text file (you will see no output).
I would execute the command every 10-30 seconds for about 5-10 times to get a good baseline prior to running the game and then again (in a new filename) while running the game with the symptoms.

You will be looking for the error counter fields. There may be low level numbers but this happens, the key thing is to show if they are rising actively. (these number are reset when the NIC is reset or you powerdown).

Just baseline stuff really, may assist in diags ....

Hope this helps ...

Last edited by Ravalox Darkshire : 09/09/15 at 04:02 PM.
  Reply With Quote
09/09/15, 08:37 PM   #11
Sp00sty
Join Date: Apr 2014
Posts: 67
Here's the specs:

ASUS Maximus VII Hero/I7-4790K Stock/16GB DDR3-1600/ Gigabyte GTX970 G1 4Gb/ACER GD235hz@120Hz 1920x1080 24"/WIN7x64 Professional

All game files on Micron M500 960gb SATA3 SSD with 340+ Gb free - all trim auto by system/ssd

Suddenlink 50Mbs Wired Cable Modem - sitting behind 3COM Cable Router w/Firewall 10/100 - only receiving 30Mbs but have a new EdgeRouter Lite ready to install for full 10/100/1000 50Mbs - Software Online Armor Firewall (ESO allowed on all settings)

Out of game only active connections are Bitdefender on local host.

Ran Netstat -s -a 10 for 2 minutes outside of game and then for logon/off all 8 characters

Compared side by side in Notepad++ they are absolutely identical for the first 2 min except for:

IPV6 STATS - OUT Output Requests = 595 GAME IPV6 Output Requests = 588
UDP IPV6 STATS - OUT Datagrams Sent = 571 GAME OUT Datagrams Sent = 564


There appear to be no major errors in or out of game - stats are identical in and out across 2 min. Stats do not change at all over time.

Warning: Spoiler

Last edited by Sp00sty : 09/10/15 at 05:15 AM.
  Reply With Quote
09/09/15, 09:52 PM   #12
Ravalox Darkshire
Ravalox of Lake Superior
 
Ravalox Darkshire's Avatar
Premium Member
AddOn Author - Click to view addons
Join Date: May 2014
Posts: 22
Originally Posted by Sp00sty View Post
Here's the specs:
Warning: Spoiler

PS is there any kind of Hide or Spoiler code for this forum so I can save you all these long posts.
Yep, the spoiler code is spoiler inside of [] with a /spoiler inside of [] to end.

Good to see your rig has a good setup and the LAN/WAN looks not to be a bottleneck either. (just good things to get out of the way).

I too agree that the reduction of addons would be a good thing. but I also understand how hard it is to get all the features you like without bloat.
  Reply With Quote
09/10/15, 05:17 AM   #13
Sp00sty
Join Date: Apr 2014
Posts: 67
yeah I tried that and must have goofed it but got it now.

Thanks ALL, yes time to go through my categories and see what I can trim down to.

Remainder of last night was getting better and better with even a few logins to Alts not even having the 999+ pause. Hit or miss though.
  Reply With Quote
09/10/15, 05:49 AM   #14
SnowmanDK
 
SnowmanDK's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 161
Hi Sp00sty

I noticed your Destinations.lua 529,287 in size.
It should be around 38,500 for global values + about 40,000 per character in size.
Mine is a total of 284,934 with 7 characters.

My guess is that you activated quest registration at some point, but never extracted the data.
  Reply With Quote
09/10/15, 07:42 AM   #15
ZOS_ChipHilseberg
ZOS Staff!
Premium Member
Yes this person is from ZeniMax!
Join Date: Oct 2014
Posts: 551
57MB is a pretty large amount of saved variable storage. We have an improvement in the works to increase the performance of saved variable write out by 3 or 4 times which should help with that (this would impact /reloadui and logout times).

Do you see significant FPS differences when you have all of your addons on and all of your addons off? We tried installing 10 of the most popular addons internally on a live build including minimap and only saw a 1-2 FPS difference between having them all on and all off.
  Reply With Quote
09/10/15, 11:44 AM   #16
Sp00sty
Join Date: Apr 2014
Posts: 67
Okay just ran some FRAPS numbers. Sorry very unscientific, using watch and timing key press of Character PLAY-Reloadui-Zone-Zone.

NO ADDONS (all installed but not activated):

Character IN (Belkarth Bank) - 11s - No 999+ Ping - No Lockup
Reloadui - 5 sec - No 999+ ping - No Lockup
Mount Sprint to Wayshrine
Zone (Daggerfall) - 32s - No 999+ ping - No Lockup
Zone (Belkarth) - 25s - No 999+ ping - No Lockup
FRAPS: Frames: 23968 - Time: 283314ms - Avg: 84.599 - Min: 9 - Max: 101


ALL ADDONS (all installed and activated):

Character IN (Belkarth Bank) - 58s - 999+ Ping - No Lockup
Reloadui - 11 sec - 999+ ping - GUI/Screen Lockup then cleared and brief 999+ ping
Mount Sprint to Wayshrine
Zone (Daggerfall) - 40s - No 999+ ping - No Lockup
Zone (Belkarth) - 31s - No 999+ ping - No Lockup
FRAPS: Frames: 19005 - Time: 288899ms - Avg: 65.784 - Min: 0 - Max: 101

The 999+ Ping and the lockups can be random and then every time with no pattern.

FRAPS appears to show a loss of FPS, but the game is still playable for me in all zones as long as the 999+ ping and lockups dissipate and don't lose sync and kick me out.

There is definitely no 999+ ping or lockups with no ADDONS loaded.

Thanks for looking at this and let me know if any more data would help from me.
  Reply With Quote
09/10/15, 12:51 PM   #17
Sp00sty
Join Date: Apr 2014
Posts: 67
and @SNOWMANDK

Nope I just checked and no Quest Data in that file.
  Reply With Quote
09/13/15, 08:06 AM   #18
Arterion
Join Date: Apr 2015
Posts: 10
Originally Posted by ZOS_ChipHilseberg View Post
We have an improvement in the works to increase the performance of saved variable write out by 3 or 4 times which should help with that (this would impact /reloadui and logout times).
Getting rid of all the space characters used for indentation? Because that definitely needs to be done! I don't even need tab characters. I can configure my text editor to indent appropriately. I would even be okay with going so far as omitting the newline characters! One long line is fine. The data format is unambiguous.

I also find that when I am profiling performance, I have to look at frametime rather than fps. I may be getting a playable 40 fps, but if my average frametime is 20 ms with a spike of 110 ms twice a second, then the game will seem jerky.

Here is some test data I gathered on ReloadUI times. My SavedVariables folder is 198 MB for 76 Addons, so this is an extreme case. 102 MB of that is a single file: PriceTracker.lua. So disabling the Addon PriceTracker nearly halves the SavedVariable size. I am on a fast SSD, so actual time spend waiting on the disk is minimal. This is almost all engine performance. For reference, My CPU is a AMD FX 6300 running at 4.4 GHz.

I tested three states:
1) No Addons Enabled
2) All Addons Enabled except PriceTracker (96 MB, 75 Addons)
3) All Addons Enabled (198 MB, 76 Addons)

Below I list the average ReloadUI times going from one state to another (or simply reloading the same state). There was very little variation running the the same test multiple times.

67 sec, All Reload
31 sec, All -> None
55 sec, All -> All except PriceTracker

7 sec, None Reload
31 sec, None -> All except PriceTracker
43 sec, None -> All

41 sec, All except PriceTracker Reload
11 sec, All except PriceTracker -> None
55 sec, All except PriceTracker -> All

It seems like ReloadUI writes the files then re-reads them back into memory. It also seems like writing is quite fast, while reading takes more time.
  Reply With Quote
09/14/15, 08:52 PM   #19
Spudd
Join Date: Nov 2014
Posts: 1
Originally Posted by ZOS_ChipHilseberg View Post
We tried installing 10 of the most popular addons internally on a live build including minimap and only saw a 1-2 FPS difference between having them all on and all off.
Sorry to pipe in on this, but I wanted to share my recent findings. Hopefully its helpful.

I don't think the issue is with most of the AddOns at the start, but they become the issue as their SavedVariable .lua file sizes grow.

Recently in a desperate attempt to improve game performance (after lots of LUA memory tweaks) I wiped my MasterMerchant saved variables (multiple < 6MB files totaling 75mb) along with my Thurisaz GuildInfo Saved variable (9mb). After doing that my performance was pretty much completely back to normal. No hitches, no lag, no fps crashes, nothing.

However, as MM has begun loading more data (just 12MB so far), I'm already starting to notice the first hints of performance degradation. Nothing major yet, but just a little bit more every hour or so as the files build. I have no doubt that once I reach about 25-30mb it will probably start being a significant issue again.

Just installing and running an add on (or group of them) probably won't reveal a lot of issues. You need to build a large history up and see how things behave. It sounds like you are already looking at improvements in how the lua files are handled, so that's good news. Just make sure your testing involves the more significant directory sizes that are probably pretty common out there.

My uninformed opinion says that something must have fundamentally changed in how .lua files are read and processed by the client or in how memory is allocated by the client for these operations. My guess is that it's not just the size of the .lua files, but how often they are accessed that contributes to the problem. Either way, its how the client is handling these files and how the memory is being used for those operations that needs to be addressed first.

Thanks, and I hope you are able to find the problem. Even if you can't completely fix it, if new parameters (maybe no files larger than 2mb) can be communicated to the addon devs, that might be helpful for everyone...at least in the short term.
  Reply With Quote
09/14/15, 10:04 PM   #20
Sp00sty
Join Date: Apr 2014
Posts: 67
Well I had hoped that today's patch might have helped resolve whatever has changed but apparently not.

I didn't patch until about 5PM CST and then logged in. First few logins did not have any 999+ ping or lock but by the 3rd Alt-Char for Hireling mails, the ping and locks returned.

As before once cleared the game is completely playable even with lower FPS with addons activated.

BUT, I logged out for dinner around 7:30 PM and I have not been able to get character load since then, now 11PM.

The load wheel churns for a bit and looks like its about to give that last chunky turn right before char load-in but then stops and never moves again. I can see my Cable Router rapid pinging trying to sync with the server but then kick to game login screen with either "Unknown Error" or "Server 'bad' try again later".

As before if I uninstall ALL addons I can get back in without fail. I don't know what changed between logging out for dinner (1hr maybe) but I'm done for the night.

They fixed something today on PC and Console EU, with emergency resets prior to patches, maybe they need to do that to NA PC as well.
  Reply With Quote

ESOUI » AddOns » AddOn Help/Support » 1.7 FPS and loading screen death - ZOS says all addons fault?

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off