Master Merchant and Update 41

Sharlikran · 03/04/24, 10:21 PM

Preface: Understanding the Zenimax Database Changes

Before delving into the details, it's crucial to clarify that the transition from id64 to id53 for sales identifiers is not the main reason behind their reset. Even if the identifier type remained unchanged, the same issue would persist. Zenimax made this change to enhance the efficiency of handling sales identifiers and introduced a new database structure for storing guild history data on the server.

I will focus on why this matters to you as a Master Merchant user. If you notice duplicate sales in the interface, rest assured that it's not an error caused by the id64 to id53 transition. This unavoidable situation stems from the changes in how Zenimax stores sales data. I'll help you understand the impact and your options.

Important Reminder!

Because the changes to the Zenimax server's database are significant, it's recommended to have a backup made prior to Update 41 to restore if changes are made to Master Merchant functionality after launch. Likewise, if Zenimax makes any changes that would directly impact how the data is obtained from the local binary file stored on each user's hard drive.

Introduction

In the upcoming Elder Scrolls Online Update 41, Zenimax is making changes to the Guild History data. The server database for sales data has been updated, and due to that, the unique sales identifiers will not continue from where they left off prior to the update. One significant consequence for users is the inability to check for duplicate sales between the old and new database entries.

Using the image as an example, without the timestamp (1709606076) and without the unique identifier, the sales would be duplicates because they are all from the same seller, buyer, for the same item, and the same amount.

Transition from id64 to id53

In Update 41, Zenimax is implementing significant changes to the Guild History data, introducing a new guild history format and resetting the server's database. Consequently, unique sales identifiers are starting over, and the system will now use the id53 format. This shift is not solely about the transition from id64 to id53; it's part of a broader move toward a more efficient guild history database structure.

Paraphrasing the UI developer Dan Batson:

Luainteger53 is superior because it allows for straightforward mathematical computations and comparisons without the need for complex functions. It can handle values as high as 2^53 (9,007,199,254,740,992) without precision issues. Id64 will still be used when needed such as character IDs. Luainteger53 was introduced a few updates ago and continuing to utilize id64 would be like holding onto an outdated and problematic system just for tradition's sake, which isn't a philosophy I support.

Adapting Master Merchant to the Changes

With these changes, and because the id53 identifiers have started over, Master Merchant will be separating data using the id64 and id53 identifiers. Master Merchant will be able to check duplicates only when the same type of data is compared, meaning id64 and id64 or id53 and id53 because the Zenimax server database is different.

All item averages used for evaluating item pricing and weekly sales totals for Guild Masters will only be accurate when using data from the new Zenimax cache and the id53 identifiers.

While there will be a toggle to utilize the old sales data, without the ability to properly check for duplicate sales, data using id53 mixed with the older id64 data will mean those totals will not be accurate.

Impact on Master Merchant Users

For the majority, the changes will have minimal impact because the binary cache should already contain data. Users that retain 30 to 90 days of data will not have a significant loss of data, and Guild Masters will be able to accurately track recent weeks for evaluating user contribution in sales.

However, users archiving extensive sales data, trying to be the library of congress of data, will have inaccurate averages when the id53 and id64 data overlap.

Additionally, I would like to remind users that maintaining an excessive amount of sales data in the Zenimax binary cache will negatively impact load times during login, even if Master Merchant is disabled.

**Baertram** · 03/05/24, 04:32 AM

Hi Sharlikran

Thanks for that info. Especially the login load times are confusing me:
Means having the new ZOs local binary saved guild history data will always be active, even if addons are disabled?
And thus each login, and reloadUI and zoneing, will be slowed down as the data is read?

Or is it only processed after login, and as you explicitly open the guild history pages or use API functions to load/read data to/from it?

sirinsidiator · 03/05/24, 06:08 AM

The binary cache is always on, but it should be a lot more efficient than what any addon could ever accomplish, so the effect on loading times should be negligible.
By default the game will limit the amount of data in the cache to 30 or 180 days depending on the category and it won't actually do anything with it until an api call is made to read that information.
Due to it being a custom binary format, it is also going to be a lot faster to load than addon saved variables that have to be parsed by the Lua interpreter.

DakJaniels · 03/05/24, 07:22 AM

Noticed this new path when I was cleaning out old AppData.

C:\Users\Dack\AppData\Local\Elder Scrolls Online\live\CachedData\GuildHistory

So, is the data not going to be in a saved variable format then? If it's saved to the AppData folder, addon's won't have access to it right?

**Baertram** · 03/05/24, 07:52 AM

Thanks for the clarification Sid.

@DakJaniels
If so: Holy god, why AppData, and not the same live/ folder in documents...
But as this is on your "live" and not "PTS": Maybe that's a bug, as sometimes even addons move over to the AppData folder by accident.

Sharlikran · 03/05/24, 09:26 AM

Originally Posted by Baertram

But as this is on your "live" and not "PTS": Maybe that's a bug, as sometimes even addons move over to the AppData folder by accident.

I don't have anything in my live folder just the pts. The .\AppData\Local\Elder Scrolls Online\live\CachedData folder was created but for me it is empty. One way or another it will be stored somewhere once U41 is released.

sirinsidiator · 03/05/24, 01:31 PM

The folder is correct and sadly they won't reconsider moving it into the other live folder. They seem to have some reason for it, but not something they will share.

**Baertram** · 03/06/24, 04:49 AM

I've found the folders of live and PTS and one of this folders (live) was created even in September 2023 already o0

Sad it's going to stay in that AppData/Local "hidden folder". Whatever reason ZOs got here, it's disturbing backups and access (explain casual users how to access it if they cannot see it? You always have to use a direct link or the %appdata% constant, and even this brings you to the Roaming folder and not the Local one).
-> Or are backups not really possible with those local cache history files? Like copy the files today to another directory and paste them back tomorrow, and then get only missing events/entries from the server of yesterday to today.

One must use something like this to access it then:
%USERPROFILE%\AppData\Local\Elder Scrolls Online\live\CachedData\GuildHistory

Sharlikran · 03/06/24, 08:56 AM

I guess we will have to see how it goes but for the most part I don't see the need to backup the binary cache. There are already functions to clear specific categories.

I also don't see this as much of an issue because while I don't use OneDrive it probably won't interfere with that folder like it does for the documents folder.

S3V3RIAN · 03/06/24, 08:05 PM

I believe everyone has received warnings to back up data - but what data needs to be backed up - is it just LibHistoire files or others as well.

Sharlikran · 03/06/24, 08:28 PM

Originally Posted by S3V3RIAN

I believe everyone has received warnings to back up data - but what data needs to be backed up - is it just LibHistoire files or others as well.

https://esouimods.github.io/3-master...GuildStoreData

Some people are backing up the entire SavedVariables folder but if you would like to focus on only what you need for MM that link has an image and file names.

**Baertram** · 03/11/24, 10:36 AM

To find the local AppData Elder Scrolls Online -> local guild cache files of your logged in Windows user,
Windows 11 users can use the following variable in the file explorer

Code:

%localappdata%\Elder Scrolls Online

Windows 10 does not support this, so you need to go this way:

Code:

%USERPROFILE%\AppData\Local\Elder Scrolls Online\

or

Code:

%appdata%\..\Local\Elder Scrolls Online

-> This will first go to the Users\Windows user\AppData\Roaming folder, then switch back one folder to \Local and go on to Elder Scrolls Online

In the folder live\CachedData\GuildHistory the local history of the live server is stored, and in PTS\CachedData\GuildHistory folder you'll find the Public Test Server data.

The local cached guild history files are the *.dat files there

Known facts

Logging out/ReloadUI doesn't create the AppData\Local\Elder Scrolls Online\live\Cached\GuildHistory folder. Only a fresh game-restart does.
/reloadui should immediately stores data into .dat files

evilspock · 03/11/24, 02:24 PM

Thanks for this and the amazing work you do on your addons!

Sharlikran · 03/11/24, 09:43 PM

No problem. Things are a little hectic on day one but it will get better. Everyone is always working on something.

I wold like to thank sirinsidiator because he worked really hard on LibHistoire and listened to all my feedback to get it up and running before the patch. While sirinsidiator has quite a todo list I'm happy it is working well enough that MM is working. Day one has been hectic because there are some odd behaviors with guild history and it's causing some confusion. I'm sure everything will get straightened out and working better soon.