ESOUI - [open] Archived Data Storage

While writing LibHistoire I've been thinking how to best store the history data in a way that won't require to delete anything in the future, regardless of how many events are stored. The conclusion I have come to is that it's best to segment it by timeframe and then put it into a compressed form. However with the current way saved variables work, this will only get me so far.

There are currently several issues that make it work less than optimal.

saved vars will escape strings on serialization
- which means we cannot use a binary format for the data
string are limited to a length of up to 1999 characters
- due to this we have to split longer strings and store them in a table
table indices on numeric tables are explicitly written out
- this is a huge waste of space and without it, we could easily save 15% and more on the file size
- due to the way Lua works, it also causes the data in the table to be loaded into the hashed part of a table instead of the indexed part
- this means accessing values after reloading will have worse performance than if they were still stored in the indexed part
- it also means the entries on disk will be scrambled when looking at the file (e.g. index 1 is written in a line after index 2)
saved vars are pretty printed
- removing whitespaces from the file (like what the autosave does), the file would be another 15% smaller
all data is stored in the same file
- this could be somewhat worked around by creating additional addons for archiving, but the game still loads the data on login

This is why I'd like to ask for a new set of apis which allows addons to store rarely needed data in a way that will minimize impact on loading time and help save disk space.
The idea is to allow addons to store an arbitrary amount of raw (binary) strings as files inside a zip-archive named after the addon.

In order to enable this feature, an addon can put the following directive into its manifest:

Code:

## EnableSavedVariableArchive: 1

The game would then maintain such an archive in memory and only read from or write to disk when regular saved variables would do the same.
An addon can then request specific files to be loaded or saved via the following new functions:

Code:

*boolean* success = LoadArchivedFile(*string* addOnName, *string* pathInArchive)

*boolean* success = SaveArchivedFile(*string* addOnName, *string* pathInArchive, *string* content)

This process should be asynchronous for obvious reason and if it's already running, the methods would just return false. The result will then be delivered via the following events:

Code:

EVENT_ARCHIVED_FILE_LOAD_FINISHED(*string* addOnName, *string* pathInArchive, *string* content)

EVENT_ARCHIVED_FILE_LOAD_FAILED(*string* addOnName, *string* pathInArchive, *ArchivedFileLoadFailureReason* reason)

EVENT_ARCHIVED_FILE_SAVE_FINISHED(*string* addOnName, *string* pathInArchive)

EVENT_ARCHIVED_FILE_SAVE_FAILED(*string* addOnName, *string* pathInArchive, *ArchivedFileSaveFailureReason* reason)

To avoid someone writing something into the archive that could be opened as an executable file, I'd advice to not allow any dots in "pathInArchive". Other than that there should not be any limitations on what can be stored.

In my specific use case I'd like to be able to archive the history per year and month and create a structure like this inside the archive:

2020 (folder)
- 10 (file)
- 11 (file)
- 12 (file)
2021 (folder)
- 01 (file)

And of course the archive should just be a regular zip file, so it can be opened or modified outside the game.