Thread Tools Display Modes
08/20/15, 11:49 AM   #1
domintheus
Join Date: Aug 2015
Posts: 8
Question Addon connect to external database?

Preface
  • I've seen some threads in regards to global auction houses, yet that's not the purpose for the addon I'm working on.
  • I understand the ui would need reloaded to get updates, I am fine with that. The target group of people are fine with that.
  • There won't be a very large number of people using the addon, so this shouldn't be a problem.
  • This isn't a question of should I do this, it is a question of how to do this.


The goal is to have an addon that can get data from an external database and display it ingame, have a field for text entry and be able to save the data to be posted via ui reload. The database connection will be set up in PHP, obviously it's not a good idea to put database usernames/passwords in the .lua files for all to find. Hopefully this will open up new possibilities for future addons from all creators.


Does anyone know how to connect to an external resource, specifically a database, webpage, or even an image on a website, with an addon in ESO?

Last edited by domintheus : 08/20/15 at 11:55 AM.
  Reply With Quote
08/20/15, 11:55 AM   #2
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
The only possibility would be to:

-Create an external tool which connects to the database, reads some information and saves the DB data into the SavedVariables lua file of an addon
-Reloadui ingame and activate the addon before
-The addon is able to use the data from its own SavedVariables then (which came from the DB)

Same thing the other way around if you like to update data from the SavedVariables to the database:
-Update information
-Reloadui
-External tool read the SavedVariables lua file and updates it into a database
  Reply With Quote
08/20/15, 02:08 PM   #3
domintheus
Join Date: Aug 2015
Posts: 8
Thanks! Would it be possible for you to provide an example of this?

I feel not only would it help me, but it would also help many others who may be seeking to do something similar.

**EDIT**

I apologize if my request seems to be too forward, but I'm very eager to figure this out and any additional help would be highly appreciated as my fingers need a break after making the API on the wiki a little more convenient to navigate:

http://wiki.esoui.com/w/index.php?ti...action=history

Last edited by domintheus : 08/20/15 at 02:13 PM. Reason: Felt too forward in my request.
  Reply With Quote
08/20/15, 03:35 PM   #4
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
What kind of example do you want to see for this?
I'm not able to actually write an addon for it :-)

Here is an example tutorial abou the SavedVariables:
http://wiki.esoui.com/Circonians_Sav...ables_Tutorial

Maybe it helps you to understand the addons and how they save the data to the harddisk.

If you just take a look at the SavedVariables folder (one folder above your "Addons" folder) on your local harddrive, you'll find the stored lua files for the different addons.
The path should be something like:
Code:
C:\Users\<your windows user name>\Documents\Elder Scrolls Online\live(eu)
The structure of the SavedVariables lua files look like this:
Code:
Addon_SavedVariables_Name =
{
    ["Default"] =
    {
        ["@AccountName"] =
        {
            ["CharacterName1"] =
            {
                ["variableName"] =
                {
                    ["value1"] = 1,
                    ["value2"] = true,
                    ["value3"] = {
                    	"1" = 1,
                        "2" = true,
                        "3" = "Hello"
                    },
                },
            },
            ["CharacterName2"] =
            {
                ["variableName"] =
                {
                    ["value1"] = 12,
                    ["value2"] = false,
                },
            }
            ["$AccountWide"] =
            {
                ["variableName"] =
                {
                    ["value1"] = -1,
                    ["value2"] = "Hello World",
                },
            }
        }
    }
}
The "Addon_SavedVariables_Name" can be any string as a description which addon' saved variables this table contains. It's sepcified inside the according addon as you can see in the tutorial above.

The "default" table entry is the default branch to follow.

@AccountName is a placeholder for the @account ingame where these settings are stored for.
Below the @AccountName there can be different "CharacterNames" if you store the settings differently for each of your characters, or a variable "$AccountWide" which will contain the settings for ALL your characters.
Settings for all characters AND for each character can be combined, so there mustn't be just either each character or all characters.
Example: Positions of windows will be stored differently for each character, but the content will be stored the same for all characters.


The data saved will be stored as different types.
Either boolean, integer, String, array, mixture of all, etc like shown in the example structure above.



You'd need some tool where you specify the SavedVariables .lua filename to read from, parse the structure of this lua file and use the information gained to upload it to a database somewhere.

Same way for writing to the SavedVariables file.
But: If you write to the SavedVariables file and the game client is running at this moment, all written data to the SavedVariables will be overwritten by the current game values if the user logouts/closes the game/does an /reloadui ingame AFTER you wrote to the file.
  Reply With Quote
08/20/15, 06:12 PM   #5
domintheus
Join Date: Aug 2015
Posts: 8
Very helpful. Thanks.

I should have been more specific, I apologize. I was getting a little hungry after spending all day on the wiki, should have had a snickers with me.

I was more curious about how one would go about making an external tool for connecting to a webserver. I understand PHP and can set up a page to translate sent data and do all the processing on the webserver side. I am just a little undereducated on the LUA side. If anyone could help bring some insight to this, it would be greatly appreciated.

Once figured out I will go back and edit the initial post in this thread with an example on how to make a simple addon send data to a webserver, and an example on what a php page should be set up like to retrieve it.
  Reply With Quote
08/20/15, 06:38 PM   #6
Ayantir
 
Ayantir's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 1,019
depends what you're going to do, what's your project? (i'm currently working on same thing, but a way bigger).
  Reply With Quote
08/20/15, 06:58 PM   #7
domintheus
Join Date: Aug 2015
Posts: 8
Well, I'd really like to be able to bring a limited number of threads in a guild's forum to the game. One of the biggest problems these days is the increased sense of rush, where no one seems to have time to utilize a guild website anymore. Guild websites are great for structure and organization of Guild Events, information, and help where each member can benefit greatly. Nobody seems to want to Alt-Tab out anymore, so if a thread has been updated, it would be among the few threads that the addon can pull from the forum. If a person is interested in what has been said, and wants to contribute to that conversation, they can then press a button to update topics (reload ui), read the new stuff, write a comment, post the comment (reload ui), and move on. All while in the game.

It would have to be configured with the web address of the php translating file on the guild's website, and with the individual's forum login info.

I say a few threads only because initially I'd like to see how much of an impact it would have on the memory usage. I'd likely keep it to the initial thread post, and the last 3 comments for up to 5 total threads to begin.
  Reply With Quote
08/20/15, 08:00 PM   #8
haggen
 
haggen's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2015
Posts: 137
I'm working on such software http://www.esoui.com/forums/showthread.php?t=4934

I haven't updated the thread yet but I have had some real progress.

The current concept is this:

The add-on manifest gets a new keyword "ArgonianUploadEndpoint", see an example:

Code:
## Name: ...
## ...
## ArgonianUploadEndpoint: <endpoint>
Once the user has the software -- Argonian -- running, it scans add-ons with said configuration and sends its saved variables to back to the add-on owner. Then it saves the last-modified-time of the file to avoid sending the same data twice, and awaits the files to change, when it repeats the operation.

It'll even accept URL or e-mail as endpoints, where it'll upload or send the file as an attachment accordingly.

I believe I'm somewhere around 70% of completion, maybe 2 weeks until a first version.

It's taking me sometime because of a couple of reasons; a) it's been a while since I last worked with .NET, so I'm still getting the hang of things; b) I work over full-time on my own job which leaves me with rare times during the week and most of the weekends to work on it; and c) I realized in the middle of the work that I needed to figure out some things before continuing (as you can see in my thread, I considered spinning up a server, which now I discarded).

Last edited by haggen : 08/20/15 at 08:07 PM.
  Reply With Quote
08/20/15, 08:04 PM   #9
Ayantir
 
Ayantir's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 1,019
Please check first if your db can be accessed remotly. It cannot be done generally with web hosting. You can see this easily with phpmyadmin per exemple. If not, you'll need to do an HTTP POST and the webform will parse and insert data .

Last edited by Ayantir : 08/20/15 at 08:26 PM.
  Reply With Quote
08/20/15, 08:13 PM   #10
haggen
 
haggen's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2015
Posts: 137
Originally Posted by domintheus View Post
Well, I'd really like to be able to bring a limited number of threads in a guild's forum to the game. One of the biggest problems these days is the increased sense of rush, where no one seems to have time to utilize a guild website anymore. Guild websites are great for structure and organization of Guild Events, information, and help where each member can benefit greatly. Nobody seems to want to Alt-Tab out anymore, so if a thread has been updated, it would be among the few threads that the addon can pull from the forum. If a person is interested in what has been said, and wants to contribute to that conversation, they can then press a button to update topics (reload ui), read the new stuff, write a comment, post the comment (reload ui), and move on. All while in the game.

It would have to be configured with the web address of the php translating file on the guild's website, and with the individual's forum login info.

I say a few threads only because initially I'd like to see how much of an impact it would have on the memory usage. I'd likely keep it to the initial thread post, and the last 3 comments for up to 5 total threads to begin.
Lua, with a few modules maybe, would be able to this just fine. But for security measures, ZeniMax has implemented a limited version of Lua in the game, where handling files and/or communicating with external services is impossible, and otherwise illegal to do.

Last edited by haggen : 08/21/15 at 09:23 AM.
  Reply With Quote
08/20/15, 09:58 PM   #11
Fyrakin
 
Fyrakin's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 129
If the data exchange is fine between gaming sessions (not requiring real time communications) then this scenario is more than doable. Add-on database can accept, consolidate and redistribute data it collects between each add-on user, so if you plan to make something similar its not a rocket science hard to make .
  Reply With Quote
08/21/15, 09:24 AM   #12
haggen
 
haggen's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2015
Posts: 137
Originally Posted by Fyrakin View Post
If the data exchange is fine between gaming sessions (not requiring real time communications) then this scenario is more than doable. Add-on database can accept, consolidate and redistribute data it collects between each add-on user, so if you plan to make something similar its not a rocket science hard to make .
How would you send the data to another user ?
  Reply With Quote
08/21/15, 11:25 AM   #13
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
Please also keep in mind that the maximum upload speed should be configurable as otherwise (if larger SavedVariables files will be sent) your ingame experience will be very laggy
Not every connection to the internet is uploading synchron with multiple mbits. Most of them still are asynchron and only send a maximum of ca. 100kb / s with maximum upload speed.
So sending a 600 kb file in the wrong moment (boss fight) could be really crap :-)

Originally Posted by haggen View Post
I'm working on such software http://www.esoui.com/forums/showthread.php?t=4934

I haven't updated the thread yet but I have had some real progress.

The current concept is this:

The add-on manifest gets a new keyword "ArgonianUploadEndpoint", see an example:

Code:
## Name: ...
## ...
## ArgonianUploadEndpoint: <endpoint>
Once the user has the software -- Argonian -- running, it scans add-ons with said configuration and sends its saved variables to back to the add-on owner. Then it saves the last-modified-time of the file to avoid sending the same data twice, and awaits the files to change, when it repeats the operation.

It'll even accept URL or e-mail as endpoints, where it'll upload or send the file as an attachment accordingly.

I believe I'm somewhere around 70% of completion, maybe 2 weeks until a first version.

It's taking me sometime because of a couple of reasons; a) it's been a while since I last worked with .NET, so I'm still getting the hang of things; b) I work over full-time on my own job which leaves me with rare times during the week and most of the weekends to work on it; and c) I realized in the middle of the work that I needed to figure out some things before continuing (as you can see in my thread, I considered spinning up a server, which now I discarded).
  Reply With Quote
08/21/15, 11:58 AM   #14
domintheus
Join Date: Aug 2015
Posts: 8
Originally Posted by haggen View Post
How would you send the data to another user ?
each user will have to configure their login info to the guild website within the addon settings, to see new posts after having been online for awhile, they would have to request new topics (reload ui)

Unless there is a way to distribute data between players in the game??
  Reply With Quote
08/21/15, 01:05 PM   #15
haggen
 
haggen's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2015
Posts: 137
Originally Posted by domintheus View Post
each user will have to configure their login info to the guild website within the addon settings, to see new posts after having been online for awhile, they would have to request new topics (reload ui)

Unless there is a way to distribute data between players in the game??
I mean technically how would you sync this information ? Or are you saying that the player would have to download the file and place it in the correct folder manually ?

About that, I wonder if we could hack our way to sync data between players in-game... What if we send the data serialized via mail/chat/private message, and then the add-on would detect such message and deserialize it. That'll definitely be ugly, but it could work. Is it possible to automate messages between players in the game ? It could work like a peer-to-peer or decentralized data transmission system

Last edited by haggen : 08/21/15 at 01:08 PM.
  Reply With Quote
08/21/15, 01:05 PM   #16
haggen
 
haggen's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2015
Posts: 137
Thumbs up

Originally Posted by Baertram View Post
Please also keep in mind that the maximum upload speed should be configurable as otherwise (if larger SavedVariables files will be sent) your ingame experience will be very laggy
Not every connection to the internet is uploading synchron with multiple mbits. Most of them still are asynchron and only send a maximum of ca. 100kb / s with maximum upload speed.
So sending a 600 kb file in the wrong moment (boss fight) could be really crap :-)
Nice point Baertram, thank you (again)
  Reply With Quote
08/21/15, 01:21 PM   #17
Baertram
Super Moderator
 
Baertram's Avatar
WoWInterface Super Mod
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 4,912
Originally Posted by haggen View Post
I mean technically how would you sync this information ? Or are you saying that the player would have to download the file and place it in the correct folder manually ?

About that, I wonder if we could hack our way to sync data between players in-game... What if we send the data serialized via mail/chat/private message, and then the add-on would detect such message and deserialize it. That'll definitely be ugly, but it could work. Is it possible to automate messages between players in the game ? It could work like a peer-to-peer or decentralized data transmission system
Emails sometimes will not show until you logout and in again. And the email texts are only accessible as you open the email inbox "manually". Answer: No

Chat messages can be sent by the help of addons as far as I know. But sometimes you are only able to put the text into the chat edit box, and you'll manually have to press the RETURN key to send the message.
Answer: Maybe (but still ugly!!!! )

Instand messages are the same like chat messages.


About the synch:
An external tool needs to read the data from a website/database and must update the information to the SavedVariables file of your addon (which is displaying the messages ingame).

Problems:
If the ueser is currently doing a reloadui ingame as the external tool is trying to update the SavedVariables file there are 2 sources trying to access the SavedVariables file at the same time -> How do you detect this? (Maybe preHook the /reloadui function and set a variable to TRUE inside the SavedVariables of your addon, read this variable by help of the external tool and only if it is FALSE update the SavedVariables file from external sources. To reet the variable to FALSE just do it at the EVENT_PLAYER_ACTIVATED).
But the other way around it is not possible to detect from the addon ingame if the external tool is currently accessing the SavedVariables file

All updates from external will be overwritten again if you do a reloadui! The game will always take the internally used variables and data and store it into the SavedVariables. So basically there is no way to update it from external as long as the user is logged in :-(
Maybe if you work with 2 different SavedVariables files somehow.
  Reply With Quote
08/21/15, 02:44 PM   #18
haggen
 
haggen's Avatar
AddOn Author - Click to view addons
Join Date: Jul 2015
Posts: 137
Originally Posted by Baertram View Post
Emails sometimes will not show until you logout and in again. And the email texts are only accessible as you open the email inbox "manually". Answer: No

Chat messages can be sent by the help of addons as far as I know. But sometimes you are only able to put the text into the chat edit box, and you'll manually have to press the RETURN key to send the message.
Answer: Maybe (but still ugly!!!! )

Instand messages are the same like chat messages.


About the synch:
An external tool needs to read the data from a website/database and must update the information to the SavedVariables file of your addon (which is displaying the messages ingame).

Problems:
If the ueser is currently doing a reloadui ingame as the external tool is trying to update the SavedVariables file there are 2 sources trying to access the SavedVariables file at the same time -> How do you detect this? (Maybe preHook the /reloadui function and set a variable to TRUE inside the SavedVariables of your addon, read this variable by help of the external tool and only if it is FALSE update the SavedVariables file from external sources. To reet the variable to FALSE just do it at the EVENT_PLAYER_ACTIVATED).
But the other way around it is not possible to detect from the addon ingame if the external tool is currently accessing the SavedVariables file

All updates from external will be overwritten again if you do a reloadui! The game will always take the internally used variables and data and store it into the SavedVariables. So basically there is no way to update it from external as long as the user is logged in :-(
Maybe if you work with 2 different SavedVariables files somehow.
Well, I figured that it would be damn hard if not impossible to do something like this, since it's a big security hole for the game. :P Thanks for the insight.

About the sync, yea that's an issue. On hindsight I see 1 possible solution and one workaround;

I noticed that my editor, Atom, could not read the the saved variables file while the game was changing it, MAYBE the game take control, the tool could detect this and delay the sync accordingly. But I feel that it'd be more difficult than that.

The workaround would be what you already mentioned; it'd befall on the add-on author to make an additional data-only add-on that could store like a version/last-modified timestamp and periodically check for update from the actual add-on.
  Reply With Quote
08/21/15, 03:26 PM   #19
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,566
Originally Posted by Baertram View Post
Problems:
All updates from external will be overwritten again if you do a reloadui! The game will always take the internally used variables and data and store it into the SavedVariables. So basically there is no way to update it from external as long as the user is logged in :-(
Maybe if you work with 2 different SavedVariables files somehow.
When you use an external tool, nothing is going to stop you from replacing one file in your addon with updated data from your website. On UI reload it will load the new data as part of the addon.
  Reply With Quote
08/21/15, 03:29 PM   #20
domintheus
Join Date: Aug 2015
Posts: 8
Couldn't the reload action both be a POST then GET operation? Operating on initial game load, similar to how MasterMerchant updates on initial game load, then only POSTing/GETting when the content is submitted or fetched via the buttons that do the reloadui?

The PHP side, or .NET in your case, could be configured with variables, just as you see in this forum.

In example:

Code:
newreply.php?do=newreply&noquote=1&p=22879

$do = newreply;
$noquote = 1;
$p = 22879;
This will require some work depending on the forum software being used, but it is definitely possible. All of the info will need to be passed as variables to the PHP translation file, it will handle the login, comment posting, then fetching the few topics and comments to replace the existing ones in the saved variables.

Variables would be something like:

Code:
$email = '[email protected]';
$password = 'somepass';
$topics = array(); // storing the topics/threads
$comment = 'some content to post as a comment';
$commenttopic = 38948; // the topic to post the comment too
If anyone is willing to help with the idea, I'm willing to put a forum up to test with.
  Reply With Quote

ESOUI » AddOns » AddOn Help/Support » Addon connect to external database?

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off