Quantcast
Introducing: ESO Market Data Relay - ESOUI
Thread Tools Display Modes
10/17/15, 10:37 PM   #1
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
Lightbulb Introducing: ESO Market Data Relay

Hello Folks!
Alpha download available here.
I have been working the last few weeks on a replication of EMDR (Eve Market Data Relay) for ESO. I am currently around alpha on the relay and looking for feedback. My focus now is verifying the data formats and working on an example site (http://www.antihax.net) on what is possible using the data. In a few days I hope to have the addon and uploader packaged for alpha testing.

What Is This?!
Short version; the data relay allows users to automatically upload market data as they observe it in game. Developers can connect to the relay and receive data in real time as the users change zone, /reloadui, logout, or quit game.

What's the point?
To allow third party development of market related apps and sites to flourish through shared interest which in turn creates useful tools and resources for players such as:
  • finding complete itemsets for sale on the market across zones,
  • finding the cheapest items for sale,
  • data for users without access to large trade guilds,
  • finding the best items to trade telvar/AP for gold,
  • understanding market volumes for specific items in various areas, etc.
  • finding the best gear for your build without running around.
  • third party REST API for google sheets or other sites to further help developers (i.e. http://eve-marketdata.com/developers/index.php or https://eve-central.com/home/develop.html)
  • etc, etc.

What do you need?
To start, i will need testers and other developer's ideas and thoughts to build upon work already done.

I'm Interested, what do i do?
Contact me in game @croakroach or reply with constructive feedback.

Checkout the code below
  1. example site code - Most of the work around the relay is in the datarelayconsumer. Written in Go but zmq is available for most languages.
  2. Addon and Uploader Lua and C#.
  3. Master/Relay code - Down the road we will need more relay operators to provide redundancy and regionalization. Written in Go.

FAQ
  • What about stale data?
    We can work out from other uploads and age of items about how stale they are. Also we know the expiration of items for sale. More people uploading, fresher data.
  • Why would people contribute?
    Because it benefits them to. I.e. how many addon developers have fielded questions about features that this will provide?
  • Wont this destroy the market and drive down prices?
    Not really, having been involved with markets in many other games i see the same thing over and over: people pay extra to save time, make mistakes, are misinformed, or are just plain lazy. Those who search out the deals will still find them.
  • What about duplicates?
    The developer receiving the data is responsible for handling duplicates; the uploader will intentionally send duplicate data between sessions by design. This is so if anyone connects to the relay late, they will not miss much data and will have a complete set within a short amount of time. This also provides a method of detecting freshness of data, allowing sold items to become stale.

Last edited by antihax : 02/28/16 at 08:13 AM.
  Reply With Quote
10/18/15, 08:38 AM   #2
merlight
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 679
Originally Posted by antihax View Post
[*] What about duplicates?
The developer receiving the data is responsible for handling duplicates; the uploader will intentionally send duplicate data. So far I have found that rounding some of the timestamps to 10 help resolve the off by one or two issues and have other identifiers to handle most duplicates. Help here to simplify this would be appreciated.[/list]
If I were to implement this -- and I was pondering that idea, but wasn't enough into trading to take on such an enormous task certainly not since two trading guilds I was in died overnight with ESO update 3 or 4 -- I'd start with a request for ZOS to add SaleId to every store-related API (search, listing, history). The client must have it, it's just a matter of passing it to add-ons. Then there would be no issues with duplicates.

Even replacing the "timeRemaining", "secsSinceEvent" and similar parameters with solid timestamps would simplify detection of duplicates.
  Reply With Quote
10/18/15, 02:30 PM   #3
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
Originally Posted by merlight View Post
If I were to implement this -- and I was pondering that idea, but wasn't enough into trading to take on such an enormous task certainly not since two trading guilds I was in died overnight with ESO update 3 or 4 -- I'd start with a request for ZOS to add SaleId to every store-related API (search, listing, history). The client must have it, it's just a matter of passing it to add-ons. Then there would be no issues with duplicates.

Even replacing the "timeRemaining", "secsSinceEvent" and similar parameters with solid timestamps would simplify detection of duplicates.
I would agree this is a way to go, the current data is a bit lacking for tools like this, but I am not going to let that hold up work. If activity increases, it will prove the necessity for such changes on the API.
  Reply With Quote
10/19/15, 02:52 AM   #4
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,474
I think your stance that the recipient is responsible for handling duplicates might be a huge turnoff for many developers, as this is one of the most difficult problems in here and if everyone has to solve this problem for themselves, you will also get a lot of duplicate solutions.

Rounding the timestamp to 10 is a bad idea, because you will a) loose precision and b) the timestamp varies by a few seconds between session. In the past I also started writing something similar to get store data out of the game and entries that had a stamp that ended on a 5 in one session had one ending on 4 or 6 in another which would then round differently.
I had an idea of using the order of the entries to identify duplicates, but never implemented it.
Search for the entry by timestamp and then see if the previous and next entries have the correct item ids. If there is one wrong, it would be a duplicate.

There is also the problem that you will have to handle manipulated uploads in some way. When you allow anyone to upload data, anyone can manipulate your data freely in any way they want by sending in fabricated entries.

Developers can connect to the relay and receive data in real time as the users change zone
I am not quite sure how you manage to get data on zone changes? The saved vars are only written out when you reload ui, logout or quit to desktop.
Getting data into the game is even harder and in my experience most ESO players won't use any data source outside of the game while playing. That's why I gave up my own project which did exactly what yours is doing and started making AwesomeGuildStore instead.
__________________
https://www.imperialtradingcompany.eu/ - My Addons - The Vault (Early updates and experimental projects) - My patreon - My blog
>siri.exe MyAddon
Does your addon work? [y/n] n
There is a typo in there.
  Reply With Quote
10/19/15, 07:14 AM   #5
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
Good feedback!

I did see yesterday that zone changes do not count. Will have to find some trick to get this to change.
I think I misworded about the duplicates. The system is intended to send duplicates, how do you know data is valid in an anonymous system? How do you calculate probability that an item is still listed? What if someone connects after an item was observed? This is what sending duplicated data is for.

I agree a better solution is to have a unique Id, but we do not. I'm going to see how well rounding to 10 works. Not to worried about losing 10 seconds. What other options are available cross client?
  Reply With Quote
10/19/15, 10:01 AM   #6
dopiate
AddOn Author - Click to view addons
Join Date: Jun 2014
Posts: 142
Originally Posted by antihax View Post
I agree a better solution is to have a unique Id, but we do not. I'm going to see how well rounding to 10 works. Not to worried about losing 10 seconds. What other options are available cross client?
where are you getting your data?

in what format is it uploaded?

You can ask for quite a bit about a sale and put all that info together (with a separator so you can split it later) and get a fairly useful ID

in my previous post I said I have two routines to remove dupes

I parse a specific savedvariabls file and each sale is put into this format:

Code:
Amazing Deals of Tamriel,Argentum,15,1437713676,150,@Medusanomi,@dOpiate,1:0:1:0
taken as a single string in an array that is fairly unique (yours would be even better since you are rounding the time)

That is the data I need to make my program work.

But if the user deleted their savedvariables and this time it comes back with a slightly different time stamp then I drop that from the string and check for

Code:
Amazing Deals of Tamriel,Argentum,15,150,@Medusanomi,@dOpiate,1:0:1:0
now that's not perfect but what are the chances of seeing that exact string twice? you have guild,item,ppu,total sale amount,seller,buyer,"details"

the "details" are created by this code
Lua Code:
  1. local levelReq = GetItemLinkRequiredLevel(itemLink)
  2.   local vetReq = GetItemLinkRequiredVeteranRank(itemLink)
  3.   local itemQuality = GetItemLinkQuality(itemLink)  
  4.   local itemTrait = GetItemLinkTraitInfo(itemLink)
  5.   return levelReq .. ":" .. vetReq .. ":" .. itemQuality .. ":" .. itemTrait

so you can get a fairly reliable amount of data on a sale to make into some sort of identifier from.

They even added lots of new info into the itemLink with the recent patch.

-d
  Reply With Quote
10/19/15, 10:59 AM   #7
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
I am using a similar method as you describe in the lua side.

It looks like the saving of the saved vars file was intentionally locked due to using it as a signaling method for cheats. So I may change the approach.
  Reply With Quote
10/19/15, 05:56 PM   #8
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
Originally Posted by dopiate View Post
where are you getting your data?

in what format is it uploaded?

You can ask for quite a bit about a sale and put all that info together (with a separator so you can split it later) and get a fairly useful ID

in my previous post I said I have two routines to remove dupes

I parse a specific savedvariabls file and each sale is put into this format:

Code:
Amazing Deals of Tamriel,Argentum,15,1437713676,150,@Medusanomi,@dOpiate,1:0:1:0
taken as a single string in an array that is fairly unique (yours would be even better since you are rounding the time)

That is the data I need to make my program work.

But if the user deleted their savedvariables and this time it comes back with a slightly different time stamp then I drop that from the string and check for

Code:
Amazing Deals of Tamriel,Argentum,15,150,@Medusanomi,@dOpiate,1:0:1:0
now that's not perfect but what are the chances of seeing that exact string twice? you have guild,item,ppu,total sale amount,seller,buyer,"details"

the "details" are created by this code
Lua Code:
  1. local levelReq = GetItemLinkRequiredLevel(itemLink)
  2.   local vetReq = GetItemLinkRequiredVeteranRank(itemLink)
  3.   local itemQuality = GetItemLinkQuality(itemLink)  
  4.   local itemTrait = GetItemLinkTraitInfo(itemLink)
  5.   return levelReq .. ":" .. vetReq .. ":" .. itemQuality .. ":" .. itemTrait

so you can get a fairly reliable amount of data on a sale to make into some sort of identifier from.

They even added lots of new info into the itemLink with the recent patch.

-d
I just duplicated the same format as MM for the unique identifiers, it is very similar to yours. This should help a bit.
  Reply With Quote
10/20/15, 06:26 AM   #9
dopiate
AddOn Author - Click to view addons
Join Date: Jun 2014
Posts: 142
a brief history of ESO market addons...

Originally Posted by antihax View Post
I just duplicated the same format as MM for the unique identifiers, it is very similar to yours. This should help a bit.
very cool - I hope it helps

just a little background...

Shopkeeper is based off of Trade Sales History and MM is based off of ShopKeeper (the version Garkin and I fixed and updated - the core of the code was written by Khaibit and awesomebilly).

Philgo68 used ShopKeeper as the base of MM, also some code from ShopKeeper made it's way into Price Tracker Updated when Garkin took it over.

Master Merchant is a great upgrade to ShopKeeper and to make my addon seamlessly integrate with his, Philgo68 was kind enough to work out a system that if my addon is detected then he produces a saved variables file specific to GSA. If you aren't using GSA then this saved variables file does not exist.

Why I am telling you all this? Because when you said you duplicated the MM format (It's very similar to the ShopKeeper format, but he collects a bit more information*) I wanted you to know how that format evolved and who was involved.

I like to give credit where credit is due. If this is something you are duplicating from MM I just wanted to let you know how MM came to be.

Master Merchant is a great addon, really a must have addon along with Awesome Guild Store (they also work together, which is a really nice feature).

This post isn't so much as feedback but background info so the authors of these great addons get the proper credit for getting us to this point.

I'm just providing feedback and suggestions based on my experiences from working on ShopKeeper and GSA and then the integration with MM that I hope will help you in some way.

-d

*Hindsight being 20/20, I wish I had added more info on each sale to make each sale as individual as possible, but right now GSA works 100% and I have no desire to make any big changes. But, I learned to never say never. We will just have to see how interesting the next DLC is.

Last edited by dopiate : 10/20/15 at 06:41 AM. Reason: me fail English? That's unpossible!
  Reply With Quote
10/20/15, 07:23 PM   #10
dopiate
AddOn Author - Click to view addons
Join Date: Jun 2014
Posts: 142
Quick Suggestion

If you leave the executable in the AddOns folder it's less likely to get used.

If you want you can use my Desktop Icon vbs.

Put it in the zip as MakeDesktopShortCut.vbs.remove

That way the user can rename it minus the .remove and run it so it puts a shortcut on the desktop.

This is the way mine does it - feel free to use this if you want...

Code:
Set oWS = WScript.CreateObject("WScript.Shell")  
Set objFSO = CreateObject("Scripting.FileSystemObject")
homedrive = oWS.Environment("PROCESS").Item("HOMEDRIVE")
homepath = oWS.Environment("PROCESS").Item("HOMEPATH")
CurrentDirectory = objFSO.GetAbsolutePathName(".")
shortCut = homedrive & homepath & "\desktop\Guild Sales Assistant.lnk"   
Set oLink = oWS.CreateShortcut(shortCut)   
oLink.TargetPath = CurrentDirectory & "\Guild Sales Assistant.exe"
oLink.Save
The more convenient you make it, the more likely people are to use it.

-d

Last edited by dopiate : 10/20/15 at 07:27 PM.
  Reply With Quote
10/21/15, 10:56 AM   #11
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
Sounds like a good idea. Right now it runs at Windowsboot.
  Reply With Quote
10/21/15, 07:10 PM   #12
dopiate
AddOn Author - Click to view addons
Join Date: Jun 2014
Posts: 142
Originally Posted by antihax View Post
Sounds like a good idea. Right now it runs at Windowsboot.

Ummm, I'm a bit confused.

You wrote an addon that runs at startup?

Exactly how do you let the user decide if they want to do this or not? Why would any user want to have an ESO addon running at startup?

Does it stay in memory? If so that is not a good idea, it's actually a terrible idea.

I'll wait for your reply before I say more, but your program is a guest on the users computer. It should be specific to the game, only work to enhance the game for them and should not be have any "auto start" capabilities whatsoever outside of ESO.

The end users of this game puts their trust in the developers because our only goal is to make their gaming experience better. If you wrote something that stays active all the time you are taking up resources (regardless of how small the amount) and you are no longer a guest but an intruder.

-d
  Reply With Quote
10/21/15, 09:46 PM   #13
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
Originally Posted by dopiate View Post
Ummm, I'm a bit confused.

You wrote an addon that runs at startup?

Exactly how do you let the user decide if they want to do this or not? Why would any user want to have an ESO addon running at startup?

Does it stay in memory? If so that is not a good idea, it's actually a terrible idea.
-d
It uses a very small amount of resources and is common for such applications to do this. It does no processing unless the Lua file is presented to it and the run at boot is an optional feature which is turned on or off via a menu option. Even Minion has this feature.

The back end is starting to shape up and the example front end is coming along. Based on our discussions prior though, I may end up offering an hourly or daily database dump and/or an API for getting data from the database, while leaving the option for connecting to the relay open.

Check: http://www.antihax.net use the search box on the left.
  Reply With Quote
11/03/15, 11:06 PM   #14
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
So bit of an update here, been adding functionality to the site and realized it is fully possible to insert data back into the game... so, done that. The latest version of the uploader will also pull weekly statistics from the site API and create a data file which is loaded by ESO. Also created some public helper functions so other developers can access this within lua.

Our major benefit here is that the more users we have, the more guild sales are observed and the more guild kiosks are seen. This is combined and fed back to addons so more than 5 guild stores histories make up the statistics.

What other data can we take out, merge and put back that would be useful and not provide unfair advantages?

Here is the Lua API available:
  • {sum, count, p5th, p25th, median, mode, p75th, p95th}= ESODR_StatisticsForWeek(itemID, uniqueID, week) - Get statistics for a specific week. Week format is YYYYWW as a string
  • YYYYWW = ESODR_CurrentWeek()
  • YYYYWW = ESODR_PreviousWeek()
  • uniqueID = ESODR_UniqueIDFromLink(itemLink)
  • itemID = ESODR_ItemIDFromLink(itemLink)

The site also has a few JSON API URLs available now. E.g.:
This is all still alpha and subject to change, but focusing more now on polishing, documenting, and adding more features where it makes sense.

Also don't forget to check out the progress on the main example site. antihax.net
  Reply With Quote
11/04/15, 05:33 AM   #15
merlight
AddOn Author - Click to view addons
Join Date: Jul 2014
Posts: 679
Why return week stamps as string? Numbers would be better IMO. Besides, you need to string.format("%04d%02d", 1970 + epocYears, epocWeeks) if you want correct strings for the first 9 weeks.
  Reply With Quote
11/04/15, 06:40 AM   #16
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
Thanks for the feedback. I was thinking the same last night and will change it later today. Was rather late.
  Reply With Quote
11/04/15, 06:56 PM   #17
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
This is done, it is now a number and expresses 1-9 correctly.
  Reply With Quote
03/05/16, 05:18 PM   #18
antihax
AddOn Author - Click to view addons
Join Date: Sep 2015
Posts: 23
I have uploaded a simple addon to access the data in game from the return feed: GuildStoreTools

This requires ESODataRelay to be installed and running.

I am looking at the possibility of an MM extension to provide basic statistics, and possibly replace the graphing if i get time aside from the relay code.
  Reply With Quote

ESOUI » AddOns » Alpha/Beta AddOns » Introducing: ESO Market Data Relay

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off