Introducing: ESO Market Data Relay

antihax · 10/17/15, 10:37 PM

Hello Folks!
Alpha download available here.
I have been working the last few weeks on a replication of EMDR (Eve Market Data Relay) for ESO. I am currently around alpha on the relay and looking for feedback. My focus now is verifying the data formats and working on an example site (http://www.antihax.net) on what is possible using the data. In a few days I hope to have the addon and uploader packaged for alpha testing.

What Is This?!
Short version; the data relay allows users to automatically upload market data as they observe it in game. Developers can connect to the relay and receive data in real time as the users change zone, /reloadui, logout, or quit game.

What's the point?
To allow third party development of market related apps and sites to flourish through shared interest which in turn creates useful tools and resources for players such as:

finding complete itemsets for sale on the market across zones,
finding the cheapest items for sale,
data for users without access to large trade guilds,
finding the best items to trade telvar/AP for gold,
understanding market volumes for specific items in various areas, etc.
finding the best gear for your build without running around.
third party REST API for google sheets or other sites to further help developers (i.e. http://eve-marketdata.com/developers/index.php or https://eve-central.com/home/develop.html)
etc, etc.

What do you need?
To start, i will need testers and other developer's ideas and thoughts to build upon work already done.

I'm Interested, what do i do?
Contact me in game @croakroach or reply with constructive feedback.

Checkout the code below

example site code - Most of the work around the relay is in the datarelayconsumer. Written in Go but zmq is available for most languages.
Addon and Uploader Lua and C#.
Master/Relay code - Down the road we will need more relay operators to provide redundancy and regionalization. Written in Go.

FAQ

What about stale data?
We can work out from other uploads and age of items about how stale they are. Also we know the expiration of items for sale. More people uploading, fresher data.
Why would people contribute?
Because it benefits them to. I.e. how many addon developers have fielded questions about features that this will provide?
Wont this destroy the market and drive down prices?
Not really, having been involved with markets in many other games i see the same thing over and over: people pay extra to save time, make mistakes, are misinformed, or are just plain lazy. Those who search out the deals will still find them.
What about duplicates?
The developer receiving the data is responsible for handling duplicates; the uploader will intentionally send duplicate data between sessions by design. This is so if anyone connects to the relay late, they will not miss much data and will have a complete set within a short amount of time. This also provides a method of detecting freshness of data, allowing sold items to become stale.

merlight · 10/18/15, 08:38 AM

Originally Posted by antihax

[*] What about duplicates?
The developer receiving the data is responsible for handling duplicates; the uploader will intentionally send duplicate data. So far I have found that rounding some of the timestamps to 10 help resolve the off by one or two issues and have other identifiers to handle most duplicates. Help here to simplify this would be appreciated.[/list]

If I were to implement this -- and I was pondering that idea, but wasn't enough into trading to take on such an enormous task

certainly not since two trading guilds I was in died overnight with ESO update 3 or 4 -- I'd start with a request for ZOS to add SaleId to every store-related API (search, listing, history). The client must have it, it's just a matter of passing it to add-ons. Then there would be no issues with duplicates.

Even replacing the "timeRemaining", "secsSinceEvent" and similar parameters with solid timestamps would simplify detection of duplicates.

antihax · 10/18/15, 02:30 PM

Originally Posted by merlight

If I were to implement this -- and I was pondering that idea, but wasn't enough into trading to take on such an enormous task

certainly not since two trading guilds I was in died overnight with ESO update 3 or 4 -- I'd start with a request for ZOS to add SaleId to every store-related API (search, listing, history). The client must have it, it's just a matter of passing it to add-ons. Then there would be no issues with duplicates.

Even replacing the "timeRemaining", "secsSinceEvent" and similar parameters with solid timestamps would simplify detection of duplicates.

I would agree this is a way to go, the current data is a bit lacking for tools like this, but I am not going to let that hold up work. If activity increases, it will prove the necessity for such changes on the API.

sirinsidiator · 10/19/15, 02:52 AM

I think your stance that the recipient is responsible for handling duplicates might be a huge turnoff for many developers, as this is one of the most difficult problems in here and if everyone has to solve this problem for themselves, you will also get a lot of duplicate solutions.

Rounding the timestamp to 10 is a bad idea, because you will a) loose precision and b) the timestamp varies by a few seconds between session. In the past I also started writing something similar to get store data out of the game and entries that had a stamp that ended on a 5 in one session had one ending on 4 or 6 in another which would then round differently.
I had an idea of using the order of the entries to identify duplicates, but never implemented it.
Search for the entry by timestamp and then see if the previous and next entries have the correct item ids. If there is one wrong, it would be a duplicate.

There is also the problem that you will have to handle manipulated uploads in some way. When you allow anyone to upload data, anyone can manipulate your data freely in any way they want by sending in fabricated entries.

Developers can connect to the relay and receive data in real time as the users change zone

I am not quite sure how you manage to get data on zone changes? The saved vars are only written out when you reload ui, logout or quit to desktop.
Getting data into the game is even harder and in my experience most ESO players won't use any data source outside of the game while playing. That's why I gave up my own project which did exactly what yours is doing and started making AwesomeGuildStore instead.

antihax · 10/19/15, 07:14 AM

Good feedback!

I did see yesterday that zone changes do not count. Will have to find some trick to get this to change.
I think I misworded about the duplicates. The system is intended to send duplicates, how do you know data is valid in an anonymous system? How do you calculate probability that an item is still listed? What if someone connects after an item was observed? This is what sending duplicated data is for.

I agree a better solution is to have a unique Id, but we do not. I'm going to see how well rounding to 10 works. Not to worried about losing 10 seconds. What other options are available cross client?

dopiate · 10/19/15, 10:01 AM

Originally Posted by antihax

I agree a better solution is to have a unique Id, but we do not. I'm going to see how well rounding to 10 works. Not to worried about losing 10 seconds. What other options are available cross client?

where are you getting your data?

in what format is it uploaded?

You can ask for quite a bit about a sale and put all that info together (with a separator so you can split it later) and get a fairly useful ID

in my previous post I said I have two routines to remove dupes

I parse a specific savedvariabls file and each sale is put into this format:

Code:

Amazing Deals of Tamriel,Argentum,15,1437713676,150,@Medusanomi,@dOpiate,1:0:1:0

taken as a single string in an array that is fairly unique (yours would be even better since you are rounding the time)

That is the data I need to make my program work.

But if the user deleted their savedvariables and this time it comes back with a slightly different time stamp then I drop that from the string and check for

Code:

Amazing Deals of Tamriel,Argentum,15,150,@Medusanomi,@dOpiate,1:0:1:0

now that's not perfect but what are the chances of seeing that exact string twice? you have guild,item,ppu,total sale amount,seller,buyer,"details"

the "details" are created by this code

Lua Code:

local levelReq = GetItemLinkRequiredLevel(itemLink)
  local vetReq = GetItemLinkRequiredVeteranRank(itemLink)
  local itemQuality = GetItemLinkQuality(itemLink)  
  local itemTrait = GetItemLinkTraitInfo(itemLink)
  return levelReq .. ":" .. vetReq .. ":" .. itemQuality .. ":" .. itemTrait

so you can get a fairly reliable amount of data on a sale to make into some sort of identifier from.

They even added lots of new info into the itemLink with the recent patch.

-d

antihax · 11/03/15, 11:06 PM

So bit of an update here, been adding functionality to the site and realized it is fully possible to insert data back into the game... so, done that. The latest version of the uploader will also pull weekly statistics from the site API and create a data file which is loaded by ESO. Also created some public helper functions so other developers can access this within lua.

Our major benefit here is that the more users we have, the more guild sales are observed and the more guild kiosks are seen. This is combined and fed back to addons so more than 5 guild stores histories make up the statistics.

What other data can we take out, merge and put back that would be useful and not provide unfair advantages?

Here is the Lua API available:

{sum, count, p5th, p25th, median, mode, p75th, p95th}= ESODR_StatisticsForWeek(itemID, uniqueID, week) - Get statistics for a specific week. Week format is YYYYWW as a string
YYYYWW = ESODR_CurrentWeek()
YYYYWW = ESODR_PreviousWeek()
uniqueID = ESODR_UniqueIDFromLink(itemLink)
itemID = ESODR_ItemIDFromLink(itemLink)

The site also has a few JSON API URLs available now. E.g.:

This is all still alpha and subject to change, but focusing more now on polishing, documenting, and adding more features where it makes sense.

Also don't forget to check out the progress on the main example site. antihax.net

merlight · 11/04/15, 05:33 AM

Why return week stamps as string? Numbers would be better IMO. Besides, you need to string.format("%04d%02d", 1970 + epocYears, epocWeeks) if you want correct strings for the first 9 weeks.

antihax · 11/04/15, 06:40 AM

Thanks for the feedback. I was thinking the same last night and will change it later today. Was rather late.

antihax · 11/04/15, 06:56 PM

This is done, it is now a number and expresses 1-9 correctly.

antihax · 03/05/16, 05:18 PM

I have uploaded a simple addon to access the data in game from the return feed: GuildStoreTools

This requires ESODataRelay to be installed and running.

I am looking at the possibility of an MM extension to provide basic statistics, and possibly replace the graphing if i get time aside from the relay code.