ESOUI

ESOUI (https://www.esoui.com/forums/index.php)
-   Bug Reports (https://www.esoui.com/forums/forumdisplay.php?f=187)
-   -   [notabug] "invalid string value (was your string larger than 2000 characters?)" (https://www.esoui.com/forums/showthread.php?t=8051)

Phuein 09/05/18 09:39 PM

[notabug] "invalid string value (was your string larger than 2000 characters?)"
 
A user of my addon Notebook 2018 just commented that she saved a page, and after checking the SavedVars file for it she saw:

Quote:

["text"] = nil, -- invalid string value (was your string larger than 2000 characters?)
Now, that comment didn't come from my addon, nor did she get an error. I had put the characters' max at 2048, not knowing this was an issue. So maybe we can clarify this...

Will the game not save any one string at over 2,000 chars? Without error?

Why 2,000? Seems arbitrary. I couldn't find info on this issue anywhere. Is this new?

Thanks for helping out. I'd appreciate a game dev' that can post an official response about this, to avoid rumors.

EDIT: Also brought up was the issue of counting characters when using unicodes wider than 1 char, and how they count towards this. I'd love a response to that, too.

Rhyono 09/06/18 07:15 AM

Any string over 2k is automatically changed to that warning in the saved vars. So if you want to save larger than that you need to split it and save it as a table of strings.

SDPhantom 09/06/18 08:20 AM

Quote:

Originally Posted by Phuein (Post 35973)
Will the game not save any one string at over 2,000 chars? Without error?

Technically, that is the error. It doesn't show up in a dialog because currently, savedvars are only written when you logout or reload the UI. Both of which, the game is in the process of unloading the "Ingame" UI and associated Lua environment.

Quote:

Originally Posted by Phuein (Post 35973)
Why 2,000? Seems arbitrary.

It is completely arbitrary, but "Why 2000?" isn't really the right question to ask. More importantly is, "Why does this limit exist?" There can be many reasons for that ranging from keeping performance in check to how the C code handles writing to the files. Long story short, buffer overflows can be quite destructive.

Quote:

Originally Posted by Phuein (Post 35973)
I couldn't find info on this issue anywhere. Is this new?

The length of time an issue has been in effect is not the only factor for its exposure. Very few addons have a need to write 2000+ characters into a string, so this is a very niche problem.

Phuein 09/06/18 02:55 PM

I would expect an error in-game, or at least a warning, saying that "a string currently has over 2,000 characters and will note save in savedvars!" sorta thing.

And a notice about this in the wiki would have been nice, too.

Still waiting on official game dev' to put down the gavel on this one. For example, will this change? No guesses please.

SlippyCheeze 09/06/18 04:54 PM

Quote:

Originally Posted by Phuein (Post 35985)
I would expect an error in-game, or at least a warning, saying that "a string currently has over 2,000 characters and will note save in savedvars!" sorta thing.

The string is replaced with text that is pretty explicitly clear about what happened. Not sure, given that it is interacted with only when shutting down, that you could do much better.

Quote:

Originally Posted by Phuein (Post 35985)
And a notice about this in the wiki would have been nice, too.

Please add one. :)

ZOS_DanBatson 09/07/18 09:20 AM

Yes that limit is set by us, no the limit is not new, and no the limit is not likely to change. We need to define the limit of the buffer somewhere, and as was already mentioned if you need more characters you can make a table.

Phuein 09/07/18 02:49 PM

Quote:

Originally Posted by ZOS_DanBatson (Post 35993)
Yes that limit is set by us, no the limit is not new, and no the limit is not likely to change. We need to define the limit of the buffer somewhere, and as was already mentioned if you need more characters you can make a table.

Thanks.

How do you do the character count? Referring to unicode wide-characters. Do they count as 1 character or as many bytes as they take? If you can point me to the actual function you use in-game that'd work, too.

Rhyono 09/07/18 03:33 PM

From my tests (about a year ago), ZOS appears to do it by character count, not byte count.

Rhynchelma 09/07/18 04:07 PM

Librarian stores book contents in this manner. It has some code you may find useful, maybe.

SDPhantom 09/07/18 07:13 PM

Quote:

Originally Posted by Phuein (Post 35994)
How do you do the character count? Referring to unicode wide-characters. Do they count as 1 character or as many bytes as they take? If you can point me to the actual function you use in-game that'd work, too.

Considering we're talking about buffers, they only operate at the byte-level. A simple length operation (#string) will give you this number.

SlippyCheeze 09/08/18 12:09 PM

Quote:

Originally Posted by SDPhantom (Post 35998)
Considering we're talking about buffers, they only operate at the byte-level. A simple length operation (#string) will give you this number.

Just checked, because I didn't know, and they (a) use a two-byte encoding for "áéúíó", so utf8, and (b) #"áéúíó" is 10, the byte count.

Phuein 09/08/18 11:29 PM

Thinking it over, I do use SetMaxInputChars(1984) which is a game function, so it's probably okay. (Untested.)

Notices topic changed. This may not be a bug then, but it's definitely not good. Just letting people save and lose their data.

Great info out here guys. Thanks for the #length tip, very useful.

Code:

a = "∀∁∂∃∄"
print(#a)
15

I should test it with my actual control "ZO_DefaultEditMultiLineForBackdrop", to see how SetMaxInputChars() behaves. Hopefully the official dev' will clarify this issue from game code, instead of us testing randomly, only to lose data later.

Phuein 09/09/18 01:20 AM

Just tested it thoroughly.

Apparently SetMaxInputChars() does NOT work with counting unicode wide characters, as #string does. I propose that this is a bug that should be fixed in the game. :)

I implemented my own counter and tested it. Everything seems okay. 2,000 characters does NOT save, but 1,999 does. I limited to 1,900 just in case.

Code:

local textLen = #text
if textLen > 1900 then
        NBUI.NB1SavePage_Button:SetHidden(true)
end
NBUI.NBUI_NB1RightPage_CharacterCounter:SetText(textLen .. ' / ' .. 1900)

Btw, max displayed (at all, in element, on screen) characters (non-unicode) count is about 3,600 - more will vanish! Bug? Maybe. I implemented a display limit with SetMaxInputChars(3000) to play it safe.

sirinsidiator 09/09/18 06:27 AM

I just did my own tests and the string functions don't count unicode characters as others have suggested.

I tested it with the string "これは日本語の文字列です" with string.len, the # operator, two methods proposed on the lua-users wiki and the utf8len function in this gist.
Lua Code:
  1. local input = "これは日本語の文字列です"
  2. local count1 = input:len()
  3. local count2 = #input
  4. local _, count3 = string.gsub(input, "[^\128-\193]","")
  5. local count4 = 0
  6. for uchar in string.gmatch(input, "([%z\1-\127\194-\244][\128-\191]*)") do
  7.     count4 = count4 + 1
  8. end
  9. local count5 = utf8len(input)
  10. d(count1, count2, count3, count4, count5)
The output is 36, 36, 36, 0, 12, meaning that the string functions use byte count, gsub and gmatch do something unexpected and the only way right now is to use some library to count how many characters you have left.

As for your original problem, you can always use a function that manages saving and reading the saved string and split it up dynamically. Here is an example that I just slapped together:

Lua Code:
  1. local MAX_SAVE_DATA_LENGTH = 2000 -- buffer length used by ZOS
  2. local function WriteLongString(obj, key, value)
  3.     local output = value
  4.     local byteLength = #value
  5.     if(byteLength > MAX_SAVE_DATA_LENGTH) then
  6.         output = {}
  7.         local startPos = 1
  8.         local endPos = startPos + MAX_SAVE_DATA_LENGTH - 1
  9.         while startPos <= byteLength do
  10.             output[#output + 1] = string.sub(value, startPos, endPos)
  11.             startPos = endPos + 1
  12.             endPos = startPos + MAX_SAVE_DATA_LENGTH - 1
  13.         end
  14.     end
  15.     obj[key] = output
  16. end
  17.  
  18. local function ReadLongString(obj, key)
  19.     local value = obj[key]
  20.     if(type(value) == "table") then
  21.         return table.concat(value, "")
  22.     end
  23.     return value
  24. end

Phuein 09/09/18 01:05 PM

Your MAX_SAVE_DATA_LENGTH should be = 1999, as I mentioned above at 2000 it will not save.

I don't think this should be an addon-side issue. Silent save-fail is terrible. The game length functions don't work with unicode correctly.

This is a bug in the sense of unexpected arbitrary behavior.

sirinsidiator 09/09/18 01:49 PM

It was explained why it happens, that is expected behavior and that there are ways to easily work around this necessary technical limitation -> not a bug. That you cannot use string functions in Lua to count multi-byte characters is also common knowledge and not a bug (google lua + unicode or just search the esoui forum).

The save also doesn't fail silently. It prints a comment into the save data why it failed. This is actually just a corner case that you as a developer should have tested when you set the limit for the input field and saved it to disk.

If you want to insist on them changing the buffer limit to something higher (which won't resolve anything), or have some error message ingame (which you can easily implement yourself) or want functions to count unicode characters (which you can get from the link I posted), please create an appropriate request on the wish list. Closing this thread now.


All times are GMT -6. The time now is 01:09 AM.

vBulletin © 2024, Jelsoft Enterprises Ltd
© 2014 - 2022 MMOUI