Thread Tools Display Modes
09/05/18, 09:39 PM   #1
Phuein
 
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 132
[notabug] "invalid string value (was your string larger than 2000 characters?)"

A user of my addon Notebook 2018 just commented that she saved a page, and after checking the SavedVars file for it she saw:

["text"] = nil, -- invalid string value (was your string larger than 2000 characters?)
Now, that comment didn't come from my addon, nor did she get an error. I had put the characters' max at 2048, not knowing this was an issue. So maybe we can clarify this...

Will the game not save any one string at over 2,000 chars? Without error?

Why 2,000? Seems arbitrary. I couldn't find info on this issue anywhere. Is this new?

Thanks for helping out. I'd appreciate a game dev' that can post an official response about this, to avoid rumors.

EDIT: Also brought up was the issue of counting characters when using unicodes wider than 1 char, and how they count towards this. I'd love a response to that, too.

Last edited by Phuein : 09/06/18 at 03:50 PM.
 
09/06/18, 07:15 AM   #2
Rhyono
AddOn Author - Click to view addons
Join Date: Sep 2016
Posts: 659
Any string over 2k is automatically changed to that warning in the saved vars. So if you want to save larger than that you need to split it and save it as a table of strings.
 
09/06/18, 08:20 AM   #3
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by Phuein View Post
Will the game not save any one string at over 2,000 chars? Without error?
Technically, that is the error. It doesn't show up in a dialog because currently, savedvars are only written when you logout or reload the UI. Both of which, the game is in the process of unloading the "Ingame" UI and associated Lua environment.

Originally Posted by Phuein View Post
Why 2,000? Seems arbitrary.
It is completely arbitrary, but "Why 2000?" isn't really the right question to ask. More importantly is, "Why does this limit exist?" There can be many reasons for that ranging from keeping performance in check to how the C code handles writing to the files. Long story short, buffer overflows can be quite destructive.

Originally Posted by Phuein View Post
I couldn't find info on this issue anywhere. Is this new?
The length of time an issue has been in effect is not the only factor for its exposure. Very few addons have a need to write 2000+ characters into a string, so this is a very niche problem.
 
09/06/18, 02:55 PM   #4
Phuein
 
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 132
I would expect an error in-game, or at least a warning, saying that "a string currently has over 2,000 characters and will note save in savedvars!" sorta thing.

And a notice about this in the wiki would have been nice, too.

Still waiting on official game dev' to put down the gavel on this one. For example, will this change? No guesses please.
 
09/06/18, 04:54 PM   #5
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 53
Originally Posted by Phuein View Post
I would expect an error in-game, or at least a warning, saying that "a string currently has over 2,000 characters and will note save in savedvars!" sorta thing.
The string is replaced with text that is pretty explicitly clear about what happened. Not sure, given that it is interacted with only when shutting down, that you could do much better.

Originally Posted by Phuein View Post
And a notice about this in the wiki would have been nice, too.
Please add one.
 
09/07/18, 09:20 AM   #6
ZOS_DanBatson
ZOS Staff!
 
ZOS_DanBatson's Avatar
Yes this person is from ZeniMax!
Join Date: Jul 2015
Posts: 171
Yes that limit is set by us, no the limit is not new, and no the limit is not likely to change. We need to define the limit of the buffer somewhere, and as was already mentioned if you need more characters you can make a table.
 
09/07/18, 02:49 PM   #7
Phuein
 
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 132
Originally Posted by ZOS_DanBatson View Post
Yes that limit is set by us, no the limit is not new, and no the limit is not likely to change. We need to define the limit of the buffer somewhere, and as was already mentioned if you need more characters you can make a table.
Thanks.

How do you do the character count? Referring to unicode wide-characters. Do they count as 1 character or as many bytes as they take? If you can point me to the actual function you use in-game that'd work, too.
 
09/07/18, 03:33 PM   #8
Rhyono
AddOn Author - Click to view addons
Join Date: Sep 2016
Posts: 659
From my tests (about a year ago), ZOS appears to do it by character count, not byte count.
 
09/07/18, 04:07 PM   #9
Rhynchelma
Premium Member
Premium Member
Join Date: Mar 2014
Posts: 43
Librarian stores book contents in this manner. It has some code you may find useful, maybe.
 
09/07/18, 07:13 PM   #10
SDPhantom
 
SDPhantom's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2014
Posts: 47
Originally Posted by Phuein View Post
How do you do the character count? Referring to unicode wide-characters. Do they count as 1 character or as many bytes as they take? If you can point me to the actual function you use in-game that'd work, too.
Considering we're talking about buffers, they only operate at the byte-level. A simple length operation (#string) will give you this number.
 
09/08/18, 12:09 PM   #11
SlippyCheeze
AddOn Author - Click to view addons
Join Date: Jul 2018
Posts: 53
Originally Posted by SDPhantom View Post
Considering we're talking about buffers, they only operate at the byte-level. A simple length operation (#string) will give you this number.
Just checked, because I didn't know, and they (a) use a two-byte encoding for "áéúíó", so utf8, and (b) #"áéúíó" is 10, the byte count.
 
09/08/18, 11:29 PM   #12
Phuein
 
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 132
Thinking it over, I do use SetMaxInputChars(1984) which is a game function, so it's probably okay. (Untested.)

Notices topic changed. This may not be a bug then, but it's definitely not good. Just letting people save and lose their data.

Great info out here guys. Thanks for the #length tip, very useful.

Code:
a = "∀∁∂∃∄"
print(#a)
15
I should test it with my actual control "ZO_DefaultEditMultiLineForBackdrop", to see how SetMaxInputChars() behaves. Hopefully the official dev' will clarify this issue from game code, instead of us testing randomly, only to lose data later.
 
09/09/18, 01:20 AM   #13
Phuein
 
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 132
Just tested it thoroughly.

Apparently SetMaxInputChars() does NOT work with counting unicode wide characters, as #string does. I propose that this is a bug that should be fixed in the game.

I implemented my own counter and tested it. Everything seems okay. 2,000 characters does NOT save, but 1,999 does. I limited to 1,900 just in case.

Code:
local textLen = #text
if textLen > 1900 then
	NBUI.NB1SavePage_Button:SetHidden(true)
end
NBUI.NBUI_NB1RightPage_CharacterCounter:SetText(textLen .. ' / ' .. 1900)
Btw, max displayed (at all, in element, on screen) characters (non-unicode) count is about 3,600 - more will vanish! Bug? Maybe. I implemented a display limit with SetMaxInputChars(3000) to play it safe.
 
09/09/18, 06:27 AM   #14
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,566
I just did my own tests and the string functions don't count unicode characters as others have suggested.

I tested it with the string "これは日本語の文字列です" with string.len, the # operator, two methods proposed on the lua-users wiki and the utf8len function in this gist.
Lua Code:
  1. local input = "これは日本語の文字列です"
  2. local count1 = input:len()
  3. local count2 = #input
  4. local _, count3 = string.gsub(input, "[^\128-\193]","")
  5. local count4 = 0
  6. for uchar in string.gmatch(input, "([%z\1-\127\194-\244][\128-\191]*)") do
  7.     count4 = count4 + 1
  8. end
  9. local count5 = utf8len(input)
  10. d(count1, count2, count3, count4, count5)
The output is 36, 36, 36, 0, 12, meaning that the string functions use byte count, gsub and gmatch do something unexpected and the only way right now is to use some library to count how many characters you have left.

As for your original problem, you can always use a function that manages saving and reading the saved string and split it up dynamically. Here is an example that I just slapped together:

Lua Code:
  1. local MAX_SAVE_DATA_LENGTH = 2000 -- buffer length used by ZOS
  2. local function WriteLongString(obj, key, value)
  3.     local output = value
  4.     local byteLength = #value
  5.     if(byteLength > MAX_SAVE_DATA_LENGTH) then
  6.         output = {}
  7.         local startPos = 1
  8.         local endPos = startPos + MAX_SAVE_DATA_LENGTH - 1
  9.         while startPos <= byteLength do
  10.             output[#output + 1] = string.sub(value, startPos, endPos)
  11.             startPos = endPos + 1
  12.             endPos = startPos + MAX_SAVE_DATA_LENGTH - 1
  13.         end
  14.     end
  15.     obj[key] = output
  16. end
  17.  
  18. local function ReadLongString(obj, key)
  19.     local value = obj[key]
  20.     if(type(value) == "table") then
  21.         return table.concat(value, "")
  22.     end
  23.     return value
  24. end
 
09/09/18, 01:05 PM   #15
Phuein
 
Phuein's Avatar
AddOn Author - Click to view addons
Join Date: Mar 2018
Posts: 132
Your MAX_SAVE_DATA_LENGTH should be = 1999, as I mentioned above at 2000 it will not save.

I don't think this should be an addon-side issue. Silent save-fail is terrible. The game length functions don't work with unicode correctly.

This is a bug in the sense of unexpected arbitrary behavior.
 
09/09/18, 01:49 PM   #16
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,566
It was explained why it happens, that is expected behavior and that there are ways to easily work around this necessary technical limitation -> not a bug. That you cannot use string functions in Lua to count multi-byte characters is also common knowledge and not a bug (google lua + unicode or just search the esoui forum).

The save also doesn't fail silently. It prints a comment into the save data why it failed. This is actually just a corner case that you as a developer should have tested when you set the limit for the input field and saved it to disk.

If you want to insist on them changing the buffer limit to something higher (which won't resolve anything), or have some error message ingame (which you can easily implement yourself) or want functions to count unicode characters (which you can get from the link I posted), please create an appropriate request on the wish list. Closing this thread now.
 

ESOUI » Developer Discussions » Bug Reports » [notabug] "invalid string value (was your string larger than 2000 characters?)"

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off