Thread Tools Display Modes
04/17/20, 08:34 AM   #1
Scootworks
 
Scootworks's Avatar
AddOn Author - Click to view addons
Join Date: Nov 2014
Posts: 312
[open] utf8.char() game crash

UTF8 functions dosn't work properly



utf8.codepoint
-> works "/script d(utf8.codepoint("Г"))" == 1043

utf8.charpattern
-> doesn't work properly "/script local str = "TЖ" for w in str:gmatch(utf8.charpattern) do d(w) end" == "T" and ignores the other char

utf8.offset
-> works "/script d(utf8.offset("Test", 1, 4))" == 4

utf8.len
-> works (or using ZoUTF8StringLength())

utf8.codes
-> doesn't work, returns: Attempt to access a private function 'anonymous' from insecure code. The callstack became untrusted 1 stack frame(s) from the top.

utf8.char
-> "/script d(utf8.char(1043))" CRASH



Ingame ticket# 200417-003814

Last edited by Scootworks : 04/17/20 at 09:20 AM. Reason: added examples
  Reply With Quote
04/19/20, 11:44 AM   #2
Sharlikran
 
Sharlikran's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 626
I thought of something and then felt I was way off base. However, does anyone know what code page the game runs in? Because UTF8 is not a code page.

Also what language is that from? What are you trying to represent? Maybe just find the correct value rather then having the game tell you. Because Bethesda does not have crash logs of any kind. I feel maybe ESOUI doesn't either, sadly. The game could be crashing because the value is out of range.

Another reason I ask is because in UTF8 1043 is a MYANMAR DIGIT THREE

Last edited by Sharlikran : 04/19/20 at 11:51 AM.
  Reply With Quote
04/19/20, 01:00 PM   #3
sirinsidiator
 
sirinsidiator's Avatar
AddOn Author - Click to view addons
Join Date: Apr 2014
Posts: 1,566
It looks like the implementation of utf8.char uses string.format with %U internally, which is not implemented in Lua5.1. When calling "string.format("%U", 1)" I get a proper error: "Illegal %U at the end of the format", but when it is called via utf8.char it will just crash for whatever reason.

And for utf8.codes it seems the secure context functionality doesn't recognize that iter_aux is a public function.

Last edited by sirinsidiator : 04/19/20 at 01:05 PM.
  Reply With Quote
04/20/20, 08:37 AM   #4
Drummerx04
AddOn Author - Click to view addons
Join Date: Sep 2017
Posts: 54
Oh yeah, I've ran into plenty of weird behaviors when dealing with utf8 in ESO. I haven't messed with it for about a year, but when trying to write a match expression for international characters in profile names, I tried something like this.
Lua Code:
  1. for match in testString:gmatch("@[a-zA-Z0-9_\128-\255]+")
The string library in lua/ESO only works on single byte characters, so what this actually does is assumes that any byte over \127 is part of a valid utf8 multibyte character, and for my purposes that would have been perfectly fine.

The issue is that it DIDN'T WORK. The \128-\255 expression was ignored (but something like \97-\123 was matched correctly as a-z).

Even more interestingly I came up with an alternative approach which DOES work
Lua Code:
  1. for match in testString:gmatch("(@[^\1-\47\58-\64\91-\96]+)") do
Basically, just create a character set you DON'T want and then invert it with '^'

I don't remember the utf8 module being in ESO, maybe that was added after I left, but if they didn't fix their regex parser ignoring any character value over \127, then you'd have more problems anyway.

Last edited by Drummerx04 : 04/20/20 at 11:25 AM.
  Reply With Quote

ESOUI » Developer Discussions » Bug Reports » [open] utf8.char() game crash

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off