If you just use a direct event callback to e.g. EVENT_CHAT_MESSAGE_CHANNEL you create a function which will be executed as the event fires ingame. If you return true or false in there this won't block other addon's event callback functions on the same event, nor the vanilla UI callback functions.
So returning true or false won't change the call behaviour.
If you PreHook (using ZO_PreHook -> will register a function that will be called before the "hooked" function) a function you are able to return with true, which will prevent the original function code to run afterwards.
In your example the code line would need to be this then:
Lua Code:
if IsInGroupInstance() then
--return true!! to prevent the original function you have PreHooked to runa fterwards. Return false or nil to let the orig. function run normally afterwards
return true
end
Read more about PreHooks and PostHooks at the Wiki:
https://wiki.esoui.com/How_to_run_yo...eHook/PostHook)
For your chat related use-case check this addon for an example e.g.:
https://www.esoui.com/downloads/info2862-AdBlock.html
It's not the best way to do but for the moment it's using a ZO_preHook on the CHAT_ROUTER function FormatAndAddChatMessage.
This might break other addons which rely on this function, e.g. building a message history, as the messages are stripepd before the other addon might notice it.
But you can use your txt file of your addon and add a line like e.g. ## OptionalDependsOn: pChat rChat to assure these other addons run before your addon and register theire prehook etc before your's then (asuming they do this in EVENT_ADD_ON_LOADED! If they do it later in e.g. EVENT_PLAYER_ACTIVATED, as the chat is special and msot chat related stuff is first ready at this event, it might not change a thing and your PreHook could still break them).
Lua Code:
ZO_PreHook(CHAT_ROUTER, "FormatAndAddChatMessage", AB.spamFilter)
Within AB.spamFilter it returns true then to prevent the execution of the vanilla UI chat message function "FormatAndAddChatMessage", if a spam message was detected in the function AB.spamFilter
Currently the CHAT_ROUTEr only allows 1 chat mesasge handling and formatting function. This makes it hard to add multiple filter functions etc. You can only prehook it, or all addons would need to use the same library which would handle the message formatters internally and call all registered addon's filter/formatting funtions then after another + provide the results.
LibChatmessage was created to do this but is currently not working for that usecase as ZOs had changed the chat stuff some patches ago.
So we need to wait for LibChatMessage2 e.g.
