yes you have to browse your bag each time, would be the same with the tooltips though I just got another idea:

you can do your own "popuptooltip" like this :
MyItemTooltip = WINDOW_MANAGER:CreateControlFromVirtual("MyItemTooltipName", parent , "ZO_ItemIconTooltip")

(if SetLink doesn't work with this try "ZO_PopupTooltip" in 3rd arg)

then you can anchor it exactly as you want with no risk to make a mistake with the original one and use a SetLink to generate the raw itemtooltip of your baits. it'll have the image and the description but no hooks from other addons. advantage of this : the number of possed items is already on it in the corner : bag and bank with no extra processing

the display would a bit overkill for just a number, but at least not the background processing. Maybe just stuck the popup in a corner and use the control to display whatever extra fancy info you need.

good luck!