I'm not sure if I'm in the right forum but...
 

I'm looking to create a mod that:

(all of these are if possible)

1- dismount if you take a certain % damage
2- target closest

I'm an old WoW add-on creature but in the beginning you could change thing like if you got hit with damage you went into "z" attack mod - or auto-pot based on HP. Eventually most of that was impossible after a few patches.

is this quest futile or can this be done in ESO

Addons are not allowed to ToggleMount, Fire abilities, Fire Attacks or even consume potions.
I am not even sure we can set the marked target, but that is the one part might just be possible.

I am pretty sure a mod with those abilities/functions would be agaisnt the TOS.

In general, taking actions, especially combat actions, cannot be automated by the addon API on purpose.