Yes thats possible and pretty easy too.

Each addon component has its own File. E.g. the MiniMap uses minimap.lua and xml/minimap.xml.
There is a lui.txt that defines which files will be loaded by ESO.

To remove a component, open your lui.txt and remove its file descriptions there. Reload your UI and the component wont get loaded and you're fine

Example:
To remove the MiniMap and the InfoPanel, just remove the blue marked lines from your lui.txt


## APIVersion: 100003
## Title: |cE12000L|rUI - By |cE12000LoPony|r
## Author: LoPony
## Description: MiniMap, InfoPanel, CombatEvents and Buffs AddOn
## SavedVariables: LUI_SavedVariables

LUI.lua

config/bindings.lua
config/bindings.xml
config/config.lua
config/fonts.xml

combatinfo.lua
unitinfo.lua
infopanel.lua
minimap.lua

xml/combatinfo.xml
xml/unitinfo.xml
xml/infopanel.xml
xml/minimap.xml