The UI of my Xbox One got very sluggish and it was a pain to use. Games were running fine but moving from tile to tile and menu to menu was tremendously slow. A quick search on the InterTubes led me to this solution:
Clear the Blu-ray cache
Select Settings/All Settings/Disc & Blu-ray/Blue-ray/Persistent Storage and select Clear persistent storage. Note that the console won't give you any kind of notification after you've done that.
Select Settings/All Settings/Network/Network Settings/Advanced Settings/Alternate MAC Address and select Clear. The console will warn you that this will require a startup.
Clearing these two settings solved the problem on my Xbox.