The UI of my Xbox One got very sluggish and it was a pain to use.  Games were running fine but moving from tile to tile and menu to menu was tremendously slow.  A quick search on the InterTubes led me to this solution:

Clear the Blu-ray cache
Select Settings/All Settings/Disc & Blu-ray/Blue-ray/Persistent Storage and select Clear persistent storage.  Note that the console won't give you any kind of notification after you've done that.

Network Settings
Select Settings/All Settings/Network/Network Settings/Advanced Settings/Alternate MAC Address and select Clear.  The console will warn you that this will require a startup.

Clearing these two settings solved the problem on my Xbox.