What the service manual doesn't make clear is that the gear shifter is removed by pulling straight up with a lot of force. For obvious reasons, don't have your face over the shifter as you pull up. After you remove the shifter and its boot, you pry up the trim below it. You may be able to just pry up the trim enough to clean around it without having to remove the shifter. In the following video of an Azera in a different market with a slightly different console arrangement, you can see them pulling up the shifter and the boot and trim come up with it, probably from the upward force of the shifter boot being removed. Keep this video bookmarked in case you ever have to remove the center stack components.
Here... these instructions ultimately tell you how to get the center stack out, but they start with the console. I just followed them a few weeks back when I had to replace my vents. They are perfect. FYI... the secret of removing the shift knob is pulling up hard!