I've heard a lot of people complaining about swipe functionality of the touchpad on the new Apple TV remote. The first time we tried ours at home the remote was out of sync and we had to reboot the device to get the swipe functionality to work properly.
(In fact, we used the old remote to do all the setup.)
Thing is, I don't think we would have known that something was wrong if I hadn't already played with a dev unit. My wife just commented that swiping wasn't very responsive or intuitive and said that the new remote would take some getting used to.
The other thing I'd mention that I think a lot of people are experiencing is accidental triggering of the touchpad. I do this thing all the time where I put the remote under my leg to hide it from our 18-month-old son. Now when I try to do that I'm accidentally pausing the show or fast-forwarding, or otherwise interrupting our viewing experience.