It does matter on the router table... and this is where i'd perform this operation.
What you want is so that the part of the bit that's cutting should be heading toward the fence - that way the wood is not trying to pull away from the fence while you're feeding. That is to say if you're facing the fence, with the bit in the center, you would be feeding from right to left. I don't think it's wise to try to remember "clockwise/counter clockwise" or "left to right, right to left" because it depends on the cut.
NOW ...
If you're cutting a slot in a single pass, the above advice is all you need. But if you're going to be taking it in multiple passes, you gotta pay attention! To widen the groove, be sure to move the fence AWAY from the bit and you can feed the same direction as before - right to left. If you move the fence CLOSER to the bit, you will still widen it, but then you will need to feed from the other direction otherwise the bit will be pinching and grabbing the wood and will be very grabby and try to over feed. And feeding the other direction isn't nearly as good because now the cutter is pulling AWAY from the fence, which could affect your cut quality (or require feather boards and other gizmos to keep it against the fence).
I am always careful to feed so that my control surfaces (fence, table, jig, pattern, etc) are not being pushed away by the cutting action - it's very difficult to control a cut when that's going on. It's all about restricting how many directions the wood/cutter can move.