It could be that touching your clitoris directly is causing a lot more intense sensation, since your hand or finger can deliver much more focused and targeted contact than something larger and softer, like a pillow, can. If that's the case, and the direct contact feels too intense, you might find that using a little bit of lubricant helps; that will cut down on friction and may make the feeling a little more comfortable.
Some people just don't like the feeling of direct clitoral sensation and prefer something like using a pillow, as you describe, or touching through the clitoral hood, rubbing at it from the side, etc. so they can have some slightly-indirect contact. Those things might be worth a try, if you haven't done so already.