So, medically speaking, phimosis just means having a very tight foreskin, which could be what's happening here. While many people's foreskins are able to retract, some people's foreskins are just really tighter in general; they might not retract below the head of the penis at all even once the person reaches adulthood. Some people with phimosis do just fine without any medical intervention. Some people might experience some discomfort at times, and for others, it may be very painful.
Given all that, and the fact it doesn't sound like this is causing you much--if any--discomfort, when or if to ask a healthcare provider about this is up to you. If you do decide to ask them, this article has a good overview of what to expect from the experience: A Foreskin Foray: Questions, Concerns, & Clarifications