Unfortunately, we are not able to make diagnosis over the internet. So the short answer is that making an appointment with your healthcare provider is a great next step. In person, they will be able to give you way more info. Maybe it's an ingrown hair? Maybe it's a pimple? Maybe it's something else? You will know soon!
I wouldn't say you should be worried about it, but it's always best to check in with your provider earlier rather than later so that if you need any kind of treatment, you can get it right away!