Welcome to the forum, and thank you for sharing this with us.
Yes it could be what has been called the hat man, or it could be a similar looking person from around 100 years ago who looks the same.
I would do a little investigation into the mill, and see if you can find a picture of the owner, as well as any other deaths related to the mill or that bridge. Particularly regarding the death of a teenager while the mill owner was still around.
It is sounding like it may be a spirit that is feeling responsible for a death and is keeping a close eye on you to make sure you are ok, as you may have a similarity to the person who died, and he feels responsible somehow.
Just a hunch. But an event like that could lead to a haunting if he felt responsible.