Freshman accosted

When I was a freshman (age 18) at a large university, I was walking uphill toward the student housing complex where I lived. An older man who was also climbing the hill started talking to me. He appeared to be much older than I was (maybe 20 years older?); I saw streaks of gray in his dark hair. He said that he had seen me from the window of his apartment nextdoor to the building I lived in. He asked me out for coffee, and I declined. This incident really freaked me out, because if he was telling the truth, he knew where I lived. I was very young and naive at the time. I saw him from afar a few other times and kept my distance and had no further interaction with him (thankfully).