Dude, looks shouldn't matter and neither should others opinions. If you are happy screw what others think. And it is possible to fall for someone you aren't physically attracted to. I.E. I met this friend of a friend back in high school and didn't think much of her. I mean there was basically no attraction at all. We started hanging out and I fell for her and her awesome personality.
If you are happy, which is what it sounds like, and can get over the fact that other's opinions don't matter, stick with it. The personality attraction will overtake the lack of physical attraction. Good luck.
