Why won't he have with me?

I've been with my fella for 7 years and for the past 3 years or so, he hasn't laid a finger on me! I ask him "can we do it tonight" and I get "yes" with a huff!!! He says he loves me everyday so don't understand why he won't make love to me? Around 3 weeks ago I "caught" him watching on the net and getting his pleasure that way!!! I was so hurt and felt somewhat betrayed seeing as he promises me we'd do it but never does. When I ask him in bed, he just turns around saying he's too tired!! I know he's not having an affair with anyone else as he never goes out..we're pretty much together 24/7 when not at work. He is very rubbish at communicating when it comes to matters of the heart so I sent him an email instead (as always!) telling him how I felt. I know he's read it but hasn't responded...instead I got a hug the next day and I know that's his way of saying sorry! I've been patient with him for 3years and even feeling sorry for him thinking he simply lost his libido or genuinely was too tired. And for that reason, i've kind of let it go though it hurts me very much especially when I get rejected the nights I ask him to do it with me. I just don't know what to do now as I love him but with recent "events" finding out he was getting off watching , it's making me move slowly away from him.....but I love him? What do I do? Do I sacrifice having a less relationship or do I simply move on. The only physical contact we have with each other is hold hands (alot) even when watching tv.. and cuddling up with each other. I so miss kissing intimately too, that's just the best thing ever!!! I'm 38 he's 41 both young at heart and look a lot younger than our age. I get plenty of attention from other blokes and sometimes just wonder......!!!! Someone please help me, I feel like i'm coming to those infamous crossroads!!!