If it feels good, then whatever. Not like you're going to be doing anything productive during that time, anyways. You'll just be bored. I remember hearing that it makes a man more tense if he never gets off, too. (though that was from the internet, so who knows how reliable that is)
If you feel that fapping is preventing you from getting important things done or getting to bed or whatever, then that just means that you need to pull yourself away from the computer when you feel that something needs to get done.