That's because your metabolism is designed so that you're able to fall asleep when you're supposed to, but if you go too far past that point it assumes an active operation again and makes it difficult for you to fall asleep until the next low point.
Losing sleep by taking advantage of your metabolism to stay up later IS bad for you as there are certain vital chemical balancing processes which cannot reach equilibrium state with rest alone. Meaning you will accrue a sort of sleep debt which strangely is NOT the exact number of hours lost but rather a function of said hours. Which is why you might feel like sleeping as little as one hour extra for multiple hours of sleep lost or possibly many hours for only a small amount of sleep lost. You'll die anyway if it reaches a certain point.