Trust me, I used to think this. I did the whole works- I had a drug problem as a teenager, I underate for most of my teen years (I mean, I never ate lunch at school because I had no money), had depression (which also spoiled my appetite), didn't sleep properly (during my growth spurt), didn't get out in the sun, didn't do exercise...
I had the luck to meet a student at my university recently doing his PhD in bone development. Well, I can't remember exactly what it was on, but it was something very similar. He put all my worries to rest.
Basically, yes, it does have an effect, but the body's catch-up strategies for growth are pretty remarkable. Apparently there are all sorts of ways in which it compensates for deficiencies in something or other.
If it's a matter of some normal, unhealthy habits, (frequently missing sleep, not getting enough calcium, binge drinking on weekends) you might have lost a few millimetres off your maximum potential height (if that). Nothing at all noticeable.
If it's a matter of being very unhealthy (like really, very unhealthy), then perhaps you could lose an inch or so at the most. But nothing that would really make a difference.
If you have anorexia or are badly malnourished during a growth spurt, then yes, you can lose a few inches- perhaps up to 4 or 5. But this is starvation, not missing meals or whatever.