Your brother just lucked out a lot.
There is a formula I've seen, but it's far from perfect. The genetics of height are far from being understood.
In Asian populations, the heritability of height is much lower than 80 percent. For example, in 2004 Miao-Xin Li of Hunan Normal University in China and his colleagues estimated a height heritability of 65 percent, based on a Chinese population of 385 families. In African populations, height heritability is also lower.
[...]
Heritability allows us to examine how genetics directly impact an individual's height. For example, a population of white men has a heritability of 80 percent and an average height of 178 centimeters (roughly five feet, 10 inches). If we meet a white man in the street who is 183 cm (six feet) tall, the heritability tells us what fraction of his extra height is caused by genetic variants and what fraction is due to his environment (dietary habit and lifestyle). The man is five centimeters taller than the average. Thus, 80 percent of the extra five centimeters, or four centimeters, is due to genetic variants, whereas one centimeter is due to environmental effects, such as nutrition.
[...]
Heritability can also be used to predict an individual's height if the parents' heights are known.
So 65% of height in Asian males is due to heritability, and 35% is due to environmental factors.
Using your parents' heights:
0.35*((173-170)+(165-160))/2 = 1.4cm
So you could have only lost 1.4cm due to environmental effects.
I find the heritable height formula useless (even if it's inferable from here), so I'd just rather not post it. You deviated just a little bit from your expected height (according to that, anyway) and your brother deviated drastically from his expected height.
EDIT: removed a word from a sentence.