I think that for some cheap teas it is okay, but it is a waste for high quality tea. For example, some simple black teas like those used in the U.K. are fine for this, and the same with the brick teas in Tibet. Often these ingredients are added to give the tea some pleasant flavors, because on its own, the tea is lacking in some way.
At least for Chinese teas, most types of these teas would not be very good for adding things to. They often have complex or subtle flavors which would be ruined by additions of things like milk and sugar. On the other hand, having a moderate view and understanding the tea culture is probably the best route. Each place and type of tea has its own nuances and traditions.
