It depends on how the tea was designed to be drunk. For instance, Assam tea, grown, processed, and brewed to British tastes, is basically undrinkable without milk. With milk, however, it transforms into a unique and special beverage. Mongolian brick tea was selected for its ability to perform well after clotted cream was added as is the custom there. On the other hand, adding milk to a delicate Chinese green would obliterate it. Adding it to a tarry Lapsang Souchong would lead to the flavors constantly fighting with one another.
The same is true for sugar. A white tea is naturally sweet, and the delicate flavors would be taken away from with extra sugar. On the other hand, Ceylon teas destined for the Middle East are designed to stand up to sugar, because it is customary to add a generous amount there.
So really it just depends on what sort of end consumer the tea-maker had in mind.
