The downside is that a lot (I dare say, most) people play on private servers for nostalgia reasons, and this would definitely change the image of the server quite a bit, and for me and I think a lot of other people would hinder the nostalgia feeling you get from playing here. Personally, I like that when I see someone in a full set of gear I know about where their character is at in progression, I don't think that's "silly" I think that's a legitimate feature about TBC that I don't like about live today. On live, my brand new alts look like they just went 12 rounds with the baddest raid content and pvpers in the game, yet they are in full garbage and I may or may not even know how to press the buttons correctly yet. When I see an orc warrior on here in full season 3 and a stormherald, immediately I know i'm dealing with a bad MF'er, and same goes for when other people see my priest in full gear in a bg, they know i'm healing for more than tooltip and enemies know it's not going to be an easy task to take me down. I think that's great, and surely impacts the game.
I can see why a lot of people like it, because it does add something to do, but in reality we are only dealing with 2 expansions worth of content here. I think there's clearly more downside to transmog than there is for something like dual-spec which I think most people find to be a welcome quality of life change.
Honestly, the more I think about it. I don't even think its arguable.