No downside? Hunters rolling swords, mages rolling 2handers, palas rolling cloth (why not if pole to sword/axe and not cloth > plate.) which would increase prices for a lot of items and would complicate who rolls on what.
If you think they d implement this (pole>axe/mace/sword), they would implement those (cloth>plate etc.) aswell since its the same.
Though I liked the idea of pet xmog.