Latency Spike Only During PVP/BGs

My latency while hanging out in Ironforge is around 125ms, the past couple of days when I go into AV or WG, the BG starts out fine. I can heal and fight with no issues, but after a little bit of time. Roughly after Galv is killed, is about the time my latency starts to climb. I first noticed when it shot up to 2000ms, when I died out of nowhere. From there I watched it climb to a peak of 12000ms. I end up getting kicked from the BG, due to afk, and load back in Ironforge. However, once back in Ironforge I see the my latency slowly comes back down and after a minute or so, I am back at my 125ms. This just started yesterday, even though this past weekend I had no issues in BGs, grinding about 40k honor with no lag.

Any ideas on why there would be this issue?

Same thing tonight, just during the last WG, started out fine, then about 15 minutes in ping went to 5000ms, lost shortly after (due to a push by MXBA). Left the raid flew to do the ghostfish quest and by the time I landed in the Basin, I was back to green with roughly a 125ms ping.