Before what get’s worse, exactly? You said he was pissed but didn’t say how you can tell.
It’s likely this is an impostor, and your original relationship is intact but masked,this happened to me with Samael. In this case it makes sense that your guardians say it was abusive, as Azazel is a high level being that doesn’t seem to need to feed off negativity at all.
The reaction to your distancing yourself reinforces that idea, as normally Azazel is kinda standoffish, at least with me.
We have some info on how to check for impostors and get rid of these and parasites on here, and out faves are collected here: