I'm not too concerned if he goes really. The defense has been average for the past 6 weeks or so and there are plenty of really good candidates who we could hire to replace him who I'm sure would love to coach guys like Parsons and Diggs.
Off the top of my head you have Steve Wilks, Ejiro Evero, Vance Joseph (assuming Kingsbury is fired), Gus Bradley (assuming they don't keep Saturday) and possibly Ron Rivera depending on what happens there. You can even throw Zimmer in there, I know most won't like that name but he was one of the top defensive minds in the league for about 10 years before he was fired in Minnesota.