Neighbor died and her family is trying to take my property by adverse possession so they can gain land for $$?

The house was built 48 years ago. Once the owners ped, their children took over. The house 3 feet from their property line, which isn't in setback limits (10 or 15 ft I think). They then installed a fence at least 20/30 years ago onto our property (which was vacant at the time). This fence also resides several feet onto another aing property. We are currently building our house and would like to install a 6 ft fence and remove their chain link fence that is on our property. The owner is threatening us with "adverse possession". The problem is this lawsuit or whatever can take a long time. I know they want to sell their house and are upset because they believed their lot line lied within the existing fence which it does not. If they "win" we will not only lose 5 feet, but we'll then have to put up our fence another 15 feet from their "illegal" fence. Again, the other issue is that their house on my site plan shows that they are not within the legal setbacks. Any advice??