Get rid of the fixed neighbor structures