It seems you face to Current Transformers saturation due to DC offset earth fault current and its bad effects. For more information you can refer to Protection Riddle No.38 and Mr. K B Santharam excellent reply. Advising with numerical relay manufacturer and using the higher tap of current transformer or increasing of relay time setting can be some solutions for mentioned mal-operation.