Found the solution, the issue was that the field was referencing nodes that are not published so kept throwing the error. So I updated the field to default allow unpublished content as well and it worked.