I have to admit I probably would have read yellow in that context to mean ‘cowardly’ rather than with racist connotations but I can see how it could be taken that way.