Small, teeny tiny, seemingly inconsequential comments, jokes, etc. work to DE-sensitize us. The next time we hear an offensive joke, we are less surprised. Eventually, these teeny tiny jokes form a stereotype and any effort to break that stereotype is met with hostility.

While our silence created this nightmare just as surely as those who told the jokes you are right. The only thing that matters to us, is defending our own reputation - "Its not my fault".