Outsiders

Not everyone in the world fits into the four-caste system. There were people who left the system voluntarily. And there who left involuntarily. Finally, there were people who never in the system to begin with.

The panchamas are the famous outcastes of India. These are people, or descendants of people, who committed some serious infringement of caste regulations that led to their being ostracized from orthodox Hindu society. A person could become a pariah, for example, by marrying outside their social group. (Remember that until about the twentieth century, this kind of thing wasn't so unusual in Western countries either.) Hindus who adopted a new religion, like Buddhism or Jainism, met with disapproval, too.

People who performed unsanitary types of jobs were also considered "untouchable". These were people who cleaned dirty like lavatories or who handled dead animals or dead people. Perhaps the ancients noticed that these people were a source of contagion (of viruses and bacteria we'd say today) and started keeping a distance.

Mlecchas are foreigners, people who never were Hindus to begin with. Generally Hindus have tended to be somewhat suspicious of foreigners. Until only recently, many Hindus considered it a calamity to have to leave India and live with mlecchas. This makes sense because for most of their history Indians enjoyed a high standard of living compared to many other cultures.