I was reading R. E. Howard’s The People of The Black Circle a few days ago and was struck by two interesting segments:
Khemsa's sorcery was based on hypnotism, as is the case with most Eastern magic. The way has been prepared for the hypnotist for untold centuries of generations who have lived and died in the firm conviction of the reality and power of hypnotism, building up, by mass thought and practise, a colossal though intangible atmosphere against which the individual, steeped in the traditions of the land, finds himself helpless.
But Conan was not a son of the East. Its traditions were meaningless to him; he was the product of an utterly alien atmosphere. Hypnotism was not even a myth in Cimmeria. The heritage that prepared a native of the East for submission to the mesmerist was not his.
Howard is presenting an idea that cultural tradition makes certain types of magic have greater effects on certain people. This is actually a neat idea. It can be expanded too; maybe certain cultural traditions lessen the effects of certain types of magic.
For example, maybe the Iron Tribe of the East Hills believe that pyromancy burns not only the body but also the soul because in their religion it’s the way that their demigod ancestor died. On the other hand, the Sun-hawk Tribe of the East Hills are fond of pyromancy and believe themselves to be descended from a fire-spirit that took mortal form. The Irons might have a chance of outright death or destruction of the soul when damaged by fire magic while the Sun-hawks might receive reduced damage when hit by fire magic. Maybe the urban populace of the Imperial Colonies finds it harder to resist the rural indigenous magic that conjures poison and disease while the Northmen of the Frozen Wastes receive greater benefits (and perhaps greater harm) from magic that involve/create something drinkable.
Of course, if you’re gonna run with this I think there have to be some fundamental factors worked in to the setting.
The tradition is social
By which I mean that this isn’t just one individual afraid of soul-devouring magical fire but a group of hundreds at the bare minimum.
The tradition is old
These ideas have been around among the group for at least a century if the people in question have human-length lifespans, longer if they’re kinda like dwarves or elves.
(Optional) The tradition has a deep cultural/mythic/religious origin
For example, the Northmen drink a lot and drinking is central to their legends. For this reason drinking a healing potion has a greater effect on them. For the Colonies, fantastical accounts of the land's deadly maladies and a conspiratorial fear of the natives weaken the Colonists' resistances.