How can downdraft be prevented or cured?

Haviin alot of trouble with downdraft in my chimney and smoke comming into the house. Only happens in high winds, none the less it happens. I do live close to trees, but i know others who live close to trees as well and do not have this problem. Anyone got any suggestions on how i can stop this from happening or solving the problem