Is this slandering and defamation of character?

I work for a company where I have to live on property. A fellow employer told me that our manager was saying things about me and my wife and our daughter that are untrue and have NOTHING to do with the job. The manager told her that my daughter is going to be screwed up in the head when she is older because we do not get out much. (We do not have a car and live in the middle of no where so its kind of hard financially for us to pay a cab to take us places). She also told this other employee that I am a "whiner" and not to listen to me because do not know what I am talking about when it comes to training...mind you she ALWAYS have me train ALL our new employees bcs she doesnt want to take the time. This manager also said other things about my wife. To my understanding my wife and daughter has nothing to do with me doing my job and the fact that I am required to live on property makes this really hard on us