文法的にわかりません

My mother is 92 and lives alone in the Midwest; I’m here in Southern California. She lives around 18
miles from town on 65 acres. I phone her twice weekly and sometimes more often. After I retired, I asked her to move here and found a lovely place for her as she prefers her own space. She refused and I tried my best to offer all the reasons why it would be the best option. I visit her twice yearly and my sister and brother visit as well, but not as often. She doesn’t drive and thankfully my niece lives close to her, shops and helps her whenever my mom asks her for help.

I know she’s lonely and it upsets me, but I feel like she made the decision to stay in her home. Here’s the dilemma: when I call, I talk about my life and what’s going on with my daughters and grandsons, etc. She has such a narrow life without much to talk about, so the conversation is one-sided.

She reads but doesn’t want to discuss books or what she’s watching on television. I try to bring up memories and she enjoys this angle of conversation. Sometimes, after talking for an hour, I try to end our conversation, and her response is scolding and negative that I don’t have time for her.

her response is scolding and negative that I don’t have time for her.の部分が文法的にわからないのですが、thatは何にかかっているのでしょうか？scoldingの前にsoがついていれば文法的に理解できるのですが。よろしくお願いします