I honestly don’t know what I need to do with people speaking Spanish at me.

In H-town, I often interact with people speaking Spanish to me saying they do not speak English. I tell them I do not speak Spanish and they won't even budge or care. I am bit confused what they expect from me. Like, do they want me to kindly use a translator for them or start learning Spanish on duolingo even though I technically live in the US? Should I just reply back to these people in my native Asian tongue so hopefully these people can understand my frustration?