Home buying in an AE flood zone

Looking at buying a place in southwest Houston. In the option period I found out it’s in an AE flood zone.

The house is over 30 years old, and I’m told there has never been a flood claim on the house. It’s elevated a bit, though I don’t know by exactly how much; the driveway feels really steep at least. Everytime I bring up my concern to anyone I’m told all the above as well as “flood zone AE means nothing, most houses that flooded during Harvey were in zone X” or “if it didn’t flood in Ike or Harvey, then odds are it will never flood”.

I just can’t shake the uneasiness about the flood zone. A ton of houses never flooded until Harvey, and those that still haven’t may flood in the next big storm. At this point, I know nowhere in the Houston area is guaranteed to be safe from flooding but it comes down to the fact that it’s in a spot recognized for potentially being high risk…

All things considered, am I worrying about something that will never happen? Would you go through with buying in a flood zone if you were told it never flooded and knew it was a little elevated?