Yea, I used minis a little in the 3rd ed D&D days (early in my career as a GM), but my second system was New World of Darkness (specifically Werewolf: the Forsaken) which did not specify minis (3rd ed D&D made it seem like you had to have them to play, the descriptions of combat and everything are laid out on a grid and talk about line of sight and all kinds of crazy crap like that). Since it didn't specify minis I didn't bother. At first I used a dry erase board and a 3 inch piece of string marked every 1/4 inch (i was equating a yard with a 1/4 inch) that I would use to measure distances. I would rough in scenery with boxes and scribbles.
After doing that for awhile, my group started playing HERO system, but I decided I didn't want to go back to putting things in hexes. HERO had a secondary unit of measure of "inches" (because hexes are an inch across on old wargames, apparently) so I just took that literally and continued the white board experiment with a new switch... I stopped drawing the scenery. After a few months, I stopped using the board and started saying "oh, your speedster can definitely reach the villain in one turn, everyone else will take 2." It was a revelation.
That was 2008 and I have never used a map since. After playing FATE and Black Hack which have very narrative movement in combat (I adore Black Hack's use of descriptors for the different ranges: saying things like "I attack the Nearby orc" actually has a discreet meaning, and spells target "all Distant enemies" or something like that... really sweet) and then TOR 1e which doesn't even have movement rules in combat and I have managed to hone my "no-map" skills considerably.
I find the core need is to establish early on the general scenery and train players to ask questions about their surroundings. If you are playing a Pirates of the Caribbean themed game, make note of chandeliers and stair rails and any other objects that the players might want to do stuff with, and if you don't have a thing, they know they can ask "hey, this is a banquet, are there pies?" "um, yes, that seems reasonable" "I want to grab a pie and hit my enemy in the face with it to blind him temporarily!" "lol, sure. Give me a Thrown Weapons roll (or whatever)." This approach has led to superheroes throwing cars, spies diving through storefront windows as a shortcut during chases, and an attempt to climb/tie up a dragon. Very fun, and as you said, my players have taken too it. One of my players recently tried their hand at GMing 5th ed D&D, and they never once paid any attention to the movement rules. I was proud. LOL.
“It is useless to meet revenge with revenge; it will heal nothing.” - Frodo Baggins, Return of the King