The biggest difference that people mistake, IMO, for a physics improvement is sound in V. If you T-bone a car in IV, it looks like you T-boned a car...but it doesn't sound like it. IV had crap for sound effects. But in V, if you even tap a car, you FEEL it. The subwoofer lets you know when you hit a car, my god, the sound feel in V is amazing. But your car will drive away like nothing happened, like an arcade game. I'd rather destroy the entire car because of a realistic accident, inconvenient sure, but I prefer it.
Now you're definitely right in one sense, the out-of-the-box handling for GTA IV vehicles was appalling, a horrific experience that no one should have to experience. Like driving a wet sponge. But it was also very realistic for most cars... How ever bad it may have been.
But, GTA IV handling is almost my favorite, EVER. I've spent over a year completely rewriting the handling code for every single car, it's utter and complete perfection the way I have my babies driving. Hours, days, weeks perfecting one car alone. Making it handle and perform identical to it's real life counterpart. The original handling may have been bad, but they give you the fundamental tools to create a work of art, a breathtaking driving experience that I had a very difficult time finding anywhere else. And of course I have my silly cars, my 900HP AWD drift missiles that set track times you wouldn't believe.
90% of driving games, any driving game (NFS Shift, Forza...), makes the car handle like an arcade racer in some way. Not GTA IV, it reminds me of Beamng really.
It's all personal opinion though.