I think the most telling explanation is simply that Speed measures the ship's movement over the course of an entire day, as JohnK pointed out. Over that kind of extended period, one ship's speed can be noticeably different than another's, even though they're both using graviton drives; during the course of a combat maybe lasting only a few minutes, though, the speed difference isn't going to be that noticeable to two ships already close to each other. The image makes sense if you think of big ships at sea; there can be a difference in speed, but it's only appreciable after a few hours of travel.
You could always houserule that the difference in Speeds acts as a modifier to the Maneuver roll for the pilot, though. This would make Speed affect piloting in a practical manner during combats.