It's inconsequential so long as you have plenty of time to rest and recover. If there is something urgent you must do as soon as you arrive (Combat, Council, start another leg of the Journey, etc.) then it becomes a bigger deal.I agree, but in my case the journey was interrupted by a combat (that came about as the consequence of a travel roll). In a combat endurance could be lost and fatigue should play a part in becoming weary. But as I understand the RAW you don't apply fatigue until the end of the journey, which seems....inconsequential.
This is also a good argument for breaking up long Journeys into several shorter ones. It allows accumulated Fatigue to catch up with you. You must then decide if you press on immediately or rest for a while.