At the end of the mystery, a character has their 3 starting items and they can keep one extra that they picked up in the mystery.
What about after the next mystery?
Can they add one item per mystery to what they carry, or is it limited to only one extra forever?
You can keep one extra item besides your three personal items. If the extra item is a pistol, you will have your three personal items plus a pistol at the start of the next mystery and at the end of that mystery you will again keep your three personal items plus one more. This can be the pistol again or another item. Another character might decide to choose that pistol as his extra item if you don't.
If the maximum limit is 4, how to all of the extra items from HQ upgrades influence this?
If a character has 4 permanent items, and they pick up the holy water from the chapel, can they carry 5 items? or do they have to swap one?
The above rule is only what you do when you end a mystery and how you start a mystery. During a mystery you can have any number of items. During the preparation phase you and your group can pick up items from the HQ (it is assumed that you return those items to the HQ after the mystery).
If the characters can store 3 items in the cellar vault, can that same character pull out the 3 of them and be carrying 8 items - 3 archetype, 1 from mystery, holy water from chapel, and 3 items from vault?
Since this isn't an item hoarding game, you can during an adventure have any number of items. Well, if you and the GM think you can carry them. So at the start of a mystery, you have you three personal items plus perhaps one more. During the preparation phase of the mystery you can pick up any item in the HQ that you are allowed to pick up or technically the group can pick them up because you can of course discuss who will get what item for this mystery.
Also, if a character has 2 starting items, for example, the occultist's poison from the background rolls at the back of the book, and they later get stronger poison, can they swap that out to upgrade their personal items?
Another example could be the servants' bandages being swapped for medical equipment, or if the Priest upgrades their musical instrument?
Your three personal items are your three personal items and will be for the rest of his life. There are ways to have more items through the HQ though.