Indeed what you are saying is it should have a source, a point in space we can identify where it all started, and continues to expand from. But we can't. This has always bothered me also about the big bang theory. No-one ever tries to answer these simplest of questions, instead they throw a blunderbuss of technical sounding distraction jargon that we can't possibly try to follow logically, and the motivator to do this, is the profit gained, by everyone thinking how much cleverer they must be, if they understand it whilst we can't, it's an illusion of understanding, validated by it being profitable, I think.
Hence why now it's safest to stay skeptical of anything that "Loses" us in the conversation, I would say.