What caused dense matter to exist and proceed to the big bang?

We can rephrase your question to ask, "What was there in that superdense space that exploded and became the Big Bang?" and the short answer is that no one knows how the Big Bang came about. And matter, as we know it, didn't come into existence until well after the Big Bang had banged.

While searching the internet for a response to your query, I came across this page, which, although it is a little tongue-in-cheek at points, provides probably one of the best explanations of what is known and not known about the Big Bang: https://tutor.hix.ai

In summary, the article argues that the Big Bang theory attempts to explain the Universe's history after the Big Bang, rather than explaining how the Universe came into being.

Though there is currently no conclusive answer within the framework of current understanding of physics, the question of why dense matter existed before the Big Bang is still a subject of theoretical speculation in the field of cosmology. Some hypotheses suggest that the conditions for the Big Bang emerged from a singularity, while others propose ideas such as quantum fluctuations or the collision of higher-dimensional branes in string theory.

The existence of dense matter prior to the Big Bang is a topic of theoretical speculation in cosmology. Some hypotheses suggest that prior to the Big Bang, the universe was in a highly dense and hot state known as the singularity. This singularity is thought to have contained all the matter and energy of the universe in a state of infinite density and temperature. However, our current understanding of physics breaks down at the singularity, and it is unclear what physical processes or conditions may have led to its existence.