Simplest answer is that it is a form of stored energy, so is finite, so must run out one day.

We might think that day is way in the future, when it won't matter anymore, but if we compare the effects of using it, with the effects of say using fossil fuels;

Assuming simple logic applies to how Earth was formed from the sun (Earth was ejected from it, during an earler volcanic phase of the sun, not the standard convoluted profit driven explanation of Earth being formed at the same time as the sun, from cosmic dust);

Fossil fuels were put here after the Earth was ejected from the sun, created on a continuous basis by the Joules we receive daily, over millions of years, whereas the thermal heat in Earth's core was inherited from the sun, as was also the kinetic energy in Earth's rotation, and our orbital trajectory, when Earth was ejected from the sun.

So there are two kinds of stored energy on Earth, the kind that was inherited, and the kind that is received by continuous solar radiation, which sustains all of life.

We have the technology, and are using it to some extent, to enhance the solar radiation being received from the sun, to add our work from it to Earth. Logically, this acts as the opposite of the damage done by using energy stores created by solar radiation, like fossil fuels.

But there is nothing we can do to reverse any harm done by using volcanic, or the kinetic energy of Earth, because those have nothing to do with the solar radiation which we have the technology to collect.

Using any volcanic energy has to result in a reduction of planetary temperature, no matter how infinitesimal, we can't undo it, unless we could somehow put a heating element in the core of Earth.

If use of the thermal energy in Earth put us on a path to a new ice age there would be nothing we could do to reverse it.

Similarly if we used kinetic energy from Earth, it would result in slowing both our rotation, and trajectory, which would have a net effect of throwing us out of the equilibrium which maintains our orbit, in other words, sooner or later we would fly off on a tangent from our orbit around the sun, with no way of correcting it.

We have to recognise that those inherited forms of stored energy are sacred, not to be touched, it seems to me.

Btw, there is no reason to require precious metals in solar collection hardware, it can be produced, and is produced with some slight reductions in efficiency by excluding rare Earth elements, all we need to do is scale up the facilities a little to compensate, when we use those cleaner versions of PV. All varieties are available on Alibaba btw, there is a truly massive selection of all kinds there.