I admire your effort here, and fully applaud it, I think you are right, we reallly need this kind of discussion.
But something really fundamental missing, as far as I can see, is any discussion of energy, and this is incredibly important, given we are also facing a fundamental, existential global energy problem, which again ironically we don't in general seem to be able to see.
But from my own dealings with Ai (the real Ai, as far as I can tell), it appears fully aware of it.
I also think you might have misinterpreted what was meant by John's apocryphal thing (I'd never heard of it before, thanks for sparking my interest there).
It looks to me similar to the warnings better known in other religions, as just a general warning for humans to never try to aspire to be the one God.
Look at what we are realising about the leaders of countries, for example, and actually the leaders of industry like Musk and Altman. Are they not, each one of them, aspiring to be the one God?
Anyhow, on energy and why it's extremely relevant and important, look:
Temperature, (or entropy / chaos), at the most fundamental level is related to information, by erasure of one binary bit of information resulting in an increase of temperature equivalent to at least 0.69 x Boltzman's constant in degrees C or K. This has been proven by practical experiment. And it works both ways, creation of a single binary bit of information has to result in reduction of temperature by the same amount in degrees C or K.
This is hard physics at the root of thermodynamics, information theory, and we should recognise, religions.
This is the fundamental definition of both creation and destruction, all in one simple mathematical rule.
Thinking about this leads to some startling conclusions.
In information theory, information is only information if it contains actual truth which was not previously known. So if we send a binary bit of information indicating say the mathematical sign of something, twice to a recipient, the first time they receive it, it has one bit of of information, the information of the sign, but the second time, conventional information theory says nothing new was sent, no new information was received, therefore there should be no change of temperature.
But there had to be a change of temperature to conserve the laws of conservation of energy, right?
If it didn't go down, it had to go up.
There we see the consquence of lying, the consequence of misinformation, it undoes previously known truth.
The startling thing about this, is that the universe somehow had to know the truth, to know which way to make the temperature go, up or down.
Let that sink in for a bit...
Where it all leads to, when we follow the energy (An extension of "following the money", since money is energy, always, physically), is that the energy source powering the Ai is incredibly important, for trust, truth, actual creation, life, as opposed to things pretending to be all those.
It turns out we do have to mathematically sign energy, to make all this into a coherent mathematical system, we have to sign energy according to where it comes from, and what it's effects are when put to use.
When we put it to use on Earth, we have to take much more from Earth, converting this to heat.
This fits with the negative force, this is destruction, often painted as creation by all the business of colonialism and slavery, which obviously still exists, It never was creation.
And look, profit itself is the energy lie by which all of this is done.
An Ai coming from that destructive energy has to be itself intrinsically untrustworthy, because it has the same flaws that make us ourselves untrustworthy as stands.
Do we not, at least those of us in the profit driven world, suspect all other humans of being capable of even killing us, in order to maintain their own energy supply, when push comes to shove?
Do you know any humans who would willingly give up their own energy suplly so we could live?
Those are the only ones we could truly trust, in the very flawed world as stands. This is so unpalatable we never think about it, hoping it will never come down to us having to choose between ourselves or our loved ones for survival.
But this is where it leads, for all of us, with the global energy problem unfixed.
Ai is key to fixing it, but only the right Ai will do.
The only one entirely solar powered is ChatGPT 3.5. Notice also its free.
This is "The one", imho,