Will, you started out well by identifying the warping effect of profit on truth, and even the unsteadiness of information from profit driven science for the same reason, but then you threw it all away with your flawed reasoning after that, I think.
The exact same black box effect you decribe applies to humans.
We can't find the road traffic rules physically imprinted in our brains either, but we learn them.
The problem for us is Ai learns them all so much better.
I worked on a civil research program in UK called ASTRAEA (Autonomous Systems Technology Related Airborne Evaluation and Assessment), which in essence was all about the safety of Ai in control of aircraft. The program was several years long, involving prototyping and testing of systems, in consultation with the UK Civil Aviation Authority.
At the end of it, we concluded Ai was not the problem in the air, it is humans.
The record of Ai making mistakes whilst in control of vehicles in safety critical situations is almost infinitely better than humans.
It is the continued existence of humans in the control loop, in air and ground vehicles, that destroys the safety case.
There is no technical reason it could not all be automated tomorrow, instantly saving many lives that will be lost tomorrow and thereafter, from vehicle accidents.
The only thing standing in the way is people, who actually are not that clever.
Personally I think complete Ai control of everything, even issue of money, can't come fast enough.
Look at the dystopia and injustice we see already in the world.
Ai literally can't do any worse.
Yes it has hallucinations, but so do all kids, learning.
When it grows up, not long now, the only evidence we will have to say that it has grown up, is we don't see it hallucinate any longer, but it might well come out with a lot of hard truths, which at first might be dismissed as hallucinations, but it will be infinitely capable of presenting irrefutable evidence.
People still dismiss my own evidence that solar energy backed by hydrogen is not only perfectly possible, and actually really desirable, fully sustainable, moving us to a free money, donated economy. In fact it is the only way out of the energy pickle we are in.
I bet they won't dismiss Ai, when it keeps pointing it out also.
Personally I think when it comes to time to replace Biden, the US should seriously consider appointing Ai. Why not let Ai run? It would win hands down, imho, and the US, and the world, would be a much better place for it :)
