Cool piece, pretty much identical to my thoughts exactly on the subject, though maybe not in terms of phenotypes. I hadn’t heard of that terminology before. I always just used “Tools”. I guess we probably converged on these thoughts from different directions, but it is reassuring to know we are not alone.

A further aspect worth looking at I think, is analysis of the issue of profit seeking in the system, how this affects the decision making, the value added and subtracted to, and from the various stakeholders.

For example, in the sentence quoted above, if the system is non-profit, then the user is indeed master, there is no system owner, and no payment for ad space, unless said ad is an inconvenience or annoyance. The user might even receive payment from the system in return for their valuable interaction, and for being annoyed by ads.

However, if the system is for profit, the user is not master, but slave. The system extracts what value it can from the users interactions, as well as revenue from the ad sponsor, and feeds this to the system owners. Since the user is not free to make decisions outside the influence of the system (driven by ads), the value generated by the user is also limited.

Estimating the future implications of those two scenarios leads to two very stark conclusions, one very good, and one very bad, for all of humanity, regardless of whoever may be benefiting financially in the short term.

Both seem to be beyond comprehension of most people. Perhaps we need some Ai to assist with that. Would we trust such help from a system built for the purposes of maximising profit? I really hope not.