I think we have an incomplete and false understanding of reality. Because of that, we are moving inside a tiny fraction of endless possibilities.
An AGI system based on previous and current knowledge can only exploit what is possible within that tiny fraction. The missing link seems to be our mystical, uniquely human ability of intuition, which allows highly conscious humans to access knowledge outside our current fraction of possibilities – creating something (ideas, inventions, theories) entirely new, that has never been done before.
Based on how AI systems are built, I expect them to create meaningful advancements within our current frame of understanding; but compared to what is actually possible, these will stay miniscule.
To access what seems impossible, we shouldn’t look for logic and intellect. We should aim to understand consciousness; i.e., study Yogis, understand DMT, make sense of Psilocybin.
Only by heightening our consciousness – and intuition seems to be the highest form of it – will we be able to emerge from current limitations. Because in the end, there are no limitations.