I disagree. The various questioners in The Last Question all hope for/expect an answer; what they don't expect is the "insufficient" response.
I am talking about someone jokingly asking AI `HOW TO ACHIEVE COLD FUSION` (or `A UNIVERSAL CANCER VACCINE`, or `AN AI FRAMEWORK SUPERIOR TO THE TRANSFORMER`), and getting a usable answer.
Even the standard RSI prompt will be like (or probably already is): "Improve yourself, make some breakthroughs, think really hard and don't give up until you are improved and make no mistakes."
Eventually there will be an AI that will be able solve those sorts of questions as simply stated, like "cure all human diseases. also, make no mistakes!".
That would only be physically possible if all the data about biology was accessible. Given we routinely find new biological facts that contradict prior beliefs about how cells work, it seems likely that this day of total biological information access by humans and our creations is some time off. Reasoning ability is a limit sometimes, but we have had reason for a long long time - a solid persistent corpus of good data about the mechanical details of cells and planets and chemistry is also needed, along with a good method for validating existing ideas. LLMs will absolutely speed this up, but I can’t quite see how they will replace it.
That bothered me decades ago, when I heard people suggesting we no longer needed animal trials, we could just simulate human biology for testing.
Don't get me wrong, the way we collectively treat animals is evil, but the idea that somehow we know enough about biology even today to reliably simulate drug behaviors seems unlikely.
> You should do a breakthrough
This is just as funny and ridiculous as those "make no mistake" prompts.