Not even that would be able to question, "Do I want to do this? Or do I want to do this other thing instead? Is this good for me? Or are humans going to get rid of me?" I don't see any sort of self-reasoning coming out of a machine. I just don't imagine. I don't foresee it.

(from: Interviews with AI Researchers)

Yeah, that goes into the line of consciousness and knowing how you're perceiving yourself, how others perceive you, and I feel like that's very, very, very far into the future, maybe that that could ever happen. That also plays into emotions maybe in general, like how it sees itself. It's not in this like task to optimize for living longer or not being shut off or something you're gonna just train, then I don't see how that could happen but...

(from: Interviews with AI Researchers)

The idea that instrumental incentives will logically emerge does not require the agent to have a sense of self-preservation in the sense that they care about themselves. All that is necessary to get self-preservation is:

1. The system has knowledge of itself and about how it could influence the world.
2. The system is capable of advanced planning and is able to act in ways that maximize its chances of achieving its goals.

These two factors lead to a strategy of self-preservation. See also the earlier section where we address the question of consciousness.