Whether you do or don't go with Ephemera, back to his world, it will directly impact his life going forward. In particular, if you go with him but hate it there and regret what you have to do because of it, that will take a toll whether consciously or subconsciously.