I think you want to invoke into your physical body, but you go to the astral temple only with your mental/astral body.
Try invoking just in your room at home, sitting in meditation and inviting the entity to come into you and lend you it’s intellect and strength.
Did you see this tutorial here and did it help at all?