If I loved them, I see no reason not to. Even if you do not remember it when you return to your world, at least for a brief span you would have had the happiness of it. If nothing else, I believe that feeling will remain, even if you never find the memories connected to them.

And if it makes the other person happy as well, then that seems even more reason to.

Besides, you could keep the ceremony small if you tried.