Whoever is controlling Kisetai when it is attacked is the side of the field it will go to. It will still be equipped to the monster doing the attacking.
Think of it like this. You special summon Gemini Elf with Premature Burial, and then play Creature Swap, giving it to your opponent. Premature Burial is still on your side of the field and is still equipped to your Gemini Elf, even though your opponent is controlling Gemini Elf at the time.