Interesting that the Shilla Kings political rival was exiled to Japan. Even though Japan during the time is known as Wa, all of Japan was not under a single political state,
and it is likely Shilla had barely noticeable enclaves on Japan's northern areas, judging from the discovery of Shilla style stone architecture in these regions. I am curious where exactly The prince was exiled to, since it would have to be an area under tight control by Shilla ruling faction to ensure that the exiled cannot escape. Usually those exiled are sent to a part of a country's own territory, not a foreign country, for reasons of security.
Even if Yamato court figured out they had been cheated by sending them an unwanted hostage - they could send the hostage back with a Japanese army to make trouble, but meanwhile the hostage would have been cut off from making connections and recruiting supporters in Silla. An exile coming back with a foreign army but no Sillan supporters might be rejected by Sillans.