I never quite understood why they bothered murdering his family. It was petty, unnecessary, and excessive in its brutality. They posed no threat to the conspirators. The daughter could have been raised to adulthood, and then married off to some Roman client king, like Augustus did with Cleopatra Selene.
A change of power in Rome by conspiracy often demanded violent action and retribution.
With him died his wife Caesonia, stabbed with a sword by a centurion, while his daughter's brains were dashed out against a wall. Life of Caligula (Suetonius)
As soon as his power was firmly established, he considered it of foremost importance to obliterate the memory of the two days when men had thought of changing the form of government. Accordingly he made a decree that all that had been done and said during that period should be pardoned and forever forgotten; he kept his word too, save only that a few of the tribunes and centurions who had conspired against Gaius were put to death, both to make an example of them and because he knew that they had also demanded his own death. Life of Claudius (Suetomius)
It isn't just about replacing the figurehead, but potentially about replacing the entire dynasty to prevent revenge or civil war, or in the case of Caligula, even to the extant of getting rid of the Caesars and returning to the older form of republic. Claudius made no bones about it - he was Caesar by the demand of the Praetorian Guard, not by choice, and in many respects he was very lucky to have been found by Praetorians who saw him as a means of preserving their lucrative existence in Rome.
As much as I love the story about his horse, that is often wrongly (IMO) put forward as an example of insanity. Caligula knew exactly what he was doing and it was a bit of humor at the expense of the Senate. It was in effect a message that they were less worthy than his horse.
The bit with Neptune and the sea shells was likewise likely intended to humiliate mutinous soldiers.