I think the main reason may be in their stance towards conversion to Islam. Under the Umayyad reign, the mawali (non-Arab Muslims) were denied privileges of Muslims (like not paying the kharaj). The rebellions of the mawali, like that of al-Ashras ibn ʿAbd Allah as-Sulami in Khorasan, that coincided with foreign invasion (in this case, of Turks), resulted with loss of territory and embarrassment. It is significant that the leaders encouraged Ummayad governors to be firm against mawali demands, fearing that they would not be able to meet financial demands imposed by the government if they lost their people. The tradition holds that Caliph ʿUmar II was unlike most Ummayads, being somehow a champion of Muslim equality; however, it seems quite improbable.
The denial of conversion to Islam was used by the enemies of the Umayyads to present them as enemies of Islam and their enemies like pious Muslims. One must be warned that most of the sources of tradition concerning the Ummayads got its final form under the ʿAbassids.
There are many other un-Islamic things ascribed to the house of Ummayads. One can note that there are literary works that deal exclusively with Ummayad crimes. An example of such work is Risala fi bani Umayya, written by al-Jahiz. Often did the writers quote a verse of Qu'ran the tree cursed in Qu'ran (ash-shajara al-malʿuna fi 'l-qur'an; XVII, 60) when referring to them.
As one may know, the Hashimid branch (to which Muhammad ibn ʿAbd Allah belonged) and the Umayyad branch of the Quraysh tribe were enemies before Muhammad ibn ʿAbd Allah was born (I believe many know of the story of siamese twins, Hashim and ʿAbd Shams, sons of ʿAbd Manaf). Therefore, Abu Sufyan, the father of the first Umayyad caliph (Caliph Muʿawiya I), was shown as the leader of the Quraysh opposition to Islam in Mecca, and the leader of the armies that attacked the army of Medina at Uhud and al-Khandaq. One might know the story by which mother of Caliph Muʿawiya (Hind bint ʿUtba) chewed on the liver of Hamza ibn ʿAbd al-Muttalib, who was an uncle of Muhammad ibn Abd ʿAllah, at Uhud. The Umayyads were referred to as at-tulaka', which was supposed to denote that they became property of Muhammad ibn Abd ʿAllah after he conquered Mecca, but he magnanimously chose to set them free (tulaka' meaning liberated ones). Muʿawiya is frequently shown as a scheming and capable man, whose role in murder of Caliph ʿUthman is often implied.
As for the refusal of the title of caliph, there are multiple alleged reasons. The first and foremost reason was that Muʿawiya introduced hereditary succession, which was contrary to the established practice of electoral Caliphate, like in Rashidun times. It is also ascribed to the Umayyads that they changed the title of khalifat rasul allah (deputy of the God's Prophet) into khalifat allah (Deputy of God), motivated by arrogance.
Individual caliphs, like Caliph Yazid, were known as drunkards. Caliph Yazid was infamous for bombarding Medina and Mecca with catapults, practically destroying Kaʿba. An Umayyad army besieged Mecca ten years later (led by al-Hajjaj ibn Yusuf). Killing of al-Husayn ibn ʿAli, the grandson of Muhammad ibn ʿAbd Allah, as well as the largest part of his family and supporters in Karbala' is usually ascribed to the Umayyads.
__________________________________________________________________________
My principal source for all of this is: G. R. Hawting, "Umayyads", The Encyclopaedia of Islam. Volume X: T-U, Leiden-Brill 2000, pp. 840-847.