The sage’s major beef with the duke was the disrespect he showed to King Xiang of Zhou after restoring him to the throne after he was deposed by his brother. He regarded the summons Duke Wen gave to the king to attend a hunt with him at his own court as a flagrant breach of ritual propriety. Even if he was the most powerful man in the kingdom, the duke was still subordinate to the Zhou king and had no right to ask or tell his sovereign to do anything. The summons was an unforgivable display of arrogance and a blatant attempt to usurp the legitimacy of the Zhou court.
By riding roughshod over the sovereignty of the king, Duke Wen was showing the world that he was the true power in the land and challenging the very foundations that the Zhou dynasty was built on. If the duke was allowed to get away with his egregious misconduct, others would follow his example and the Zhou kingdom would collapse into chaos.
Even though Confucius may have been technically correct in his condemnation of Duke Wen for his admittedly serious violation of the rules of propriety, his singling out of just one ruler for criticism suggests an unwillingness or inability on his part to acknowledge the harsh reality of his age. The Zhou dynasty had already entered its death spiral thanks to a collective breakdown of traditional norms and values rather than as a result of the actions of a single individual and it was already too late to save it.