Syria's new government has formed a committee to investigate reported mass revenge killings of a minority group - after the United Nations human rights chief called for "swift actions".