Please pay attention to the following points:

Firstly: your claim about that Prophet and his successors appointed representatives in cities so that people learn their religion from them, does not have any base; because they did not appoint representatives for this purpose, rather they appointed representatives only to handle taxes and to collect Zakat and to establish prayers and to confront the enemy and to fulfil their commands and hence, many of their representatives were not the most knowledgeable people in religion and some of them gravitated to betrayal and were reprimanded or fired by them. Therefore, your criticism is invalid from the very first hypothesis.

Yes indeed it was possible that some individuals have been praised by prophet and his Caliphs for being truthfull in narrating from them, but even such individuals were not considered as their representatives such that people learn their religion from them, rather they were just considered as trustworthy narrators whose narrations from prophet and his Caliphs were authentic for people of their time, with due regards to the fact that their personal opinions had no validity for them nor they had necessarily a narration for every single needed issue such that they could satisfy the need to refer to the prophet and his Caliphs. Because verification of authenticity of these narrators by prophet and his Caliphs could only mean that they do not intentionally attribute lies to the prophet and his Caliphs and hence if they narrate an act or a saying from them during the time of their lives, one can trust their narrations as long as there is access to them.

Secondly the responsibility of those people who were in distant locations from the Prophet and his successors was to migrate toward them in first place; as God has said: ﴿إِنَّ الَّذِينَ آمَنُوا وَهَاجَرُوا وَجَاهَدُوا بِأَمْوَالِهِمْ وَأَنْفُسِهِمْ فِي سَبِيلِ اللَّهِ وَالَّذِينَ آوَوْا وَنَصَرُوا أُولَئِكَ بَعْضُهُمْ أَوْلِيَاءُ بَعْضٍ وَالَّذِينَ آمَنُوا وَلَمْ يُهَاجِرُوا مَا لَكُمْ مِنْ وَلَايَتِهِمْ مِنْ شَيْءٍ حَتَّى يُهَاجِرُوا﴾[1]; “Indeed those who believed and migrated and fought for the Cause of God with their wealth and their lives in the path of God and those who sheltered and those who helped, shall be each other’s friends and those who believed but did not migrate, you have no friendship with them unless they migrate” and has said: ﴿إِنَّ الَّذِينَ تَوَفَّاهُمُ الْمَلَائِكَةُ ظَالِمِي أَنْفُسِهِمْ قَالُوا فِيمَ كُنْتُمْ قَالُوا كُنَّا مُسْتَضْعَفِينَ فِي الْأَرْضِ قَالُوا أَلَمْ تَكُنْ أَرْضُ اللَّهِ وَاسِعَةً فَتُهَاجِرُوا فِيهَا فَأُولَئِكَ مَأْوَاهُمْ جَهَنَّمُ وَسَاءَتْ مَصِيرًا﴾[2]; “Indeed those whose lives are taken by angels while they are oppressors to themselves, will hear angels saying: “What status were you in?” They will reply “We were oppressed in the land” (such that we did not have the possibility to learn about beliefs and commandments of religion) They (the angels) will say: “Was the earth of God not wide enough for you to migrate in it?” Their shelter will be hell, and that is a miserable arrival”! But in second place those people who were not truly at fault because of their failure and could not migrate were responsible to dispatch a party from themselves to the Prophet and his successors to learn about beliefs and commandments of religion; as God has said: ﴿إِلَّا الْمُسْتَضْعَفِينَ مِنَ الرِّجَالِ وَالنِّسَاءِ وَالْوِلْدَانِ لَا يَسْتَطِيعُونَ حِيلَةً وَلَا يَهْتَدُونَ سَبِيلًا﴾[3]; “Except for those men, women, and children who are weakened and find no path” and has said: ﴿وَمَا كَانَ الْمُؤْمِنُونَ لِيَنْفِرُوا كَافَّةً فَلَوْلَا نَفَرَ مِنْ كُلِّ فِرْقَةٍ مِنْهُمْ طَائِفَةٌ لِيَتَفَقَّهُوا فِي الدِّينِ وَلِيُنْذِرُوا قَوْمَهُمْ إِذَا رَجَعُوا إِلَيْهِمْ لَعَلَّهُمْ يَحْذَرُونَ﴾[4]; “And the believers should not go forth altogether, rather why (at least) a party from each group did not migrate so that they achieve knowledge about religion and warn their people when they return to their nation so that they may beware?!” and it is clear that narration of this “party” on one hand is not a Wahid narration and on the other hand it is only valid for their people who are at their time and not for those who come after them in future centuries; because their responsibility is also to migrate to the successor of their time or to dispatch a party from themselves to him and this is a rule that runs up to the day of judgment.

To conclude, no one can be a reference for people to learn their religion except for an infallible person and it is inevitable that any reference other than him should lead to him with certitude and this is a rule that absolutely cannot be allocated.