Short answer is that it is problematic.

It’s rather suspicious that Naofumi’s party members are slaves and pets. The audience can say at a glance that when you decide to become the Shield Hero’s slave you gain more power. On the surface that seems alright doesn’t it?
So there are a few problems with the logic behind it. As viewers we need to remember the reasons why Naofumi got his first slave Raphtalia to begin with. The shield hero was falsely accused of rape, and adventurers refused to join his party. Now this would be fine, the shield hero could solo potentially, but the shield hero lacks the ability to actually damage his targets. So as a last resort Naofumi was forced to buy a weak slave, demihuman girl, and force her to fight his battles. A slave could not refuse Naofumi’s order to fight without it inflicting harm on the slave.
While it is morally wrong for Naofumi to own a slave, especially by modern standards, there is justification within the context of the story for him needing to resort to immoral methods.
While Naofumi ended up taking care of his slave and making her stronger in the process, it did prove that he could care for the needs of himself and his party. But due to political intrigue Naofumi was humiliated and his slave was freed by the order of the King. When all others would not help Naofumi, Raphtalia did. She could see that there was more to Naofumi than met the eye. As a demihuman Raphtalia could understand Naofumi’s rejection by others, since she had experienced it herself. She is the only one who can at that moment in time care for Naofumi’s emotional needs.
But why does Raphtalia agree to become Naofumi’s slave again?
So this is where things become murky. Naofumi can’t accept that Raphtalia would help him altruistically. In his emotional breakdown he doesn’t think she would willingly help him. In his mind, she would only help him to betray him again. His illogic flies in the face of her actions. She literally stood up for him and pledged to help him. No others would do this for Naofumi. And due to Naofumi’s deep distrust of others, Raphtalia decides to become his slave again to earn his trust. She feels indebted to him to do so.
So this story becomes more problematic. The shield hero is incapable of trusting anyone. And while there were many instances of people he could not rely on, there were also cases of people who did help Naofumi. And instead of thinking of the people who were reliably and trustworthy to him, primarily Raphtalia, his distrust forces her hand to go to return to slavery. The shield hero is a moral failure in this regard, and it does strengthen the argument that the shield hero is scum. The silver lining here is that Naofumi was driven to it in part because of the horrible actions of others, so he can’t be completely condemned for breaking down in an unfair process. Naofumi’s inability to trust others unless he can 100% control their actions is a problem.
Granted after Raphtalia decides to become his slave, Naofumi does return to a state of rationality and normalcy. He does after all choose to trust the Queen, but only after she effectively strips the king and princess of their positions and titles. While the shield hero on the surface is the come back story of an underdog, there are big question marks regarding Naofumi and if he is ought to be the hero we should be cheering for.
The Rising of the Shield Hero: How does the Second Anime Season Compare to the Light Novels?
Hero’s Dilemma – Episode 23 Rising of the Shield Hero
Actually Impossible – Rising of the Shield Hero: Episode 17 A Promise Made