I am formerly Catholic and I can distinctly remember things being said about the immoral actions carried out by the church. These things were quickly dismissed as I concluded as most do, it’s the radicals.
The issue is that the majority of religious people see the good that religion does, while rejecting the bad. The bad are usually just attributed to radicals. Although it can be reasonably argued that the problem with radicals is that they are following their holy book too closely, there is another issue.
The good that comes from religion can be (and is often) done without religion. There is no act of good that a religious person can commit, that I could not. On the other hand, evil acts committed by religion are almost entirely predicated by the belief in the supernatural.
Think of an evil action committed by a religious group. Whether it be suicide bombing, genital mutilation, or the denial of contraceptives, such actions lose place when taken out of a religious context.
What this means is that if religion is taken out of the picture, we could recognize kind actions previously attributed to religion, simply as humanistic values.
If you believe that you or others would give up your actions of kindness as a result of losing your religion, then I believe your moral character is suspect.