I think you have it right. We must always pray for people even if we do not feel led to do it. When we know of people who suffer or know of people in need, we not only have to pray for them, but we must also help them where we can, without feeling condemned if we don’t.
We have our own lives to live and we help where we can. We should not feel condemned about this unless God specifically led us to pray for or to help people and we refuse to do so or fail to do so.