Utilitarian Calculations in non-Utilitarianism

Does our ethical judgment always rest on calculation of what is most useful/ most good? Even in decisions involving rights to assign to (all) people, do we not decide in our minds the value to society of that right and compare it to that same society without that right. We are dealing with a quasi- mathematical process I think. We are comparing quantities of good/utility. Even in the assessment of rights. Our minds are performing these types of calculations quite a lot in our survival - what do I do now to be safe/healthy/wise? It should not seem so implausible that calculation is at the center of ethical thinking itself.

If calculation is the common process in all ethical thinking, why can we not reduce all ethical thinking - including thinking about rights - to utilitarian calculations. What else is being decided when we examine the usefulness to society of say, free speech. In societies and thinking that champion rights, are these rights not chosen for their benefit to society and the individual as well. How is this different than the utilitarian approach? When we calculate the good of free speech, are we not considering its value to society and the individuals of that society?

The reason I want to reduce rights-based thinking to utilitarian thinking is based on my greater confidence in utilitarianism.