This isn’t at all difficult. Robert Heinlein explained it more than fifty years ago:
Political tags – such as royalist, communist, democrat, populist, fascist, liberal, conservative, and so forth – are never basic criteria. The human race divides politically into those who want people to be controlled and those who have no such desire.
Libertarians are the second sort, and they understand that allowing others’ freedom to be diminished will in time cost them their own.