As I understand it, building:levels is the number of levels, not the index of the highest level. In your example, level=23 in a building:levels=20 building does not bother me. For example, in France and other countries, the ground level is level 0, therefore building:levels should be 1 + the index of the highest level.
I think the level tag, when possible, should contain the level index as indicated in the building.