There can be different sources of OSM data - the most complete is the whole planet, but if you need just daily dumps of several countries (or the whole continent), you can use place like this:
http://download.geofabrik.de/africa.html
and filter only the data you need, for example using Osmosis ( https://wiki.openstreetmap.org/wiki/Osmosis ).
It should be quite clear to what country does such building belongs just by checking if it’s located inside country area and probably the same for states. Things get tricky when coming down to the street level. Some towns are not tagged as the administrative border, so you can only guess where the building belongs to. And definitely streets are just lines, so there’s no way to tell if it belongs to this street if there is no address data added to the building, in general.
For example this building:
https://www.openstreetmap.org/way/181530441
lies in Nigeria for sure, also it is in Federal Capital Territory:
https://www.openstreetmap.org/relation/3717259
but Abuja is tagged only as a node:
https://www.openstreetmap.org/node/31203257
So it’s hard to say if it is inside the city in the automated way. And you can’t say if it belongs to this street or another one:
https://www.openstreetmap.org/way/582886061#map=18/9.08003/7.48735
https://www.openstreetmap.org/way/169164687#map=18/9.07988/7.48721
You can also rely on what reverse geocoding engines are saying, for example Nominatim says:
Peniel Apartments, Adetokunbo Ademola Crescent, Wuse District, Abudża, Municipal Area Council, Federal Capital Territory, 900001, Nigeria
However it looks like a bit of guessing, because Wuse District is also a node and it is not tagged as part of Abuja:
https://www.openstreetmap.org/node/31384766
and the street is just close to the building and it might be the other street as well, as I have shown above.
Here you can look at what admin boundaries are available for different countries in OSM:
https://wambachers-osm.website/boundaries/
And, depending on your needs, you might just use Nominatim API (be it on this server or have your own deployment):
https://nominatim.org/release-docs/latest/api/Reverse/
https://wiki.openstreetmap.org/wiki/Nominatim