Mapnik uses Cairo or AGG for rendering (I guess one can choose) [1]. Both are able to handle the same functions with another output (PNG, SVG, PDF, …)
Osmarender uses XSLT. Therefore the OSM-XML is directly transoformed into an SVG (which ist XML as well). After that Tiles@Home (which produces the osmarender-layer) uses Inkscape [2] for rendering PNG-tiles out of the SVGs.


[1] http://trac.mapnik.org/wiki/MapnikRenderers
[2] http://www.inkscape.org

Hope this helps,
Malte