Hello thug_n_g, I don´t believe most dutch houses look alike. I believe, we need some thousands of textures for this reason.
Maybe something about history: 1995-1998 i developed one of fisrt german 3D city models (Hamburg) and was scientifical consultant of German government ( I´m from Poland).
One of 2 first city models in Google Earth is even Hamburg. After them i developed the first 3D navigation systems ( see: http://www.spiegel.de/fotostrecke/fotostrecke-19901.html - there are some of 1600 (not 1400) 3d models which my staff have made).

Now we have also Google street view and I believe, we can make it better than Google, because they have only automaticly added pictures and we have the possibility to generate true, textured vector data. Of course, the expected amount of data is damn big, but tis is an other problem.
We ( some OSM user from Germany, Poland and Russia) are working on 3D software with the possibility of multitexturing ( e.g.: http://wiki.delphigl.com/index.php/Tutorial_MultiTexturing )by which is mean putting of Elements like typical windows, doors and other elements over background (like brick). With this approach the library is limited to some hundert of basic elements.

You wrote already: You´re not alone, so I belive it is possible to make it. Of course, at teh Beginning it sounds crazy, but OSM was definitly crazy too at the beginning. In my eyes is the impementation of new ideas logical way of innovative projects.
Well, the firs step is the collection of textures. Could You help?
Cheers,
Marek