I have a 30MB .osm file with about 205k nodes. JOSM is terribly slow (with -xmms1024) and actually unusable…
The slowness is normally caused by rendering.
If you zoom in enough, it should be as fast as always. In addition there are filters to disable drawing of certain features, e.g. all untagged nodes. Make sure to set the “hide” flag of the filter, so the objects don’t appear as a “shadow”, but are actually skipped.
Also disable painting of areas or use a map style that doesn’t draw areas.