You can try to allocate more space to josm using the -Xmx call, but ultimately you can’t open really big maps in JOSM no matter what you try. The biggest “usable” large map I’ve opened is a 13 Mb bz2 file (not sure how big that is uncompressed).

For example,


java -Xmx1500m -jar josm-tested.jar 

will allocate 1.5 Gb of heap space to JOSM. Obviously your system must have this amount of RAM to start with.