How convert OSM to OBF with command line in OSMAndMapCreator new versions

chris38 · October 27, 2023, 10:04pm

have two more questions about this topic :

Could you update this answer : It seems net.osmand.data.index.IndexBatchCreator no more exists in 2023 versions
I noticed that the software increased from 2014 to 2023 from a size of 24MB to 189MB. What are the reasons for such great growth when the software of 2014 made it possible to do everything that the software of 2023?

thanks a lot for any answer

whb · November 3, 2023, 8:17pm

It is possible to generate the POI-OBF and the MAP-OBF separately and then merge them:

OsmAndMapCreator/utilities.sh generate-map ~/Downloads/my.osm.pbf --poi-types=/home/<user>/Downloads/poi_types.xml && mv My.obf My-MAP.obf && OsmAndMapCreator/utilities.sh generate-poi ~/Downloads/my.osm.pbf --poi-types=/home/<user>/Downloads/poi_types.xml && mv My.obf My-POI.obf && OsmAndMapCreator/utilities.sh merge-flat-obf My-MAP_and_POI.obf My-POI.obf My-MAP.obf && OsmAndMapCreator/inspector.sh My-MAP_and_POI.obf

whb · October 27, 2023, 10:50pm

Please have a look at:
Create Offline Raster & Vector Maps | OsmAnd
The utilities.sh script is include in OsmAndMapCreator.

OsmAndMapCreator contains essentially all the functions that OsmAnd also contains, such as routing, rendering, and additionally map creation, so it is also at least as big as OsmAnd.
As the size of OsmAnd increased, so did OsmAndMapCreator.

chris38 · October 28, 2023, 10:13pm

I had already seen this page which is very succinct and refers to the code for more detail, code which I had briefly browsed and seen among other things a “generate-obf-files-in-batch” option which I tested ( keeping the same structure for the XML file as in previous versions) but without success (file generated but empty). I also tested by launching “utilities.sh generate-poi” but then the question arises of knowing where to put the OSM sources and where the OBF destinations arrive.
In short, a lot of questions for a simple OSM to OBF conversion of POI.
Examining all the code (if I can grasp all the mysteries) is not an option for me

whb · November 2, 2023, 2:21pm

There is nothing special about it.

mkdir ~/pbf_download
cd ~/pbf_download
wget https://download.geofabrik.de/europe/malta-latest.osm.pbf
mkdir ~/oamc_test
cd ~/oamc_test
OsmAndMapCreator/utilities.sh generate-poi ~/pbf_download/malta-latest.osm.pbf

The obf file is created in the current working directory:
~/oamc_test/Malta-latest.obf

Java 11 recommended.

chris38 · November 2, 2023, 3:51pm

Thank you for your reply. This allows me to run the commands without being in the fog because the example taken in the documentation (OsmAndMapCreator/utilities.sh generate-poi albania_europe.pbf --chars-build-poi-nameindex=3) was not very explicit (was ‘albania_europe.pbf’ a source or a destination?).
However my tests are still not conclusive:
I took a simple OSM test.osm file (which generates an OBF without problem with the previous version of osmandmapcreator) below


<?xml version='1.0' encoding='UTF-8'?>
<osm version='0.5' generator='TopoCad 6.23'>
  <node id='-1' visible='true' lat='45.3814634071118021552' lon='4.82919853794681674941'>
    <tag k='name' v='DT0810027'/>
    <tag k='user_defined' v='344'/>
  </node>
  <node id='-2' visible='true' lat='45.3816080370158445589' lon='4.8220665502973929506'>
    <tag k='name' v='DT0810047'/>
    <tag k='user_defined' v='344'/>
  </node>
  <node id='-3' visible='true' lat='45.3947353268311175611' lon='4.80709102324775283677'>
    <tag k='name' v='PC0810023'/>
    <tag k='user_defined' v='344'/>
  </node>
</osm>

The command produces an empty 54 byte file. For greater safety, I took another test2.osm file with the same structure of almost 200 points and this produced a file of 55 bytes, all with no error announced.
I specify that the file is indeed UTF8 and I tested with LF and CR+LF line endings which gives the same result.
I don’t think it’s the OSM file generator that is the cause since the generated files work very well with the previous version.

A curiosity also exists: the generated OBF file sees the first letter of its name transformed into capital letter (?)

Thank you in advance for any information that could resolve this issue.

Best regards

PS: java version is
openjdk 11.0.20.1 2023-08-24
OpenJDK Runtime Environment (build 11.0.20.1+1-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.20.1+1-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)

whb · November 2, 2023, 7:47pm

Is a modified rendering_types.xml used with the previous version?
I’m not sure if user_defined POIs will work without a modified rendering_types.xml.

https://raw.githubusercontent.com/osmandapp/OsmAnd-resources/master/obf_creation/rendering_types.xml:

<category name="user_defined" >
	<!-- Here user defined categories could be added -->
	<!-- <type tag="user_defined" value="wsl" minzoom="10" /> -->
</category>

Yes, the OBF file name begins with a capital letter. For whatever reason.

chris38 · November 2, 2023, 8:13pm

Hum… I don’t see what connection there could be with the type of rendering. And there’s no rendering_types.xml file in osmandmapcreator (only .jar files and only in root .bat and .sh files with one batch.xml file) ??

PS : there’s only in the batch.xml file a line renderingTypesFile=“” both the same for the old and new version…
PS2 : And the same file without tag user_defined give the same bad result…

whb · November 2, 2023, 8:49pm

lib/OsmAnd-java-master-snapshot.jar contains the rendering_types.xml.
An external rendering_types.xml can be specified using the --rendering-types= parameter:
Create Offline Raster & Vector Maps | OsmAnd
Only types that are defined in the rendering_types.xml are included in the OBF file.

chris38 · November 2, 2023, 9:37pm

OK ! It’s plausible. I have not modified the rendering_types.xml but it is possible that the default version has been modified making the two versions incompatible… However, this does not explain why if I provide it with the same file without the lines with the ’ tag’ ‘user_defined’, I still don’t have any results (?)
And how then can we create a map with a particular ‘tag’? By modifying the rendering_types.xml file and provide it to the conversion tool?

PS: If I translate this ‘user_defined’ OSM file with old version of osmandmapcreator and give it to the actual version of osmand (with his default rendering_types I have not modified), I can see POI “User defined” with all names and tags …

whb · November 3, 2023, 9:38am

A modified poi_types.xml is required, the rendering_types.xml is irrelevant here.
https://raw.githubusercontent.com/osmandapp/OsmAnd-resources/master/poi/poi_types.xml

For example, this line could be added to the user_defined_other category:
<poi_type name="my_custom_poi" tag="user_defined" value="344" no_edit="true"/>

(...)
		<poi_category name="user_defined_other" default_tag="amenity" no_edit="true" excluded_poi_additional_category="fee,payment_type,wheelchair_accessibility,internet_access_type,opening_hours">
			<!-- Add additional combination for UK postcode POI files-->
			<poi_type name="user_defined_other_postcode" tag="user_defined_other" value="postcode" no_edit="true"/>
			<poi_type name="my_custom_poi" tag="user_defined" value="344" no_edit="true"/>
		</poi_category>
(...)

Generate the OBF using the modified poi_types.xml:
OsmAndMapCreator/utilities.sh generate-poi ~/Downloads/test.osm --poi-types=/home/user/Downloads/poi_types.xml

The OBF can now be inspected with the OAMC inspector:
OsmAndMapCreator/inspector.sh -vpoi ~/osmand/Test.obf

The user-defined POIs should be listed:

Binary index Test.obf version = 2 edition = Fri Nov 03 12:00:00 CET 2023
1 Poi data Test - 398 bytes
	Region: Test
		Bounds (left top - right bottom) : 4.8071, 45.3947 NE - 4.8292, 45.3815 NE
		Categories:
			user_defined_other
				my_custom_poi
		Subtypes:
			name text

user_defined_other: my_custom_poi DT0810027 Lat 45.381466 Lon 4.829194 osmid=-1 
user_defined_other: my_custom_poi DT0810047 Lat 45.38161 Lon 4.8220644 osmid=-2 
user_defined_other: my_custom_poi PC0810023 Lat 45.394737 Lon 4.807087 osmid=-3

chris38 · November 3, 2023, 1:29pm

I begin to understand :
The old procedure allowed (thanks to batch.xml) to generate any OBF with the possibility of generating routing or not, the same for POI, Map…etc.
The new procedure (utilities.sh) seems to only be able to generate POIs (generate-poi) or OBFs (generate-obf but what it contents?) so it is logical that the POIs are “framed” according to the existing types and therefore that, if we provides an OSM for which no POI type matches, it generates an empty file.
But then, how can we generate an OBF in which we don’t want any routing, just in order to have a map?
The advantage of the old procedure was that it allowed you to generate a map with “user_defined” POIs in a single command.
I will first try to generate these POIs with the method you suggest for the new version.

Thanks again for this information.

PS: I see there’s an “generate-map” option : can we chain commands in one line like utilities.sh generate-map generate-poi osmfile.osm --poi-types=./my_poi_types.xml ?

chris38 · November 3, 2023, 1:49pm

I just saw that the sequence of commands on a single line seems impossible.
However, I have a question: a batch.xml file still exists in the new version. How then to use it?

whb · November 3, 2023, 2:37pm

Maybe:
generate-obf-files-in-batch <path to batch.xml>

All indexes: Create Offline Raster & Vector Maps | OsmAnd

Use the GUI (all combinations possible), try generate-obf-files-in-batch or modify code:

github.com

osmandapp/OsmAnd-tools/blob/master/java-tools/OsmAndMapCreatorUtilities/src/main/java/net/osmand/MainUtilities.java#L64


      
          import net.osmand.util.Algorithms;
          import net.osmand.util.CombineSRTMIntoFile;
          import net.osmand.util.ConvertLargeRasterSqliteIntoRegions;
          import net.osmand.util.CountryOcbfGeneration;
          import net.osmand.util.FixBasemapRoads;
          import net.osmand.util.GenerateExtractScript;
          import net.osmand.util.IndexBatchCreator;
          import net.osmand.util.IndexUploader;
          import net.osmand.util.ResourceDeleter;
          import net.osmand.util.TileListsForRegions;
          import net.osmand.util.WeatherPrepareRasterSqliteRegions;
          import net.osmand.wiki.WikiDatabasePreparation;
          import net.osmand.wiki.WikipediaByCountryDivider;
          import software.amazon.awssdk.core.sync.RequestBody;
          import software.amazon.awssdk.services.s3.S3Client;
          import software.amazon.awssdk.services.s3.model.PutObjectRequest;
          
          public class MainUtilities {
          	private static Log log = PlatformUtil.getLog(MainUtilities.class);
          
          	public static void main(String[] args) {

generate-map

Maybe that works with merge-index (merge map and poi).

chris38 · November 3, 2023, 4:36pm

generate-obf-files-in-batch : no, this throws an error.
according to the code this should be it, but at this time, there seems to be a bug (exception)

so we will not know how and if we can use this batch file…
and new version can only do one thing at a time from command line

thank you for all those informations

best regards

chris38 · November 3, 2023, 9:37pm

yes