To remove apostophe what should I do?

[Varun_28]

Which is preferable to modify the ICU normalization rule to remove apostrophes during search instead of changing the python query preprocessing?

[InsertUser]

There are 125 pages in the Python category on the OSM wiki and a score of libraries mentioned on the Software Libraries page that reference Python.

I suspect it would be easier for people to help with some context about what you’re using and what you’re trying to do.

[Varun_28]

I am studying about nominatim and investigating how apostrophes are handled in search normlization. Currently, punctuations are handled as - ,so peter’s is read as peter s instead of peters. I would like to understand the recommended way to adjust so both are treated equally..

[SomeoneElse (Andy Townsend)]

@Varun_28

I would like to understand the recommended way to adjust so both are treated equally

It depends what you want to do. When I needed Nominatim to do something different I just treated it as a black box and modified the input data. That won’t help if your goal is to “understand what Nominatim does internally” rather than “achieve a desired result”.

[Varun_28]

I am trying to solve issue #3703 so that words words ending with s and ‘s are treated equally. But I have a doubt in which will be the correct place in the pipeline to implement this which is helpful for both import and query processing.

[Vinzenz_Mai]

@Varun_28

to solve issue #3703

Here’s the link to this issue:
Ignore apostrophes when searching · Issue #3703 · osm-search/Nominatim · GitHub with thoughts of @lonvia about the options to solve this problem:

Currently the apostroph is already removed during normalization and replaced with a “token space”. That somewhat limits our options.

Option 1: contract 's to s during normalization in indexing and query time. That means “Saintbury’s” and “Saintburys” will work but “Saintbury” will stop working. I’m sure there are cases where that is inconvenient. It surely wouldn’t be a good solution for French articles.

Option 2: create variants during indexing time. We’d having “Saintbury’s” and “Saintburys” and could find all three variants at the expense of making our already huge word list even larger. Similar to the suggestions in #3578, just the reverse variant creation process.

Option 3: create variants during search time. That’s tricky because to have a general approach for the genitive s, you’d have to create variants for every word that ends in s.

I added some tags to this topic too.

[Varun_28]

Thanks for outlining the options. For option 2 ,is it efficient to store more versions of the search key during indexing compare to changing the search logic?