Android Question OpenStreetMap using .PBF?

[adrianstanescu85]

Hello,

I'd like to ask if there's a solution to use the maps from OpenStreetMap project using directly the .PBF files (offline) that are freely available for download.

For instance, http://download.geofabrik.de/europe.html has every country in Europe being updated daily in that vector format so using such a file would give an immense advantage over the OsmDroid tiles approach, which may be downloaded or generated but take up a huge amount of space and time.

I would also like to ask if using the alternative SHP or OSM formats is possible in a similar way, if PBF is not (which I would expect).

Any info over this please?

Thank you!
Adrian

 

[warwound]

Hi there.

You can convert an OSM .pbf file to a MapsForge .map file and use the MapsForgeTileProvider to display tiles on a GoogleMap.

You can also do much the same using an OSMDroid map, i've not uploaded the library that is required to do this to the forum yet, but you can download the library and an example project from here: http://b4a.martinpearman.co.uk/osmdroid/MapsForgeTileSource.zip.

A MapsForge .map file is a map database, and you can either:

• Download map databases from: http://download.mapsforge.org/maps/.
• Create your own map databases using the Mapsforge Map-Writer.
  

  The plugin requires an input stream of valid OSM data. Use for example the MySQL, PostGIS, XML or PBF-Binary tasks of Osmosis to read OSM data.

  If you have map vector data that is in a format not directly supported by the Map-Writer you might be able to convert it to a format that is supported and then create a map database from it.
  Have a read through the MapsForge Google Group: https://groups.google.com/forum/?fromgroups&hl=en#!forum/mapsforge-dev.

Martin.

 

[adrianstanescu85]

Martin, thank you for your extremely welcome reply!

I downloaded your sample but when enabling the MapsForge library included there, I get the following error message:

What should I do to make that work?

I do have to say I'm using B4A 3.20 and I do have the OSMDroid 3.0.8 pre-enabled for the project.

Adrian

 

[warwound]

Ah yes!
I made a mistake with a recent compilation of the GoogleMaps MapsForgeTileProvider library and the xml file causes this error.
I fixed the problem but must have uploaded the broken version of the GoogleMaps MapsForgeTileProvider instead of the OSMDroid MapsForgeTileSource!

First if you delete the already downloaded archive MapsForgeTileSource.zip, now download the fixed archive.
Same link as before: http://b4a.martinpearman.co.uk/osmdroid/MapsForgeTileSource.zip

Martin.

 

[adrianstanescu85]

Thank you, that works perfectly now!!

I tried converting a PBF file to MAP with Mapsforge Map-Writer (i.e. installed the mapwriter plugin to Osmosis) and it outputs an error. Is it possible their conversion tool is faulty?
I took the PBF file from http://download.geofabrik.de/europe.html (being the smalles size among them) and renamed it to monaco.osm.pbf for easier use.

Thank you!

 

[warwound]

I've always found that setting up the Map-Writer took a little time and patience to fix the bugs.
A lot depends on which operating system you're using and which instructions you used to install both osmosis and the Map-Writer.

Here on Ubuntu 12.04 for example installation of both osmosis and the Map-Writer always need the fix described here: http://code.google.com/p/mapsforge/... Status Priority Stars Owner Summary Modified, after that it works perfectly.

It's been a long while since i tried to run the Map-Writer on Windows - and i can see you're using Windows.
Have a read through the reported issues http://code.google.com/p/mapsforge/...rity+Stars+Owner+Summary+Modified&cells=tiles for 'cannot load jpf-plugin'.

A fix is reported here: https://code.google.com/p/mapsforge/issues/detail?id=439.

Martin.

 

[warwound]

One interesting thing though is that I replaced

Dim MapsForgeDatabaseName As String="kings_lynn.map"

with

Dim MapsForgeDatabaseName As String="monaco.map"

and added the monaco.map file from http://download.mapsforge.org/maps/europe/ to the same folder as the original map. Then reinstalled the app.

However, after a large zoom out and seeing a big white square on the map that would correspond to Monaco, when zooming back in.. that square is big white space, no info inside. Is their map misfit? Or am I doing something wrong?

Thank you!

Did you see map tiles before zooming out?
I've found that sometimes a map database can contain geometries that cause MapsForge to crash but only when the area that contains the problem geometry is loaded.
That is - the loaded map database renders fine until an area containing a problem geometry is loaded - after that the tile rendering generally fails for all areas.

It's a know bug on the MapsForge issues page, here's a typical report: https://code.google.com/p/mapsforge... Status Priority Stars Owner Summary Modified.

I'm not 100% sure if your monaco.map file contains an invalid geometry or not but would suspect that to be the cause.
These bugs originate from the raw open street map data - that data changes often and occasionally buggy geometries get included.

I've just downloaded monaco-latest.osm.pbf and converted it to monaco.map and tested on the demo project - zooming in and out repeatedly everything works fine.
I'd say that the pbf file i downloaded did not contain any invalid geometries and that the monaco.map file that you downloaded was created using data that did contain invalid geometries.

I've attached the working monaco.map file.

Martin.

 

[adrianstanescu85]

Martin, I got it all solved, it seemed that Monaco was very small and from very above it took a lot of zooming in.

I'll try to convert the maps conversions under Ubuntu right away.

So far I've succeeded with other maps from http://download.mapsforge.org/maps/europe/ and they work fine, which is really great!

One question though: the actual rendering is somewhat different from Mapnik style, i.e. the roads are white not yellow, and I get no traffic info like traffic lights on their spot, driving directions (ex. one way street) and so on. Is there any way to enable such info (like one way streets to be shown)?

Thank you!

 

[adrianstanescu85]

Also, does this support overlays? Like putting a picture/sign over a certain location or so.

 

[warwound]

https://code.google.com/p/mapsforge/wiki/RenderThemeAPI

There are two things to look at if you want to change the default MapsForge tile rendering style:

• When osm data is converted to a map database the Map-Writer only includeds osm data that is defined in the default 'tag-mapping.xml' file:
  http://code.google.com/p/mapsforge/...ge-map-writer/src/main/config/tag-mapping.xml
  

  You need to be aware that this configuration only defines what data is to be included in the map file.

• When tiles are rendered on the device the default 'render theme' is used.
  The render theme defines how data that is in the maps database is rendered on tiles.
  You can read about render themes here: https://code.google.com/p/mapsforge/wiki/RenderThemeAPI.
  The default render theme can be found here: https://code.google.com/p/mapsforge.../src/main/resources/osmarender/osmarender.xml.

The GoogleMaps MapsForgeTileProviderOptions object has a method:

SetRenderTheme (InputStream1 As InputStream) As MapsForgeTileProviderOptions
Optionally set a custom RenderTheme to be used to render tiles.
Default value is Null which means that the built in default RenderTheme will be used.

So you can set your own render theme with GoogleMaps.

The OSMDroid MapsForgeTileSource has no such option (yet!), but i'm sure i could soon add such a feature to the OSMDroid library - let me know if you'd like me to add such an option.

Also, does this support overlays? Like putting a picture/sign over a certain location or so.

OSMDroid version 3.0.8 does not support such a feature.
The new (not yet completed and not uploaded) OSMDroid version 4.1 has support for adding any View to the map.
You can add one or more Labels or ImageViews for example, any View can be anchored to the map.

Until version 4.1 is completed you have another option - create a Marker and set it's icon to an image you want to overlay to the map.

Martin.

 

[adrianstanescu85]

Martin,

The OSMDroid MapsForgeTileSource has no such option (yet!), but i'm sure i could soon add such a feature to the OSMDroid library - let me know if you'd like me to add such an option.

That would be extremely super useful, since there are Android devices that do not allow access to Google services. Among other devices I'm using one like that - the Motorola TC55 - which is an industrial use device and Motorola cut off the access to everything that is from Google. Fortunately I use other devices as well, so I can test options.

The OSMDroid is an excellent replacement, so if you could add such an option (SetRenderTheme) then that would help tremendously!

And I can't wait until version 4.1 comes up, those new features are really quite fantastic and would come in very useful!

Thank you!

 

[warwound]

I'll look at adding the SetRenderTheme method to the OSMDroid library over the next few days.

Meanwhile you might want to read this thread: http://www.b4x.com/android/forum/threads/googlemapsextras.26277/page-6#post-221952.
Forum member scrat reports that using map databases downloaded from http://www.openandromaps.org and making a minor edit to the default render theme fixes the bug where tiles that contain part land and part sea are not rendered correctly.
I've not had time to try and openandromap.org map database yet but it's worth bearing in mind if your application displays part land and part sea tiles.

Martin.

 

[adrianstanescu85]

I didn't know about http://www.openandromaps.org but fortunatelly they have good maps that can also work with OSMDroid (which I've tested).

As for

There are two things to look at if you want to change the default MapsForge tile rendering style:

• When osm data is converted to a map database the Map-Writer only includeds osm data that is defined in the default 'tag-mapping.xml' file:
  http://code.google.com/p/mapsforge/...ge-map-writer/src/main/config/tag-mapping.xml

That brings back the issue with making Map-Writer actually work. Looking back at the fix you pointed at, I modified the plugin.xml which is inside mapsforge-map-writer-0.3.0-jar-with-dependencies.jar (I put that under /bin/plugins) the following way:

<?xml version="1.0" ?>
<!DOCTYPE plugin PUBLIC "-//JPF//Java Plug-in Manifest 1.0" "http://jpf.sourceforge.net/plugin_1_0.dtd">
<plugin id="org.mapsforge.map.mapfile-writer"
    version="0.3.0">

    <requires>
        <!--
            these are plugins we require. Usually only
            org.openstreetmap.osmosis.core.plugin.Core will be required.
        -->
        <import plugin-id="org.openstreetmap.osmosis.core.plugin.Core"
            plugin-version="0.36" reverse-lookup="false" />
    </requires>

to

<?xml version="1.0" ?>
<!DOCTYPE plugin PUBLIC "-//JPF//Java Plug-in Manifest 1.0" "http://jpf.sourceforge.net/plugin_1_0.dtd">
<plugin id="org.mapsforge.map.mapfile-writer"
    version="0.3.0">

    <requires>
        <!--
            these are plugins we require. Usually only
            org.openstreetmap.osmosis.core.plugin.Core will be required.
        -->
        <import plugin-id="org.openstreetmap.osmosis.core.plugin.Core"
            reverse-lookup="true" />
    </requires>

    <!--These are library-files we require including our own code. -->
    <runtime>
        <library id="mycode" path="/" type="code" />
    </runtime>

so in the end when running Osmosis the previous error disappeared and was replaced by:

Which is actually almost disappointing... If only I could have a working Map-Writer then I could also be able to tweak the included osm data in tag-mapping.xml and also be able to generate any conversion from PBF to MAP. I don't know if anyone has a working... whole Osmosis folder I'd say for Windows, or an online tool (I tried the Linux approach and stumbled). If necessary I could add here my Osmosis folder which I'm keeping under C:\Program Files (x86), maybe someone could help...

Thank you!

 

[warwound]

I only use Windows for stuff i can't do in Ubuntu - that's mainly running the B4A and B4J IDEs.
And Ubuntu 12.04 makes so many things easy - including using osmosis and the Map-Writer.

A virtual Ubuntu 12.04 computer using either VMWare Workstation (costs £££s) or VirtualBox (free) would be your best solution.
I could soon help you set up and configure such a virtual computer and once set up it'd make all these command line utilities so much easier to work with.

I've just downloaded the Great Britain map database from http://www.openandromaps.org/ and am gonna give it a test with the render theme modification.
If it works i'll add the render theme modification to both GoogleMaps and OSMDroid MapsForge libraries and release it as an update.

Martin.

 

[adrianstanescu85]

I do have Ubuntu 13 installed alongside Windows 8, is there a clear way to install both Osmosis and Map-Writer there?

Thank you

 

[warwound]

MapsForgeTileSource updated to version 1.10
Completely incomapatible with code written using the previous version but much more configurable:

OSMDroid_3_0_8_MapsForgeTileSource
Comment: MapsForgeTileSource enables you to render a MapsForge map database file as a TileSource on an OSMDroid MapView.
Version: 1.10

• MapDatabase
  Methods:
  • IsInitialized As Boolean
  Properties:
  • MapFileInfo As MapFileInfo [read only]
• MapFileInfo
  Methods:
  • IsInitialized As Boolean
  Properties:
  • BoundingBox As BoundingBoxE6Wrapper [read only]
    Get the bounding box of the area covered by the map database file.
  • Comment As String [read only]
    Get the comment field of the map database.
    The comment field can be set when the map database is created.
    It may not have been set, in which case this method will return an empty String "".
  • CreatedBy As String [read only]
    Get the created by field of the map database.
    The created by field can be set when the map database is created.
    It may not have been set, in which case this method will return an empty String "".
  • FileSize As Long [read only]
    Get the size of the map database, measured in bytes.
  • FileVersion As Int [read only]
    Get the file version of the map database.
  • LanguagePreference As String [read only]
    Get the preferred language for names as defined in ISO 3166-1.
    The preferred language for names can be set when the map database is created.
    It may not have been set, in which case this method will return an empty String "".
  • MapDate As Long [read only]
    Get the date of the map database in milliseconds since January 1, 1970.
  • PoiTags() As Tag [read only]
    Get the POI Tags from the map database.
  • StartPosition As GeoPointWrapper [read only]
    Get the map database start position.
    The start position can be set when the map database is created.
    It may not have been set, in which case this method will return a GeoPoint that is NOT initialized.
  • StartZoomLevel As Int [read only]
    Get the map start zoom level
    The start zoom can be set when the map database is created.
    It may not have been set, in which case this method will return a value of -1.
  • WayTags() As Tag [read only]
    Get the way Tags from the map database.
• MapsForgeTileSource
  Methods:
  • GetMapsForgeTileSourceOptions As MapsForgeTileSourceOptions
    Gets the MapsForgeTileSourceOptions that will be used when the MapsForgeTileSource Initialize method is called.
  • Initialize
    Initialize the MapsForgeTileSource.
    You must have set any MapsForgeTileSourceOptions required options before calling Initialize.
  • IsInitialized As Boolean
  • SetTextScale (TextScale As Float)
    Sets the text scale.
    Default value is 1.
  Properties:
  • MapDatabase As MapDatabase [read only]
• MapsForgeTileSourceOptions
  Methods:
  • SetDebugSettings (DrawTileCoordinates As Boolean, DrawTileFrames As Boolean, HighlightWaterTiles As Boolean) As MapsForgeTileSourceOptions
    Optionally set the DebugSettings.
    Default values are:
    DrawTileCoordinates - False.
    DrawTileFrames - False.
    HighlightWaterTiles - False.
  • SetMapDatabaseFile (Dir As String, FileName As String) As MapsForgeTileSourceOptions
    Set the path to the MapsForge map database file.
  • SetRenderTheme (InputStream1 As InputStream) As MapsForgeTileSourceOptions
    Optionally set a custom RenderTheme to be used to render tiles.
    Default value is Null which means that the built in default RenderTheme will be used.
  • SetTextScale (TextScale As Float) As MapsForgeTileSourceOptions
    Optionally set the text scale.
    Default value is 1.
  • SetTileSourceName (TileSourceName As String) As MapsForgeTileSourceOptions
    Sets the name to use for this tilesource for example "England".
• Tag
  Methods:
  • Equals (Tag1 As Tag) As Boolean
  • IsInitialized As Boolean
  Properties:
  • Key As String [read only]
  • Value As String [read only]

Library files and updated demo project available from:
http://b4a.martinpearman.co.uk/osmdroid/MapsForgeTileSource 20140308.zip

Martin.

 

[warwound]

PS The new version of the MapsForgeTileSource contains the same modified render theme file as the MapsForgeTileProvider update i just uploaded:
http://www.b4x.com/android/forum/threads/googlemapsextras.26277/page-6#post-228681
Fixing the problem with tiles that contain both land and sea when used with an andromaps map database.

martin.

 

[adrianstanescu85]

Absolutely fantastic update! It works right from the start!

Now since RenderThemes is there, I set it up on my OSMDroid project.

This is what I tried:

MapView1.Initialize("")
  
    Dim CurrentTilesSources As List=MapView1.GetTileSources
    If CurrentTilesSources.IndexOf(MapsForgeTileSourceName)=-1 Then
        '    the MapsForgeTileSource has not been added to the MapView so create it and add it to the map
      
        Dim MapsForgeTileSource1 As MapsForgeTileSource
        '    set the required MapsForgeTileSourceOptions MapDatabaseFile and TileSourceName
        MapsForgeTileSource1.GetMapsForgeTileSourceOptions.SetMapDatabaseFile(MapsForgeDatabasePath, MapsForgeDatabaseName)
        MapsForgeTileSource1.GetMapsForgeTileSourceOptions.SetTileSourceName(MapsForgeTileSourceName)
      
        '    example of setting an optional MapsForgeTileSourceOptions
        '    MapsForgeTileSource1.GetMapsForgeTileSourceOptions.SetTextScale(2.0)
        MapsForgeTileSource1.GetMapsForgeTileSourceOptions.SetRenderTheme(File.OpenInput(MapsForgeDatabasePath, "andromaps_light.xml"))
      
        '    now that the required MapsForgeTileSourceOptionshave been set we can call the MapsForgeTileSource Initialize method
        MapsForgeTileSource1.Initialize
        MapView1.AddTileSource(MapsForgeTileSource1)
    End If

I took a RenderTheme (I guess) from inside an archive at http://www.openandromaps.org/en/downloads/europe where they also put the maps that do work with OSMDroid - updated every 4 weeks for Europe (!). Unzipped those to the File.DirRootExternal where I also put the map, and tried it... nothing happens! Actually what does happen is the whole map that otherwise would work fine is now gone. I tried both of the RenderTheme types inside the archive (from themes_locus.zip and from themes_orux_new.zip) with the same result.

Am I doing something wrong when setting the InputStream, or is there anything missing?

Is there a place to get or build RenderThemes? Even a sample to prove it does something new compared to the default Theme would help!

Thank you!

 

[adrianstanescu85]

I tried extracting the osmarender.xml from OSMDroid_3_0_8_MapsForgeTileSource.jar together with its two corresponding folders ("patterns" and "symbols") and used this arrangement like this:

MapsForgeTileSource1.GetMapsForgeTileSourceOptions.SetRenderTheme(File.OpenInput(MapsForgeDatabasePath, "osmarender.xml"))

So it does work, that is if I'm trying to modify colors, line sizes, font sizes, zoom levels and so on.

What it does not work is changing the patterns and symbols from the two folders. I've updated some files with the same size and filename, just to see if it works, also adjusted the path for them in osmarender.xml like:
from

<area src="jar:/assets/rendertheme/patterns/military.png" stroke="#e4e4e4" stroke-width="0.2" />

to

<area src="/patterns/military.png" stroke="#e4e4e4" stroke-width="0.2" />

When I set this modification, the map doesn't load at all. I figure this happens because the xml calls for a file (military.png) that was not called/loaded as an InputStream. I may be very wrong about this, too. So far I've tried all combinations I could think of, and the only way to change a pattern on that map is actually changing the specific png inside the MapsForgeTitleSource jar file.

Any help on how could this be done easier? Basically, if changing the colors and sizes works by loading a new xml from somewhere, the files that are called withing the xml (png-s) should load too considering the xml paths have been changed.

Thank you!

 

[warwound]

https://groups.google.com/forum/m/?fromgroups&hl=en#!topic/mapsforge-dev/wUMudj8-zHM

Reply #2 suggests changing jar: to file:

Martin.