(Carefully) adding ArcGIS data to iD – Transcription

Okay, let’s keep moving forward here. So Sam Libby from ESRI is going to talk about ArcGIS. Yes, are you okay in the back? Okay. I need to switch to duplicate. I’m Sam Libby, I work in the Boulder area for Esri, on the professional services team, I do projects for our customers. I assume that many of you are familiar with Esri and ArcGIS, we will talk about work we are doing in general with the OSM community in a couple directions, and this specific project that Nick has been helping out with. I’m Nick Doren, I’m working with Esri on this project, I have OpenStreetMap and I’m really excited to share this project. I’m talking about the business case for adding ArcGIS to iD, and Nick will talk about adding that and we will get feedback as well. So what is it? We are trying to add a button in the iD in the map data tab that allows you to add geoservice. What is that? What it allows you to do is review and add geometries and attributes to the iD interface in their browser from existing data that is exposed through one of our ArcGIS services. We produce a server infrastructure that people can use to expose their geo data from ArcGIS systems, it is used in counties, state governments, and federal governments, and internationally. You may be familiar with this end point that looks like this. This is the rest API documentation, this is an open doc that we have, and the rest API looks like this, basically our publicly-facing HTML interface that you can query for different applications and use cases. I will get into it later. This gives a guided interface through iD that allows you to add this to the map for location, map this into the data source for OSM tags, and review the change feature before it is submitted. This is a contribution that we have been trying to make to the community. The next question is what we will cover in the presentation, I will try to go over why this is a problem we want to solve, how it will improve the quality of OSM data, how people will use it, and how are we trying to prevent incorrect or unlicensed edits, and a brief coverage of what we are trying to do with OSM. We are trying to solve, this is my opinion but a fair statement, a lot of time and money has been spent in the last 50 years to build high quality geo data all over the world. We heard an hour ago from the City of Boulder in a panel discussion they did about how the city spent a lot of time and money, in this case taxpayer dollars, to give geo data that should be available to more. The interface that I showed earlier is not the most user friendly, they can build apps, but we wanted open in places like OSM. And people are asking for places to submit it like OSM. If they are using our stuff to manage the data, how do they get it into OSM reliably and carefully. And this data is made available more and more through open APIs and friendly licenses. We have lot of work to do on that to educate our customers and our community about licenses, it is growing as an area of interest. They benefited from close local review and represent key local knowledge from the community. Getting the data into OSM is difficult from a technical and policy perspective, that came up in the last talk from the city of Boulder. For an analyst for the city, how do you start with the import process? There’s a lot of Wikis to read, a lot of mailing lists to read up on in the background, and it is intimidating, speaking as a person who looked at it in the past. Improvements are beneficial in large parts of the OSM dataset. We had a tool in the past 5 years that is a desktop editor for OSM data that allows people to download a chunk of data, make changes, and then submit changes back to the community. And there is one example of an area in New Zealand on the coast Near Wellington that shows the current OSM base map on the left, with an example of overlaping building data that we have from the community that they published out and digitized through a vendor or their purposes. I would love to see those maps come closer together without having somebody draw those buildings when they already exist. I will let Nick talk about development as well. Nick: So it is difficult approaching this project, there is no plug-in standard for iD. I think we heard throughout the day, a couple different people, including the Facebook machine learning presentation, people are trying to integrate with iD to connect to different processes. There is no existing standard. What we wanted to do was, as much as possible, reduce our branch to one JavaScript module and compare it to other SVG layers. When you add a GPX layer, it resolves file and over lays, so trying that system and modifying the existing UI to extent integrations into the editor where you select different tag and pre-sets. Also, in order to do some of the work that we wanted to do, avoiding, for example, overlapping data, we have GIS modules added to the package.JSON. The only existing method in iD right now to create a vector data within OpenStreetMap is drawing it. When you put the GPX in, it is only a Raster tile. It is not when you upload GPX file, you cannot start editing it like it is OpenStreetMap data. We want to make sure that the data that was added was in the same environment and with the same editability as any other data in the iD editor, once it is imported in. Then we – so, sort of finding the right methods to attach that, and put them together in the same way that when you draw things, they are added and edited in the OSM system. And because of that, we were able to use tools for drawing, editing, undo, and redo. We were able to use existing pre-sets, so any pre-set you can select in the iD editor for something you are drawing, you can start that for something more complex. For example, if you cannot remember the tag for a school or a driveway, you can find the preset in the interface and get the same benefits that they would, had you been drawing that manually. And sort of trying to come up with the idea of, you know, there’s a lot of different fields in a geoservice and trying to make sure that we can map them to OpenStreetMap fields, and leave out fields that we find are unnecessary, instead of going through the process and trying to figure out what’s the gap between the geoservice and the OpenStreetMap data model.

All right, cool. We will do a little demonstration. So I will try to put my tabs up one at a time. Hopefully it works. This is the rest API documentation, this is a version of iD running on the server, this is live on GitHub, you can see it right now. I will share links at the end. What we added in is this additional button on the map data tab. Hopefully that is big enough, I tried to make it big enough and not too big to not see the screen. This is an additional button that we added in the existing interface that said add geo service layer. When I open that, it opened a model that allows me to put in the url. I identified a layer from the county of Boulder, as well as a location that serves for my purpose, this is a planned demo. But these are footprints in Boulder counter. We have some nice streets here, this is a relatively new development without building footprints. I can start to draw these, but it will take me to do 20 minutes to do the whole neighborhood just in this small area. And I know that the county data exists for that. If I go here and copy my url from this layer, I will be asked first to paste that in, and then to confirm that this is a proper license for this data. In this case, it is not, the county does not provide an open license for that, I will not make edits. But we would pull up that info if it was available in the services page, I will pretend that I have communicated with them or see a different page that grants the license to be used properly in OpenStreetMap. We provided a Wiki page open that allows you to import your documentation, and we have a couple different tasks for people to provide relatively small imports in this case. I’m looking for feedback on this kind of stuff as well. And what you see next is the option to pick a feature type. If I type building, I will see values in the lower section populated, this is the preset image. We are seeing the image attribution for the features, this is a sample feature. You can see that Boulder county has a source field, structure type, and the source is a demolished building, and then the height of the building. I can take any of these and choose to include them in my mini-import. I will take the height, put it in the name field, the source, and they are pre-populated with the right fields from the building preset and I will put this information into the address. This dataset does not have great attribution for addressing, we use the geometries. In the bottom, there are almost 141,000 buildings in this data set, I have 50 in the map. I load and view. Whether I finish, I close the modal and you will see an orange – all of these buildings I pulled in. If I zoom out, the area of interest, these have been added to the map, full iD features, I can drag and move them, the same iD tools as well. And if I click, you can see this button at the top for approving or rejecting a feature. We are trying to hit the middle between making it easy to import geo data without importing all of them. I can see in my tags I have the area mapped in, the area and city that I put in from that data set. If you put in the address, it will map at attribute. I will go one by one and approve if I’m happy, they turn green and are edits in the OSM submission. I can approve all or one, and submit the same way with iD. Another example in San Juan County, Washington, and I do have permission to submit the data from Nick Peal. Are you here? So thanks to him, I will give examples. This is a folder of services in counties in the islands of Puget Sound. They have soil maps, parcels, transportation features, some low or less relevant to OSM, and I saw they had nice drive ways data. If I copy that, I will go back and show you something else. If I copy the url and open up another interface here, when I add that, I will be asked to pick a layer from the service and I will pick the drive ways layer. As Nick was saying, we did a lot of work to add in the presets here. If I type in drive way, I will have a nice preview image and map the attributes to the drive way tags. If I take this iD and map it to the name, and add the 16 drive ways in my view. I tried to do that without confirming the license, and I get a prompt, check that, stored in the local storage and every time I come back I don’t get a prompt from this server url. When I add that to the map, I will see if it works. I will try that one more time. So I added those features, nice driveways, adding rich, local detail and highly detailed data to OSM. I don’t think driveways are a feature to routing, but it is a nice feature in OSM, it makes the map look better, and you will get high resolution data that you will not see people going and doing across an entire country. This is how I’m looking to better control this, this is a better automation and computer problem, we are looking at you to look at these individually and improve those. If I tried to improve one of these and tried to save it, the iD would warn me and have me review during the submission process. If I have enough stop baps on this functionality to submit the quality data. And again here, you will see automatically tagged information from the drive way preset so I don’t submit data that is missing these required tags, and the preset, this is part of the integration with it being directly in iD. In terms of how it will help the community, it will help fill in the map where you do not have a large local OSM community, or very active community. It requires a lot of help to digitize data across large areas, not all areas have that community, or a natural disaster occasion to do that. But a lot of areas in the U.S., at least, have high quality geo data because counties have to manage this stuff in existing systems. It allows you to incorporate data layers that are difficult or not common to select. We will not take a GPS of every drive way in the county, but that data has been collected by somebody and can be used. We hope to provide correct attribution that is not easy to find. A lot of times, these organizations maintain official addresses and building heights, all of these information that you have to find on your own or ask about. Hopefully it will jump-start local mappers that are discouraged by features in their area. And for me, it is about adding high quality data to OSM carefully that has been submitted carefully again to that community. These municipalities are paying good money to companies to collect it, we should take advantage of it. And those who benefit would be journalists, civic hackers, and open data advocates. I like to think of local professionals, we have many that could make use of this quickly. There’s a good use case for disaster response, hopefully working with mapathons as well. I did a fair amount of digitizing in South Florida as part of the hurricane response, and there were cases where I knew the existing imagery was in the database. That data was created by somebody and I didn’t want to re-draw buildings that existed. So this is an opportunity to look at hot to look at certain cases where it is a fitting use case. We added a lot of back and forth on the iD repository on how we can look to prevent incorrect low-quality or unlicensed edits. I have shown you a few of the ways we tried to do that. One feature at a time is a part of that, we want individual review of features, for them to mark it as approved before adding it to the mapathons, and give the chance to inform users about licensing. A lot in the public thinks that the data is open and available, but the fact that the government does not say it prevents it from being OSM. We say, if you would update your license on the service, it could be added to OSM by the community. We are hoping that users import the discussion on GitHub, or on Wiki, we are trying to prevent overlaping buildings from being added, and doing vertex integration of those features. I have no magic bullet to solve that, but we are looking at that in future releases. And to cover a few OSM things, we have an ad hoc approach to working at OSM. We have a number of people that are integrated into the community and do a lot of work on that, and we have added a lot that is exciting. Brian mentioned his iD talk, we have provided the world imagery service now for access in iD editor, and all other OSM editors, which is a tremendous licensing under taking. We have imagery for all kinds of providors, and a lot of licenses that we signed from a legal perspective to make it available. To get it available in OSM was a two and a half year process, thank you for your patience in that, but it provides a lot of cool options in iD. I will show you an example of that – out of time. Sorry. So, I will just cover that. There are areas where we have good resolution and some interesting information that is not normally available, as well as this little add-in, a control shift B bonus inside of iD that shows you the detail level for that area. Happy for questions, I have more topics here, we will be here all week at the booth. I’m out of time, I appreciate you listening. Thank you. (Applause).
Any questions? (Speaker far from mic). A lot of times, there’s a systematic difference between the municipal government data and OpenStreetMap data. I work for King County, Washington, all of our street types are abbreviated and OSM wants full types. Have you considered a text replacer when you are mapping your fields to the tags? Yeah, I have thought about that. One area would be concatenating existing tags into one address tag, or doing string replacements, that kind of thing. We set it up in a way where we can easily add that, that is something we are trying to develop in the community. That is a good example of something that takes lot of knowledge to do. If we have a good import, that would be awesome.
Okay, we are going to cut it off here and keep moving forward. (Speaker far from mic).