Skinning the Community Cat – Transcription

Our next speaker is Rahul Pratap. I will give everyone a minute for the title slide, I won’t say it aloud because it is too morbid, but hopefully it got your attention.
I’m Rahul Pratap, and I work with Foursquare in New York. I work with the database and the edits for other users. I have done this for three years now, and I gathered perspective on a way to build and sustain a large global community of geospatial data on the previous slide, and I would like to share that perspective in the next large global community. A high-level comparison if you are not familiar with Foursquare, it has a place database, it is just POIs. It is a POI database with 105 million places world-wide, and our products have 15 million active users, and 44,000 of them are super users. Between them, they move about 4 million, yeah, edits per month, including automated edits. I will have something to say about that in a bit. And the data is directly consumed by hundreds of millions of users, to over 125,000 third-party developers that use the Foursquare API, including a few pretty big names on that slide. And OSM is a map of the world that also has POI database, these are the potential number of POIs, but not all. You have 14.2 million contributors making 80 million edits a month. And your data is being directly consumed by many more users than you have contributors, including Foursquare users on the web, where we use Mapbox tiles to show people maps. And both these communities, it is obvious, are powered by volunteers who are giving their time, resources, and expertise for free and for the common good. When I think about that, it is remarkable and a testament to the value of what Foursquare has created. It is a closed dataset, owned by a private company, is thought to create a common good. Not everybody has that position, I’m grateful that we have that. So we are a smaller community working on a smaller dataset than OSM, but we are every bit as passionate. If you do the math in terms of edits processed for editor, that is one metric, it does not count for much, we are a much more productive per editor. These are international communities and spread across the world. Foursquare users are users of our apps, they are passionate users, and that causes their geographic distribution to reflect the popularity of our apps. And super users, in many ways, they are the ideal user of our products, in that they prove the thesis on which Foursquare was founded, this is something that is spoken of a lot. The mechanics helps to hook users, and super users were hooked by the game mechanics, and they started to get value out of the data and the content, like tips and photos and became invested in the quality. That tapped into this inner obsessive level of perfectionism and attention to detail, and they wanted to make that data clean and consistent, like OSM editors. And then they self-organized and self-policed. And now, Foursquare as a community supports the community in every way we can, but I have to admit, they are entirely of their own making. And the game mechanics as a motivator has its own problems, users were incentivized to create junk duplicates to get badges, or to be a mayor of a place, the same way they would check into a place they were not physically presently at. So there was this tension in the early days between acceling at the game, which is tempting, and acceling at editing and maintaining the data quality at the high level you want it to be. Some of our super users today, some of them, are reformed cheaters, which is what we called these users back then, because of the stronger incentives we built around editing. Think about it, we have a handful of super users that made more than a million lifetime edits, individually, each of them. That is telling you something, that is telling you that these people believe that these rewards for putting in that kind of time and effort is compelling, to say the least. It is worth pointing out, while we are on the topic, for the post part, OSM has a different incentive structure. And to the extent that it does not rely on game mechanics as much, I think you have avoided the issue. And now, there is vandalism, and that is differently-motivated, but we will not talk about that now. What are the incentives OSM offers? Apart from editing an open map, you can see the history of edits, and OSM is focused on mapping vulnerable areas of the world, especially disaster-ridden ones. And I would suggest to introduce leader boards and badges, and to see how that is working out for OSM. And as far as super users, it is hard to progress three 10 users, which brings data editing powers, in terms of the editing you can make and the weight your votes carry. There are secret super-user only features, they are invited to private events online or in person that either Foursquare organizes, or supports. We send them swag from time to time, and they love it. Moving on to what is probably OSM’s greatest trend, the ecosystem of tools and platforms you communicate. You have an enviable comprehensive Wiki with a code of conduct. I wish we had an equivalent of this. There is a list of topics from geocoding, accessibility, legal, and regional discussions. And you have discussions for individual change sets, and mapping partys and other such forms of real-world collaboration. Foursquare has the equivalence of some of those things. We have global discussions by topic, we have talk pages for individual venues that are analogous to the discussions, and previously, communities have written fastidiously and enforce their own style guides, and work is under way to produce a consolidated global style guide. And given that we are a company and own the data, Foursquare is centralized in a way that OSM probably cannot be. So access to HQ, or the headquarters, is an additional component of SU privileges, and that functions as an incentive. And super users at level 9 and 10 get access to a mailing list that includes Foursquare employees, and super users at level 10 have access to a company Slack channel channel. The super users are the eyes and ears of the company, they let us know when things are broken, they have amazing ideas on improving the tools. So giving them that direct line to communicate with the coders at Foursquare gives them ownership. And Foursquare has both first-party official tools, at the heart of which is a set of pending local edit cues that are defined by type. And a lot of the high-level SUs find themselves resorting to using third-party bulk editing tools. These are called sweep tools that are written by high-level super users. 4 Sweep was by SU9 in the U.S. And so, very quickly, this is how a user on the web would suggest an edit on Foursquare, the most sophisticated-looking native views in the apps as well. This is what it looks like when you go to the Foursquare SU tools, you have your own stats, there’s a leader board, you can jump into one of these queues, and a queue looks like this. There is information to suggest whether a suggestion is valid or not, warnings, and showing you places on the map. And this is where we get to what is probably the code of this entire discussion. In comparing these two data sets and communities, what it comes down to is the fundamental editing philosophies we have. OSM has no reputation system, all users are trusted equally. Regardless of their experience or track record. And there is no moderation, edits are immediately approved and the community is relied on to report bad edits. This makes sense for mapping and fast response to disasters, for instance, or to map areas of the world where there are not that many contributors to begin with. This is similar to wickpedia, but it embraced the system of detecting and reverseing vandalism. Foursquare’s approach is optimized for quality. The idea is that we draw the delay edits until they can be shown they are good, just in case something slips through, we support reverting bad edits. We trust users different amounts depending on their past edits, and we allow super users to accept or reject edits. And at the heart of the reputation and moderation system is SU football, although made by Americans with American football in mind, involves a metaphor like a ball, with two goals, which is like the real football, which you call soccer. And this is interesting, there is an American football stadium out here, but I refuse to commit to a non-circular shape of the ball. If you squint at it – but I could talk for 15 minutes on the ins and outs of SU football. It starts at 0, we compute the positive access goal and the negative reject goal. These are not equidistant from 0. The accept goal is a higher value, and the way these values are determined is by what is at stake, what edit is being suggested to determine how popular a venue. And every user as a reporter goal, which tracks the accuracy of the previous edit. So the ball moves ahead an amount equal to the reporter score. Sometimes the user is so trustworthy that it is immediately accepted, in other cases, it awaits moderation, and users can accept or reject the rejection. When the super user accepts an edit, the ball moves forward equal to the voter score, or backwards equal to the voter reject score, and these are a measure of their tendency to be trigger happy in one direction or the other. When the ball crosses the goal, the edit is accepted or rejected. Putting together the material for this presentation, I have drawn on the experiences and opinions of members of both communities. I pulled in the 9 and 10 mailing list, asking it people regularly contribute to OSM, and these are the responses. And as far as basic methodology goes, it with stands basic scrutiny, but I got great insight. And while the survey covers a lot of ground, and it is used to identify what Foursquare users can learn from OSM, I want to focus on the things they felt OSM could stand to learn from Foursquare. First of those, unsurprisingly, is this idea of levels. Some people think that OSM is democratic to a fault. I have seen that echoed by people that are more a part of the community than I am. They think their discussions have a tendency to become never-ending arguments, and the idea of levels would establish leadership, or authority and help drive these discussions toward a consensus. Similarly, the wish that OSM had more moderation, special in edits from new and experienced users. And the moderation is key to stopping or slowing vandals. If not that, they believe that some version of Foursquare super user short listing process, which is where you place new or potential users under probation and only allow them to review edits at venues we know they edited, because we know where they have been. And the respondents to the survey also acknowledge and welcome OSM charts, which is this QA tool from Mapbox to flag and queue edits before they are published to Mapbox. But the fact of the matter is that OSM still updates instantly. They wish that OSM would have a more liberal stance as far as automated edits and imports go. This is something that could be easily fixed, especially with modern machine learning techniques. I’m not talking as ambitious as Facebook, but simpler stuff than that. Foursquare is pretty good at automated edits, we don’t have bots, but we have the consistency fairy that loops over all the events, we have the patterns, and we make the edits that the super users get to review, or if we are confident, we push through. This is something worth replicating. And what is probably the least actionable, but we still felt strongly about, is that OSM needs a better way for end users to suggest edits. We are 44,000 super users, but all of our 15 million users, and hundreds of millions of users that use or consume our data via the third party apps that use the API are editors. This is hard to do with data, but as the Mapbox goes to show, this is possible and maybe the foundation needs to push harder for using OSM data, and rendering it on a map, and making it possible for people to suggest edits somehow. That is pretty much everything that I had, and they give me plenty of material. They are more plugged into the community than I am, and I appreciate their insights and opinions. I’m free to take questions. (Applause).
Okay, we will take a break until 4:45.
Live captioning by Lindsay @stoker_lindsay at White Coat Captioning @whitecoatcapx.