Mapping SMS Incident Reports: A Review of Ushahidi and Managing News

Posted by MelissaLoudon on May 03, 2010
Author: 
Melissa Loudon
Abstract: 

In this how-to, we test out two systems for SMS incident mapping: Ushahidi and Managing News. Incident mapping is a simple but powerful concept that does what it says - using SMS to report a given incidence and mapping the data geographically. This article compares the two platforms, their pros and cons, and outlines when to use either.

In this how-to, we test out two systems for SMS incident mapping. Incident mapping is a simple but powerful concept that does what it says - using SMS to report a given incidence and mapping the data geographically.

It has been used in various scenarios ranging from reports from natural disasters to tracking violent crime, citizen reporting in elections.

Ushahidi, a platform for map and time-based visualizations of text reports, has been used most prominently in crisis mapping. The first instance of Ushahidi tracked the post-election violence in Kenya in 2007, closely followed by an instance covering outbreaks of xenophobic violence in South Africa in early 2008. Following the Haiti earthquake in early 2010, an Ushahidi deployment at Tufts University provided a platform for aggregating, translating and disseminating incident reports and requests for assistance. Ushahidi is an open-source  PHP/Javascript platform.


DevelopmentSeed's Managing News originated as a news aggregation and republishing platform, but also offers SMS and mapping functionality. It's been used to visualize election results in Afghanistan, monitor security-related news reports in Afghanistan and Pakistan, and track incidences and reports of H1N1 in the United States. Managing News is based on the open-source content management system Drupal, and makes extensive use of existing modules, as well as related open source projects for SMS and mapping.

First Impressions

While both platforms provide SMS mapping functionality, Ushahidi and Managing News are different.

Ushahidi's focus on crisis mapping makes it easy to see how it can be used to visualise discrete pieces of time-sensitive information - 'headlines' on incidents, for example, or requests for assistance. Managing News, on the other hand, is quickly recognizable to anyone who's ever used a feed reader, or a social bookmarking service like Delicious. The map visualization in Managing News is just one way to organise feeds, and in the default instance, the global map view offers and example of how this could be used by organisations working on an issue with international reach.

In addition to the great introductory videos made available by both projects (here's Ushahidi's video, and here's the video for Managing News), the example projects on each site are a useful way to find out how others have used each system. Ushahidi has been around for a while and has a number of public deployments, some with greater uptake than others. Managing News is newer and has few public deployments to date, but builds on other DevelopmentSeed work in both SMS and mapping, some of which MobileActive.org has previously covered.

The Install

Both systems are web applications, and both are written in PHP. Before you start, you'll need a web server running, with PHP (with some common extensions - the system requirements below have more details) and access to a MySQL database. Full system requirements are here for Ushahidi, here for Managing News.

The download process for Managing News was completely straightforward - click a link, get a .zip archive, wait a short while and you're ready to go. Getting Ushahidi was slightly more involved, with an extra 'gatekeeper' step that asks for your name and email address along with some optional details of your project, and then promises to forward your request on for 'approval'. Fortunately this appears to be automatic, and I was emailed a download link within minutes. I understand why projects do this, but isn't the implied threat of non-approval a little disconcerting?

Once downloaded, both systems should be relatively simple to install. Ushahidi's instructions (available on the wiki here) are clear and easy to follow, even without much web experience, and there are some links to alternative install instructions too. Managing News assumes a more technical user, so it might be a little more difficult if you aren't used to reading the instructions several times and googling a word here or there, but it's certainly not complicated either. One thing to note is that the default Drupal install instructions are included in the Managing News download - if you read INSTALL.txt and find yourself downloading Drupal, you're on the wrong track! Correct install instructions are on the download page if you scroll down. In particular, note that you'll need to set up a cron job (scheduled task) to regularly refresh your feeds.

Both systems have very nice graphical installers, which should work for most configurations, particularly if you're on shared hosting. Once you're up and running, you'll access an admin link to configure your site.

With Ushahidi, this involves navigating a dizzying array of settings, many of them optional. Aside from SMS configuration and map views, which I'll discuss further below, there are also appearance and categorization settings, news feeds, and various incoming data options (email, twitter, laconica) that can be configured. There's lots to try, all centering on the theme of incident mapping.

Managing News has a much cleaner admin page, with configurations for general site data, feeds, mapping and additional features. It's unapologetically technical, though, and quite intimidating if you're not familiar with the technologies involved. More documentation would help, - it's quite easy to set up a themed news agregator, but hard to know what else could easily be configured.

SMS Integration

Before comparing SMS integration in Managing News and Ushahidi, it's worth recapping the options for any system that needs to send and receive SMSs. Mobileactive has a longer guide on how to set up an SMS system, but in general you have two options:

  1. Pay a commercial provider to receive SMSs and forward them on to your system, and allow you to send mesages out. You'll need to do this if you plan on using a shortcode or zero-rated line, and it's also the most scalable option.
  2. Set up your own SMS server using a GSM modem (or mobile phone) and SMS Gateway software. Popular SMS gateways include the open source platforms FrontlineSMS, Kannel, RapidSMS and SlingshotSMS.

Ushahidi uses both options. For incoming messages, FrontlineSMS integration provides a desktop SMS server solution that can be run off a laptop with an Internet connection. For outgoing messages (requests for more information, or alerts), you need to sign up for an account with Clickatell, a large international gateway provider with operations in many countries. The rationale for this setup is explained here - essentially, Clickatell is cheaper for outgoing messages, but cannot provide a local number for incoming messages in all countries. With FrontlineSMS (or another of the gateways), a local number can be used for receiving reports.
 


Source: http://blog.ushahidi.com/index.php/2008/11/05/sms-reporting-through-ushahidi/

Managing News doesn't currently support commercial gateways, although presumably this would be quite simple to implement. Instead, it integrates with SlingshotSMS, another DevelopmentSeed product that provides an open source SMS gateway small enough to run on a USB stick. Although the documentation only lists a Mac build, SlingshotSMS is cross-platform, with Linux and Windows versions available as source code. To get it to work, you'll also need to get the Managing News/SlingshotSMS integration feature, available here. The USB stick configuration means SlingshotSMS can be set up without having to install anything - just be sure to secure your SMS server somewhere where you won't have people 'borrowing' the stick.
 

Source: http://developmentseed.org/blog/2009/aug/14/slingshotsms-alpha-code-released-lightweight-sms-gateway-stick

One of the other advantages of the Managing News/SlingshotSMS setup is that it uses public key cryptography to make sure the messages you receive are actually coming from the correct SMS server. Ushahidi uses a shared key for the same purpose when receiving messages through FrontlineSMS, which is less secure (and sent in the URL in plain text...) but possibly less critical since all incoming messages have to pass human verification. There's also the option of developing a module to parse SMSs in a specific format, described here.

In general, the FrontlineSMS/Ushahidi link seems like a great way to test things out - particularly for existing FrontlineSMS users - while SlingshotSMS offers more configuration at the expense of a bit of technical complexity. FrontlineSMS sync with Ushahidi might also be a little tenuous in a production setting, particularly with the hacked-together solution involving a cron job and python script that you're meant to set up to get around the fact that FrontlineSMS won't retry sending messages if the Internet connection goes down.

My take is that in the long term you'd be better off investigating commercial providers in your target country, or failing that setting up one of the many SMS gateways designed for reliability on the server side - all of which should be able to feed messages into Ushahidi with very little additional work.

Mapping

The map displays in both Ushahidi and Managing News use OpenLayers, a fully open source javascript library that displays a dynamic map. Ushahidi defaults to displaying map tiles from Google Maps, but this can be easily changed by editing the OpenLayers code for the main page - more details in this forum post. The only caveat, which applies to both systems, is that OpenLayers might significantly increase the download size of your pages.



Managing News comes with a default world map layer, as well as extensive configuration options in the admin section (courtesy of the OpenLayers Drupal Module). The default world map, as well as several other preset layers, come from MapBox. You can read more about MapBox here, and there's also a tutorial on how to use MapBox layers in Managing News. You need to be quite technically proficient to make full use of the mapping capabilities in Managing News, but the results can be impressive.
 

The other important aspect of SMS mapping is geocoding - translating a location given as text in a message into co-ordinates on a map. Ushahidi does this by comparing words in the message against a database table of city names, exported from Geonames for one or more countries configured in the admin section. The import process didn't work on my test system (I had this problem), but editing the database table manually is easy enough. Managing News also has a built-in list of locations, starting with countries and major cities in the default install. You can add new locations by uploading a text file in the admin section.

Obviously, this only works if each message contains a location that is also in your database - and if that location is correctly spelled. There are various web services that will take a street address or place name and attempt to return a location, although neither system implements them out of the box for SMS.

In Ushahidi, the person creating an incident report from an incoming message can use the location search box or map view to pinpoint the location of the message - assuming they are able to identify it. Even so, many places have different names in different languages, street name records in many countries are inaccurate and misspellings are common. One of the interesting bits of technology put together in the wake of the Haiti earthquake is InSTEDD's SMS to Geo location crowdsourcing platform, which relies on local knowledge to map locations from text messages. Provided such knowledge is available, the best solution is probably a combination of automated and manual geocoding.

In Summary...

Use Ushahidi if you need to get set up fast (or remotely), aren't looking to do extensive customization of the workflow, and have requirements that fit the crisis mapping paradigm of reports that are approved by a human moderator. As for SMS, FrontlineSMS integration will get you started and Clickatell is easy to set up for outgoing messages, but reliability may dictate investigating other options in the long term.

Use Managing News if you're interested in building a significantly customized system, or have requirements that fit best with a news feeds paradigm. Managing News has been built with a view to being a platform as well as a product, and makes good use of Drupal to do this. SlingshotSMS looks like a good option for a DIY SMS server, and as with Ushahidi, the OpenLayers-based mapping is highly configurable. For slick and beautiful maps, Managing News' MapBox integration is hard to beat.

Lastly, don't forget that there's nothing magical about displaying SMS on a map. There's no point in re-inventing the wheel, but if you need to do something really different, get an SMS server set up to forward messages to a database, research geocoding options and set up a map to display your newly geocoded points!

Ushahidi Resources

  • Existing deployments listed here
  • Download it here
  • Install directions here
  • Developers/implementers can access help on a wiki and friendly forum, as well as a skype chat and an IRC channel - see the wiki for details.
  • User documentation (Haiti-specific) is available on how to submit reports as well as how to access reports

Managing News Resources

  • Examples listed here
  • Download it here, install directions on the same page
  • Developers/implementers can use ManagingNews's issue queue to report problems. There's also a lot of good technical information on the DevelopmentSeed blog, but more user and developer documentation is sorely needed
  • Customization of the Managing News platform is described here, using the Afghanistan Election Data site as an example

Screenshots courtesy of Ushahidi and Managing News/Development Seed.

 

Mobile Tools: 
Mapping SMS Incident Reports: A Review of Ushahidi and Managing News data sheet 17465 Views
Author: 
Melissa Loudon
Abstract: 

In this how-to, we test out two systems for SMS incident mapping: Ushahidi and Managing News. Incident mapping is a simple but powerful concept that does what it says - using SMS to report a given incidence and mapping the data geographically. This article compares the two platforms, their pros and cons, and outlines when to use either.

In this how-to, we test out two systems for SMS incident mapping. Incident mapping is a simple but powerful concept that does what it says - using SMS to report a given incidence and mapping the data geographically.

It has been used in various scenarios ranging from reports from natural disasters to tracking violent crime, citizen reporting in elections.

Ushahidi, a platform for map and time-based visualizations of text reports, has been used most prominently in crisis mapping. The first instance of Ushahidi tracked the post-election violence in Kenya in 2007, closely followed by an instance covering outbreaks of xenophobic violence in South Africa in early 2008. Following the Haiti earthquake in early 2010, an Ushahidi deployment at Tufts University provided a platform for aggregating, translating and disseminating incident reports and requests for assistance. Ushahidi is an open-source  PHP/Javascript platform.


DevelopmentSeed's Managing News originated as a news aggregation and republishing platform, but also offers SMS and mapping functionality. It's been used to visualize election results in Afghanistan, monitor security-related news reports in Afghanistan and Pakistan, and track incidences and reports of H1N1 in the United States. Managing News is based on the open-source content management system Drupal, and makes extensive use of existing modules, as well as related open source projects for SMS and mapping.

First Impressions

While both platforms provide SMS mapping functionality, Ushahidi and Managing News are different.

Ushahidi's focus on crisis mapping makes it easy to see how it can be used to visualise discrete pieces of time-sensitive information - 'headlines' on incidents, for example, or requests for assistance. Managing News, on the other hand, is quickly recognizable to anyone who's ever used a feed reader, or a social bookmarking service like Delicious. The map visualization in Managing News is just one way to organise feeds, and in the default instance, the global map view offers and example of how this could be used by organisations working on an issue with international reach.

In addition to the great introductory videos made available by both projects (here's Ushahidi's video, and here's the video for Managing News), the example projects on each site are a useful way to find out how others have used each system. Ushahidi has been around for a while and has a number of public deployments, some with greater uptake than others. Managing News is newer and has few public deployments to date, but builds on other DevelopmentSeed work in both SMS and mapping, some of which MobileActive.org has previously covered.

The Install

Both systems are web applications, and both are written in PHP. Before you start, you'll need a web server running, with PHP (with some common extensions - the system requirements below have more details) and access to a MySQL database. Full system requirements are here for Ushahidi, here for Managing News.

The download process for Managing News was completely straightforward - click a link, get a .zip archive, wait a short while and you're ready to go. Getting Ushahidi was slightly more involved, with an extra 'gatekeeper' step that asks for your name and email address along with some optional details of your project, and then promises to forward your request on for 'approval'. Fortunately this appears to be automatic, and I was emailed a download link within minutes. I understand why projects do this, but isn't the implied threat of non-approval a little disconcerting?

Once downloaded, both systems should be relatively simple to install. Ushahidi's instructions (available on the wiki here) are clear and easy to follow, even without much web experience, and there are some links to alternative install instructions too. Managing News assumes a more technical user, so it might be a little more difficult if you aren't used to reading the instructions several times and googling a word here or there, but it's certainly not complicated either. One thing to note is that the default Drupal install instructions are included in the Managing News download - if you read INSTALL.txt and find yourself downloading Drupal, you're on the wrong track! Correct install instructions are on the download page if you scroll down. In particular, note that you'll need to set up a cron job (scheduled task) to regularly refresh your feeds.

Both systems have very nice graphical installers, which should work for most configurations, particularly if you're on shared hosting. Once you're up and running, you'll access an admin link to configure your site.

With Ushahidi, this involves navigating a dizzying array of settings, many of them optional. Aside from SMS configuration and map views, which I'll discuss further below, there are also appearance and categorization settings, news feeds, and various incoming data options (email, twitter, laconica) that can be configured. There's lots to try, all centering on the theme of incident mapping.

Managing News has a much cleaner admin page, with configurations for general site data, feeds, mapping and additional features. It's unapologetically technical, though, and quite intimidating if you're not familiar with the technologies involved. More documentation would help, - it's quite easy to set up a themed news agregator, but hard to know what else could easily be configured.

SMS Integration

Before comparing SMS integration in Managing News and Ushahidi, it's worth recapping the options for any system that needs to send and receive SMSs. Mobileactive has a longer guide on how to set up an SMS system, but in general you have two options:

  1. Pay a commercial provider to receive SMSs and forward them on to your system, and allow you to send mesages out. You'll need to do this if you plan on using a shortcode or zero-rated line, and it's also the most scalable option.
  2. Set up your own SMS server using a GSM modem (or mobile phone) and SMS Gateway software. Popular SMS gateways include the open source platforms FrontlineSMS, Kannel, RapidSMS and SlingshotSMS.

Ushahidi uses both options. For incoming messages, FrontlineSMS integration provides a desktop SMS server solution that can be run off a laptop with an Internet connection. For outgoing messages (requests for more information, or alerts), you need to sign up for an account with Clickatell, a large international gateway provider with operations in many countries. The rationale for this setup is explained here - essentially, Clickatell is cheaper for outgoing messages, but cannot provide a local number for incoming messages in all countries. With FrontlineSMS (or another of the gateways), a local number can be used for receiving reports.
 


Source: http://blog.ushahidi.com/index.php/2008/11/05/sms-reporting-through-ushahidi/

Managing News doesn't currently support commercial gateways, although presumably this would be quite simple to implement. Instead, it integrates with SlingshotSMS, another DevelopmentSeed product that provides an open source SMS gateway small enough to run on a USB stick. Although the documentation only lists a Mac build, SlingshotSMS is cross-platform, with Linux and Windows versions available as source code. To get it to work, you'll also need to get the Managing News/SlingshotSMS integration feature, available here. The USB stick configuration means SlingshotSMS can be set up without having to install anything - just be sure to secure your SMS server somewhere where you won't have people 'borrowing' the stick.
 

Source: http://developmentseed.org/blog/2009/aug/14/slingshotsms-alpha-code-released-lightweight-sms-gateway-stick

One of the other advantages of the Managing News/SlingshotSMS setup is that it uses public key cryptography to make sure the messages you receive are actually coming from the correct SMS server. Ushahidi uses a shared key for the same purpose when receiving messages through FrontlineSMS, which is less secure (and sent in the URL in plain text...) but possibly less critical since all incoming messages have to pass human verification. There's also the option of developing a module to parse SMSs in a specific format, described here.

In general, the FrontlineSMS/Ushahidi link seems like a great way to test things out - particularly for existing FrontlineSMS users - while SlingshotSMS offers more configuration at the expense of a bit of technical complexity. FrontlineSMS sync with Ushahidi might also be a little tenuous in a production setting, particularly with the hacked-together solution involving a cron job and python script that you're meant to set up to get around the fact that FrontlineSMS won't retry sending messages if the Internet connection goes down.

My take is that in the long term you'd be better off investigating commercial providers in your target country, or failing that setting up one of the many SMS gateways designed for reliability on the server side - all of which should be able to feed messages into Ushahidi with very little additional work.

Mapping

The map displays in both Ushahidi and Managing News use OpenLayers, a fully open source javascript library that displays a dynamic map. Ushahidi defaults to displaying map tiles from Google Maps, but this can be easily changed by editing the OpenLayers code for the main page - more details in this forum post. The only caveat, which applies to both systems, is that OpenLayers might significantly increase the download size of your pages.



Managing News comes with a default world map layer, as well as extensive configuration options in the admin section (courtesy of the OpenLayers Drupal Module). The default world map, as well as several other preset layers, come from MapBox. You can read more about MapBox here, and there's also a tutorial on how to use MapBox layers in Managing News. You need to be quite technically proficient to make full use of the mapping capabilities in Managing News, but the results can be impressive.
 

The other important aspect of SMS mapping is geocoding - translating a location given as text in a message into co-ordinates on a map. Ushahidi does this by comparing words in the message against a database table of city names, exported from Geonames for one or more countries configured in the admin section. The import process didn't work on my test system (I had this problem), but editing the database table manually is easy enough. Managing News also has a built-in list of locations, starting with countries and major cities in the default install. You can add new locations by uploading a text file in the admin section.

Obviously, this only works if each message contains a location that is also in your database - and if that location is correctly spelled. There are various web services that will take a street address or place name and attempt to return a location, although neither system implements them out of the box for SMS.

In Ushahidi, the person creating an incident report from an incoming message can use the location search box or map view to pinpoint the location of the message - assuming they are able to identify it. Even so, many places have different names in different languages, street name records in many countries are inaccurate and misspellings are common. One of the interesting bits of technology put together in the wake of the Haiti earthquake is InSTEDD's SMS to Geo location crowdsourcing platform, which relies on local knowledge to map locations from text messages. Provided such knowledge is available, the best solution is probably a combination of automated and manual geocoding.

In Summary...

Use Ushahidi if you need to get set up fast (or remotely), aren't looking to do extensive customization of the workflow, and have requirements that fit the crisis mapping paradigm of reports that are approved by a human moderator. As for SMS, FrontlineSMS integration will get you started and Clickatell is easy to set up for outgoing messages, but reliability may dictate investigating other options in the long term.

Use Managing News if you're interested in building a significantly customized system, or have requirements that fit best with a news feeds paradigm. Managing News has been built with a view to being a platform as well as a product, and makes good use of Drupal to do this. SlingshotSMS looks like a good option for a DIY SMS server, and as with Ushahidi, the OpenLayers-based mapping is highly configurable. For slick and beautiful maps, Managing News' MapBox integration is hard to beat.

Lastly, don't forget that there's nothing magical about displaying SMS on a map. There's no point in re-inventing the wheel, but if you need to do something really different, get an SMS server set up to forward messages to a database, research geocoding options and set up a map to display your newly geocoded points!

Ushahidi Resources

  • Existing deployments listed here
  • Download it here
  • Install directions here
  • Developers/implementers can access help on a wiki and friendly forum, as well as a skype chat and an IRC channel - see the wiki for details.
  • User documentation (Haiti-specific) is available on how to submit reports as well as how to access reports

Managing News Resources

  • Examples listed here
  • Download it here, install directions on the same page
  • Developers/implementers can use ManagingNews's issue queue to report problems. There's also a lot of good technical information on the DevelopmentSeed blog, but more user and developer documentation is sorely needed
  • Customization of the Managing News platform is described here, using the Afghanistan Election Data site as an example

Screenshots courtesy of Ushahidi and Managing News/Development Seed.

 

Mobile Tools: 

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd><p><br> <b><i><blockquote>
  • Lines and paragraphs break automatically.

More information about formatting options