Development seed

Drupal4Gov, government transparency, and semantics in research this week in Washington, DC

According to the forecast, we'll (finally) get a reprieve from the rain later today and see the sun tomorrow! That's something to look forward to, and the sunny spring weather will surely make heading out of the office to check out a technology event or two more appealing. Below are the events that caught our eye this week - including two presentations that folks from Development Seed will be giving. You can find a full schedule of the week's events at DC Tech Events.

Tuesday, March 16

3:00 - 4:00 pm

Drupal4Gov Webcast: Features and Exportables in Drupal: Are you a developer or IT manager for the government using - or interested in - Drupal? Check out this group. At this month's meeting Jeff will present via webcast on the development -> staging -> production workflow in Drupal and how using the Features module can save you time and hassle.

7:00 pm

NSCoderNightDC: Into mac and iphone development? Then check this meetup out, where people gather to talk about Cocoa, Cocoa Touch, Objective-C, and mac and iphone from the week - over tasty french desserts.

How you can save time and drastically improve your Drupal experience with the Features module

Tomorrow Drupal4Gov is holding their monthly meeting, bringing together developers and IT managers who work in the federal space to talk about Drupal best practices specifically in the U.S. government space. This is a new group, led by Kirsten Burgard at the Department of Veterans Affairs, and one that has already generated a lot of interest and participation. Jeff is headlining this month's meeting and will present via webcast on managing a clean development -> staging -> production workflow in Drupal. His discussion will be targeted to IT decision makers and to developers who are interested in streamlining their site development and deployment with the Features module.

There are some large issues and headaches that come from following the traditional approach of keeping configuration in the database, and these get exasperated when working on complex sites that require a solid development and staging system. What happens when your production database gets out of sync with the configurations changes that just passed QA on the staging site? Do you really have to go in by hand and reconfigure the changes again on the live database?

This is where the Features module comes to the rescue. Jeff will explain how Features let you export your code, check it into version control, and then just push it out to the production site, saving you valuable time, reducing the possibility of errors, and drastically improving your Drupal experience. Jeff will also talk about the benefits of using Features when building a site, give some examples of how exporting our code to Features has helped us build faster at Development Seed, and prepare developers to start making their own Features.

The presentation is at 3:00 pm tomorrow (Tuesday, March 16). If you'd like to join, contact Kirsten Burgard at Veterans Affairs to get access to the webcast.

Some resources to prep for the talk

• The Development -> Staging -> Production Workflow Problem in Drupal
• Making and Using Features in Drupal
• Spaces: A Paradigm for Reusable Drupal Features

Some ideas recapped from the SXSW presentation on open source business models

Yesterday while speaking on the SXSX panel Selling Your Milk When the Cow is Free, I talked about some of our plans on making Open Atrium development sustainable and for the first time publicly shared ideas on what an Open Atrium Partners Program would look like, in addition to plans for commercial support to follow. In this post I'll recap that and go into more detail on what we are seeing in a Partners Program and get feedback.

Why partners are a good thing

About two months ago I starting emailing different shops that are building on Open Atrium to ask what would be helpful for them if we were to move into the commercially supported space. What came back was a unanimous request for an "Open Atrium Partner Program" that would help teams build and deploy Open Atrium better and more efficiently. Honestly, I've always been cynical of partner programs because I had only seen a few that had done it really well. But after really looking at other companies that offer support and the feedback we got from different shops, we believe we are in an amazing position to offer real value that will help shops build better sites and sell more projects.

We want to make it easy for teams to prove their Open Atrium expertise and ensure Open Atrium custom development is always successful. This is why the main focus of our partners program will be on training and capacity building. Here are the main components we are seeing now:

• Training courses
• Developer channels for third tier support
• Partner meetings
• Partner section on OpenAtrium.com
• Partner bling

Our goal is to create a training course and support system that will bring in shops as real partners and give them real value - and not have something that is a a pay to play program. Below are more details on our current draft plan. We are very open to adjusting this to provide more value to partners, so if you have ideas please talk to us.

Looking at Open Atrium in the light of open source business models

This afternoon I am speaking on the SXSW panel Selling Your Milk When the Cow is Free about what it has been like to make the transition to selling products with Open Atrium.

We only have one way to sell milk right now, and it ain't scaling, but we have plans :)

On the panel, I'll talk about the bootstrapping process we undertook when Open Atrium first launched last July. I'll share how we communicated about the work we wanted to be doing in a way that let us book a set of clients that had similar needs, which allowed us to pool resources to build a good foundation for Open Atrium.

This was a great way to start this work, but now we're facing the question "how the hell are we going to support this?" Last week Open Atrium passed 90,000 downloads. At this stage it's no longer something that we can keep supporting only through high end customization work. This leaves us with the other two prominent open source business models - support and/or SaaS. For the first time publicly, I'll share what we are planning and talk about the obstacles for us in making the transition to a new business model, from obtaining start up capital to fostering team growth and our culture.

The facilitator, Jeff Eaton, organized this talk to "educate others about OSS opportunities and pitfalls", so the Open Atrium angle is only a small part of a much larger and diverse conversation. The panel also includes:

• Evan Prodromou, the founder Status.net, an open source microblogging service.
• Tiffany Farriss from Palantir.net, a digital design and web shop that does a lot of work with Drupal.
• Brad Fitzpatrick, who worked for LiveJournal and is now at Google working on standards like OpenID.

The panel is at 3:30 pm in the Hilton A/B room.

Sidenote: I have only been to Austin once before and it was a really quick stop, so I would love to get some good bbq while I'm here. If you have any recommendations, please hit me up on twitter @ericg.

A simple tool for offline mapping and custom data overlays

We just released Maps on a Stick, a simple client-side mapping tool that lets you access map tilesets and mash them up with information while you're offline. Maps on a Stick runs on either a USB drive or directly from a hard drive and allows you to browse custom baselayer maps and add KML overlays to them without connecting to the internet.

Here's a look at Maps on a Stick. The data overlay is from a U.S. Geological Survey KML file showing recent earth quake data, and the baselayer map is the World Light tile from MapBox.

You can zoom in on the maps, just like you can with any slippy map.

Some examples of what you can do with the latest Drush

Over the last few years Drush has matured significantly and has seen an incredible uptake in usage. It's become indispensable in the day to day workflow of innumerable Drupal users and has been accepted with open arms by contributed module developers who are finding new and wonderful functionality to expose via its clear command line interface.

What not many people realize is that beneath this simple command line API beats the heart of a far more flexible and powerful beast. Drush was written with re-use and scriptability in mind, with this entire concept deeply ingrained in its design, and this is a large part of what gives it its power and flexibility. This will be even more apparent in Drush 3.0.

Below is a rundown of some useful things you'll be able to do with Drush 3.0.

Remote procedure calling

Each Drush command can be called from within any other Drush command, spawning a new process and returning structured and meaningful information to the calling script. A more accurate terminology would be that Drush commands all operate on a somewhat RESTful API , which is similar to XMLRPC and other RPC mechanisms that you may already be familiar with. Because of how this feature is implemented in Drush it can not only call Drush commands locally, but it can also call Drush commands on a remote server via SSH.

Up until now this functionality has been hidden within the API and only systems such as Aegir really made use of it. For example, in the Aegir hosting system we re-use the drush updatedb command when we do site migration and to run updates, and we have a single drush provision-backup command we can simply call when we need to make a backup.

What about security on remote calls?

Remote Drush calls run over SSH and require you to have your key already added to the remote server's list of authorized keys. Therefore Drush isn't capable of doing anything that you don't have the necessary permissions to log in manually and do on the server itself. It just does what it does best - automate away some of the steps so you get your results quicker.

DAI using Open Atrium to help their GIS teams in Pakistan communicate

Development Alternatives Inc. (DAI), one of the leading international development contractors, has deployed Open Atrium to help its GIS team in Islamabad communicate with its office in Peshawar and headquarters back in Bethesda on a capacity building project. In addition to improving team communications, they decided to use Open Atrium to boost the project's transparency for USAID and to get a sustainability win by using open source software that can stay behind with the local team in Pakistan long after the mapping project is completed.

Below is a short Q&A I had with Andrew Ross from DAI's GIS office about his work and how Open Atrium is helping.

Q: Can you tell us about your project and your team in Pakistan?

A: We are working on a capacity building project to assist organizations in Peshawar and develop capacity in many areas. Specifically our project currently has four GIS staff based in Peshawar, and we also have an office in Islamabad. We produce GIS products for a number of clients and share development statistics with other organizations. We are also developing systems to help share this data. Workflows are difficult to establish as our staff are frequently move between offices.

Lightning talks on the Boxes module and a new Drupal site and its data migration

The March Washington, DC Drupal meetup will be tonight (3/8) at 7:00 pm at Stetson's. After missing last month's meetup due to the snowmaggedons, it will be great to catch up with everyone.

As usual there will be a round of lightning talk where anyone can take the floor and talk - for five minutes or less - on a Drupal related topic. So far two people have volunteered:

• Jeff will talk about the Boxes module, which produces exportable and Spaces-friendly blocks

• StevenC has volunteered to talk about a website he built for a local construction firm that included a data migration from a legacy MS SQL database and some complex back-end programming.

If you'd like to give a talk, post about it in the comments here and come ready to talk. More details on the meetup are in the Washington DC Drupal group. Hope to see you tonight!

Drupal, dot gov, and a data challenge this week in Washington, DC

After a positively beautiful weekend here in Washington, DC, we're set to have even warmer weather through most of the week. Dare I say that spring is finally here? While it will be tempting to be outdoors as much as possible this week, there are some interesting technology events scheduled that may just convince you to head into a bar. Below are the events that caught our eye, and you can find a full like at DC Tech Events. Have a great week!

March 8

7:00 - 9:00 pm

DC Drupal Meetup: Interested in learning more about Drupal - the increasingly popular open source framework - or want to geek out with other Drupal developers? Then come out to tonight's meetup for good conversations about the content management system and how it's being used by developers in town. We'll be there :)

Wednesday, March 10

5:30

New Tech Happy Hour: This networking event brings together people working in new technology and ICT4D to talk and share ideas, and usually draws a big group.

6:00 pm

Developers Roundtable + AppShare: This meetup is an opportunity to pick the brains of other local technologists, with topics of technology, marketing, and operations all on the table. Also, there will be pizza and beer.

To better tag content in multilingual versions of Managing News

We just added support for Yahoo! Placemaker's geo-recognition service to Managing News. This is an optional replacement for the default place-term lookup feature and can be enabled and configured in the administration section of Managing News. For those new to automatic geotagging services, Placemaker is a free API that leverages Yahoo's GeoPlanet location database to identify where in the world a particular piece of content is relevant.

Here is a look at Managing News using Placemaker to tag content in Chinese:

There are a few reasons we wanted to do this. The first is that we've been working on multilingual versions of Managing News and found that the core geotagger is not able to correctly identify word breaks in languages like Chinese and Arabic. The Placemaker API is able to receive and parse content in 21 languages and return place names in the language you request. This flexibility greatly extends the accessibility of Managing News to non-English users.

Second, it's not always easy to find a list of location terms with latitude and longitude for the places for which we want to track news. Will White has a great post about customizing maps on Managing News, but we wanted to make it even easier for end-users to download the software and get going.

To get set up using Placemaker geotagging with Managing News check out the DRUPAL-6--1 branch of the Managing News profile on Drupal.org. The included makefile will let you build all the necessary components from source. This feature will be included in the next release of the project from ManagingNews.com, but we wanted to let folks who couldn't wait access it now.

Citizen Technology and Election Accountability Conference in Nairobi, Kenya

National Democratic Institute's conference on Citizen Technology and Election Accountability here in Nairobi just wrapped up. The two days brought together election experts and technologists from all over the region. Much of our discussions focused on how to move away from relying on impressionistic and anecdotal data and start properly integrating technology into on the ground operations to provide more real time, representational data that will help teams better communicate about complex situations.

I got to talk about real time election observations and mobile data collection (specifically our recent integration of Slingshot SMS into Managing News) and led a session on publishing data for "Results Management and Visualizing Election Information" where I discussed our Afghanistan Election work. In addition to walking through the communications strategy behind our Afghanistan election data project from last fall, I showed the first public example of "Maps on a Stick", a client-side mapping tool that we will be releasing later this week. Maps on a Stick makes it easy to carry around custom map tilesets and mash them up with information (read KML data) and is designed for low and no bandwidth environments. The bad internet at the conference venue made the demo awesome.

Create a single sign-on experience based on Drupal's OpenID stack

Small pieces loosely joined is one of the defining paradigms of the web and in many cases we recommend investing in two or three smaller websites rather than a single big one. Smaller sites are cheaper to build and maintain and by keeping sites independent you give yourself maximum flexibility for future decisions.

A big piece that has been missing in making loosely coupled sites work well is a solution for scattered identities. Imagine the following scenario. In a set of five independent websites you would like to have users sign in once and be able to access any of these sites. Further you would like to have a central location where you can see user activity on all of the five sites. And of course you'd like to allow users to update information like their email address centrally and have the next notification from any of the five sites go to the new address.

This is exactly the challenge that one of our clients - Sandusky Register - faced with several loosely joined Drupal sites. After a survey of available single sign-on solutions, we decided to go with an OpenID based approach since we needed to support different domains, wanted to avoid sharing user tables and did not want to add complex system requirements for browser clients or the server.

Making using OpenID really simple

The great advantage of this scenario is that we know which five sites need to play nicely together and all of them are Drupal sites under the client's control. This premise allows us to add an additional site as a designated OpenID provider that we call "Hub" and make all five sites point to the Hub as their default identity provider. The resulting user experience is somewhat similar to what we are used to from Google or Yahoo. When users click 'sign up' on a site, they are kicked over to the Hub to authenticate and then transferred back and automatically logged in to the original site. Check out the screencast:

A demo of OpenID simple sign-on can be downloaded from github in the form of two Drupal install profiles. The two modules that are at the core of the simple sign-on experience are OpenID SSO and OpenID Provider SSO. If you take a quick look at the source, you will see that they are quite simple - both modules simplify only slightly the behavior of the OpenID and OpenID Provider module (props to walkah for both modules at this point!).

Details on the latest release of Aegir and what it will allow you to do

The Aegir development team has just released our latest piece of tech out onto the world. Aegir 0.4 Alpha 6 is the first major step in a series of milestones reaching toward the goals we have for 0.4. The biggest change in this release is that Aegir now supports multi-server use. You can now see a listing of all the servers that your Aegir site has access to provision sites on, letting you more easily configure your environment correctly.

In previous releases of Aegir, there were two separate node types that represented database servers and web servers respectively. We discovered pretty early on that this was problematic when trying to represent the real network topology we wanted to manage. If you had two web servers that each had a database server installed, we were required to share the single localhost database node between them, and even then this only worked if the two database servers were configured in EXACTLY the same way.

The major change in this release is that we've created a single new node type - named "server". We've then abstracted each of the previous servers or daemons into a new layer called "services". Each conceptual server node may have multiple services it provides (such as mysql or apache).

PHP, startups, elearning, and photography this week in Washington, DC

Finally, some warmer weather! It's still a bit premature to say that spring is on the way with snow in the forecast for Wednesday, but we will get a nice break from the freezing temperatures for most of the week. It's a great excuse to head out of your apartment/house to check out a local technology event. Below are the events that caught our eye, and I have to say that I'm most excited for the DCist Exposed photography exhibit on Saturday - though I'm definitely biased from working with one of the exhibitors. A full calendar of events is up over at DC Tech Events. Have a great week!

Tuesday, March 2

7:00 pm

NSCoderNightDC: Do you develop for macs or the iphone, or are you looking to get started? Come out to meet other developers and talk technology and the latest random Mac/iPhone news of the week, over some tasty desserts.

Wednesday, March 3

6:00 - 8:00 pm

DC PHP Developers Beverage Meetup: This monthly meetup is a chance for php developers to get together to talk code casually over a few beers. Just look for the fuzzy blue elePHPant once you get to the bar.

We are sharing our latest work @DrupalconSF - vote for the sessions you would like to see

Our presentation proposals this year for DrupalCon SF focus on the processes and tools we use in our products like Open Atrium and Managing News. A lot has moved in the last six months. Here is a run down of what we are planning on sharing. If you like what you see, vote for them!

Jeff will cover Managing and deploying configuration with exportables and the Features module configuration, set-up, and administration.

Young will present The Heart of Open Atrium: Context, PURL, and Spaces Under the Hood.

Alex will share his new aggregation work in Aggregate and import with Feeds!, which will also have a real time web angle looking at his recent PubSubHubbub Support for Drupal.

Adrian and the Aegir team will of course present on the Aegir Hosting System - one Drupal to rule them all.

On the business side, Eric is joining a few panels to talk about:

• Strategies for Community Contributions
• Drupal as a Web Application/Product Platform
• Harnessing Drupal for Your Next Web Project
• Best Practices For Managing a Drupal Firm

New map tiles are approximately 75-80% smaller

We're currently in the process of updating our set of world maps for MapBox to update data, improve visuals, and increase speed. Here's what will change:

• Move from 24-bit PNGs to 8-bit PNGs with a 32-color palette
• Adjustments to labels to improve how different-sized countries are handled
• Additions of provincial and state labels that were missing in the previous version

After testing various palette sizes for efficiency and attractiveness, we settled on a maximum of 32 colors per image. This gives us a major file size reduction from full-color PNGs, while still giving us good flexibility for smooth text and lines. With our initial tests we had hoped that a palette with as few as 16 colors would be sufficient, but in practice it caused too many oddities and jaggy lines. Increasing the palette to 32 colors was a very small sacrifice in average file size, especially compared to our old tiles, and gave us the appearance we wanted.

Efficiently aggregate news in realtime with Feeds

The latest version of Feeds has realtime web feed subscriptions based on PubSubHubbub (PSHB). Here I explain when this could benefit you and how you can set up your site to aggregate news efficiently in realtime.

The polling problem

If you have ever worked with an RSS or Atom aggregation engine like Drupal core's Aggregator, SimpleFeed, FeedAPI, or Feeds, you will be very familiar with their common shortcoming: they use the "are we there yet?" approach. They poll a feed every 30 or 60 minutes and see whether there is something new to be imported.

This approach is a waste of resources. Take for instance this blog, which publishes around three posts a week. If you poll it every 30 minutes, you ask it "are we there yet?" 336 times a week in order to get just three positive responses. In other terms, you would download around 11MB a week from developmentseed.org to get the 9K in blog posts you care about.

If you don't aggregate one blog but instead 100, you'll download over a gigabyte of data a week just in polling. What's worse is that once you start adding many feeds to a single system, you find another common problem with this approach: feed polling can back up. When this happens your system doesn't visit a feed every 30 minutes - instead visiting it every 60 minutes or at even less frequent intervals, resulting in articles coming in sometimes hours after they have been posted.

PubSubHubbub

PubSubHubbub addresses the polling problem. It notifies a feed subscriber when new content is available, thus rendering the "are we there yet?" question virtually obsolete.

Here is how it works. Feeds that support PubSubHubbub include a link tag that specifies a hub that distributes update notifications. See for example the hub link in the Haiti Innovation News feed:

<atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="hub" href="http://pubsubhubbub.appspot.com/" />

Upon pulling this feed for the first time, a subscription request can be sent to the hub specified in this link ("http://pubsubhubbub.appspot.com/"). When new content is added to the feed or when existing content is changed, the hub will send a notification containing the new and changed content to a notification URL specified upon subscription. This is the part of the magic that matters for the subscriber - if you are interested in the details or the publisher and hub side, check out PubSubHubbub's website.

Right now blogs like ReadWriteWeb and Gawker support PubSubHubbub. Google (no surprise as the PSHB inventors are Google employees) threw its weight behind the protocol by supporting PSHB for all Feedburner feeds and in Google Buzz. The Bay Area startup SuperFeedr offers a blanket subscription service that even enables PSHB notifications for feeds that don't support PSHB natively.

Lightweight systems for data collection meet powerful tools for analytics

We can now dynamically map and visualize real time SMS messages in Managing News using the new SlingshotSMS feature. All the code is available on github. This release is particularly exciting for us because it ties together two of our core projects in such a way that each is made better. SlingshotSMS is a lightweight SMS gateway that can be run off of a USB drive, needing only a GSM modem and an internet connection to act as a bridge between mobile phones and the web. Managing News is a powerful data aggregator and visualization tool that lets distributed teams work together to make large amounts of information useful. Together, they provide an extensible framework for teams conducting mobile data collection projects in the field.

Extensibility is key here because we need this to meet a wide variety of use cases in order for it to be useful. We have been particularly focused on use cases related to election monitoring, but this is just one of many possible applications. Here's a graphic that Saman made illustrating how the system works:

This technology is meant to accompany your existing processes of data collection. You have people in the field, they have cellphones, you have a phone back in headquarters, and they can text in messages to you that are then relayed to a visualization space, which helps keep you and your team on the same page.

Since SlingshotSMS runs on a USB drive, you just plug it in, plug in your phone, and set up what website you want to have the SMS messages sent to. The SMS messages are turned into RSS 2.0 and PUSHed, like as a fat ping. You computer just needs internet to send these messages.

Usability, Crisis Camps, and photography this week in Washington, DC

The thaw (or melt) is on, and the piles of snow are (finally!) starting to disappear. That's as good of a reason as any to go out after work to check out a fun technology event. Below are the week's events that caught our eye, and you can find a full list over at DC Tech Events. Have a great week!

Tuesday, February 23

6:00 - 8:00 pm

DCPHP Beverage Subgroup: Want to talk php code over a few beers? Then come out to this meetup at the Irish Pub Four Courts.

6:30 - 8:30 pm

UX Show and Tell DC: Come out to talk about usability, show off some sites you've recently built, and get and give feedback on the usability of other websites and designs.

Wednesday, February 24

6:30 pm

DC Media Makers: The Who-What-How: Impact of Community Technology and Crisis Camps: Have you been amazed with how community technology is helping humanitarian relief efforts after the earthquake in Haiti? Wish more was done? Come out to this meetup to talk about how CrisisCampHaiti is helping the relief efforts and its effectiveness.

Using custom map tiles and open geo data to build a hyper local news tracker

Managing News ships with its maps and geotagging features preconfigured for tracking global news on a world map. But user interface improvements in Managing News Beta 7 make it easy to use an alternate map layer and import custom location data so you can track news in a specific geographic region.

This screencast shows how I was able to tailor my maps to focus on the city of Washington, DC and plot the stories published on various neighborhood blogs. I used open geographic data from the DC government and the DC Nightvision tileset from MapBox.