I spent some time over the weekend really digging into Adam Lofting’s work with the metrics around #teachtheweb looking for insights into Marc Surman’s call for a Mozilla Academy by the year 2017.
You know family is town if you would rather spend the holiday looking at analytics rather than aunties. Still the numbers, and the KPI’s that came out of Portland shine a light on why a global movement is needed if webmaker….sorry….Mozilla Learning Networks is to be successful.
I am not a click counter by trade so I apologize in advance for my ignorance.
Retention
The problem, unlike relatives, users of webmaker tools leave pretty quickly.
The legacy webmaker tools, empowered users to read, write, and participate on the open web. Yet the conversion and retention rates are awful. After sixty days just over a half of one percent are still active and after 90 days the number might as well be zero.
Empowering tools means nothing without community.
People of #TeachTheWeb
The primary goal of the Mozilla Academy has to be connecting people. This will be done through both the Learning Networks and the Products.
The number of people within the Learning Networks is growing. The club data is still blank as that is a new initiative and the click counters are still debating what constitutes and club and a city.
In many ways I am trying to understand the vision and role of the Learning Networks. I kind of see a hierarchy of involvement. Maker Party evolves to club, club morphs into Hives, Hives become relatively independent and self sustaining.
All four of these levels fit under Mozilla Academy. Yet where should we put our efforts when discussing this global classroom?
Maker Party, due to this new vision, and a shift in funding, now are year round affairs. I am seeing this being conflated with the Mozilla Web Clubs. I disagree. Year round Maker Parties should be more flash events, that can be done with limited support.
Club First Strategy
Mozilla Academy is a lofty goal but when I look at the metrics that matter I think it becomes clear that building club capacity has to come first. Marc talked about the literacy, skill, and craft involved in the Learning Networks but these are simply metaphors to how enculturated one is in the discourses of the Open Web.
In a people first strategy you recognize that social capital, especially when you rely on volunteer contributors such as myself, is your greatest asset. We need to have a feeder system to move folks from skill to craft. We need people to use and evangelize the webmaker products and the Manifesto Values if we want to reach the KPI’s. More importantly you need people living on the Open Web.
Community Centered Design
The first goal of the Learning Networks should be the cultivation of community. The success of the Mozilla Foundation in the last few years has been nothing short of amazing (the fundraising especially). This growth rate has required massive investments of time and treasure and has centralized much of the effort.
What I do not see is an uptick in social media impressions and sustained involvement as well. I think the metrics team needs to build (they probably do but its not in the public dashboard) and pay attention to the social media metrics as much as the tool use.
When you look at the #teachtheweb hashtag, the #mozacademy hashtag, discourse.webmaker.org, the IRC chat, and Google+ you see few interactions or the cultivation of relationships. Engagement is notoriously impossible to measure but I know when I see silence in Web spaces.
I have seen new clubs from India start telling their story on Twitter. There has been an uptick of individuals on Google+ seeking entrepreneurial help (microbusinesses and the Academy..hmmm?), but it is quiet.
Without people the KPI’s are meaningless
Hi Greg,
Thanks for reading my blog and for digging into so many of the details. It’s extremely valuable to have another pair of eyes on this.
I think you’ve pulled together our overarching conversations in a really clear way and called out the main challenges we’re working on.
I’ll comment on some of the metrics pieces, as that’s where my head is…
* The Webmaker legacy tools work well in face-to-face teaching scenarios (i.e. in tandem with Webmaker events) but we also have many people who come to Webmaker independently (through Mozilla referrals and elsewhere). The existing retention metrics are so low because of the combination of these two things: (1) lots of people find Webmaker and think it’s a tool they can use on their own to make and learn and sign-up (2) that’s not what we have been building in the past. Making a remote-learning product ‘sticky’ is tough, but we’ve made retention (aka sustained engagement) a KPI for that reason. If we get this right it’s a way to engage deeply with hundred’s of thousands of people over a long period of time in the nuances that make the open web so great.
* As you note, The Learning Networks data points are still being worked out (along with branding and naming challenges which make all of these conversations a bit more confusing in the meantime). But overall, the KPIs we’re optimising for are all about people, and building sustained relationships with them. The details we’re working out now for metrics are to ensure that what we’re counting meaningfully reflects what we care about. The KPIs are the ‘means’ rather than the ‘end’.
* For social metrics. Actually, no, I’m not tracking this right now. There are things to learn from this data (I’ve worked a lot with it in the past) but it’s a secondary priority in relation to direct engagement through the tools and programs. In short, lots of people talking about the product won’t make up for people not using it, and there are lots of ways to game social metrics without having real impact on the goals. That’s not to dismiss it entirely, but just to say it’s further down my priority list right now. And on that note, the ‘metrics team’, for the Foundation at least, is just me 🙂
I hope that’s all useful context, and think it’s already in line with your understanding. And I’d love to keep this conversation going throughout the year.
Adam,
Thank you for the comment. Your work is so fascinating. Especially the A/B testing post. Had no idea what a few words in a snippet could do.
I was looking the metrics through the lens of the #MozAcademy so the view on retention is flavored by my last three years.
It looks like then we need a plan to focus on the conversion rates of visitors to users. I saw that metric in the dashboard.
The tools do work in tandem with the clubs. Learning was always at the center of the design. I love the new webmaker prototypes. They will get the mass market making. It will be up to us to figure out how to apply them to teaching. These seems like a fundemental shift.
The legacy tools have issues. The gallery didn’t make learning pathways noticeable, the teaching kits weren’t easy to find and there were 14,000 different places to go. Every visit to webmaker could end in a different reader experience.
I still love the tools. Thimble taught me everything I know about the web. A year ago, literally today, I tweeted the most basic question about HTML and CSS. Now I can mess up web pages in ways I never thought possible.
I was able to learn by myself because I stumbled into a community. I can’t wait to see what we can do with the new tools. I think they need to be a major part of the UX in the Academy.
Yes we will not be the first to try and make a global classroom. In fact MoFo has tried a few times already. I went back and read the GitHub issue on defining a club and a city. To count on the dashboard clubs will have to actively engaged. Once the definition is solidified I would add it to the dashboard.
I agree the KPIs make sense for now. Atleast until after the “softest of launches” and the breaking of ground on the Academy.
I consider connections to be the most important content in any connected learning environment. I disagree with you that you can have folks talking about tools without using them. Tools shape human activity. More importantly I do not think they would be gained because I do not see these social metrics or marketing metrics. The impressions on social media are learning metrics.
I believe #MozAcademy should be a distributed classroom and not a destination classroom. So I was pulling in your work to highlight a need in the Mozilla Learning Networks. In the 2-3 years I have been involved with #teachtheweb there have only been a handful of people (super deadicated people) networking. This may have just as much to do with low retention as it does with tools designed for too specific of a use case. People talking about tools, means people are using tools, which means higher retention. Twitter, Tumblr, Google+ are where people so to learn how to read, write, and participate on the web.
I didn’t realize you are an army of one on the metrics team. Your squad rocks.
@jgmac1106 thanks Greg! And I’ve only just seen you replied in your post too (weirdly I didn’t get an email notification)
@adamlofting older post on discovering dashboard Got me thinking after talk w/ @kevinzawacki & @HPaulJ we have tools to make awesome happen