School of Information Blogs

Yesterday, Pew Internet and American Life Project (in collaboration with Berkman) unveiled a brilliant report about “Teens, Social Media, and Privacy.” As a researcher who’s been in the trenches on these topics for a long time now, none of their finding surprised me but it still gives me absolute delight when our data is so beautifully in synch. I want to quickly discuss two important issues that this report raise.

Race is a factor in explaining differences in teen social media use.

Pew provides important measures on shifts in social media, including the continued saturation of Facebook, the decline of MySpace, and the rise of other social media sites (e.g., Twitter, Instagram). When they drill down on race, they find notable differences in adoption. For example, they highlight data that is the source of “black Twitter” narratives: 39% of African-American teens use Twitter compared to 23% of white teens.

Most of the report is dedicated to the increase in teen sharing, but once again, we start to see some race differences. For example, 95% of white social media-using teens share their “real name” on at least one service while 77% of African-American teens do. And while 39% of African-American teens on social media say that they post fake information, only 21% of white teens say they do this.

Teens’ practices on social media also differ by race. For example, on Facebook, 48% of African-American teens befriend celebrities, athletes, or musicians while one 25% of white teen users do.

While media and policy discussions of teens tend to narrate them as an homogenous group, there are serious and significant differences in practices and attitudes among teens. Race is not the only factor, but it is a factor. And Pew’s data on the differences across race highlight this.

Of course, race isn’t actually what’s driving what we see as race differences. The world in which teens live is segregated and shaped by race. Teens are more likely to interact with people of the same race and their norms, practices, and values are shaped by the people around them. So what we’re actually seeing is a manifestation of network effects. And the differences in the Pew report point to black youth’s increased interest in being a part of public life, their heightened distrust of those who hold power over them, and their notable appreciation for pop culture. These differences are by no means new, but what we’re seeing is that social media is reflecting back at us cultural differences shaped by race that are pervasive across America.

Teens are sharing a lot of content, but they’re also quite savvy.

Pew’s report shows an increase in teens’ willingness to share all sorts of demographic, contact, and location data. This is precisely the data that makes privacy advocates anxious. At the same time, their data show that teens are well-aware of privacy settings and have changed the defaults even if they don’t choose to manage the accessibility of each content piece they share. They’re also deleting friends (74%), deleting previous posts (59%), blocking people (58%), deleting comments (53%), detagging themselves (45%), and providing fake info (26%).

My favorite finding of Pew’s is that 58% of teens cloak their messages either through inside jokes or other obscure references, with more older teens (62%) engaging in this practice than younger teens (46%). This is the practice that I’ve seen significantly rise since I first started doing work on teens’ engagement with social media. It’s the source of what Alice Marwick and I describe as “social steganography” in our paper on teen privacy practices.

While adults are often anxious about shared data that might be used by government agencies, advertisers, or evil older men, teens are much more attentive to those who hold immediate power over them – parents, teachers, college admissions officers, army recruiters, etc. To adults, services like Facebook that may seem “private” because you can use privacy tools, but they don’t feel that way to youth who feel like their privacy is invaded on a daily basis. (This, btw, is part of why teens feel like Twitter is more intimate than Facebook. And why you see data like Pew’s that show that teens on Facebook have, on average 300 friends while, on Twitter, they have 79 friends.) Most teens aren’t worried about strangers; they’re worried about getting in trouble.

Over the last few years, I’ve watched as teens have given up on controlling access to content. It’s too hard, too frustrating, and technology simply can’t fix the power issues. Instead, what they’ve been doing is focusing on controlling access to meaning. A comment might look like it means one thing, when in fact it means something quite different. By cloaking their accessible content, teens reclaim power over those who they know who are surveilling them. This practice is still only really emerging en masse, so I was delighted that Pew could put numbers to it. I should note that, as Instagram grows, I’m seeing more and more of this. A picture of a donut may not be about a donut. While adults worry about how teens’ demographic data might be used, teens are becoming much more savvy at finding ways to encode their content and achieve privacy in public.

…

Anyhow, I have much more to say about Pew’s awesome report, but I wanted to provide a few thoughts and invite y’all to read it. If there is data that you’re curious about or would love me to analyze more explicitly, leave a comment or drop me a note. I’m happy to dive in more deeply on their findings.

I’m a big believer in taking breaks from work, research, and my mediated life to travel, trek, and explore. Years ago, I implemented email sabbaticals to give myself space to take time off without being overwhelmed by the ongoing flow of emails and expectations. I documented the process because too many people assumed that I just disappeared, unannounced, leaving my colleagues and friends in a lurch. Actually, email sabbaticals are a process because they have stages where I make sure that everyone who depends on me is taken care of before I disappear into the void. I typically announce them to colleagues 3-6 months ahead of time and give a public warning 6 weeks ahead of time. And then I have a process upon return where I check in with folks. When I disappear, my goal is to never leave people that depend on me faltering. This isn’t a perfect process, but I’ve found that letting people know well ahead of time guarantees that I finish out projects and account for commitments before taking time off without the lingering stress of unfinished business.

Normally, when I plan on taking a massive break, I know the dates well ahead of time and can help everyone plan and prepare. I’m now facing a different dilemma. I know that sometime in the next n weeks, I’m going to take some serious time off. The problem is that I’m not quite sure exactly when that will occur nor am I exactly sure how long I will be gone or how much I will want to be off email. Y’see, rather than taking a vacation this summer, I’m planning on having a baby.

As folks who have seen me in recent months know, I’m pregnant, due August 3. I look pretty silly, particularly given that I’m a klutz and managed to obtain a lisfranc injury on top of my bulging belly which means that I look like a whale on a scooter. But I’m still chugging along, finishing out writing projects (the book is in copyediting right now!) and starting new ones (nom nom trouble). I’m planning on taking parental leave when the kiddo arrives, but I’m not yet sure how long I’ll want to be focused solely on being a parent. Given my love of research, I expect that I’ll ease back in to email and projects throughout the fall. And I plan to start traveling again with a spectacular book tour when the book launches in early 2014. But I also know that I’m entering a world of uncertainty.

I don’t intend to detail my life as a mother on this blog because, frankly, I don’t want to. But I do want to give y’all a heads up that I intend to disappear for a bit when the time comes. So if there are things you need from me this summer or early fall, perhaps ping me sooner rather than later. And please understand that when I do get to finally meet the kiddo, I will probably be pretty inattentive to email and the internets for a while. I suspect that this will look and feel differently than my email sabbaticals, but I still don’t want to leave people in a lurch. So I wanted to give y’all a heads up.

A new adventure awaits!  Thanks for your patience!

It takes this producer eight hours to get his coffee down from the mountains to ADISA, the coop. #noteasy

Ariel at work with Ismael from Frutos de Selva, a coffee coop in Northern Peru. Our work: Acopio.

(This post was originally written for LinkedIn.  Go to the LinkedIn version to engage in the conversation.)

I’ve been scratching my head trying to think about how to understand the different facets of labor that are shaping contemporary life. I don’t have good answers; I only have some provocations and a few questions, but I would love to hear your thoughts.

As a teenager, I was a sandwich artist. I’d arrive at work, don my uniform and clock in. I had a long list of responsibilities – chopping onions, cleaning the shop, preparing the food, etc. Everything was formulaic. I can still recite how to ask a customer if they want onions, pickles, lettuce, green peppers, or black olives. The job paid minimum wage and was defined by doing pre-specified tasks in an efficient and predictable manner with a smile. When my compatriots got fired, it was almost always for being late. In-between making sandwiches and doing the rote tasks, we would gossip and chat, complain about regulars and talk about run-ins with cops (who demanded free food which meant a dock in pay for whoever was working). And when my shift was over, I’d clock out and leave, forgetting about Subway even though the scent lingered and filled my car.

Today, I have my dream job. I’m a researcher who gets to follow my passions, investigate things that make me curious. I manage my own schedule and task list. Some days, I wake up and just read for hours. I write blog posts and books, travel, meet people, and give talks. I ask people about their lives and observe their practices. I think for a living. And I’m paid ridiculously well to be thoughtful, creative, and provocative. I am doing something related to my profession 80-100 hours per week, but I love 80% of those hours. I can schedule doctor’s appointments midday, but I also wake up in the middle of the night with ideas and end up writing while normal people sleep. Every aspect of my life blurs. I can never tell whether or not a dinner counts as “work” or “play” when the conversation moves between analyzing the gender performance of Game of Thrones and discussing the technical model of Hadoop. And since I spend most of my days in front of my computer or on my phone, it’s often hard to distinguish between labor and procrastination. I can delude myself into believing that keeping up with the New York Times has professional consequences but even I cannot justify my determination to conquer Betaworks’ new Dots game (shouldn’t testing new apps count for something??). Of course, who can tell if my furrowed brow and intense focus on my device is work-focused or not. Heck, I can’t tell half the time.

In the digital world, the line between what is fun and what is work is often complicated. There are people whose job it is to produce tweets and updates as a professional act, but they sit beside people in a digital environment who produce this content because it’s connected to how they’re socializing with their friends. Socializing, networking, and advertising are often intertwined in social media, making it hard to distinguish between professional and personal, paid labor and career advancement.

There are are people who understand that they’re “on the job” because of where they are physically, but there are also people whose model of work is more connected to their interaction with their Blackberries or the kinds of actions that they’re taking. And then there are people like me who have lost all sense of where the boundaries lie.

There is tremendous anxiety among white collar workers about how blurry the boundaries have gotten, but little consideration for how that blurriness is itself a mark of privilege. More often than not, those with more social status have blurrier boundaries around space, place, and time. Sometimes, this privilege comes with a higher paycheck, but freelance writers have a level of class privilege that is not afforded to the punch-in, punch-out workforce even if their actual income is paltry.

Often, there’s rampant financial and status inequality between those whose careers are defined by blurred boundaries and those who work in a prescribed manner. Many C-level execs justify their exorbitant salaries through the logic of risks and burdens without accounting for the freedoms and flexibility associated with this kind of work or recognizing the physical, psychological, and cultural costs that come with manual, service, or rote labor in prescribed environments. The freedom to control one’s own schedule has value in and of itself. Yet, not everyone with economic resources feels as though they are in control of their lives. And it’s often easier to blame the technology that tethers them than work out the dynamics of agency that are at work.

What’s at stake isn’t just that work is invading people’s personal lives or that certain types of labor are undervalued. It’s also that the notion of fun or social is increasingly narrated through the frame of work and productivity, advancement and professional investment.

Labor is often understood to be any action that increases market capitalization. But then how do we understand the practice of networking that is assumed as key to many white collar jobs? And what happens when, as is often the case in the digital world, play has capital value?

In academic circles, debates are raging over the notion of “free labor.” Much of what people contribute to social media sites is monetized by corporations. People don’t get paid for their data and, more often than not, their data is used by corporations to target them – or people like them – to produce advertising revenue for the company. The high profitability of major tech companies has prompted outrage among critics who feel as though the money is being made off of the backs of individual’s labor. Yet most of these people don’t see their activities as labor. They’re hanging out with friends or, even if they’re being professional, they’re networking. Accounting for every action and interaction as labor or work doesn’t just put a burden on social engagements; it brings the logic of work into the personal sphere.

Most of these dynamics predate the internet, but digital technologies are magnifying their salience. People keep returning to the mantra of “work-life balance” as a model for thinking about their lives, even as it’s hard to distinguish between what constitutes work and what constitutes life, which is presumably non-work. But this binary makes little sense for many people. And it raises a serious question: what does labor mean in a digital ecosystem where sociality is monetized and personal and professional identities are blurred?

As you think about your own professional practices, how do you define what constitutes work? How do you think labor should be understood in a networked world? And what does fairness in compensation look like when the notion of clocking in and clocking out are passe?

(This post was originally written for LinkedIn.  Go to the LinkedIn version to engage in the conversation.)

Starting another year – this time in Lima, Peru.

Ulanowicz (2000) proposes in information-theoretic terms several metrics for ecosystem health, where one models an ecosystem as a for example a trophic network. Principal among them ascendancy , which is a measure of the extent to which energy flows in the system are predictably structured weighted by the total energy of the system. He believes that systems tend towards greater ascendancy in expectation, and that this is predictive of ecological ‘succession’ (and to some extent ecological fitness). On the other hand, overhead, which is unpredictability (perhaps, inefficiency) in energy flows (“free energy”?), are important for the system’s resiliency towards external shocks.

At least in the papers I’ve read so far, Ulanowicz is not mathematically specific about the mechanism that leads to greater ascendancy, though he sketches some explanations. Autocatalytic cycles within the network reinforce their own positive perturbations and mutations, drawing in resources from external sources, crowding out and competing with them. These cycles become agents in themselves, exerting what Ulanwicz suggests is Aristotelian final or formal causal power on the lower level components. In this way, freely floating energy is drawn into structures of increasing magnificence and complexity.

I’m reminded on Bataille’s The Accursed Share, in which he attempts to account for societal differences and the arc of human history through the use of its excess energy. “The sexual act is in time what the tiger is in space,” he says, insightfully. The tiger, as an apex predator, is flame that clings brilliantly to the less glamorous ecosystem that supports it. That is why we adore them. And yet, their existence is fragile, as it depends on both the efficiency and stability of the rest of its network. When its environment is disturbed, it is the first to suffer.

Ulanowicz cites himself suggesting that a similar framework could be used to analyze computer networks. I have not read his account yet, though I anticipate several difficulties. He suggests that data flows in a computer network are analogous to energy flows within an ecosystem. That has intuitive appeal, but obscures the fact that some data is more valuable than others. A better analogy might be money as a substitute for energy. Or maybe there is a way to reduce both to a common currency, at least for modeling purposes.

Econophysics has been gaining steam, albeit controversially. Without knowing anything about it but based just on statistical hunches, I suspect that this comes down to using more complex models on the super duper complex phenomenon of the economy, and demonstrating their success there. In other words, I’m just guessing that the success of econophysics modeling is due to the greater degrees of freedom in the physics models compared to non-dynamic, structural equilibrium models. However, since ecology models the evolutionary dynamics of multiple competing agents (and systems of those agents), its possible that those models could capture quite a bit of what’s really going on and even be a source of strategic insight.

Indeed, economics already has a sense of stable versus unstable equilibria that resonate with the idea of stability of ecological succession. These ideas translate into game theoretic analysis as well. As we do more work with Strategic Bayesian Networks or other constructs to model equilibrium strategies in a networked, multi-agent system, I wonder if we can reproduce Ulanowicz’s results and use his ideas about ascendancy (which, I’ve got to say, are extraordinary and profound) to provide insight into the information economy.

I think that will require translating he ecosystem modeling into Judea Pearl’s framework for causal reasoning. Having been indoctrinated in Pearl’s framework in much of my training, I believe that it is general enough to subsume Ulanowicz’s results. But I have some doubt. In some of his later writings Ulanowicz refers explicitly to a “Hegelian dialectic” between order and disorder as a consequence of some of his theories, and between that and his insistence on his departure from mechanistic thinking over the course of his long career, I am worried that he may have transcended what it’s possible to do even with the modeling power of Bayesian networks. The question is: what then? It may be that once one’s work sublimates beyond our ability to model explicitly and intervene strategically, it becomes irrelevant. (I get the sense that in academia, Ulanwicz’s scientific philosophizing is a privilege reserved for someone tenured who late in their career is free to make his peace with the world in their own way) But reading his papers is so exhilarating to me. I’ve had no prior exposure to ecology before this, so his papers are packed with fresh ideas. So while I don’t know how to justify it to any of my mentors or colleagues, I think I just have to keep diving into it when I can, on the side.

[error] /home/sb/ischool/cs294/hw3/src/main/scala/TestScript.scala:32: type mismatch;
[error] found : Array[wikilearn.TestScript.parser.Page]
[error] required: Array[wikilearn.WikiParser#Page]
[error] Note: wikilearn.TestScript.parser.Page <: wikilearn.WikiParser#Page, but class Array is invariant in type T.
[error] You may wish to investigate a wildcard type such as `_ <: wikilearn.WikiParser#Page`. (SLS 3.2.10)

wtf, Scala.  You know exactly what I’m trying to do here.

EDIT: I sent a link to the above post to David Winslow. He responded with a crystal clear explanation that was so great I asked him if I could include it here. This is it, below:

It’s a feature, not a bug :) This is actually the specific issue that Dart had in mind when they put this note in the language spec:

class Array[T] {
  def apply(i: Int): T // "producing" a T
  def update(i: Int, t: T): Unit // "consuming" a T
}

String[] strings = new String[1];
Object[] objects = strings;
objects[0] = Integer.valueOf(0); // the runtime error occurs at this step, but even if it didn't: 
System.out.println(strings[0]); // what happens here?

Anyway, the upshot is that immutable collections only use their types in covariant positions (you can get values out, but never insert) so they are much handier. Does your code work better if you replace your usage of Array with Vector? Alternatively, you can always provide the type parameter when you construct your array. Array(“”) is an Array[String], but Array[AnyRef](“”) is an Array[AnyRef].

On May 3rd, the Boy Scouts are considering lifting their ban on gays, and are putting a vote to the local and national councils. This means that it's easy to influence the vote by calling in and expressing your opinion. It's simple to do so, and more voices could change the direction of the Boy Scouts of America, allowing all boys to be included and accepted.

Here's what to do:

1. Find your local council
2. Give them a call or send them an email telling them your opinion.
3. Contact the national council via email: feedback@scouting.org.

Most important is to contact your local council. We need their phones to be ringing off the hook with people expressing their opinions. It truly takes no more than two minutes. Here's the San Diego council: (619) 298-6121, and the East Bay council: (925) 674-6100.

If you prefer a form letter, you can just do this one through the Human Rights Campaign (34,000 people already have).

Here's a simple transcript for you to follow:

I'm [calling/wrriting] to express my desire that the Boy Scouts immediately lift their ban on gays. This ban is discriminatory, outdated and pointless. Scouting teaches many great lessons to thousands of adolescents across the U.S. One lesson that Scouting should not teach is that homosexuality is somehow wrong or means for discrimination.

On May 3rd, I hope that you will help the BSA finally makes the right decision so it can continue to lead boys into being mature and accepting men.

Send this to the email above, and call your local council. This could make a difference.

I’ve been working with music files lately trying to get Steve Morrell‘s music online. In the process I’ve had to convert his albums, which I’ve ripped in .wav format, to .mp3.

To accomplish this, I’ve written a short bash script. It’s requires a number of tricks I wasn’t familiar with and had to look up.

#!/bin/bash

SAVEIF=$IFS
IFS=$(echo -en "\n\b")

for file in $(ls *wav)
do
  name=${file%%.wav}
  lame -V0 -h -b 160 --vbr-new $name.wav $name.mp3
done


IFS=$SAVEIFS

Though it isn’t recommended, I did the for loop on ls because I wanted to limit it to .wav files. But that means the script chokes on file names with spaces unless you swap out the IFS variable.

I used LAME for the conversion.

For our latest JayWSalon meetup, the topic is “inspiration”. Here is how the game was played: Write your inspirations (people, books, movies, technology, company) on post-its and put them up in following categories; and then talk about why.

Yiran, Connie and myself were helping with the session, and it was really fun and inspirational. To me, it certainly proves itself to be one of the best ways for people to learn about each other, and can build mutual understanding in a really short period of time.

I have encountered some quite difficult time in previous years, but I am fortunate enough that I’ve got to know numerous people that provided their sincere help. However, in this post, I am going to blog about my inspiration, the defining moment that changed my views on things.

She is Oprah Winfrey. As an African woman born into poverty in rural Mississippi in the 50s, she worked all the way up to be one of the most successful and powerful person on the planet. I have read extensively about her, but I would like to share her commencement speech @ Stanford.

* Feel: Lesson one, follow your feelings. If it feels right, move forward. If it doesn’t feel right, don’t do it.
— If it doesn’t feel right, don’t do it.
— “When you’re doing the work you’re meant to do, it feels right and every day is a bonus, regardless of what you’re getting paid.”

* Failure: Don’t react against a bad situation; merge with that situation instead.
— What matters most is what’s inside.
— What matters most is the sense of integrity, of quality and beauty.

*Happiness: You have to be in the moment. Whatever has happened to you in your past has no power over this present moment, because life is now.
— To be happy, you have to give something back.
— If you operate from the paradigm of service, I know your life will have more value and you will be happy.

I am trying to learn how to use Hadoop. I’m am trying to learn to program in Scala. I mostly forget how to program in Java. In this post I will take notes on things that come up as I try to get my frickin’ code to compile so I can run a Hadoop job.

There was a brief window in my life when I was becoming a good programmer. It was around the end of my second year as a professional software engineer that I could write original code to accomplish novel tasks.

Since then, the tools and my tasks have changed. For the most part, my coding has been about projects for classes, really just trying to get a basic competence in commodity open tools. So, my “programming” consists largely of cargo-culting code snippets and trying to get them to run in a slightly modified environment.

Right now I’ve got an SBT project; I’m trying to write a MapReduce job in Scala that will compile as a .jar that I can run on Hadoop.

One problem I’m having is there are apparently several different coding patterns for doing this, and several frameworks that are supposed to make my life easier. These include SMR, Shadoop, and Scalding. But since I’m doing this for a class and I actually want to learn something about how Hadoop works, I’m worried about having to good a level of abstraction.

So I’m somewhat perversely taking the Scala Wordcount example from jweslley’s Shadoop and make it dumber. I.e., not use Shadoop.

One thing that has been confusing as hell is that there Hadoop has a Mapper interface and a Mapper class, both with map() functions (1,2), but those functions haved different type signatures.

I started working with some other code that used the second map() function. One of the arguments to this function is of type Mapper.Context. I.e., the Context class is a nested member of the Mapper class. Unfortunately, referencing this class within Scala is super hairy. I saw a code snippet that did this:

override def map(key:Object, value:Text, context:Mapper[Object,Text,Text,IntWritable]#Context) = {
    for (t <-  value.toString().split("\\s")) {
      word.set(t)
      context.write(word, one)
    }
  }

But I couldn’t get this thing to compile. Kept getting this awesome error:

type Context is not a member of org.apache.hadoop.mapred.Mapper[java.lang.Object,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.IntWritable]

Note the gnarliness here. It’s not super clear whether or how Context is parameterized by the type parameters of Mapper. The docs for the Mapper class make it seem like you can refer to Context without type parameterization within the code of the class extending Mapper. But I didn’t see that until I had deleted everything and tried a different track, which was to use the Mapper interface in a class extending MapReduceBase.

Oddly, this interface hides the Context mechanic and instead introduces the Reporter class as a final argument to map(). I find this less intimidating for some reason. Probably because after years of working in Python and JavaScript my savvinness around the Java type hierarchy is both rusty and obsolete. With the added type magicalness of Scala to add complexity to the mix, I think I’ve got to steer towards the dumbest implementation possible. And at the level I’m at, it looks like I don’t ever have to touch or think about this Reporter.

So, now starting with the example from Shadoop, now I just need to decode the Scala syntactic sugar that Shadoop provides to figure out what the hell is actually going on.

Consider:

  class Map extends MapReduceBase with Mapper[LongWritable, Text, Text, IntWritable] {

    val one = 1

    def map(key: LongWritable, value: Text, output: OutputCollector[Text, IntWritable], reporter: Reporter) =
      (value split " ") foreach (output collect (_, one))
  }

This is beautiful concise code. But since I want to know something about the underlying program I’m going to uglify it by removing the implict conversions provided by Shadoop.

The Shadoop page provides a Java equivalent for this, but that’s not really what I want either. For some reason I demand the mildy more concise syntax of Scala over Java but not the kind of condensed, beautiful syntax Scala makes possible with additional libraries.

This compiles at least:

  class Map extends MapReduceBase with Mapper[LongWritable, Text, Text,
 IntWritable] {   

    val one = new IntWritable(1); 

    def map(key: LongWritable, value: Text, output: OutputCollector[Text,
     IntWritable], reporter: Reporter) = {
      var line = value.toString();

      for(word <- line.split(" ")){
        output.collect(new Text(word), one)
      }
    }
  }

What I find a little counterintuitive about this is that the OutputCollector doesn’t act like a dictionary, overwriting the key-value pair with each call to collect(). I guess since I’m making a new Text object with each new entry, that makes sense even if the collector is implemented as a hash map of some kind. (Shadoop hides this mechanism with implicit conversions, which is rad of course.)

Next comes the reducer. The Shadoop code is this:

def reduce(key: Text, values: Iterator[IntWritable],
            output: OutputCollector[Text, IntWritable], reporter: Reporter) = {
  val sum = values reduceLeft ((a: Int, b: Int) => a + b)
  output collect (key, sum)
}

Ok, so there’s a problem here. The whole point of using Scala to code a MapReduce job is so that you can use Scala’s built in reduceLeft function inside the reduce method of the Reducer. Because functional programming is awesome. By which I mean using built-in functions for things like map and reduce operations are awesome. And Scala supports functional programming, in at the very least that sense. And MapReduce as a computing framework is at least analogous to that paradigm in functional programming, and even has the same name. So, OMG.

Point being, no way in hell am I going to budge on this minor aesthetic point in my toy code. Instead, I’m going to brazenly pillage jweslley’s source code for the necessary implicit type conversion.

  implicit def javaIterator2Iterator[A](value: java.util.Iterator[A]) = new Iterator[A] {
    def hasNext = value.hasNext
    def next = value.next
  }

But not the other implicit conversions that would make my life easier. That would be too much.

Unfortunately, I couldn’t get this conversion to work right. Attempting to run the code gives me the following error:

[error] /home/cc/cs294/sp13/class/cs294-bg/hw3/wikilearn/src/main/scala/testIt/WordCount.scala:33: type mismatch;
[error]  found   : java.util.Iterator[org.apache.hadoop.io.IntWritable]
[error]  required: Iterator[org.apache.hadoop.io.IntWritable]
[error]       val sum = (values : scala.collection.Iterator[IntWritable]).reduceLeft (
[error]                  ^

It beats me why this doesn’t work. In my mental model of how implicit conversion is supposed to work, the java.util.Iterator[IntWritable] should be caught by the parameterized implicit conversion (which I defined within the Object scope) and converted no problemo.

I can’t find any easy explanation of this on-line at the moment. I suspect it’s a scoping issue or a limit to the parameterization of implicit conversions. Or maybe because Iterator is a trait, not a class? Instead I’m going to do the conversion explicitly in the method code.

After fussing around for a bit, I got:

    def reduce(key: Text, values: java.util.Iterator[IntWritable],                                                               
      output: OutputCollector[Text, IntWritable], reporter: Reporter) = {                                                        
      val svals = new scala.collection.Iterator[IntWritable]{
        def hasNext = values.hasNext
        def next = values.next
      }
      val sum = (svals : scala.collection.Iterator[IntWritable])\|
.reduceLeft (
       (a: IntWritable, b: IntWritable) => new IntWritable(a.get() + b.get())}
      ) 
      output collect (key, sum)
    }

…or, equivalently and more cleanly:

    def reduce(key: Text, values: java.util.Iterator[IntWritable],
      output: OutputCollector[Text, IntWritable], reporter: Reporter) = {

      val svals = new scala.collection.Iterator[Int]{
        def hasNext = values.hasNext
        def next = values.next.get
      }

      val sum = (svals : scala.collection.Iterator[Int]).reduceLeft (
        (a: Int, b: Int) => a + b
      )
      output collect (key, new IntWritable(sum))
    }
  }

I find the Scala syntax for defining the methods of an abstract class here pretty great (I hadn’t encountered it before). Since Iterator[A] is an abstract class, you define the methods next and hasNext inside the curly braces. What an elegant way to let people subclass abstract classes in an ad hoc way!

There’s one more compile error I had to bust around. This line was giving me noise:

conf setOutputFormat classOf[TextOutputFormat[_ <: WritableComparable, _ <: Writable]]

It was complaining that WriteComparable needed a type parameter. Not confident I could figure out exactly which parameter to set, I just made the signature tighter.

conf setOutputFormat classOf[TextOutputFormat[Text, IntWritable]]

Only then did I learn that JobConf is a deprecated way of defining jobs. So I rewrote WordCount object into a class implementing the Tool interface, using this Java snippet as an example to work from. To do that, I had to learn the to write a class that extends two interfaces in Scala, you need to use a “extends X with Y” syntax. Also, for trivial conditionals Scala dispenses with Java’s ternary X ? Y : Z operator in favor of a single line if (X) Y else Z. Though I will miss the evocative use of punctuation in the ternary construct, I’ve got to admit that Scala is keeping it real classy here.

…

Wait…ok, so I just learned that most of the code I was cargo culting was part of the deprecated coding pattern, which means I now have to switch it over to the new API. I learned this from somebody helpful in the #hadoop IRC channel.

[23:31]  what's the deal with org.apache.hadoop.mapreduce.Mapper and org.apache.hadoop.mapred.Mapper ??
[23:31]  is one of them deprecated?
[23:31]  which should I be using?
[23:32]  sbenthall: Use the new API
[23:32]  sbenthall: (i.e. mapreduce.*) both are currently supported but eventually the (mapred.*) may get deprecated
[23:32]  ok thanks QwertyM
[23:33]  sbenthall: as a reference, HBase uses mapreduce.* APIs completely for its provided MR jobs; and I believe Pig too uses the new APIs
[23:33]  is MapReduceBase part of the old API?
[23:33]  sbenthall: yes, its under the mapred.* package
[23:33]  ok, thanks.

Parachuting into the middle of a project has it’s drawbacks, but it’s always nice when a helpful community member can get you up to speed. Even if you’re asking near midnight on a Sunday.

Wait. I realize now that I’ve come full circle.

See, I’ve been writing these notes over the course of several days. Only just now am I realizing that I’m not going back to where I started, with the Mapper class that takes the Context parameter that was giving me noise.

Looking back at the original error, it looks like that too was a result of mixing two API’s. So maybe I can now safely shift everything BACK to the new API, drawing heavily on this code.

It occurs to me that this is one of those humbling programming experiences when you discover that the reason why your thing was broken was not the profound complexity of the tool you were working with, but your own stupidity over something trivial. This happens to me all the time.

Thankfully, I can’t ponder that much now, since it’s become clear that the instructional Hadoop cluster on which we’ve been encouraged to do our work are highly unstable. So I’m going to take the bet that it will be more productive for me to work locally, even if that means installing Hadoop locally on my Ubuntu machine.

I thought I was doing pretty good with the installation until I got to the point of hitting the “ON” switch on Hadoop. I got this:

sb@lebensvelt:~$ /usr/local/hadoop/bin/start-all.sh 
Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /usr/local/hadoop/libexec/../logs/hadoop-sb-namenode-lebensvelt.out
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
starting jobtracker, logging to /usr/local/hadoop/libexec/../logs/hadoop-sb-jobtracker-lebensvelt.out
localhost: ssh: connect to host localhost port 22: Connection refused

I googled around and it looks like this problem is due to not having an SSH server running locally. Since I’m running Ubuntu, I went ahead and followed these instructions. In the process I managed to convince my computer that I was undergoing a man-in-the-middle attack between myself and myself.

I fixed that with

$ ssh-keygen -R localhost

and successfully got Hadoop running with

$ /usr/local/hadoop/bin/start-all.sh 

only to be hung up on this error

$ hadoop fs -ls
Warning: $HADOOP_HOME is deprecated.

ls: Cannot access .: No such file or directory.

which somebody who runs an Indian matrimony search engine had run into and documented the fix for. (Right way to spell it is

hadoop fs -ls .

With an extra dot.)

There’s a point to me writing all this out, by the way. An important part of participation in open source software, or the hacker ethic in general, is documenting ones steps so that others who follow the same paths can benefit from what you’ve gone through. I’m going into a bit more detail about this process than really helpful because in my academic role I’m dealing with a lot of social scientist types who really don’t know what this kind of work entails. Let’s face it: programming is a widely misunderstood discipline which seems like an utter mystery to those that aren’t deeply involved in it. Much of this has to do with the technical opacity of the work. But another part of why its misunderstood is because problem solving in the course of development depends on a vast and counter-intuitive cyberspace of documentation (often generated from code comments, so written by some core developer), random blog posts, chat room conversations, forum threads. Easily 80% of the work when starting out on a new project like this is wrestling with all the minutia of configuration on a particular system (operating system and hardware contingent, in many cases) and idioms of programming language and environment.

The amount of time it takes to invest in any particular language or toolkit necessarily creates a tribalism among developers because their identities wind up being intertwined with the tools they use. As I hack on this thing, however incompetently, I’m becoming a Scala developer. That’s similar to saying that I’m becoming a German speaker. My conceptual vocabulary, once I’ve learned how to get things done in Scala, is going to be different than it was before. In fact, that’s one of the reasons why I’m insisting on teaching myself Scala in the first place–because I know that it is a conceptually deep and rigorous language which will have something to teach me about the Nature of Things.

Some folks in my department are puzzled at the idea that technical choices in software development might be construed as ethical choices by the developers themselves. Maybe it’s easier to understand that if you see that in choosing a programming language you are in many ways choosing an ontology or theoretical framework through which to conceive of problem-solving. Of course, choice of ontology will influence ones ethical principles, right?

But I digress.

So I have Hadoop running on my laptop now, and a .jar file that compiles in SBT. So now all I need to do is run the .jar using the hadoop jar command, right?

Nope, not yet…

Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject

OK, so I the problem is that I haven’t included scala-library.jar on my Hadoop runtime classpath.

I solved this by making a symbolic link from the Hadoop /lib directory to the .jar in my Scala installation.

ln -s /usr/local/share/scala-2.9.2/lib/scala-library.jar /usr/local/hadoop/lib/scala-library.jar

That seemed to work, only now I have the most mundane and inscrutable of Java errors to deal with:

Exception in thread "main" java.lang.NullPointerException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

I had no idea how to proceed from here. This time, no helpful folks in the #hadoop channel helped me either.

So once again I switched to a new hunk of code to work from, this time the WordCount.scala file from Derrick Cheng’s ScalaOnHadoop project. Derrick posted a link to this project on our course’s Piazza earlier, which was awesome.

Another digression. There’s a lot of talk now about on-line education. People within the university context feel vaguely threatened by MOOCs, believing that there will be a superstar effect that advantages first movers, but many are uncomfortable making that first move. Taek Lim in my department is starting to studying user interfaces to support collaboration in on-line learning.

My own two cents on this are that the open software model is about as dynamics a collaborative environment as you can get, and at the point when people started to use our course discussion management system, Piazza, as if it were a forum to discuss the assignment almost as if it was an open source mailing list, we started to get a lot more out of it and learned a lot from each other. I’m not the first person to see this potential of the open source model for education, of course. I’m excited to be attending the POSSE workshop, which is about that intersection, this summer. At this rate, it looks like I will be co-teaching a course along these lines in the Fall targeted at the I School Masters students, which is exciting!

So, anyway, I’m patching Derrick Cheng’s code. I’m not working with BIDMat yet so I’m leaving that out of the build, so I remove all references to that and I get the thing to compile…and run! I got somebody else’s Scala WordCount to run!

This seems like such a triumph. It’s taken a lot of tinkering and beating my head against a wall to get this far. (Not all at once–I’ve wrote this post over the course of several days. I realize that it’s a little absurd.)

But wait! I’m not out of the woods yet. I check the output of my MapReduce job with

hadoop fs -cat test-output/part-r-00000

and the end of my output file looks like this:

§	1
§	1
§	1
§	1
§	1
§	1
§	1
§	1
§	1
Æschylus	1
Æsop	1
Æsop	1
Æsop	1
É	1
Élus,_	1
à	1
à	1
à	1
æons	1
æsthetic	1
è	1
è	1
è	1
état_.	1
The	1
The	1

What’s going on here? Well, it looks like I successfully mapped each occurrence of each word in the original text to a key-value pair. But something went wrong in the reduce step, that was supposed to combine all the occurrences into a single count for each word.

That is just fine for me, because I’d rather be using my own Reduce function. Because it uses Scala’s functional reduceLeft, which is sweet! Why even write a Map Reduce job in a functional programming language if you can’t use a built=in language reduce in the Reduce step?

Ok, mine doesn’t work either.

Apparently, the reason for this is that the type signature I’ve been using for the Reducer’s reduce method has been wrong all along. And when that happens, the code compiles but Reducer runs its default reduce function, which is the identity function.

It’s almost (almost!) as if it would have made more sense to start by just reading the docs and following the instructions.

Now I have edited the map and reduce functions so that they have the right type signatures. To get this right exactly, I looked at a different file. I also tinkered

At last, it works.

Now, at last, I understand how this Context member class works. The problem was that I was trying to use it with the mapred.Mapper class from the old API. So much of my goose chase was due to not reading things carefully enough.

On the other hand, I feel enriched by the whole meandering process. Realizing that my programming faults were mine and not due to the complexity of the tools I was using paradoxically gives me more confidence in my understanding of the tools moving forward. And engaging with the distributed expertise on the subject–through StackOverflow, documentation generated from the original coders, helpful folks on IRC, blog posts, and so on–is much more compelling when one is driven by concrete problem-solving goals, even when those goals are somewhat arbitrary. Had I learned to use Hadoop in a less circuitous way, my understanding would probably be much more brittle. I am integrating new knowledge of Hadoop, Scala, and Java (it’s been a long time) with existing background knowledge. After a good night’s sleep, with any luck it will be part of my lifeworld!

This is the code I wound up with, by the way. I don’t suggest you use it.

Earlier this week, Mendeley was bought by Elsevier. I posted the announcement on Twitter to state that I would be quitting Mendeley. This tweet sparked a conversation between me and the head of academic outreach at Mendeley (William Gunn) that could only go so far in 140 character chunks. I was trying to highlight that, while I respected the Mendeley team’s decision to do what’s best for them, I could not support them as a customer knowing that this would empower a company that I think undermines scholarship, scholars, and the future of research.

Today, Gunn posted the following tweet: “All you folks retweeting @zephoria know who she works for, right?” before justifying his implied critique by highlighting that he personally respects MSR.

I feel the need to respond to this implicit attack on my character and affiliation. When I’m critical of Elsevier, I’m speaking as a scholar, not on behalf of Microsoft or even Microsoft Research. That said, I get that everyone’s associations shapes how they’re perceived. But I’m not asking people to buy my ?product? or even the products of my employer. I’m making a public decision as a scholar who is committed to the future for research. I believe in making my research publicly available through open access initiatives and I’m proud to work for and be associated with an organization that is committed to transforming scholarly publishing.  I’m also committed to boycotting organizations that undermine research, scholarship, libraries, and the production of knowledge.

I also think that it’s important to explain that there are huge differences between Microsoft and Elsevier.  I fully recognize that I work for a company that many people think is evil. When I joined Microsoft four years ago, I did a lot of poking around and personal soul-searching. Like many other geeks of my age, I spent my formative years watching an arrogant Microsoft engage in problematic activities only to be humiliated by an anti-trust case. Then I watched the same company, with its tail between its legs, grow up. The company I was looking to join four years ago was not the company that I boycotted in college. It had been a decade since United States vs. Microsoft and even though many of my peers are never going to forgive my employer for its activities in the 90s, I am willing to accept that companies change.

There are many aspects of Microsoft that I absolutely love. For starters, Microsoft Research (MSR) is heaven on earth. Overall, MSR offers more freedom, flexibility, and opportunities to scholars than even the best academic institutions. They share my values regarding making scholarship widely accessible (see: Tony Hey’s 6-part series on open access). And, unlike research entities at other major corporations, Microsoft Research has supported me in doing research that’s critical of Microsoft (even when I get nastygrams from corporate executives). Beyond my home division, there are other sparkly beacons of awesome. I love that Microsoft has made privacy a central value, even as it struggles to ethically negotiate the opportunities presented by data mining. I have been in awe of some of the thoughtful and innovative approaches taken by the folks at Bing, in mobile, and in Xbox. Even more than the work that everyone sees, I get excited by some of the visioning that happens behind closed doors.

Don’t get me wrong. Like all big companies, Microsoft still screws up. I’ve facepalmed on plenty of occasions, embarrassed to be associated with particular company decisions, messages, or tactics. But I genuinely believe that the overall company means well and is pointed in a positive, productive, and ethical direction. Sure, there are some strategies that don’t excite me, but I think that the leadership is trying to move the company to a future I can buy into. I’m proud of where the company is going even if I can’t justify its past.

I cannot say the same thing for Elsevier. As most academics and many knowledge activists know, Elsevier has engaged in some pretty evil maneuvers. Elsevier published fake journals until it got caught. Its parent company was involved in the arms trade until it got caught. Elsevier played an unrepentant and significant role in advancing SOPA/PIPA/RWA and continues to lobby on issues that undermine scholarship. Elsevier currently actively screws over academic libraries and scholars through its bundling practices. There is no sign that the future of Elsevier is pro-researchers. There is zero indicator that Mendeley’s acquisition is anything other an attempt to placate the academics who are refusing to do free labor for Elsevier (editorial boards, reviewers, academics). There’s no attempt at penance, no apology, not even a promise of a future direction. Just an acquisition of a beloved company as though that makes up for all of the ways in which Elsevier has in the past _and continues to_ screw over scholars.

Elsevier’s practices make me deeply deeply angry. While academic publishing as a whole is pretty flawed, Elsevier takes the most insidious practices further at each and every turn, always at the expense of those of us who are trying to produce, publish, and distribute research. Their prices are astronomical, bankrupting libraries and siloing knowledge for private profit off of free labor. As a result, many mathematicians and other scientists have begun stepping off of their editorial boards in protest. Along with over 13,000 other scholars, I too signed the Cost of Knowledge boycott.

I see no indication of a reformed Elsevier, no indication of a path forward that is actually respectful of scholars, scholarship, librarians, or universities. All I see is a company looking to make a profit in an unethical manner and trying to assuage angry customers and laborers with small tokens.

Mendeley’s leadership is aware of how many academics despise Elsevier. In their announcement of their sale, they justify Elsevier through some of the technologies they developed. There’s no indication that the “partnership” is going to make Elsevier more thoughtful towards academics. Mendeley’s reps try to explain that the company is a “large, complex organization” full of good people as though this should relieve those of us who are tired of having our labor and ideas abused for profit.

All companies have good people in them. All companies are complex. This is not enough. What matters is the direction of the leadership and what kinds of future a company is trying to create. People may not like either Microsoft or Elsevier’s past, but what about the future?

In Mendeley’s post, they indicate overlap in their vision and Elsevier’s vision as a company. This does not make me more hopeful of Elsevier; this makes me even more dubious of Mendeley. Elsevier has a long track record with no indication of change. It is the parent company. Startups don’t get bought by big companies to blow up the core company. New division presidents or vice-presidents do not have penultimate power in big companies, particularly not when their revenue pales in comparison to the parent company’s. I wish Mendeley employees the best, but I think that they’re naive if they believe that they can start a relationship with the devil hoping he’ll change his ways because of their goodness. This isn’t a Disney fairy tale. This is business.

I genuinely like Mendeley as a product, but I will not support today’s Elsevier no matter how good a product of theirs is. Perhaps they’ll change. I wouldn’t bet on it, but I am open to the possibility.  But right now, I don’t believe in the ethics and commitments of the company nor do I believe that they’re on the precipice of meaningful change. As minimally symbolic as it is, I refuse to strengthen them with my data or money. This means that I will quit Mendeley now that they’re part of Elsevier. In the same vein, I respect people who disagree with my view on the future of Microsoft and choose to not to use their products. I believe in consumer choice. I’m just startled that a head of academic outreach would try to brush off my critique of his new employer by implicating mine. I guess that’s the way things work.

I believe that the next place for me is probably Zotero, but I’m trying to figure out how to get my data (including the PDFs) over there. I’m hopeful that someone will write the scripts soon so that I don’t have to do this manually. If you’ve got other suggestions or advice, I’m all ears.

Two years ago, when I started working on issues related to human trafficking and technology, I was frustrated by how few people recognized the potential of technology to help address the commercial sexual exploitation of children. With the help of a few colleagues at Microsoft Research, I crafted a framework document to think through the intersection of technology and trafficking. After talking with Mark Latonero at USC (who has been writing brilliant reports on technology and human trafficking), I teamed up with folks at MSR Connections and Microsoft’s Digital Crimes Unit to help fund research in this space. Over the last year, I’ve been delighted to watch a rich scholarly community emerge that takes seriously the importance of data for understanding and intervening in human trafficking issues that involve technology.

Meanwhile, to my delight, technologists have started to recognize that they can develop innovative systems to help address human trafficking. NGOs have started working with computer scientists, companies have started working with law enforcement, and the White House has started bringing together technologists, domain experts, and policy makers to imagine how technology can be used to combat human trafficking. The potential of these initiatives tickles me pink.

Watching this unfold, one thing that I struggle with is that there’s often a disconnect between what researchers are learning and what the public thinks is happening vis-a-vis the commercial sexual exploitation of children (CSEC). On too many occasions, I’ve watched well-intentioned technologists approach the space with a naiveté that comes from only knowing about human trafficking through media portrayals. While the portraits that receive widespread attention are important for motivating people to act, understanding the nuance and pitfalls of the space are critical for building interventions that will actually make a difference.

To bridge the gap between technologists and researchers, I worked with a group of phenomenal researchers to produce a simple 4-page fact sheet intended to provide a very basic primer on issues in human trafficking and CSEC that technologists need to know before they build interventions:

How to Responsibly Create Technological Interventions to Address the Domestic Sex Trafficking of Minors

Some of the issues we address include:

1. Youth often do not self-identify themselves as victims.
2. “Survival sex” is one aspect of CSEC.
3. Previous sexual abuse, homelessness, family violence, and foster care may influence youth’s risk of exploitation.
4. Arresting victims undermines efforts to combat CSEC.
5. Technologies should help disrupt criminal networks.
6. Post-identification support should be in place before identification interventions are implemented.
7. Evaluation, assessment, and accountability are critical for any intervention.
8. Efforts need to be evidence-based.
9. The cleanliness of data matters.
10. Civil liberties are important considerations.

This high-level overview is intended to shed light on some of the most salient misconceptions and provide some key insights that might be useful for those who want to make a difference. By no means does it cover everything that experts know, but it provides some key touchstones that may be useful. It is limited to the issues that are most important for technologists, but those who are working with technologists may also find it to be valuable.

As researchers dedicated to addressing human trafficking and the commercial sexual exploitation of children, we want to make sure that the passion that innovative technologists are bringing to the table is directed in the most helpful ways possible. We hope that what we know can be of use to those who are also looking to end exploitation.

(Flickr image by Martin Gommel)

Since the French Revolution, we have had the Left/Right divide in politics.

Probably seven or so years ago, some people got excited about thinking about a two-dimensional political spectrum. There were Economic and Social dimensions. You could be in one of four quadrants: Libertarian, Social Democrat, Totalitarian, or Conservative.

Technology is getting more political and politicized. Have we figured out the spectrum yet?

Because there’s been a lot of noise about their beef, lets assume as a first pass that O’Reilly and Morozov give us some sense of the space. The problem is that there’s a good chance the “debate” between them is giving off a lot more heat than light, so it’s not clear if there’s a substantive political difference.

Let me try to take a constructive crack at it. I don’t think I’m going to get it right, but I’m curious to know how much this resonates and if others would map things differently.

A two-dimensional representation of the continuum of technical politics, with unscientifically plotted representatives

Some people think that “technology”, by which most people mean technology companies, should be replacing more and more of the functions of government. I think the peer progressives are in this camp, as are the institutionalized nudgers in the UK Conservative party, who would prefer to shrink the state. There’s a fair argument that the “open government” people are trying to shrink government by giving non-state actors the ability to provide services that the state might otherwise provide. Through free flow of information and greater connectivity, we can spur vibrancy in civil society and perfect the market.

Others think that the state needs to have a strong role in regulating technology companies to make sure they don’t abuse their power. There’s a lot of that going around in my department at UC Berkeley. These people see that democratic state as the best representative of citizen’s interests. The FTC and Congress need to help ensure, e.g., people’s privacy. Maybe Morozov is in here somewhere. Monopoly concentrations of technical power are threatening to the public interest; technical platforms should be decentralized and controlled so that politics is not overwhelmed by an illegitimate technocracy.

Another powerful group, the Copyright lobby, is economically threatened by new technology and so wants to restrict its use. Telecom companies would like to effectively meter flow of information. Maybe it’s a stretch, but perhaps we could include the military-industrial complex and its desire to instrument the Web for surveillance purposes in this camp as well. These groups tend to not want technology to change, or to tightly control that technology.

Then there’s the Free Software movement. And Stanford’s Liberation Technology folks, if I understand them correctly. And maybe Anonymous is in here somewhere. Pro-technology, generally skeptical of both state and corporate interests.

So maybe what’s going on is that we have a two-dimensional political space.

In one dimension, we have Centralization versus Decentralization. Richly interconnected platforms managed by an elite with tight arrangements for data sharing, versus a much more loosely connected set of networks where the lines of power are less clear.

In the other dimension, we have Unrestricted versus Controlled. Either the technical organizations should be free to persue their own interests, or they should be regulated by non- (or at least less) technical political forces, such as the state.

What do you think?

The best thing I did today was stop by for the beginning of Kay Holekamp‘s talk on “Social Complexity and the Evolution of Intelligence.”

Her work involves researching spotted hyenas.

Spotted hyenas live in clans of about a hundred hyenas, which contain several martilineal kinship groups each. Female hyenas have an observable social hierarchy that is caused by and a cause of survival “fitness”.  Male hyenas migrate to a different clan before reproducing.

This is very similar to the social structure of certain primates, like baboons.  It is nothing like the social structure of cats and dogs (hyenas are somewhere in between the two, closer to cats.)

What’s interesting about the research is that without exception, results about the social cognitive capabilities of primates is, without exception, reproducible in spotted hyenas.

That means that the same capacities for social intelligence has been achieved by multiple species through convergent evolution.

This past week I was at the kick-off meeting of the LA County Voting System Assessment Project's (VSAP) Technical Advisory Committee. The VSAP is Registrar/Clerk Dean Logan's intense and groundbreaking effort to design, develop, procure and implement a publicly owned voting system. I am honored to be asked to serve on such an important body.

LA County is the largest election jurisdiction in the US, with 5 million registered voters, 10 languages, 5,000 precincts and a very large physical area. The county currently uses the InkaVote Plus voting system (with Audio Ballot Booth for accessibility), which is essentially an overhaul of former punchcard equipment to use inked styluses for marking and to provide in-precint checks for the voter in case they make mistakes.

Here is the InkaVote Plus system and the Microcomputer Tally System (MTS) that is used to rapidly count (>1,200 ballots per minute!) ballots after they're returned to the Registrar's Norwalk, CA headquarters:

My fellow committee members include:

• Henry Balta (Senior Assoc. CIO, LA County)
• Mike Byrne (Professor of Psychology and Computer Science, Rice)
• Josh Franklin (IT Specialist, NIST)
• Diane Goldin (Policy Coordinator, AATAP)
• Joseph Lorenzo Hall (Senior Staff Technologist, CDT)
• Brian Hancock (Director, Testing and Certification, EAC)
• Jared Marcotte (Technology Manager, Pew)
• Noel Runyan (Principal, Personal Data Systems)
• Rich Sánchez (CIO, LA County)
• Pam Smith (President, VVF)
• Charles Stewart III, (Professor of Political Science, MIT)
• David Wagner (Professor of Computer Science, UC Berkeley)

The mission of the TAC is to provide technical advice to LA County through a design and development process to meet a variety of goals and principles that LA County has determined its voting system must meet.

While we'll have an official web page and other materials soon for public perusal, I was able to take a number of photos and videos during a tour of the tabulation and storage facilities that we had during the end of the day. Find them at this Flickr photo set.

I'll leave you with the following video, that shows just how fast their card reader tabulation system, the MTS, can count ballots -- a blazing 600 ballot cards in 28 seconds! This is just one example of a metric that will be difficult to match in a new system!

Information transfer just is the coming-into-dependence of two variables, which under the many worlds interpretation of quantum mechanics means the entanglement of the “worlds” of each variable (and, by extension, the networks of causally related variables of which they are a part). Information exchange collapses possibilities.
This holds up whether you take a subjectivist view of reality (and probability–Bayesian probability properly speaking) or an objectivist view. At their (dialectical?) limit, the two “irreconcilable” paradigms converge on a monist metaphysics that is absolutely physical and also ideal. (This was recognized by Hegel, who was way ahead of the game in a lot of ways.) It is the ideality of nature that allows it to be mathematized, though its important to note that mathematization does not exclude engagement with nature through other modalities, e.g. the emotional, the narrative, etc.

This means that characterizing the evolution of networks of information exchange by their physical properties (limits of information capacity of channels, etc.) is something to be embraced to better understand their impact on e.g. socially constructed reality, emic identity construction, etc. What the mathematics provide is a representation of what remains after so many diverse worlds are collapsed.

A similar result, representing a broad consensus, might be attained dialectically, specifically through actual dialog. Whereas the mathematical accounting is likely to lead to reduction to latent variables that may not coincide with the lived experience of participants, a dialectical approach is more likely to result in a synthesis of perspectives at a higher level of abstraction. (Only a confrontation with nature as the embodiment of unconscious constraints is likely to force us to confront latent mechanisms.)

Whether or not such dialectical synthesis will result in a singular convergent truth is unknown, with various ideologies taking positions on the matter as methodological assumptions. Haraway’s feminist epistemology, eschewing rational consensus in favor of interperspectival translation, rejects a convergent (scientific, and she would say masculine) truth. But does this stand up to the simple objection that Haraway’s own claims about truth and method transcend individual perspective, making he guilty of performative contradiction?

Perhaps a deeper problem with the consensus view of truth, which I heard once from David Weinberger, is that the structure of debate may have fractal complexity. The fractal pluralectic can fray into infinite and infinitesimal disagreement at its borders. I’ve come around to agreeing with this view, uncomfortable as it is. However, within the fractal pluralectic we can still locate a convergent perspective based on the network topology of information flow. Some parts of the network are more central and brighter than others.

A critical question is to what extent the darkness and confusion in the dissonant periphery can be included within the perspective of the central, convergent parts of the network. Is there necessarily a Shadow? Without the noise, can there be a signal?

There is an interesting thing happening. Let me just try to lay down some facts.

There are a number of organizations in the Bay Area right now up to related things.

• Machine Intelligence Research Institute (MIRI). Researches the implications of machine intelligence on the world, especially the possibility of super-human general intelligences. Recently changed their name from the Singularity Institute due to the meaninglessness of the term Singularity. I interviewed their Executive Director (CEO?), Luke Meuhlhauser, a while back. (I followed up on some of the reasoning there with him here).
• Center for Applied Rationality (CFAR). Runs workshops training people in rationality, applying cognitive science to life choices. Trying to transition from appearing to pitch a “world-view” to teaching a “martial art” (I’ve sat in on a couple of their meetings). They aim to grow out a large network of people practicing these skills, because they think it will make the world a better place.
• Leverage Research. A think-tank with an elaborate plan to save the world. Their research puts a lot of emphasis on how to design and market ideologies. I’ve been told that they recently moved to the Bay Area to be closer to CFAR.

Some things seem to connect these groups. First, socially, they all seem to know each other (I just went to a party where a lot of members of each group were represented.) Second, the organizations seem to get the majority of their funding from roughly the same people–Peter Thiel, Luke Nosek, and Jaan Tallinn, all successful tech entrepreneurs turned investors with interest in stuff like transhumanism, the Singularity, and advancing rationality in society. They seem to be employing a considerable number of people to perform research on topics normally ignored in academia and spread an ideology and/or set of epistemic practices. Third, there seems to be a general social affiliation with LessWrong.com; I gather a lot of the members of this community originally networked on that site.

There’s a lot that’s interesting about what’s going on here. A network of startups, research institutions, and training/networking organizations is forming around a cluster of ideas: the psychological and technical advancement of humanity, being smarter, making machines smarter, being rational or making machines to be rational for us. It is as far as I can tell largely off the radar of “mainstream” academic thinking. As a network, it seems concerned with growing to gather into itself effective and connected people. But it’s not drawing from many established bases of effective and connected people (the academic establishment, the government establishment, the finance establishment, “old boys netowrks” per se, etc.) but rather is growing its own base of enthusiasts.

I’ve had a lot of conversations with people in this community now. Some, but not all, would compare what they are doing to the starting of a religion. I think that’s pretty accurate based on what I’ve seen so far. Where I’m from, we’ve always talked about Singularitarianism as “eschatology for nerds”. But here we have all these ideas–the Singularity, “catastrophic risk”, the intellectual and ethical demands of “science”, the potential of immortality through transhumanist medicine, etc.–really motivating people to get together, form a community, advance certain practices and investigations, and proselytize.

I guess what I’m saying is: I don’t think it’s just a joke any more. There is actually a religion starting up around this. Granted, I’m in California now and as far as I can tell there are like sixty religions out here I’ve never heard of (I chalk it up to the lack of population density and suburban sprawl). But this one has some monetary and intellectual umph behind it.

Personally, I find this whole gestalt both attractive and concerning. As you might imagine, diversity is not this group’s strong suit. And its intellectual milieu reflects its isolation from the academic mainstream in that it lacks the kind of checks and balances afforded by multidisciplinary politics. Rather, it appears to have more or less declared the superiority of its methodological and ideological assumptions to its satisfaction and convinced itself that it’s ahead of the game. Maybe that’s true, but in my own experience, that’s not how it really works. (I used to share most of the tenets of this rationalist ideology, but have deliberately exposed myself to a lot of other perspectives since then [I think that taking the Bayesian perspective seriously necessitates taking the search for new information very seriously]. Turns out I used to be wrong about a lot of things.)

So if I were to make a prediction, it would go like this. One of these things is going to happen:

• This group is going to grow to become a powerful but insulated elite with an expanded network and increasingly esoteric practices. An orthodox cabal seizes power where they are able, and isolates itself into certain functional roles within society with a very high standard of living.
• In order to remain consistent with its own extraordinarily high epistemic standards, this network starts to assimilate other perspectives and points of view in an inclusive way. In the process, it discovers humility, starts to adapt proactively and in a decentralized way, losing its coherence but perhaps becomes a general influence on the preexisting societal institutions rather than a new one.
• Hybrid models. Priesthood/lay practitioners. Or denominational schism.

There is a good story here, somewhere. If I were a journalist, I would get in on this and publish something about it, just because there is such a great opportunity for sensationalist exploitation.

Last week at PyCon 2013, with the assistance of Asheesh Laroia and several other helpful TAs, I delivered an introductory Django tutorial wherein one builds a simple bookmarking application. Instead of the official Django tutorial, I used a different curriculum based on the tutorial I wrote for my IOLab students in fall 2011 (itself based on the PyStar Django tutorial).

The idea was to write a web development tutorial targeted at a group of students who were familiar with HTML, CSS, and perhaps Javascript and had had some introduction to Python, but had little to no experience with databases or server-side code. Thus, the tutorial skips over some of the traditional tutorial’s topics (e.g. the built-in admin) and emphasizes other areas: static files, CRUD, Django forms, using asynchronous Javascript with Django, and the Django templating language. It also incorporates non-core-Django topics such as git-based version control, database migrations using South, and sharing your work with others both locally and using Heroku.

The version control aspect was especially key, as it allowed me to set up “checkpoints” where everyone would be working from the same copy of the code, even if they hadn’t managed to get part of the code working in their personal branch. Given that PyCon tutorials have a single lecturer/large room structure, rather than small pods or groups, that sort of thing is kind of necessary–you have to keep a pace, even when a few students might not have gotten a step in your tutorial working yet.

I also had a goal of making it so that literally everything you had to type into the command line or your text editor to get the tutorial app to work was available for copy and paste on the tutorial wiki. (It’s not quite true–I think there are some steps in the laptop setup guide that don’t qualify for this, and most of the commits aren’t spelled out–but it’s pretty close. The cheatsheet also helped to fill in any gaps.) This was important not just for speed — to get through all the material, typing all the HTML, etc. out by hand would have been prohibitive — but also so that if a student had trouble in one of the sections, or if the class didn’t make it through all the material (we didn’t, and that was expected, and there are a bunch of “reader exercises” at the end of the tutorial that weren’t meant to be taught in class at all), she would still have everything she needed to revisit it later.

With every runthrough of the tutorial I find more things to fix and tweak, and there’s definitely cleanup and improvements I want to make post-PyCon. Still, I’m pretty proud of the tutorial Asheesh and I (and playtesters Shauna and paultag!) put together, available on the web for free via the OpenHatch wiki and licensed under a permission CC license. I hope other people make use of the tutorial curriculum and (shudder) video too, either to teach or to learn!

I have a backlog of things I’ve been planning to write about. It’s been a fruitful semester for me, in a number of ways. At the risk of being incoherent, I thought I’d throw some of my spaghetti against the Internet wall. It’s always curious to see what sticks.

One of the most fascinating intellectual exchanges of the past couple months for me was what I’d guess you could call the Morozov/Johnson debate. Except it wasn’t a debate. It was a book, then a book review (probably designed to sell a different book), and a rebuttal. It was fantastic showmanship. I have never felt so much like I was watching a boxing match while reading stuff on the Internet.

But what really made it for me was the side act of Henry Farrell taking Morozov to task. Unlike the others, I’ve met Farrell. He was kind enough to talk to me about his Cognitive Democracy article (which I was excited about) and academia in general (“there are no academic jobs for brilliant generalists”) last summer when I was living in DC. He is very smart, and not showy. What was cool about his exchange with Morozov was that it showed how a debate that wasn’t designed to sell books could still leak out into the public. There’s still a role for the dedicated academic, as a watchdog on public intellectuals who one could argue have to get sloppier to entertain the public.

An intriguing fallout from (or warm up to?) the whole exchange was Morozov casually snarking Nick Grossman‘s title, “Activist in Residence” at a VC fund in a tweet (“Another sign of the coming Apocalypse? Venture capital firms now have “activists in residence”? “), which then triggered some business press congratulating Nick for being “out in the streets”. Small world, I used to work with Nick at OpenPlans, and can vouch for his being a swell guy with an experienced and nuanced view of technology and government. He has done a lot of pioneering, constructive work on open governance applications–just the sort of constructive work a hater like Morozov would hate if he looked into it some. Privately, he’s told me he’s well aware of the potential astroturfing connotations of his title.

I got mixed feelings about all this. I’m suspicious of venture capital for the kind of vague “capital isn’t trustworthy” reasons you pick up in academia. Activism is sexy, lobbyists are not, and so if you can get away with calling your lobbyist an activist in residence then clearly that’s a step up.

But I think there’s something a little more going on here, which has to do with the substance of the debate. As I understand it, the Peer Progressives believe that social and economic progress can happen through bottom-up connectivity supported by platforms that are potentially run for profit. If you’re a VC, you’d want to invest in one of them platforms, because they are The Future. Nevertheless, you believe stuff happens by connecting people “on the ground”, not targeting decision-makers who are high in a hierarchy.

In Connected, Christakis and Fowler (or some book like it, I’m been reading a lot of them lately and having a hard time keeping track) make the interesting argument that the politics of protesters in the streets and lobbyists aren’t much different. What’s different is the centrality of the actor in the social network of governance. If you know a lot of senators, you’re probably a lobbyist. If you have to hold a sign and shout to have your political opinions heard, then you might be an activist.

I wonder who Nick talks to. Is he schmoozing with the Big Players? Or is he networking the base and trying to spur coordinated action on the periphery? I really have no idea. But if it were the latter, maybe that would give credibility to his title.

Another difference between activists and lobbyists is their authenticity. I have no doubt that Nick believes what he writes and advocates for. I do wonder how much he restrains himself based on his employers’ interests. What would prove he was an activist, not a lobbyist, would be if he were given a longer leash and allowed to speak out on controversial issues in a public way.

I’m mulling over all of this because I’m discovering in grad school that as an academic, you have to pick an audience. Are you targeting your work at other academics? At the public? At the press? At the government? At industry? At the end of the day, you’re writing something and you want somebody else to read it. If I’m lucky, I’ll be able to build something and get some people to use it, but that’s an ambitious thing to attempt when you’re mainly working alone.

So far some of my most rewarding experiences writing in academia have been blogging. It doesn’t impress anybody important but a traffic spike can make you feel like you’re on to something. I’ve been in a world of open work for a long time, and just throwing the spaghetti and trying to see what sticks has worked well for me in the past.

But if you try to steer yourself deeper into the network, the stakes get higher. Things get more competitive. Institutions are more calcified and bureaucratic and harder to navigate. You got to work to get anywhere. As it should be.

Dang, I forgot where I was going with this.

Maybe that’s the problem.

…

Comments are attached.

Nick Doty
Lecturer / Researcher
UC Berkeley, School of Information

Last night, I was inducted into the SXSW-Interactive Hall of Fame in a whirlwind awards ceremony. I only had a flash of a moment to reflect so I want to take a moment here to provide greater context and appreciation to all who made this possible.

I started attending SXSW a decade ago at a time when the conference was extraordinarily small and “social software” was a glimmer in entrepreneurs’ eyes. The conference was spring break for geeks. It only took up a hallway in the convention center and if attendees weren’t attending sessions, they were probably sitting close to a power outlet in the hallway or drinking in the Hilton bar. As the day wound down, the evening would begin with some BBQ followed by a stroll down Red River where it was easy to wander into the different parties put on by not-yet-famous tech companies. It was small, intimate, and very very geeky.

I met numerous people at SXSW who changed my life. For example, an alcohol-driven debate with Ev Williams in the Hilton bar resulted in him inviting me to apply to work with his team at Blogger/Google. And an accidental encounter at a Sleater-Kinney show introduced me to the person who would become my lover and life partner. Through SXSW, I met countless friends and eventual colleagues. The conference became an annual break from reality for me, a chance to laugh and party and be silly with folks who valued tech in the same way I did. I will never forget how, only days after learning my advisor was dying, I was surrounded by loving friends in Austin who helped me remember the beauty of life.

Because SXSW meant so much to me, I was passionate about giving back, both to the conference and to the community. I helped organize panels that brought together different intellectual and professional communities. Along with the amazing women at Blogher, I helped Hugh Forrest (the conference’s amazing godfather) diversify the attendees so as to expand the audience. I gave numerous talks and helped organize parties and flitted around, introducing new people to old people.

The truth is that at this stage, the contemporary SXSW is a bit alien to me. I’m a geek. My rudimentary social skills allow me to mostly pass in most mainstream settings, but I am dreadful at casual talk and am drained by large crowds. Long lines terrify me and my hearing is pretty crap so I can’t have conversations at parties where the music overwhelms. So I often feel like a fish out of water in this new incarnation of cool. But whenever I go to Austin, I can’t help but grin with joy at how successful SXSWi has become precisely because I wanted to see a space where diverse voices could gather and engage over the future of tech.

And it’s in this context that I was both startled by and grateful for the induction into the Hall of Fame. SXSW has meant so much to me for so long that being inducted feels like a huge gift from a conference that has given me so much. I’m so very very thankful for having this event, even if I barely know a fraction of the attendees at this stage. And I’m honored by those who see me as a central part of this community, even if I feel like an awkward alien.

As I reflected on what to say upon being inducted (in the briefest of briefest formats), I realized that I wanted to share two critical values that I feel have long underpinned the community as I see it. I did so because I see these foundational values as central to SXSW’s success and key to the future of a healthy tech community. With that in mind, I asked last night’s attendees to never forget to:

1) Build technologies and experiences that make people’s lives better.
2) Take a moment to step back and listen to diverse voices.

Now that social media is mainstream and geekery is cool, it’s often easy to get distracted by the glitz and to relish the games of status that emerge. But what makes SXSW amazing is not the tequila or the Tex-Mex (which are both awesome), but the ideas and the people. My hope is that as this community continues to grow, attendees will continue to find innovative ways of using the event to challenge their assumptions and expectations. This is what has made the event magical for me and I hope that many more can feel that magic too.

My Notes after reading “inspired”.

Pew Research Center recently put out a report on Twitter’s liberal bias. It argues that overall sentiment on certain political events on Twitter is far to the left of national surveys. Also, people on Twitter complain a lot (are more negative). That makes sense, because Twitter has only 13% of the country even looking at it, and only 3% of the population posting or retweeting. And many of these people are younger.

First published on ethnographymatters.net.

Last month on Ethnography Matters, we started a monthly thematic focus where each of the EM contributing editors would elicit posts about a particular theme. I kicked us off with the theme entitled ‘The Openness Edition’ where we investigated what openness means for the ethnographic community. I ended up editing some wonderful posts on the topic of openness last month – from Rachelle Annechino’s great post questioning what “informed consent” means in health research, to Jenna Burrell’s post about openaccess journals related to ethnography and Sarah Kendzior’s stimulating piece about by legitimacy and place of Internet research by anthropologists. We also had two really wonderful pieces sharing methods for more open, transparent research by Juliano Spyer (YouTube “video tags” as an open survey tool) and by Jeff Hall, Elizabeth Gin and An Xiao in their inspiring piece about how they facilitated story-building exercises with Homeless Youth in Boyle Heights (complete with PDF instructions!) Below is the editorial that I wrote at the beginning of the month where I try to tease out some of the complexities of my own relationship with the open access/open content movement. Comments welcome!

On Saturday the 12th of January, almost a month ago, I woke to news of Aaron Swartz’s death the previous day. In the days that followed, I experienced the mixed emotions that accompany such horrific moments: sadness for him and the pain he must have gone through in struggling with depression and anxiety, anger at those who had waged an exaggerated legal campaign against him, uncertainty as I posted about his death on Facebook and felt like I was trying to claim some part of him and his story, and finally resolution that I needed to clarify my own policy on open access.

I had worked passionately for open access in my previous life, helping educational institutions and foundations design open access policy, pushing for open government data and railing against those who didn’t ‘get’ why closing access to publicly-funded information was outdated and unsustainable. But nearing the end of my work with Creative Commons and its international offshoot, iCommons, I became jaded by the internal politics of the open content movement, and embarrassed by my previous zealousness. I started to realize that open access was definitely not revolutionising access to education in the majority of the world, and that the passion that myself and others had felt about pushing forward the openness agenda was becoming sinister as any criticism was met with aggressive denial, as definitions of openness became ever narrower and technologically defined, and as we seemed to get further and further from the goals that we started with.

In the wake of Aaron’s death, and the renewed calls by the open access community for academics to take a stand, I felt that I needed to resolve these feelings and to define my own perspective on the issue. Thinking about the openness of your research can be like going down a rabbit hole because if you’re attempting maximum accessibility for all people at all times, any open access policy looks incomplete. Open access definitions tend to be restricted to a particular medium (digital, online) and a particular definition of free (free of charge and free from most copyright licensing conditions) (see Peter Suber’s great introduction to open access here).But access, especially when it concerns the study of people and communities, needs to follow a multiply layered, contextual approach if it is to fulfill the ultimate goals of an open research process. What about accessibility standards for differently abled? What about the fact that Arabic speakers can’t read your final report about them? What about the fact that not everyone can understand academic lingo or appreciate layperson explanations? Researchers’ open access policies don’t generally cover these equally valid issues that necessarily close your work off to particular audiences.

When the ethnomatters team met on Skype a few days ago to talk about this month’s theme, we reflected on what we thought about the current debate and what we might do to contribute to it. We all agreed that, although we deeply respect academics and researchers who have written personal open access policy statements boycotting closed access publications, our own view is that the open access debate shouldn’t be limited to the intellectual property rights and/or price of final results of research.

This website is testament to the principles that we think are important: we started it so that we would have an avenue to make our research results more accessible to audiences, but perhaps more importantly so that we could have a conversation with other researchers about the changing role of ethnography. In other words, the fact that Ethnography Matters is free and openly licensed is less important than the fact that we are starting new conversations accessible to audiences outside (and in addition to) those who read academic papers – in closed or open access journals. Information, to us, is valuable not just as a neatly packaged product for journals, but as the signal of larger conversations and the enactment of social relationships.

Janet Vertesi and Paul Dourish’s 2011 CSCW paper is really instructive here. Vertesi spent three and a half years embedded with two NASA-led spacecraft teams both working in the field of Planetary Science and examined the differences between the two team’s data sharing principles and practices. She noticed that Paris team members shared raw data internally across all members of their mission and saw their data as an inherently shared resource, while Helen members did not automatically share internally and were on the whole more selective about where their data traveled. By examining the context of data production within the Paris and Helen teams, Vertesi and Dourish concluded that how the data is crafted and acquired (rather than distributed or used) is critical to understanding their members’ different data sharing values. They explained that, with changes in expectations of data sharing within NASA and the broader scientific community, there has been an imposition of Paris-like rules for data-sharing uponHelen, producing culture clashes, confusion and frustration.

Rather than ascribing the sharing regimes to generosity or selfishness, it became important to see data in its “interactional context” where data is not merely an end product, a commodity, but that data “enacts social relationships”; it is one of the interactional elements with which social relationships are performed. Vertesi and Dourish argue that the imposition of data-sharing systems that do not respect the context of production results from “a troublesome process of commodification”, where the interactional and social regimes that produced the data are obscured. In conclusion, the authors call for thinking about “data sharing” within a broader “data economy” that focuses on production as well as use and exchange, so that we might clarify the value of data within local organizational cultures.

Vertesi and Dourish’s paper resonates because it acknowledges the different stages of data sharing in research, rather than merely the end product, and ascribes different data sharing principles to local organizational cultures (rather than different research fields). It reminds us that data has deep sociotechnical origins and that those origins are essential to understanding different cultures of data sharing. ‘Data’ or academic papers or blog posts should not be merely seen as a ‘objects’ that are primed, packaged, ready for transfer, but rather as a speech acts within broader conversations, each of which are shared and ascribed with different value within particular cultures.

For the Ethnography Matters team, it is important for us to support and apply accessibility as a principle to each stage of our research and that strategies to enhance access would necessarily look different according to each of the groups (or conversations) we hope to affect with our work. For Tricia, it is important to share photographs of her fieldwith her research participants and her friends and colleagues back home. For Jenna, it is important that she write an accessible summary of each of her journal articles for Ethnography Matters or her blog. For me, it’s important to enable my research participants to speak back to the data about them. All of us recognize the importance of open access to our research materials, but we think of access somewhat differently from the way that it is traditionally defined.

What do you think ethnographers should do about open access? What does openness look like to ethnographic researchers or to any researchers who study technology? Jenna has written a thoughtful post that has attracted a lot of great feedback on open access journals relating to ethnography. Do you have other suggestions on thinking about data sharing, open access and broader accessibility issues? We’d love to hear from you. Contact us or share your comments below.

Pic: ‘Open window’ by Sharon Hall Shipp. CC-BY-NC on Flickr

Sitting with a group of graduating high school seniors last summer, the conversation turned to college roommates. Although headed off to different schools, they had a similar experience of learning their roommate assignment and immediately turning to Facebook to investigate that person. Some had already begun developing deep, mediated friendships while others had already asked for roommate transfers. Beyond roommates, all had used Facebook to find other newly minted freshman, building relationships long before they set foot on campus.

At first blush, this seems like a win for students. Going off to college can be a scary proposition, full of uncertainty, particularly about social matters. Why not get a head start building friends from the safety of your parent’s house?

What most students (and parents) fail to realize is that the success of the American college system has less to do with the quality of the formal education than it does with the social engineering project that is quietly enacted behind the scenes each year. Roommates are structured to connect incoming students with students of different backgrounds. Dorms are organized to cross-breed the cultural diversity that exists on campus. Early campus activities are designed to help people encounter people whose approach to the world is different than theirs. This process has a lot of value because it means that students develop an appreciation for difference and build meaningful relationships that will play a significant role for years to come. The friendships and connections that form on campuses shape future job opportunities and help create communities that change the future. We hear about famous college roommates as exemplars. Heck, Facebook itself was created by a group of Harvard roommates. But the more basic story is how people learn to appreciate difference, often by suffering through the challenges of entering college together.

When pre-frosh turn to Facebook before arriving on campus, they do so to find other people who share their interests, values, and background. As such, they begin a self-segregation process that results in increased “homophily” on campuses. Homophily is a sociological concept that refers to the notion that birds of a feather stick together. In other words, teens inadvertently undermine the collegiate social engineering project of creating diverse connections through common experiences. Furthermore, because Facebook enables them to keep in touch with friends from high school, college freshman spend extensive time maintaining old ties rather than building new ones. They lose out on one of the most glorious benefits of the American collegiate system: the ability to diversify their networks.

Facebook is not itself the problem. The issue stems from how youth use Facebook and the desire that many youth have to focus on building connections to people that think like they do. Building friendships with people who have different political, cultural, religious beliefs is hard. Getting to know people whose life stories seem so foreign is hard. And yet, such relationship building across lines of difference can also be tremendously transformative.

To complicate matters more, parents and high school teachers have beaten into today’s teens’ heads that internet strangers are dangerous. As such, even when teens are turning to Facebook or other services to find future college friends, they are skittish about people who are discomforting to them because they’ve been socialized into being wary of anyone they talk with. The fear-mongering around strangers plays a subtle but powerful role in discouraging teens from doing the disorienting work of getting to know someone truly unfamiliar.

It’s high time we recognize that college isn’t just about formalized learning and skills training, but also a socialization process with significant implications for the future. The social networks that youth build in college have long-lasting implications for youth’s future prospects. One of the reasons that the American college experience is so valuable is because it often produces diverse networks that enable future opportunities. This is also precisely what makes elite colleges elite; the networks that are built through these institutions end up shaping many aspects of power. When less privileged youth get to know children of powerful families, new pathways of opportunity and tolerance are created. But when youth use Facebook to maintain existing insular networks, the potential for increased structural inequity is great.

Photo by Daniel Borman

This post was originally written for LinkedIn. Visit there for additional comments.

With the month almost complete, I’m overdue for an update about what’s keeping me busy:

Drone Lab

Since attending Drone Games last fall, I’ve been crazy about personal drones and Nodecopter. I bit the bullet and bought a Parrot AR Drone, my friends have them, the School of Information bought one, and I’ve rallied a posse of fellow graduate students to participate a group directed study this semester that we’re calling Drone Lab. I was even called a ‘drone fiend’ on Twitter, based upon my enthusiasm.

The structure and idea of Drone Lab is simple: every Friday we meet to discuss and program AR Drones, some of us simply hacking while others working on more-ambitious projects. So far the development has ranged in complexity from programming basic flight patterns, to exploring computer vision, hooking up an arduino in hopes of attaching additional sensors, and ultimately I think our collective goal is to deliver a taco (see Tacocopter). Random people have even started showing up to check it out, interested in learning how to program the drones – it’s pretty amazing.

I think Drone Lab has been a great and informal way for a group of us to work together, creating a local community of hackers. I’m not sure if this will remain a small group, or turn into something more open and large, but I’m having a great time. We have a shared repo of drone scripts on GitHub if you’re interested in checking out some of the code we’re experimenting with.

Harlem Shake (Drone Edition)

With the help of Arthur Che and Gilbert Hernandez, I directed a Harlem Shake at the UC Berkeley School of Information, featuring an AR Drone! I know the meme is dead, but it’s worth checking out for lols.

Mozilla Open Badges

I’m sprinting with the rest of the Open Badges team as we approach our Mid-March 1.0 release. Working as an Integration Engineer, I’m helping partners and community developers become Open Badges-compliant while also contributing code including Facebook integration with an upcoming release of the Mozilla backpack, and merging some large community contributions for a future version of WPBadger. It’s awesome. The badges team is wildly talented, and it feels like the community has momentum as we work toward hitting 1.0.

GitHub for Collaborative Writing

As part of my Computer-Mediated Communication (CMC) course, my team is researching and prototyping a tool that uses GitHub as a backend for collaborative writing. We’ll be looking at a variety of domains: academic writing, journalism, business materials, and technical documentation in order to identify overlapping needs and features that we could integrate into our tool. Some social and community features of GitHub like issues, comments, and forking will likely be integrated into our tool in addition to the distributed version control features of git. My contribution will be a technical evaluation of GitHub’s API, and contributing to our prototype.

When explaining the implications of this project, I’ve pointed people to one of the slickest tools I’ve recently seen, called Huboard. Huboard is a Kanban-style project management tool that allows you to organize GitHub tickets at various stages of development, and creates a radically different workflow for how you normally interact with tickets. And while the details of the prototype we’re developing for CMC are still up in the air, I hope that whatever we come up with is similarly simplistic in its approach, making use of GitHub to provide a lightweight visual and organizational layer. GitHub is a tremendously powerful platform, and finding non-code uses for it will be great.

Drinkly

Last but not least, Michael Hintze and I have been busy working together on our final masters project, called Drinkly. This project is solidly in the ‘fun’ category, with the purpose of making it easy to find happy hour specials at nearby bars. Drinkly is the latest of a number of projects we’ve worked on related to the chicken and egg problems of incentivising businesses to share information with consumers in a dedicated channel. We’ve received some positive feedback so far. More details on that next month.

If you were building a new platform, a platform that you fully and confidently expected to be the future of computation, would you want its most commonly cited advantage to be "cheaper than other laptops"? Do you think that kind of positioning would motivate cutting-edge development? Would you expect consumers to be excited about a device — even one with functional advantages over its alternatives — when they've only seen its category as the cheaper and lower-build-quality version of what they have now?

Or would you want there to be a premium, interesting, even exotic version, to demonstrate the possible future of the platform? Would you want developers, early adopters and advocates to have the opportunity for something luxurious to inspire pride and comfort? Would you want to highlight that in fact what makes the Web so amazing is that it's the kind of platform that can run effectively on cheap, even disposable devices with limited form factors and also on immaculately-designed high-end machines perfectly suited to it?

I don't think the Pixel will be a top-seller. I won't recommend it to many of my friends and family, including the ones to whom I've recommended or even outright given cheaper Chromebooks. But I want one, even though I can't really afford it. And I think that's a nice feeling to have about a Web-dedicated device.

—Nick

This notice is a bit overdue, but I figured I should put something here. I'm sad to say that I've mostly retired from food blogging. I just don't have the bandwidth to give Arthur Hungry the time and attention it deserves. Go big or go home, as they say.

That said, I'm not really full... I still love traveling and eating, but I'm more focused on enjoying the experiences and the people I'm sharing them with. There are amazing photographers/writers out there documenting meals more thoroughly and eloquently than I ever did. I'll leave it to the pros.

I'll probably tweet more and perhaps occasionally write a full post. Thanks to those who have followed my blog in years past. It was great fun while it lasted, and I'll keep everything online. Keep in touch via @CheDigital.

Yesterday afternoon I was out with some friends, having a beer at Jupiter, a restaurant in Downtown Berkeley. As I took a drink, I saw a beautiful colorscape at the bottom of my glass.

I looked up and realized the color was a reflection of a billboard on the side of the building, advertising the very hefeweizen I was drinking.

I tilted my glass and tried to get a better shot.

This one is almost unrecognizable as a glass of beer, but I love the colors.

Over the last couple weeks, I’ve been reading a startup essay series on the train to and from work. It’s Blake Masters’ notes from a startup class Peter Thiel taught at Stanford last year converted to essay form.

It’s been interesting–not quite the mythmaking or buzzword dump that most talk about startups devolves into. It’s not even all about startups or even technology–there are forays into the nature of capitalism, the contemporary social and political zeitgeist, themes about heroism and independent thinking I’d previously only seen in Harry Potter and the Methods of Rationality, and the basic narratives of Western civilization. I didn’t agree with all of it, but it was worthwhile reading.

The first essay starts off like basically every frickin’ class at the School of Information seems obligated to do, with lip service to the history of technology and globalisation and whatnot. If you didn’t go to the ISchool, though, you might not find it tiresome. And even I didn’t, once I forced myself to not skim it–there’s a few new ideas in there.

My favorite essays:

• Class 9: Distribution (Why selling things is hard and matters and goddammit you need it)
• Class 11: Secrets (Relies on some concepts introduced in class 5)
• Class 13: You Are Not A Lottery Ticket (Something completely different than what I expected from the title, but still good: the role of luck in startups and other crazy endeavours and how different time periods and societies think about the future)
• Class 18: Founder as Victim, Founder as God (The role of startup founders–and iconoclastic hero-villains of all sorts–in society. Aka “What do Oedipus, Zulu kings, Britney Spears, and Sergey Brin have in common?”)

Some of you close to me will know that I spent a brief period of unpleasantness trying to do good work on Wikipedia when I was in graduate school. I haven't made serious substantive edits to Wikipedia since that time.

The "high-water mark" of that period was when I started to try and moderate some Wikipedia disputes as part of the mediation cabal. I've always been a diplomat and have a passion for helping folks come to a common understanding. One instance particularly stood out as unpleasant: it resulted in someone I can only term a "wacko" writing a 60-page letter to the Dean of my program, AnnaLee (Anno) Saxenian, about alleged misconduct on my part as a mediator. Anno, of course, paid zero heed to it as she's familiar with my integrity and diplomatic tendencies.

Fast forward to this weekend, I received a communication from the person who felt wronged, almost 7 years later (and wrote the 60-page letter to Anno). He had the gall to reinstate a demand for monetary compensation (I forget why).

Anyway, in the interest of not feeding trolls but holding them up, I give you Thomas Cool, likely still indefinitely banned from Wikipedia:

-------- Original Message --------
Subject: W.r.t. democracy
Date: Mon, 18 Feb 2013 09:51:24 +0100
From: Thomas Cool / Thomas Colignatus
To: joehall@nyu.edu,joe@cdt.org

Dear dr. Hall,

You may recall my person from an unhappy situation at wikipedia in 2006. Someone asked me what I thought about the current entry on Arrow's theorem, and this caused me to update my comments on that. It appears that wikipedia has been misleading its readers since 2006. That update of my comments is in English as a supplement to a Dutch text:

http://thomascool.eu/Thomas/Nederlands/Wetenschap/Artikelen/2013-02-14-PasOpMetWiskundeOverVerkiezingen.html#Supplement

I think that it is fair to apply 4% interest over the seven years since 2006. The sum of $150 turns into $197. I use paypal at cool@dataweb.nl (still the same). My advice is that you also buy a copy of Mathematica and a license of my Economics Pack (a professional licence now is $99), so that you can start studying on the problem.

Allow me to congratulate you on completing your Ph.D. thesis. My hope is that you are better aware of scientific integrity now. Nevertheless, allow me to advise you to discuss the issue with your director Leslie Harris, just to be sure. We must hope that a Center for Democracy & Technology really understands what democracy is about.

Sincerely yours,

Thomas Cool / Thomas Colignatus Econometrician and teacher of mathematics Scheveningen, Holland http://econpapers.hhs.se/RAS/pco170.htm

PM. The other relevant links:

https://josephhall.org/nqb2/index.php/wikidisp

http://thomascool.eu/Thomas/English/Science/Letters/2006-03-20-Comments-RfC.pdf

http://www.wikinfo.org/Multilingual/index.php/Bordafixedpoint

http://thomascool.eu/Papers/VTFD/Index.html

Please note that I switched from dataweb.nl to this different location, but otherwise the links are the same.

Last week the FBI seized the file exchange site MegaUpload through a court order. Previously, users could exchange files too large to email through the service. Now visitors to the site see this message:

The significance of a website seized by law enforcement is heightened in light of the controversial SOPA and PIPA legislation currently being considered in Congress. Given the high stakes—the open exchange of information in a free society, government interference with the Internet—I feel compelled to let the people at the FBI know what I think.

Guys, this is embarrassing. Really amateur hour. Seriously, look at this again:

Where have I seen a better looking image? I mean, other than every movie ever made that shows a computer screen?

I don’t even know where to start. Fine, the seals. Any one of these seals by itself is like starting the drive from your own five-yard line. The three together is handing the ball to the other team and carrying them into the end zone. You’re not into football metaphors? OK, a seal crammed full of text like “National Intellectual Rights Coordination Center” is the design equivalent of dividing by zero. All three is taking the limit of bad design as it approaches zero and it evaluates to negative infinity. Math isn’t your thing? No sweat—what I said doesn’t make sense anyway. The point is, the seals are ugly.

But they’re your logos, you say? I feel badly saying this, but they look like someone just slapped them together at 4PM on Friday after a lunchtime happy hour. Take the right one. It says, “Protection is our trademark.” I’m not a IP genius, but it seems to me like if protection really is your trademark, and you want people to take it seriously, you need to use that symbol. Like “Protection is our trademark™” or maybe “PROTECTION™”. But since you’re not actually selling anything or engaging in trade, maybe it would be more accurate to say that protection is your service mark. You don’t see that little SM enough.

As if the seals weren’t texty enough already, someone put “FBI ANTI-PIRACY WARNING” on top of the middle one. Is that part of the seal? Operating under the dubious assumption that there’s any design merit to this logo in the first place, the last thing you want to do is cover up your logo. Can you imagine Nike labeling clothes with its swoosh but then covering half of it up with “GARMENT CARE INSTRUCTIONS”?

Who picked the color scheme for the background? Had this person eaten a hot dog recently? That’s the only way I can figure it out. You can’t even read the complete word “seized” once in this tiled background.

The cited list of alleged crimes at the bottom is a nice touch, but, guys, what are we doing with the typography here? A big block of bold, italic, centered text. I read the first line and I think, “This is heavy stuff—they’re being changed with Conspiracy to Commit” and then I get confused until I realize that it’s Conspiracy to Commit … Copyright Infringement (18 U.S.C. § 371). I know how to continue reading on the next line, but you’re taking some serious liberties with awkward line breaks.

Let’s check out the source of the page:

<html>
<title>NOTICE</title>
<body>
<img src="banner.jpg"/>
</body>
</html>

No JavaScript. No AJAX. No CSS. Not even any tables. The image doesn’t have ALT tags. Maybe you’re not worried about Google indexing this page, or visually impaired people being able to read it, but I hope you realize you are just flushing the last 8 years of the Internet down the toilet. Interestingly, you went with the trailing slash that closes empty elements in XHTML but the DOCTYPE is…nothing. Whatever—this stuff is for nerds.

What we need to focus on is what a colossal missed opportunity this is for you. MegaUpload is down and the notice on the site is getting tons of exposure and when you go there it’s like you’re stuck watching the beginning of a movie forever, or at least that’s what it seems like for those people who paid for the movie and have to watch the FBI reminder to pay for the movie.

You must plan these operations, right? I mean, it’s not like you just randomly seize private property on a whim. This is a failure of project management. You can’t just bring in a designer at the last minute and expect them to polish your design turd. This is your chance to shine. Go wild. Animation, maybe a Matrix-style flow of numbers in the background. Ominous type. Here are some ideas:

• The user goes to MegaUpload. The site looks normal. Suddenly, the eagles from your logos swoop in and the cool one with the arrows in its feet starts attacking the site while the other one hangs a banner over it that says “Seized by the FBI” and then jail bars drop down over the entire site.
• The user goes to MegaUpload. The screen is filled with static like an old television. Then it looks like the screen is flipping through different TV channels. They’re all static. Finally, you get to a channel with a retro-looking message: “Seized by the FBI”. The retro part here probably plays to your design strengths.
• The user goes to MegaUpload. The site is covered with sheets of brushed aluminum that look very heavy duty. Etched into the aluminum is the message: “Seized by the FBI”.
• The user goes to MegaUpload. It says “HTTP Error 460” (this doesn’t exist—you would be making it up): “Seized by the FBI”.
• The user goes to MegaUpload. A video of Rick Astley singing “Never Going To Give You Up” starts playing. When the video finishes, it fades out and is replaced by the text “Seized by the FBI”.
• The user goes to MegaUpload. Suddenly, a S.W.A.T truck drives onto the screen. Fighter jets fly overhead. Missiles, bombs—BOOM—the screen explodes. DOM elements lie in a heap at the bottom of the screen. Smokes rises from the ashes and all of a sudden you realize it’s forming words: “Seized by the F.B.I.”

There are probably jQuery plugins that do all these effects already and you could use those as framework to build on. So dust off your copy of Photoshop. Use the mosaic filter. Add some lens flares. Watch Sneakers and Hackers and The Net and The Matrix and Tron and WarGames. Stay away from Papyrus. Then go and take down MegaUpload and put up something amazing. This is your moment: seize it.

Pilots will tell you that the most dangerous part of flying is takeoff and landing. This is correct, though it’s most dangerous for people who aren’t busy flying the plane. Shortly before takeoff the doors to the aircraft are closed. From this point on the FAA requires passengers to shut off their electronic devices—the primary means of entertaining oneself in the 21st century—until the aircraft has reached its cruising altitude. Your Kindle off, there’s nothing to read. Your iPad off, there are no angry birds to avenge. Your eyes drift towards the seatback pocket, the only remaining source of entertainment until 10,000 feet. Safety card? Boring. Motion sickness bag? Gross. Oh, hello there, SkyMall.

SkyMall has been in American seatback pockets forever. It has a selection of products from dozens of resellers spread over 100 pages. You can purchase products you find in the magazine by calling 1-800-SKY-MALL, or at skymall.com, both inaccessible from most aircraft.

Browsing an issue of SkyMall follows a typical path. Initially, your resistance is high. Everything in this catalog is ridiculous. These things are a total waste of space and money. Who wants a $400 bronze sculpture of Superman? A 6-foot tall Yeti garden sculpture? A birdhouse to curb your dog’s barking? If SkyMall were your first exposure to human desire, you would think that people are really crazy for pillows. A typical issue has at least a dozen different pillows. Inflatable pillows, wraparound pillows, NASA space-age cooling pillows, “the most comfortable pillow you will ever own,” a pillow that looks like a tree log “only feeling about a million times more comfortable.” You could also be excused for thinking that humans are gonzo for replica objects from fantasy movies like Aragorn’s sword from Lord of the Rings, Hermione Granger’s wand from Harry Potter, and Gandalf’s staff from The Hobbit. I hope we are not.

Next you imagine the scenarios behind some of SkyMall’s offerings. A magnetic cup holder that attaches to your ride-on lawnmower or tractor: “Mary, do you remember when I was out mowing last week and I dropped my beer? I found something that could change all that.” A grass turf platform with fire hydrant so your dog has a place to relieve himself in the apartment. “We used to have some many problems with Butch until we bought a fake hydrant for him to piss on.” A cell phone suitable for underwater use up to 100 feet: “What a beautiful triggerfish! That manta ray is so graceful. Hold on—I have to take this call from Gloria in accounting.”

You find a favorite crazy item. How about this pair of wine glasses that can each hold an entire bottle of wine? It’s perfect for the occasions when you want to impress your guests by serving them a pound and a half of wine. Parties aside, they’re great for any time you want to kick back with a bottle of wine, without the hassle of refilling your glass four, even five times.

But by the time the plane has taxied and left the ground, your resolve has weakened. Sure, most of the things in SkyMall are useless, but there are a few gems. I have been trying to improve my posture. Here on page 47 is a breakthrough waiting to happen: a velcro harness that I could wear just 30 to 45 minutes a day to train my back. Definitely dog-ear that page.

I am usually not one for travel clothing, which looks best on people who don’t care how they look. That said, I could really get into these pickpocket-proof pants on page 62. Pockets that are too hard for thieves to open make a lot of sense. I don’t love wearing a money thong on international trips. With these pockets, I could just leave it at home and carry my wallet like a normal person. No, that’s silly. I’m not going to wear the same pants every day of my trip. Oh. They come in three different colors. And shorts. If I rotated them, that could easily be 10 to 15 days worth of pants. How do I get these again?

I just want the posture harness and five or six pairs of these pickpocket-proof pants, which are actually a great deal at $109. The woman on page 73 looks like she’s sleeping really well with that neck pillow. And you know, it’s not like this phone only works underwater. Obviously I could use it above or below the ocean. I could stop worrying about getting pushed into swimming pools with my phone in my pocket. On second glance, there is a kind of quiet beauty in this bronze Superman. OK, the harness, the pants, the pillow, the Superman—two, my brother loves comic book movies too—the underwater phone and—

Ladies and gentlemen, you may now use your approved electronic devices.

What? Where am I? Woah, close one. See you on the way down, SkyMall. Be strong, Ryan.

Sigh.

Outlook should die, die, die.

Short version: if you use Thunderbird to read email and you regularly get meeting.ics files from people, install Show All Body Parts in order to open/save them.

Longer version: Lately, I've had the following problem: Using Mozilla's Thunderbird email client (v17.0.2), when I receive a meeting reminder/calendar entry from a co-worker -- meeting.ics -- no matter what I do, I cannot seem to save the attached file. When I attempt to "open" it, Thunderbird responds with an error and says there's no file and when I "save" it, no file is saved. Nothing. Frustrating.

Up until today, I had been able to reasonably cope with this by looking at the raw email message source ("View" -> "Message Source"), which includes the .ics file in text at the very end.

It's not pretty; it looks something like:

--------------030809050301030806060808
Content-Type: text/calendar;
 name="invite.ics"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="invite.ics"

BEGIN:VCALENDAR
PRODID:-//Google Inc//Google Calendar 70.9054//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REQUEST
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
X-LIC-LOCATION:America/Los_Angeles
...

Not ideal, but oh well. At least I can read it.

However, today, I did that and saw this (!!!):

--_000_C6E91EC3EACD3E44B2A8242C2663C396027136FD09exmb2phlpewpe_
Content-class: urn:content-classes:calendarmessage
Content-Type: text/calendar;
    charset="utf-8";
    method="REQUEST";
    component="VEVENT";
    name="meeting.ics"
Content-Transfer-Encoding: base64

QkVHSU46VkNBTEVOREFSDQpNRVRIT0Q6UkVRVUVTVA0KUFJPRElEOk1pY3Jvc29mdCBFeGNoYW5n
ZSBTZXJ2ZXIgMjAwNw0KVkVSU0lPTjoyLjANCkJFR0lOOlZUSU1FWk9ORQ0KVFpJRDpFYXN0ZXJu
...

OMG. That's not readable. It's our old friend base64 encoding.

Sigh. It's not hard to decode base64 and any good techie should be able to do it... but this is my email and why should I have to write a script or something just to read email!?

So, I dug a bit deeper.

It turns out this is a "feature" of MSFT's Exchange/Outlook product line. That is, the MIME type they use for this is multipart/alternative which means, "here are a few different versions of this email, pick whichever one you want, they are the same". This is useful if you want to send a plain text version of an email and also an HTML version of an email that looks prettier and includes clickable links and such.

However, Outlook and Exchange include text/plain, text/html and text/calendar... and -- here's the kicker -- not all the stuff in the text/calendar version is included in the other two! And to make things totally ridiculous, Outlook transforms email into base64 encoding!!! So, you can end up exactly with the situation I had today: I had a calendar invite but no indication from the body of the email the date and time of the meeting, and when I viewed the message source, it was unreadable without decoding from base64.

Turns out, the Thunderbird extension called Show All Body Parts will display all the alternative message formats when you ask it to ("View" -> "Message Body As" -> "All Body Parts").

Whew.

BTW, this is a 3.5 year old bug: Bug 505024.

Michael Hintze and I recently met up with Peter Merholtz to discuss our I School final masters project. With three months until graduation, we were seeking advice on product development and the decisions we’ll need to make.

One of the strategies Peter suggested was to focus around a set of design principles, referring us to a blog post by Luke Wroblewski. This has helped organize our project early on, and spurred a series of related thoughts I felt were worth writing down and sharing.

In his blog post about Design Principles at Microsoft, Luke cites his own definition of design principles:

Design principles are the guiding light for any software application. They define and communicate the key characteristics of the product to a wide variety of stakeholders including clients, colleagues, and team members. Design principles articulate the fundamental goals that all decisions can be measured against and thereby keep the pieces of a project moving toward an integrated whole.

This definition captures several important things. First of all, the audience of design principles can be diverse, simultaneously resonating with programmers, designers, project managers, and sponsors. Particularly on large teams, communication breakdowns between team members can fragment a project and sometimes force revisions to work that has already been done. Additionally, principles can help teams make better decisions.

Peter’s suggestion was to write the principles on paper and hang them from the walls of the workplace. When making a design decision, does it align with the principles? Principles can become a filter through which team decisions are made.

Principles for our project

Here’s a sampling of the design principles Michael and I identified for our final project:

1. Delight and surprise. The smallest details matter for a great experience. Use well placed animations when you can.
2. Notify users sparingly. Notifications are disruptive. Only bother users when the alert is meaningful and significant.
3. Have fun when you can. A beer mug filling up is more fun than a spinning circle, or beach ball of death.
4. Keep it simple. People love brevity. Keep things short and meaningful.
5. Get to know the user. Learn about our users over time. Don’t make them input the same thing over and over

These principles capture our user experience goals, and begin to define the personality of the project itself. Simplicity, delight and surprise, and having fun clarify both what the project is, as well as what it is not. I’ll update this post later in the semester to measure their effectiveness.

Luke’s blog post shares examples of principles that have persisted through product development at Microsoft, including Windows 7 and the Microsoft Surface. There’s some interesting crossover between their principles and ours, but also some divergence – I think it’s important to not be dogmatic about the use and definition principles, but to think of them as a management tool. No two projects will have the same principles, nor the same quantity or granularity.

Design principles in an agile process

Principles create an artifact that persists across the development cycle for a product, sometimes in the absence of a predefined plan and before a working version of software has been delivered. In an agile development model, decisions are made throughout the life of a project, instead of up front as they would be in a waterfall process. After all, the Manifesto for Agile Software Development itself is organized around a series of principles. Principles offer flexibility to make decisions later, while keeping them focused on a larger goals.

They can also establish an understanding among stakeholders about the experience of a product, which sometimes is difficult to grasp until a prototype exists. There are obviously examples of when this is less crucial – strong corporate or team cultures are other ways to focus the passion and ideas of employees, establishing consensus regarding style and qualities of a work. But the reality of many teams is one where not everyone is centrally located, possibly involving differing stakeholders including contractors, content partners, and sponsors. Principles can provide cohesiveness and certainty to what otherwise may seem distributed and chaotic.

Principle ALL THE THINGS!

Lastly, principles can be applied more broadly than to a single project, and instead to a larger body of work. One of the inspiring talks I watched last year was given by Bret Victor, Inventing on Principle. Bret asked us to define what the guiding principle of our individual work is, and I spent a lot of time trying to define my own guiding principle as a result. I’ve come to decide that I lack a single principle, although I admire Bret’s focus. I could name half a dozen that guide me, but not a only one. That is fine. The purpose of principles is simply to help you focus, provide something to look back at in the future to evaluate your progress, and offer a way to clearly communicate your purpose to others.

I’d love to discuss how principles are used in other ways than I’ve described in this blog post. Please let me know what you think.

At the ASTD TechKnowledge conference, I was asked to reflect on networked learning and how tomorrow’s workers will challenge today’s organizations. I did some reflecting on this topic and decided to draw on two strands of my research over the last decade – startup culture and youth culture – to talk about how those outside of traditional organizational culture are calling into question the norms of bounded corporate enterprises. The piece is more of a provocation than a recipe for going forward, but you might enjoy the crib of my talk none-the-less:

“Networked Norms: How Tech Startups and Teen Practices Challenge Organizational Boundaries”

(Image courtesy of victuallers2)

I was walking throug the library recently and happened to notice this post-it taped to someone's laptop. That someone was a large, burly and athletic-looking undergraduate guy; not someone I would have expected to have such a new-agey sentiment so prominently displayed on his computer. I hesitated but was so curious I had to ask him about it. 

He was happy to explain its' origins. He is a representative on a student governance organization at UC Berkeley and said he found himself repeatedly getting worked up about insensitive comments  made by others during the meetings; so much so that he'd had a couple outbursts which he was embarassed about. He thought the message on the post-it might make him think twice before saying something he'd later regret. I asked him if it worked. Yeah, he said, it actually did cause him to pause and reconsider his anger during meetings. It had been on there about 3 weeks by the time we talked. 

This is something which BJ Fogg would call this a "hot trigger"; an actionable reminder put in the path of someone who was motivated to modify their behavior. I also like that this is actually a 2-step process, implying that before you can get to get to the listening part you must first breathe. Words of wisdom for all!

Permalink | Leave a comment  »

Another of my friends is asking me questions about buying a bike. I love that they do this, but since it's become a trend, I figure I should throw my thoughts together here as a reference.

Buying a bike is actually pretty simple:

1. You want a used road bike.
2. You want gears; more is better.
3. Lighter is better.
4. Size is important.
5. Exotic isn't for you (yet).

That's pretty much all there is to it, at least on the macro level. Let's dive into each of these a tad, shall we?

First, you want a used road bike. The reason for this is pretty logical: As a commuter, you'll be riding mostly on...roads! You could get a cruiser (but that defies rule #2), or a mountain bike (defies rule #3) or a recumbant (see rule #5). But ultimately, road bikes are the best for what you want to do: travel quickly and safely to and from work and around town with minimal fuss. New ones cost a fortune, so get one used.

I wish I didn't have to mention rule number two, but for all the hipsters out there, you're wrong about fixies. If you life in San Francisco, a city with specific maps for the hills, you should use the things that were invented to get you up hills: Gears! Even if you live in a flat-ish city, you should get gears because they give you flexibility. Sure, your bike is a commuter today, but tomorrow your friend might want to go on a bike ride with you. Or maybe you move to a new city. Who knows? Get gears. Don't be trendy.

Enough said. Rule number three is that lighter is better. If you look at new road bikes, you'll quickly learn that you can spend an incredible amount on bikes. And naturally, the ligher they are, the more expensive they are. So how light is right? Well, this one is tough and somewhat subjective, so I say, find one that's as light as you can buy for your dollar. Off the cuff, I'd wager that around $400 is the point of deminishing returns for most people.

Size is important. If it doesn't fit, it's useless. Go to a bike shop and get sized before you do much shopping. It'll help you winnow the stuff you're looking at anyway. This may as well be the first step of your search.

Rule five explains itself. When you know more about bikes, branch out if you care. You probably won't, so save yourself the effort of looking at exotic stuff. It's exotic for a reason.

Bonus questions for the avid reader

What brand should I buy? Doesn't much matter, surprisingly. There are better and worse brands, but if you're buying a used road bike, and follow rule number 3, your goals will be accomplished.

What material should my bike be? Probably steel. Aluminum is good too, but probably out of your price range. Steel's a very reputable material though. If you can find Reynold's steel, all the better.

How do I know the bike I'm buying isn't stolen? Good question! This one's hard. You can look for the serial number or try to only deal with people that seem legit. There is a national bike registry (which you should use!), but otherwise there's not a whole lot you can do...yet.

Comments? Thoughts? Email me. You're probably my friend already if you're reading this...

This post is as much Public Service Announcement as anything else. I didn't realize that two-factor authentication had finally taken off. It's practically vital for your email account (you're asking for trouble without it), but in the past year or so, a bunch of other services have begun offering it.

Today I went on a little security binge, and found that I could turn on two-factor authentication at:

• Google/Gmail
• Yahoo
• Dropbox
• Charles Schwab (they send you a fob for free)
• Facebook
• Paypal
• Amazon Web Services

One note about Charles Schwab is that getting their fob is great, but it's hardly all you should do to secure your account. You should also set up what they call a "verbal password" that you have to provide whenever you call in. Without it, it's pretty easy to get into an account via their surprisingly weak phone security.

Anyway, this is a pretty good list so far. The companies are using a handful of different techniques for doing this, but they all seem pretty solid in the end. Google's, naturally, seems to be one of the most robust, but I'm impressed there's so much offered.

Go set these up!

When I ranted about how the media’s incessant desire to get the story was not allowing the people of Newtown to mourn, I did not expect that I’d be mourning the death of a friend a month later. As I’ve tried to come to terms with Aaron’s death, I’ve found myself slipping between personal grief and meta reflection, my primary coping mechanism for dealing. I’ve decided to craft this deeply personal, reflective blog post because I think that it makes sense to think about and discuss what mourning and vulnerability mean in a culture of public-ness.

Last week, I read the story of a Newtown woman who was photographed grieving at a vigil. She voiced her struggle in becoming an icon of the tragedy, understanding the importance of the story but also wanting to be respected as an individual going through her own struggles. This story resonated with me.

Aaron Swartz was a friend of mine. When I woke up three Saturdays ago to learn about his death, I went into complete shock. I spent the day talking with mutual friends, reading heartfelt stories on the internet, and crying. I woke up the next day and while lying in bed, penned my own memorial for my blog. I always struggle with what my blog is, wanting it to be a way of communicating with people who know me while also recognizing that it’s read by many strangers. Still, it’s the best vessel to share with a community of geeks that I love. So I posted my reflections there.

I should’ve known that my blog post would attract the attention of the press. But I wasn’t really processing reality and I hadn’t yet begun to realize that the story would become a mainstream one. On one hand, I was glad. People should know who Aaron was and why what he did was important. On the other hand, I watched as the story took on a life of its own, not just a rallying cry for geeks, but a portrait that I could barely recognize.

I started getting questions from journalists and I panicked. I wrote as politely as I could that I couldn’t do it, I couldn’t be their source. But I also struggled with the very fact of their asking. Yes, I blogged about him, but what right do I have to speak for Aaron? To tell Aaron’s story? I was Aaron’s friend, but like many friends in my world, we ebbed and flowed in terms of closeness over the last 9 years. I was not his lover, I was not his professional collaborator. I had only met his father once and only met his current partner briefly. I knew only a small subset of his social circle. I knew – and adored – a facet of Aaron. But who was I to tell his story? I found myself fretting over how my blog post had created a signal that I didn’t want to to convey. I wanted to express my grief and my fear over how parts of my community were reacting, not claim proximity or define Aaron.

As the funeral rolled around, I scratched my head as to whether or not to go. I hate funerals, but I also knew that many people I cared deeply about were struggling even more than me. And none of us live in the same city. And part of me wanted to support them and show respect. But then I heard people who didn’t know Aaron talking about going because it was “the right thing to do” which was basically code for the notion that everyone who was everyone was going to be there. It also became clear that it was going to be a media event with the possibility of protestors and counter-protesters. And I balked. I couldn’t stomach the idea of simultaneously trying to mourn and trying to deal with a collective performance of mourning. So I stayed put, in a daze.

And then the memorials started up and I was asked to speak. And I realized that I was so not there yet. I was still at the stage of cry and hug. And I was struck by how some of those who he spent his last days with were able to channel their own emotional struggle into a far more productive output than I could possibly muster. I was in awe of their strength, of their commitment. Because I relished the activist in Aaron and I was completely intellectually with them in their efforts to use this moment to make a difference, to speak out against the structural inequities and abuses of power that Aaron was fighting for. But I wasn’t able to be there; I was still in a dark place that wasn’t productive. So I pushed my own escape valve, a privileged valve I’ve learned to use whenever things got dark. I spent my frequent traveler points and went to Miami to hug the sun and the beach and get a grip on my mental state.

I’ve spent a lot of years struggling with my own demons and one of the best parts of growing old for me is that I’ve developed coping mechanisms to identify and deal with my emotional state before I put myself or anyone else at risk. I’m good at getting control over me before running full-speed with my activist goals. This is something that I’m very proud of. But I also look back with a lot of uncertainty because I don’t know how I would’ve gotten here without the internet as it was when I was coming of age.

As I’ve said many times, the internet has been my saving grace. I usually publicly point to the ways in which I have learned about new opportunities and been challenged to think about a world that was bigger than the one that I was inhabiting on a daily basis. But embedded in those positive-minded narratives is the fact that the internet allowed me to be vulnerable and safe in a productive way. I spent countless hours anonymously talking to strangers about my demons without being tracked or identified or outed. I used LiveJournal to be deeply vulnerable among friends in an environment where we all understood that it was a site where you shared your struggles and everyone respected that. I lashed out in ways that helped create friendships as a byproduct of getting help. And I started blogging in a context where putting yourself out there enabled a small community of caring folks to listen and be supportive in all sorts of unique ways.

Today, I don’t have that luxury. My internet is painfully public. My interactions online are heavily constrained by pressures I get from others. When I post something controversial or incomplete, I am consistently publicly attacked and I’m regularly told that this is what I get for choosing to speak in public. I am told that as a public figure, I deserve whatever I get. So when I choose to make myself vulnerable in public, I have to brace myself for attacks. Even my attempt to mourn Aaron generated messages by strangers attacking me for clearly having not been a real friend or else he wouldn’t have died. I’ve learned to roll with these attacks, but they still sting. And I don’t know that my younger self could’ve handled them well. But what I know is that I no longer have an internet where I feel safe to deal with my demons. There are still places where people can carve out that space, but somehow, it never feels safe enough to me anymore.

I realize that this nostalgia for the past is interwoven with a whole set of different issues that make this a self-absorbed lament. But Aaron’s death threw it in my face. Because, on the day that Aaron died, the geeks that I grew up being vulnerable with all let down their guards and wailed through the media that they knew best. And then his death went from creating a weekend of collective sadness and storytelling to an event that took on a life of its own. And I’m left with a discomforting question: where are geeks allowed to be vulnerable today?

With each new geek death by suicide, folks ask what we should do about the depression in our community. I can’t help but think that we’re paying the costs of the public-ness that we’ve helped create. We’ve made geek culture something to watch, an economic engine, a dependency. And in doing so, we haven’t enabled safe spaces to grow. We’ve created communities connected around ideas and actions, relishing individualistic productivity for collective good. But we haven’t created openings for people to be weak and voice their struggles and demons. In short, we don’t know how to support vulnerabilities and rather than debugging the problem, we just hope that if we don’t pay too much attention to them, they’ll go away.

But the pressures of public-ness are forcing us to pay an ugly price. In NYC alone, we’ve lost two geeks in two years because their worlds spun out of control in ways that didn’t leave them with enough strength to grapple with their demons while also trying to make the world a better place. Throughout this country, there are geeks and hackers facing serious pressures by structural power as a byproduct of the public-ness we’ve created. And many of them are also facing serious demons. I’m definitely among those who want to hold political entities accountable for capitalizing on vulnerabilities in their pursuit of the status quo, but I am also hoping that the geek community can figure out a way to make sure that those who are struggling have enough support to fight both their demons and their oppressors. Before we lose another one. The problem is that I genuinely don’t know how.

[Note: This post will be completely boring and irrelevant if you don't care about running Mystery Hunts, but Atlas Shrugged asked and Asheesh insisted I post it somewhere publicly. :P]

1.) On our team, hardly any of the art volunteers had web design experience. Many didn’t do digital art; quite a few didn’t even have steady access to a scanner. It’s a bit of a mismatch between skills and tasks given that the biggest, most visible responsibility for the art team is the hunt website. Thus, one of the biggest responsibilities for the art team lead is to divvy up and delegate any artistic elements of the site design that can be done independent of the HTML/CSS/etc, so that your web designers and web devs’ bandwidth can be saved for the stuff only they know how to do. For instance, I started working on the Ocean’s round design months ago with placeholders for Diana’s casino paintings. When the paintings were finished, everything else on the round was basically ready I could just plop them in. Same with the Indiana Jones map.

It’s easy for the art and tech teams to be composed of very different sets of people who don’t talk enough to each other. We were lucky this year that I knew most of the tech team and am a programmer myself for my day job, and thus could bridge that gap. I hope you have at least one person on your team who can take on that role as well! It might even be worth reorganizing so that there is an explicit ‘Web Team’ consisting of both designers and developers (and, later, post-production volunteers helping prepare puzzles for publication).

2.) Also along these lines, making it easy for non-programmers to get your hunt website up and running locally is key if you want people to contribute to your hunt site design–otherwise, they can’t see what they’re doing! For the Agent 99 round, DD designed it in some static HTML files, and I converted it to use Jinja2 blocks and play nice with the rest of the site myself. This worked okay, but meant that I was a bottleneck for any and all design changes; no one else on the art team had a working copy of the hunt web software and thus could work on this stuff. Which is not a great state of affairs.

What may be wise (and kind of what we ended up doing) is to spin off the front-end as a separate app with URLs and views, placeholder templates, and a json or Markdown file full of fake hunt data (a team, puzzles, rounds, whether things are solved or not). Asheesh also packaged all the dependencies he could into this app’s git repository, to minimize the amount of installation hassle people would have to go through. (Though there was still some hassle; yay gevent!) Having a quality README for getting things up and running, and someone to patiently mentor your designers in using source control, are key too.

The goal is to have a piece of software with easy cross-platform install instructions where designers can muck around in the templates and static files folders, modify the fake hunt state to simulate different parts of the hunt, and see the results immediately, without requiring them to know Python or Django or Flask or whatever and making any necessary use of the command line as painless as humanly possible. (For more on this, ask Asheesh. He is the helping-newcomers-be-useful-in-software-projects expert.)

3.) Less (a dialect of CSS) is awesome. You don’t have to install anything to use it, you can just include a JS file to interpret it, it makes your stylesheets so much cleaner, and since it’s a superset of CSS if there’s anybody on the project who insists on using vanilla CSS their code will still work fine.

4.) Google Web Fonts is also pretty sweet. However, regardless of the fonts you use elsewhere on the site, the div where you put puzzle bodies should use one of the standard boring web fonts (Verdana, Helvetica, Times New Roman, etc.) because some puzzles use crazy arcane special characters. Most of the fonts on GWF don’t have those, so they’ll either show blanks or just look weird.

5.) Browser compatibility was not a priority for us. Our rule was that we’d get things working in Chrome, Safari, and Firefox, and if it worked in IE too, so much the better. We also assumed browser widths of at least 950px, which I’d argue is pretty standard these days. Got a fair number of complaints about that, though–both puzzle authors complaining that the design was too narrow for their puzzle, and other team members complaining that the design was too wide for netbooks. Responsive design, and/or having a separate stylesheet for tiny screens, would have been ideal, but we didn’t manage it in all the rounds and there were lots of more important things to spend our time on. YMMV.

I don’t think any hunt to date has built a mobile version of their site. Probably because it’d be a giant pain and you’ve got a million other things to do. Not unthinkable, though, if your site design is already on the minimalist side. And imagine what you could do with it for the runaround!

6.) Something I completely forgot about until a few days before hunt was printability. Pretty sure some of our rounds fared lousy by that metric. For next year, it might be worthwhile to have a “printable version” link on each puzzle page that links to a unstyled HTML page version of whatever is in your {% puzzle_body %} block (or equivalent template architecture). Then you don’t have to sacrifice the round design for this consideration.

7.) Originally, many of the round designs included slots for correct answers on round or puzzle pages. I was convinced by my teammates that this was a bad idea because if the unlock code glitched or HQ accidentally marked a wrong answer as correct, whole puzzles or even rounds could be instantly spoiled for teams. Bad news. What we decided was a best practice (and eventually implemented, though it wasn’t finished until after Hunt started) was to have a call-in page for each puzzle which shows the list of answers that that team has called in. Since this page never asks the database what the real answer is–it only looks at call-ins–even if HQ were to screw up, it can never show spoilers, while still making it possible for teams to look up solved puzzles’ answers.

8.) The company that 3D printed this year’s coin (and last year’s coin) is http://milcoins.com/. If you want a metallic coin-shaped coin for next year, I recommend them.

9.) Video is hard to pull off. Really, really hard. Unless it’s something you can shoot with a cell phone, nobody owns a video camera. And even if someone does own a video camera, they probably don’t own Final Cut (or equivalent software). And even if you have people who fill these requirements, getting them in the same place as your puzzle writers and actors for a whole day (even for a short video) is hard. And then when your editor or testsolvers demand changes to your video puzzle, instead of just editing a text file like a normal puzzle author, you have to haul all those writers and actors and camerapeople into the same room *again*, and make sure not to have any red herrings due to continuity errors. Rinse and repeat for the next testsolve. Video puzzles are the opposite of agile. And fact-checking or test-solving a script is simply not good enough.

Video is ridiculously resource-intensive. Only do it if you’ve got a really, really good reason and all the people involved are living in the same place. (Opening ceremony is probably a good reason, though–especially since that doesn’t typically need testsolving.)

10.) Watch out for spoilers in your CSS files (and other static files). We missed this this year. Based on some of the id names in the CSS file that appeared on the enigmavalley.com login page (which was findable based on the Hunt invitation), at least one team was able to guess coinheist.com and found its login page–the only thing between them and a view of the Hunt website in a fully-solved state–a few days before Hunt started. Whoops. (They never managed to log in there, thank goodness, but it sent the tech team into a panic as we quickly changed our passwords to be much more secure and swept our CSS files for spoilers.)

Similarly, ideally, you should only be able to access a round’s static files when that round itself has been unlocked–otherwise, the URL should 404. This was something that this year’s hunt site architecture handled quite well. I wouldn’t worry about this until close to Hunt, though–juggling lots of different stylesheets and JS files across multiple folders is a pain when you’re just trying to get started on a design.

Oh, hm. One more bit of advice occurs to me, though who knows if this falls under the purview of the art team or the tech team:

11.) Set a deadline, after which the canonical version for all the puzzles gets moved from Puzzletron to your hunt website git repo and if authors want changes after that date, they have to make them (or request that they be made) in the site code. (Textual edits can be made via Github’s visual editor, so even non-techies can fix typical errata after this date if you show them how.) We failed to do this, which resulted in sleep madness Thursday night due to not knowing which puzzle files we could declare done and move over and which ones people were still tweaking.

Due to a php upgrade at my wonderful host Birdhouse, this blog has be inoperable for a few months... I've upgraded the b2evolution software underneath and will try to write about once a week. I've disabled comments as they're totally useless, don't read comments!

Note that my work, CDT, has its' own set of blogs I contribute to regularly.

I spun this out into a separate post because the previous post was already just too absurdly long. Anyway, I wanted to post copies of the three puzzles I worked the most on for Hunt, both to save them for posterity and to give readers some idea of what Mystery Hunt puzzles are like.

The first puzzle is named “Evolution”, and contains a bunch of screenshots of email clients.

The second puzzle is named “Will Ruin Your Life [I'm Feeling Lucky]” and has a bunch of images from various pop culture media items.

The third puzzle, “Noise Pop”, is a bonus music-based puzzle me and Asheesh put together. It did not actually appear in this year’s hunt (it was a backup in case we needed to swap out a puzzle at the last minute). So if any readers of this blog were at the Hunt, and aren’t sick of solving, well, here’s a new (short, I promise!) puzzle! :)

I’ve also posted the puzzles’ solutions, though (obviously) don’t spoil yourself if you don’t want to be spoiled!

1. Evolution

2. Will Ruin Your Life [I'm Feeling Lucky]

A maze of twisty open tabs, all alike. Or so it seems.

Only a singular path will take you from the beginning to the end.

Solution

…and here’s the new puzzle!

3. Noise Pop

MP3

_____◯___________
_____◯
◯____◯___
◯◯______
___◯
◯___◯_
__◯____◯______
____◯
__◯_____

tl;dr: I spent the last week helping organize the 2013 MIT Mystery Hunt; this year’s Hunt broke the record for the longest hunt ever, lasting 73 hours instead of the expected 48; congrats to the team whose name is the entire text of Atlas Shrugged (who we referred to as John Galt) for winning, condolences to them as well for having to do this next year; learning to appreciate bits of joy and hilarity in a sea of crushing fail. Also, check out this post for some sample Hunt puzzles and an as-yet-unseen puzzle written but not used for Hunt.

(warning: effin’ gigantic post)

(other warning: all of the below is my half-baked opinion based on incomplete information)

The Hunt We Wanted

If you aren’t familiar with the MIT Mystery Hunt, it’s a weekend-long puzzle-solving competition that’s been held at MIT for over thirty years. Each year the Hunt has a different theme or plot of some kind; last year’s Hunt was Broadway musical themed (particularly The Producers), and the year before that was video game themed, where you were Mario trying to rescue Princess Peach. Every year, to win the Hunt you must find the coin: a usually (but not always) coin-shaped object hidden on MIT’s campus that requires solving tons of puzzles and engaging in a scavenger hunt-esque “runaround” in order to find.

I’d heard of the Mystery Hunt years ago, as I had several East Coast friends and acquaintances who participated. Last year was my first time attending the Hunt, though, with a team called the Manic Sages. Since we won last year, we had the dubious privilege of organizing and writing this year’s Hunt. I volunteered in a few different capacities–writing and testsolving a few puzzles, hammering out the hunt theme, serving as the Art Director, and working closely with the tech team on this year’s Hunt website.

This year’s Hunt was heist themed. The premise:

The Mystery Hunt has run into some financial trouble. The Manic Sages took a loan from Enigma Valley Investment and Loan, and through some dodgy paperwork, the bank ended up owning the Mystery Hunt outright. Enigma Valley decided that the coin was a valuable financial instrument, so through various Coin Default Swaps they leveraged a trillion dollars on the bet that nobody would find the coin. To be sure of it, they locked the coin in a high-security vault in an undisclosed location, with security systems designed by some of the world’s finest minds.

MIT legend Alyssa P. Hacker is not happy about this state of affairs. She wants to save the Mystery Hunt, by legal means or no. Thus, she hacks into Enigma Valley’s servers to try and find information on the bank’s security systems. She also enlists IHTFP, a heist consultancy firm, to help us organize and pull off a heist to steal the coin, bring down Enigma Valley, and save the Hunt.

The Hunt was structured with the opening round on enigmavalley.com, which identified the security systems used by the bank and the experts who designed them: Danny Ocean, Richard Feynman, Maxwell Smart (later replaced by Agent 99), Indiana Jones, Marty Bishop, and Erno Rubik.

Then there were six normal rounds on coinheist.com, in which teams recruit the six experts to their heist team by finding reasons for them to turn against Enigma Valley.

When a team solved the round and recruited a new member, they would get to work through a practice obstacle, constructed by IHTFP, associated with that person. The six security system obstacles in the bank were guards, a safe, a laser maze, a puzzle box, an impenetrable door (so teams had to use ventilation shafts to get around it), and a keypad.

After solving the final round, Alyssa P. Hacker would congratulate teams, but before allowing the heist to go forward, insist they come up with a reason why they are motivated to steal the coin from the bank, so she can trust them. At the same time, teams would get an phone call with an automated recording from Enigma Valley, telling them about “some puzzling activity” on their account. The phone call would give teams the link to a final puzzle, styled like a list of security questions–sixty security questions! Ultimately, the form would never allow you to unlock your account, but you could extract the answer: ENIGMA VALLEY GIVES YOU THE RUNAROUND.

The runaround then consists of solving each of the six real obstacles, each of which is slightly harder than they were in the practice versions, due to the bank beefing up security. A corporate newsletter smuggled out by a sympathetic member of the bank gives teams information to figure out what has changed about each obstacle and how to get around the changes. Finally, teams get into the vault–and are faced with an unexpected seventh obstacle, where the coin is hidden in a bunch of “counterfeit” coins, and teams have to use a scale to determine which is the real coin before the guards come back to the safe.

The Hunt We Had

Exhaustion–physical, mental, emotional.

Not just that this year set a record for the longest hunt (73 hours long), even with us handing out free answers like candy and cutting a metapuzzle towards the end. Heck, technically we even cut the runaround, since we declared a winner before the winning team completed it (less than an hour before the wrap-up ceremony). But for me (and most Sages), Hunt started long before Friday. Hunt had been distracting me from work for months. The week before the Hunt, all I did was sleep, eat, and work on the Mystery Hunt. I barely saw the sun.

Honestly, the predominant emotion I feel right now about this year’s hunt is despair. The Hunt starting an hour behind schedule, cutting so much of the hunt material in desperate bids to end the Hunt on time, having the Hunt go until Monday afternoon–I felt like we had failed. We abused teams’ time and sanity, letting the Hunt go into Monday. There was so much cool stuff that people worked their asses off for that no one–or only John Galt (the winning team) and Palindrome (the only other team to get to the runaround)–ever got to see: the security questions puzzle, the runaround traps, the coin design, even the finished hunt home page, all effectively for naught! On Monday I laid on the floor crying in the tech room for a while, not really able to productively handle it. I skipped all of the post-hunt events except wrap-up, mainly because I couldn’t deal with being around hunt people anymore. It was all just a reminder of how badly we–I–had fucked up.

Objectively speaking, I was the art director, and art and front-end development actually went basically fine, despite miscommunications and last minute feature requests. My puzzles worked well and I got some positive feedback on them. I was a new Sage, and didn’t have authority either in terms of seniority or in terms of being a puzzle expert, so it’s not like I had grounds to have opinions about the length and structure of the Hunt. But I didn’t–don’t–feel responsible for just my little circles of contribution. I was emotionally invested in the entire Hunt, and have had difficulty recalibrating that.

When my warnings from months ago to our tech lead that writing our own brand new hunt website software was a mistake he was too intelligent to make came to fruition, I wasn’t at all happy to be right. I stayed up until 4 AM Thursday night hoping that I could somehow help fix the backend tech, and feeling completely fucking helpless that I couldn’t usefully contribute. The two main website tech volunteers hacked something like 40 hours straight, and even with that herculean effort we still had to delay the start of Hunt an hour.

After the rush up to hunt, and the tech embarrassment, I had been hopeful that the rest of Hunt would be relatively relaxing. Maybe I’d be called upon to fix minor front-end tweaks, but fundamentally once the Hunt site design is done, it’s done, and it’s unlikely to explode in sudden unexpected ways. But then we noticed that the teams were progressing much more slowly than we (and our automated projections) ever expected. Round after round unlocked based on the time-based side of our puzzle release mechanism, but nobody was completing the rounds. We’d intended the opening round to be so easy that every active team would solve it, yet even by Saturday many teams hadn’t gotten it.

By the time we realized just how badly things were going, there was little we could do about it. From Saturday afternoon through Sunday, HQ was full of the thousand-yard stares of sleep-deprived, overworked people, all wondering what on earth we could possibly do to save the Hunt.

We couldn’t cut a full round for many reasons–one, the plot already had indicated you needed to recruit six people, and two, the last round’s leader (and the people whose puzzles were in that round) would kill us if we didn’t include it. We would’ve needed to cut it before the Hunt even started, if such a decision were politically feasible even then. If we’d had better projections of how long our hunt would take to solve, and had known late Friday/early Saturday that the damn thing would go into Tuesday without intervention, there was only one thing we could have done at that point: hand out a utterly absurd number of options (free answers). Explicitly restructure the Hunt so teams could choose their favorite 90 or so puzzles out of the hunt, solve them, and use options for the rest. As it happens, that’s what we ended up doing, but we didn’t realize we were doing it at the time, and we hesitated too long for it to make a radical difference.

(The other change we could have made at that point would be to make it obvious in the as-yet-unlocked rounds which answers contributed to which metapuzzles, which would help teams be more strategic. But that would have made much less significant a difference than the above.)

It seems fairly clear why we failed.

• First, last year’s hunt was around 100 puzzles long, and it ended early. We heard lots of whining about the early end. So we thought that this hunt needed more puzzles in order to last until Sunday noon. In the early Hunt structure conversations, some individuals vociferously argued that the Hunt needed to be around 180-200 puzzles long, or else it wouldn’t be long enough. Arguably, this shifted the Sages HQ Overton Window in terms of what we thought the Hunt required–people retorted that 135 was enough, not that we needed to *lower* the number back to 100.
• Second, the team was too self-deprecating. We thought that the average Manic Sage was a below-average Mystery Hunt participant, and that the only reason we won last year was because of sheer manpower. So if a typical testsolving group only barely managed to make it through a puzzle, after a few attempts, it went in, with the thought that it would be easier for a “real” team. If a group of ordinary humans couldn’t solve the puzzle, but one of our best solvers could, it went in, hamstrung also by the belief that since most of our best solvers were spoiled on many puzzles due to being writers and editors, the sampling wasn’t realistic. If a meta was solvable by our best puzzler with only half the answers, or no answers at all, we dialed up the difficulty. For the most part, we didn’t ship broken puzzles–certainly not more than other Hunts. Every puzzle we used was solved in testsolving. We just thought they were easy puzzles when they really, really weren’t. We now know that the average Manic Sage is a slightly-above-average puzzler, and our best puzzle-solvers are as good as anyone else’s. Had we believed this before, it would have improved our estimates quite a bit.
• Third, as stated earlier, the tech team succumbed to Not-Invented-Here syndrome. The new system was designed to handle lots of connections without a lot of load, and it did succeed at that. Where last year’s load average was around 900 at the start of Hunt, ours was 0.1. We could have run the Hunt on a Raspberry Pi if we’d wanted. That part went great. But the brilliant new architecture was arcane, so when bugs appeared hardly anyone could do anything about it.

  This was a far less impactful failure, in the end, though; it “only” cost us an hour in starting Hunt, various broken links that got fixed over time, some other useful but not crucial site features for showing teams their previous answer submissions and other status information, and the tech team’s health and sanity. On Friday I remember team members worrying that we looked unprofessional, that this hunt would be remembered as sloppy. If we’d only known how right we were…

It is difficult for me to focus on the positive. It feels like everybody hated this Hunt, and I seem to be thoroughly enveloped in post-Hunt depression. But there were some successes and beautiful, hilarious moments.

Codex’s mock, Occupy-style protest at the opening ceremony filled me with glee. The only thing Codex knew (like every other team) was that there was a bank involved with this year’s hunt theme. Yet their protest turned out to be super thematic! It was actually a little *too* realistic–I had to run interference between the security guard at the opening ceremony and Codex to assure the former that it was just a prank, not a real protest, and plead with the latter to avoid alarming any cops when they left! Even with that, afterward Lobby 7 had a heavy police presence.

Scribbling on the opening ceremony bank slides as Alyssa P. Hacker: lots of fun. :D

Half the hunt website frontend–including the entirety of the Rubik round–got designed and built during the week before hunt. And it looked fucking awesome, if I say so myself–Diana’s crazy-detailed casino art, Alice’s world map calligraphy, Rob’s revisions to the Sneakers round design, DD’s shoephones and the rest of the Get Smart design, and Sean’s manipulable, browser-compatible 3D Rubik’s cube… Plus the recruit cartoons on the hunt home page and the evil bank’s logo and brand identity. Even if we had to take away a few features due to the techpocalypse, I am seriously proud of this hunt’s art and front-end design and glad I had the chance to take the lead on it.

The bank security system obstacles were a ton of work to construct, but totally fucking cool. I only wish everyone had gotten to see both versions! I was especially giddy about the “twist” in the Feynman bank obstacle, which was a sentence Blake came up with months and months ago: “DIP IT IN ICE WATER”. So we constructed a safe puzzle where the demo safe’s combo was HOT LAVA, and the real, runaround safe’s combination was OBSIDIAN.

Also, we had not one, but TWO real, working laser mazes. The second maze’s solution? CLAP OFF. Yeah.

No apologies necessary for the obstacles portion of the Hunt; they were pure fucking awesome. Darby, Sparky, and all the Sages who helped tape cardboard: you guys rule.

The coin. Good God the coin was sexy this year, even if we forgot to show it to teams at the closing ceremony. DD drew the beaver and suitcase; the rest of the design was mine. Our production lead Sparky, and the company that 3D printed the coins, did a fantastic job–the company even added fur texture to the beaver!

One bright spot midway through the hunt was Heist Test Kitchen, a puzzle wherein teams had to cook a dish containing certain ingredients (the ingredients varying depending on their team composition) and bring it to HQ. Most teams’ dishes were quite delicious–even the one that used rice cakes and Doritos as ingredients, shockingly–and between the various items it meant that we didn’t have to cook much of a separate team dinner for Saturday night! A+ puzzle, would include again. :)

Events: The decorations for Casino Lobdell looked amazing, and the pictures for the Thomas Crowne event also came out fantastic. And, as in that one psychology test, at Thomas Crowne we had a guy in a gorilla suit that only half the participants there even noticed (their focus was on watching the bag handoffs)! Even the non-hunting journalist videotaping the event thought we were joking or lying about the gorilla–”Really! Play back your video!”

We got quite a few amusing submissions to the hunt’s answer queue:

• Here is a list of people you don’t want on a heist (aka wrong answers we got for the opening round):
  • MARTY MCFLY / SEAMUS MCFLY / CALVIN KLEIN (lot of guesses along these lines for Marty *Bishop*’s puzzle)
  • FANTASTIC MR. FOX
  • RUBE GOLDBERG
  • FELIX BAUMGARTNER
  • ZEFRAM COCHRANE
  • KING OF FRANCE
  • FRANKENSTEIN
  • GLORY HOLE (what?)
• For Sages Style, Providence Transplantations submitted HOMEOMORPHICALLY IRREDUCIBLE TREES (the real answer was GRAPHS)
• For Ex Post Facto, Luck submitted HALF SOLVED (which is an instruction you get partway through the puzzle). After we told them that was incorrect, they tried SOL. *facepalm*
• Multiple teams submitted WONDERPIG for a puzzle whose correct answer is WONDERING. I get you don’t have all the letters, but why would you guess that *first*?
• Strange New Universe tried BLESSED IS THE DETERMINANT for the Feynman metapuzzle. Would have been a great answer!
• For my email puzzle, Random Thymes tried QUIT AND RENAME THE DIR. Nope.
• For Tuva or Bust, Super Team Awesome tried BUST. They also tried RUSSIA, RUSSIAN FEDERATION, and about half the dictionary in a fit of frustration. The real (unfortunate) answer, that they had all along? RENAMES.
• Luck submitted “best liquor for puzzle solving”. They immediately followed up with “Sorry, thought this was Google”. Neither of these was an answer to any puzzles in this year’s Hunt.
• There was a puzzle called “Time Conundrum”, as part of which we had to call teams and tell them that their answer to the puzzle, RED HERRING, was incorrect two minutes *before* they unlocked the puzzle. (Most teams found this merely confusing; one actually tried to argue with the caller that they hadn’t called any such thing in!) After the puzzle was unlocked, a few teams called in RED HERRING. We responded, “Don’t call in repeat answers.” They were insistent, though: “We know it’s wrong; we just wanted to prevent paradoxes.”
• John Galt tried EWE for the puzzle I Can’t Spell You.
• John Galt also called in THEREARETWOPEOPLEIN26-100HAVINGSEXRIGHTNOW. This turned out to be correct, but not for the Highlight Reel puzzle. Folks, this is not what the answer queue is for…

There were also a variety of amusing HQ moments not related to answers…

• One of the oldest teams in the Hunt, Setec Astronomy, used to win rather frequently. As a result of being forced to write so many hunts, the team now deliberately does their best to not win, usually by applying liberal amounts of alcohol if signs indicate that they are near the top of the pack. Early in the hunt–I think late Friday–Setec Astronomy solved the opening round meta. This put them in the lead for a while, both by total puzzles and total rounds solved. Upon yelling “SETEC is in the lead!” in HQ, we resolved to never ever inform them of this. :)
• When Catherine overheard the tech team talking about how the automatic time unlock feature wasn’t working (but we could still manually trigger it), she exclaimed, “What do you mean we have to hit a button every two hours or the hunt explodes? That has to get fixed. This isn’t Lost!”
• HQ ran out of plastic forks on Sunday, which worried the kitchen crew:
  

  Veronica: This is the last fork. Soon we’ll run out of food and we’ll have to start eating each other.
  Tom: Worse, we’ll have to eat each other with our hands!

• At one point in the Hunt, Random Thymes had only gotten one of the three wheels to their Enigma machine for the Sneakers metapuzzle, but thought that they could crack it anyway. One of their members had access to a supercomputer in Ohio, so they plugged the puzzle into it. As they told us, “If the British can do it with a Turing machine, we can do it with 8000 cores.” Sadly (?) the team completed the other parts of the round and had all three wheels by the time they solved the puzzle, so we will never know if this meta could have been brute-forced!
• The second place team, Palindrome, was unhappy that their number of options was uneven (they had 32 options, which at that point in the Hunt would never amount to anything, since free answers cost multiples of 100). So, they contacted HQ and offered to do something entertainingly humiliating for 68 options. This resulted. We gave ‘em the points. :)
• Late at night on Sunday, a notice popped up in the contact HQ queue from the team Immoral, Illegal, and Fattening. Its text? “um, we set our metapuzzle on fire…”.

  While basically all of HQ was incapacitated with laughter for several minutes, the team also sent an unrelated request along with “(also we are sorry about the flaming metapuzzle still)”. We finally managed to call them back, and they answered the phone: “Hello, this is Immoral, Illegal, and Flammable!”

  As the team explained, they thought the piece of paper the Get Smart metapuzzle was on was treated with the kind of invisible ink that appears if you apply heat to the paper (rather than the kind that appears if you apply vinegar, which was in fact the case). To test this hypothesis, instead of using an iron set to low, like a sane person, or a candle flame, like a less sane but more understandable person, they decided to put the paper inside a toaster.

  This was pretty much the “As God is my witness, I thought turkeys could fly” moment of the Hunt.

  Fortunately, we had plenty of duplicates — we had made 100 copies of the metapuzzle, but ended up using only four of them, including the burnt one.

A number of hunt participants tweeted great things using the hashtag #mysteryhunt. I especially enjoyed the retweets from the (fictional) @manicsages account. Most of them are compiled in the Tech’s Storify. I’m glad I didn’t see these during Hunt, though, as a couple of them are rather disheartening, but I can appreciate them now. I liked Eric Berlin’s hunt wrap-up post, as well. There are other blog posts reviewing our hunt in much more disgruntled terms… but, right, this is the positive section.

Perhaps most of all…

I don’t know yet if I’ll be back for Hunt next year. I was never a very good puzzler, and maybe we’re now the Depressed Sages? Got a whole year to recover, I suppose.

I am eagerly looking forward to once again having a life outside of the Hunt, in any case.

I just posted the article about Ushahidi and its future challenges that was published in the Index on Censorship last month (‘Crowd Wisdom’ by Heather Ford in Index on Censorship December 2012, vol. 41, no. 4 33-39 doi: 10.1177/0306422012465800) . I wrote about Ushahidi’s emergence as a powerful tool used in countries around the world to document elections, disasters and food – among others – and the coming challenges as the majority of Ushahidi implementations remain ‘small data’ projects and as tools move towards automatic verification, something only possible with ‘Big Data’.

First posted at EthnographyMatters.net

Last month’s Wired magazine showed an infographic with a headline that read: ‘History’s most influential people, ranked by Wikipedia reach’ with a group of 20 men arranged in hierarchical order — from Jesus at number 1 to Stalin at number 20. Curious, I wondered how ‘influence’ and ‘Wikipedia reach’ was being decided. According to the article, ‘Rankings (were) based on parameters such as the number of language editions in which that person has a page, and the number of people known to speak those languages’. What really surprised me was not the particular arrangement of figures on this page but the conclusions that were being drawn from it.

According to the piece, César Hidalgo, head of the Media Lab’s Macro Connections group, who researched the data, made the following claims about the data gathered from Wikipedia:

a) “It shows you how the world perceives your own national culture.“

b) “It’s a socio-cultural mirror.“

c) “We use historical characters as proxies for culture.“

And finally, perhaps most surprising is this final line in the story:

Using this quantitative approach, Hidalgo is now testing hypotheses such as whether cultural development is structured or random. “Can you have a Steve Jobs in a country that has not generated enough science or technology?” he wonders. “Ultimately we want to know how culture assembles itself.”

It is difficult to comment on the particular method used by this study because there is little more than the diagram and a few paragraphs of analysis, and the journalist may have misquoted him, but I wanted to draw attention to the statements being made because I think it represents the growing phenomenon of big data analysts using Wikipedia data to make assumptions about ‘culture’.

National culture? 

Hidalgo claims that Wikipedia can show you ‘how the world perceives your own national culture’. But Wikipedia is actually pretty bad at showing national differences because of two key reasons: a) Wikipedia divides projects by language, and languages are necessarily cross national (Portuguese, for example, is spoken by people from two different continents; Arabic Wikipedia has contributions by people on three continents), and b) Different Wikipedia language editions are not necessarily being edited by people from the place traditionally associated with that language.

In 2011/12, I did a study of Swahili Wikipedia and learned, surprisingly, that the majority of edits were being made, not by East Africans where Swahili is a lingua franca but by Europeans, with only one of the three Swahili Wikipedia bureaucrats (Muddyb from Tanzania) a native Swahili speaker. Long-time Swahili Wikipedia editor and bureaucrat from Tanzania, Muddyb, complained about the lack of local contributions to  the encyclopedia on his blog:

The most active users who are contribute on the Swahili Wikipedia is Wazungu (the White people) who coming from various areas (but mostly from Germany). Kipala is the only user who change the Swahili Wikipedia from the lower level to the highest level.  Kipala was the one who took it from the 140 to the 1,000 mark – and make it the first African Wikipedia to reach over the 1,000.

The notion of home/first languages

Similarly, there is a common misconception that editors will edit the encyclopedia in their home/first language rather than a “foreign” language. The Makmende case study showed how Kenyans (who generally speak a native, regional language like Kikuyu or Maasai as well as the “official languages” of Swahili and English) preferred to write the article in English than in Swahili (where the article still does not exist). In interviews, I learned that the majority of Kenyan Wikipedia Chapter members choose to edit in English and that more Kenyans actually read English Wikipedia than Swahili Wikipedia, despite the fact that Swahili Wikipedia is one of the largest African Wikipedias with almost 23,000 articles (Zachte, 2011) (1). In a country where the national language is Kiswahili and the official languages are Kiswahili and English, Swahili Wikipedia gets less than 1% of Wikimedia readership, while English Wikipedia receives 89.6% of all Wikimedia requests. The largest readership of Swahili Wikipedia comes from Tanzania at only 20%, followed closely by the United States at 16.5%, Kenya at 6.6% and Germany at 5.6%. Swahili Wikipedia is actually being read either by East African expats living abroad and/or by Westerners with ties to the region.

Readership and editing of Wikipedia language versions in Kenya and Tanzania as at 31 October 2011 (Source: http://stats.wikimedia.org/)

There are many reasons why this is the case – the most important is the fact that language in Kenya (as in many places on the continent) is intimately connected with an individual’s access to education, employment, and political participation (Simpson, 2008). Even though Swahili was chosen as the national language in Kenya after independence, the colonial experience had a large impact on the way that languages were perceived, since being fluent in a colonial language ultimately meant access to the best positions during colonial times (Simpson, 2008). The legacy of this (and as Simpson argues, a characteristic of African traditional society) is that different languages are used for different situations in modern day Kenya: English for government, business and science and technology discussions, Swahili (or a version, like Sheng) for everyday conversation.

So, even though many East Africans (like Muddyb) complain about the lack of local support for Swahili, there are very important reasons why people choose to edit and read the English version instead. But this also means that looking at Wikipedia data about Swahili cannot ‘show you how the world perceives your own national culture’ because Wikipedia editors of particular language versions are often not even native speakers of the language which is supposedly reflecting this culture.

The role of individuals 

The second problem with using ‘Wikipedia as a proxy for culture’ is that some language versions are very small, and that individuals can significantly skew the data. Take this example below of Mark Graham’s visualization of Swahili Wikipedia.

Wikipedia editors are made up of a particular slice of the population and this tends to reflect the kinds of articles that are representation (and not represented) on Wikipedia. Metafilter has a great discussion about the infographic and I really loved what user ‘eotvos’ had to say about why someone like Vasco da Gamma or Muhammad doesn’t show up in the graphic:

Once again, the subject as has the potential to be a lot more interesting than either the analysis or the article. While this doesn’t say much at all about “influence” or national reach or whatever else the authors suggest, comparisons of wikipedia databases in multiple language could be fascinating.

Consider Copronymus’ lovely example of Vasco da Gamma and Muhammad above. I suspect most of us have a reasonably good idea of what it means to have a wikipedia page about a person in English, or in Spanish. But, how ought we to think about wikipedia articles in languages with less reach?

To pick an example, what does it mean to have a wikipedia page in Nahuatl? The number of literate Nahuatl speakers with Internet access who don’t also speak one of the big 10 languages may not be zero, but it’s vanishingly close to zero. People who create wikipedia pages in Nahuatl are clearly doing something other than informing the monolingual Nahuatl-speaking public about Vasco da Gama. What are they up to, and what can we learn about them from their editorial choices?

Instead of measuring Vasco da Gama’s influence among Nahuatl speakers, these data probably say something about the canon of world knowledge as it is understood by a very small group of tech-savvy language activists. A detailed analysis of that could be really exciting.

November 27, 2012

Indeed. Wikipedia data about different language encyclopedias has interesting things to show us about the culture of Wikipedians themselves. The ways in which that data points to yearning for friendships across the seas, to access to what some see as greener pastures, to the things that they want people who are watching or reading to see and the things that they see should be part of the global English corpus. But Wikipedia is no proxy for national culture.

In the next post, I’ll talk about the particular mechanics of editing on Wikipedia that shape what ends up being represented.

Footnotes:

(1) Afrikaans and Swahili Wikipedias tend to periodically switch with one another as being the largest African Wikipedia. See Ian Gilfillan for more on this.

(2) Muddyb explains his editing of articles on Tanzania and Norway as an expression of friendship towards Wikipedians from those countries. See http://muddybtz.blog.com/2011/08/14/this-time-for-norway/

Featured image: Planisphæri cœleste Celestial map from the 17th century, by the Dutch cartographer Frederik de Wit. Public domain from Wikipedia.

The last 24 hours have been an emotional roller coaster. I woke up yesterday to find that a friend of mine – Aaron Swartz – had taken his life. My Twitter feed went into mourning – shock, sadness, anger, revenge. I spent the day talking with friends who were all in various states of disarray. I watched as many of them poured out their hearts on their blogs, a practice we’ve all been doing for over a decade. And yet, I couldn’t find the words to express what I’ve been feeling. When I tweeted yesterday about being angry, well-meaning friends and mental health experts who didn’t know Aaron wrote to me about how I couldn’t be responsible for someone’s depression. This made me want to scream. I decided to write this blog post instead. It is raw and imperfect, but that’s where I’m at right now.

For better or worse, I’ve known a lot of people over the years who have committed suicide. I’ve watched people struggle through serious depression and then make that choice. Having battled my own demons, I understood. Part of why Aaron’s death hit me like a rock is because this time it was different.

There’s no doubt in my mind that depression was a factor. I adored Aaron because he was an emotional whirlwind – a cranky bastard and a manic savant. Our conversations had an ethereal sense to them and he pushed me hard to think through complex issues as we debated. He had an intellectual range that awed me and a kitten’s sense of curiosity. But when he was feeling destructive, he used his astute understandings of people to find their weak spots and poke them where it hurt. Especially the people he loved the most. He saw himself as an amateur sociologist because he was enamored with how people worked and we argued over the need for rigor, the need for formal training. He had no patience for people who were intellectually slower than him and he failed to appreciate what could be gained by a university setting. Instead, he wanted to mainline books and live in the world of the mind.

I’ve known Aaron for nine years and I both adored him to pieces and found him frustrating as hell. In recent years, our connection grew more sporadic because I loved the ups but really struggled with the downs. But when the arrest happened, I grew very worried about him. We decided never to talk about the case itself, but amidst brainjams, we’d joke about him finally getting his degree in jail as a way to relieve the pressure. I promised to curate an educational plan built off of great pieces of scholarship and told him I’d send him a printout from JSTOR each day. I knew he was struggling, but he was also a passionate activist and I genuinely thought that would see him through this dark period.

What made me so overwhelmingly angry yesterday was the same thing that has been boiling in my gut for the last two years. When the federal government went after him – and MIT sheepishly played along – they weren’t treating him as a person who may or may not have done something stupid. He was an example. And the reason they threw the book at him wasn’t to teach him a lesson, but to make a point to the entire Cambridge hacker community that they were p0wned. It was a threat that had nothing to do with justice and everything to do with a broader battle over systemic power. In recent years, hackers have challenged the status quo and called into question the legitimacy of countless political actions. Their means may have been questionable, but their intentions have been valiant. The whole point of a functioning democracy is to always question the uses and abuses of power in order to prevent tyranny from emerging. Over the last few years, we’ve seen hackers demonized as anti-democratic even though so many of them see themselves as contemporary freedom fighters. And those in power used Aaron, reframing his information liberation project as a story of vicious hackers whose terroristic acts are meant to destroy democracy.

Reasonable people can disagree about tactics and where and when a particular approach pushes too far. Like Lessig, I often disagreed with Aaron about his particular approach to freeing the world’s information, even if I never disagreed with him about the goal. And one of the reasons why so many hackers and geeks spent yesterday raging against the machine is because so many people in power have been unable to see past the particular acts and understand the intentions and activism. So much public effort has been put into controlling and harmonizing geek resistance, squashing the rebellion, and punishing whoever authorities can get their hands on. But most geeks operate in gray zones, making it hard for them to be pinned down and charged. It’s in this context that Aaron’s stunt gave federal agents enough evidence to bring him to trial to use him as an example. They used their power to silence him and publicly condemn him even before the trial even began.

Yesterday, there was an outpouring of information about his case, including an amazing account from the defense’s expert witness. Many people asked why people didn’t speak up before. I can only explain my reasoning. I was too scared to speak publicly for fear of how my words might be used against him. And I was too scared to get embroiled in the witch hunt that I’ve watched happen over the last three years. Because it hasn’t been about justice or national security. It’s been about power. And it’s at the heart and soul of why the Obama administration has been a soul crushing disappointment to me. I’ve gotten into a ridiculous number of fights over the last couple of years with folks in the administration over the treatment of geeks and the misunderstanding of hackers, but I could never figure how to make a difference on that front. This was a source of serious frustration for me, even as SOPA/PIPA showed that geeks could make a difference.

So here we are today, the world lacking a prodigious child whose intellect scared the shit out of everyone who knew him. He became a toy for a government set on showing their strength. And they bullied him and preyed on his weaknesses and sought to break him. And they did. All for the performance of justice. All before he was even tried in a society that prides itself on innocent until proven guilty. Was depression key to what happened on Friday? Certainly. But it wasn’t the whole story. And that’s what makes it hard for me to stomach.

There is a lot of justifiable outrage out there. Many people want the heads of the key administrators who helped create the context in which Aaron took his life. I completely understand where they’re coming from. But I also fear the likelihood that Aaron will be turned into a martyr, an abstraction of a geek activist destroyed by the State. Because he was a lot more than that – lovable and flawed, passionate and strong-willed, brilliant and infuriatingly stupid. It’ll be easy for folks to rally cry for revenge in his name. But not much is gained from reifying the us vs. them game that got us here. There has to be another way.

What I really hope comes out of this horrible tragedy is some serious community reflection and a deep values check. Many of the beliefs that Aaron stood for – the liberation of knowledge, open access to information, and the use of code to make the world better – are core values in the geek community. Yet, as Biella Coleman astutely dissects in “Coding Freedom”, this community is not without its flaws. Nor was Aaron. He did things his way because he believed that passion and will and action trumped all. And his stubbornness made him breakable. If we want to achieve the values and goals that are core to the geek community, I don’t think that we’ll ever make a difference by creating more martyrs that can be used as examples in a cultural war. As we collectively mourn Aaron’s death and channel our anger into making a difference, I think we need to look for an approach to change-making that doesn’t result in brilliant people being held up as examples so that they can be tormented by power.

I had an interesting conversation the other day with a health care professional. He was lamenting the relationship between doctors and pharmaceutical companies.

Pharmaceutical companies, he reported, put a lot of pressure on doctors to prescribe and sell drugs, giving them bonuses if they sell certain quotas. This provides an incentive for doctors to prescribe drugs that don’t cure patients. When you sell medicine and not health, why heal a patient?

I’ve long been skeptical about pharmaceutical companies for another reason: as an industry, they seem to be primarily responsible for use and abuse of the patent system. Big Pharma lobbies congress to keep patent laws strong, but then also games the patent system. For example, it’s common practice for pharmaceutical companies to make trivial changes to a drug formula in order to extend an existing patent past its normal legal term (14 years). The result is a de facto Sonny Bono law for patents.

The justification for strong patents is, of course, the problem of recouping fixed costs from research investment. So goes the argument: Pharmaceutical research is expensive, but pharmaceutical production is cheap. If firms can freely compete in the drug market, new firms will enter after a research phase and prices will drop so the original researching company makes no profit. Without that profit, they will never do the research in the first place.

This is a fair argument as far as it goes.

However, this economic argument provides a cover story that ignores other major parts of the pharmaceutical industry. Let’s talk about advertising. When Big Pharma puts out a $14 million dollar Super Bowl commercial, is that dipping into the research budget? Or is that part of a larger operating cost endured by these companies–the costs of making their brand a household name, of paying doctors to make subscriptions, and of lobbying for a congenial political climate?

A problem is that when pharmaceutical companies are not just researching and selling drugs but participating as juggernauts in the information economy, it’s very hard to tell how much of their revenue is necessary for innovation and how much funds bullying for unfair market advantage that hurts patients.

There are some possible solutions to this.

We can hope for real innovation in alternative business models for pharmaceutical research. Maybe we can advocate for more public funding through the university system. But that advocacy requires political will, which is difficult to foster without paid lobbyists or grassroots enthusiasm. Grassroots enthusiasm depends on the participating of the media.

Which gets us to the crux. Because if big media companies are cashing out from pharmaceutical advertising, what incentive do they have to disrupt the economic and political might of Big Pharma? It’s like the problem of relying on mass media to support campaign finance reform. Why would they shatter their own gravy train?

Lately, I’ve been seeing political problems more and more as aligned with centralization of the media. (Not a new observation by any means, but here I am late to the party). There are some major bottlenecks to worldwide information flow, and economic forces battle for these like mountain passes on ancient trade routes. Thankfully, this is also an area where there is a terrific amount of innovation and opportunity.

Here’s an interesting research question: how does one design a news dissemination network with mass appeal that both provides attractive content while minimizing potential for abuse by economic interests that are adversarial to the network users?

Long-time friends will probably realize that with the coming of the new year comes a revisit to my annual donations.

This year's donations are larger than any previous year, but largely fall along similar trends as in the past. The larger donations this year (about $1,000-worth) go towards non-profit organizations. The choices this year were hard. After consulting with a few friends, I decided to donate to two new categories: Environmental and Anti-Gun.

Finding a good environmental organization to give your money is HARD. After a few hours of research, I had looked at many organizations that were doing good work. But a lot of those organizations were still trying to prove the point that climate change is an issue, or were focused on small-scale issues. These are both noble goals, but I think what we need now are big solutions on an international level. I'm no expert in this topic, by far, but I'm fairly convinced that individual decision making isn't going to solve the problem fast enough. It's great if we all learn to recycle and to consider environmental impact in our daily lives. That, I don't disagree with. But I don't think it's enough. I think we need to start forcing governments and organizations to be cleaner. I'm convinced that so long as the economic incentives are in place that have led to the current behaviors, the market will follow those incentives. I'm hopeful that my donation to the Center for Climate and Energy Solutions will help bring changes to these incentives.

Finding an anti-gun organization is easier, especially given the current state of affairs after Sandyhook Elementary School. While I'm not so sure that anti-gun legislation is going to solve any truly big problems, I hope that donating my money here will help strike while the iron is hot. I simply can't believe that the 2nd Amendment pro-gun lobby is as successful as it is, and I am hopeful that we'll be able to change the dialog around guns over the next few years. Gun ownership is trending down in the U.S., and I hope that we can accellerate that trend, bringing a cease to the needless gun deaths violence we currently live with.

The other big donations in this year's list go mostly towards organizations that I've donated to in the past. Fair Vote and Rootstrikers are organizations that work to fix the current political system. Most Americans (about 70%, I believe) agree that the current Federal legislation system is corrupt, and these organizations are working to fix that. I'm pessimistic that until these organizations find success, we won't be able to deal with the small or large issues facing the country, so these organizations continue to get the plurality of my donation ($400 between them). I think the ridiculous fiscal cliff "negotiations" are testament to how bad things have gotten. Our political system is paralyzed.

Other organizations that did well this year include a handful of open-source foundations that I rely on, but which otherwise give away their work for free. My livelihood and these very donations rely on these bits of infrastructure we take for granted, so I figure I should give them some money to keep 'em going.

Here's the nitty gritty breakdown of my donations this year (as well as last):

As always, I welcome input on these decisions, and suggestions for the years ahead. Those that made suggestions for this year, I truly appreciate your help.