# School of Information Blogs

## November 21, 2019

Ph.D. 2018

#### Big tech surveillance and human rights

I’ve been alarmed by two articles to cross my radar today.

• Bloomberg Law has given a roundup on the contributions Google and Facebook have given to tech policy advocacy groups. Long story short: they give a lot of money, and while these groups say they are not influenced by the donations, they tend to favor privacy policies that do not interfere with the business models of these Big Tech companies.
• Amnesty International has put out a report arguing that the business models of Google and Facebook are “an unprecedented danger to human rights”.

Surveillance Giants lays out how the surveillance-based business model of Facebook and Google is inherently incompatible with the right to privacy and poses a systemic threat to a range of other rights including freedom of opinion and expression, freedom of thought, and the right to equality and non-discrimination.

Amnesty International

Until today, I never had a reason to question the judgment of Amnesty International. I have taken seriously their perspective as an independent watchdog group looking out for human rights. Could it be that Google and Facebook have, all this time, been violating human rights left and right? Have I been a victim of human rights abuses from the social media sites I’ve used since college?

This is a troubling thought, especially as an academic researcher who has invested a great deal of time studying technology policy. While in graduate school, the most lauded technology policy think tanks, those that were considered most prestigious and genuine, such as the Center for Democracy and Technology (CDT), are precisely those listed by the Bloomberg Law article as having been in essence supporting the business models of Google and Facebook all along. Now I’m in moral doubt. Amnesty International has condemned Google of human rights violations for the sake of profit, with CDT (for example) as an ideological mouthpiece.

If all data controllers were information fiduciaries, that would almost certainly settle the human rights issues raised by Amnesty International. But how likely is this strong language to survive the legislative process in New York?

There are two questions on my mind after considering all this. The first is what the limits of Silicon Valley self-regulation are. I’m reminded of an article by Mulligan and Griffin about Google’s search engine results. For a time, when a user queried “Did the holocaust happen?” the first search results would deny the holocaust. This prompted the Mulligan and Griffin article about what principles could be used to guide search engine behavior besides the ones used to design the search engine initially. Their conclusion is that human rights, as recognized and international experts, could provide those principles.

The essay concludes by offering a way forward grounded in developments in business and human rights. The emerging soft law requirement that businesses respect and remedy human rights violations entangled in their business operations provides a normative basis for rescripting search. The final section of this essay argues that the “right to truth,” increasingly recognized in human rights law as both an individual and collective right in the wake of human rights atrocities, is directly affected by Google and other search engine providers’ search script. Returning accurate information about human rights atrocities— specifically, historical facts established by a court of law or a truth commission established to document and remedy serious and systematic human rights violations—in response to queries about those human rights atrocities would make good on search engine providers’ obligations to respect human rights but keep adjudications of truth with politically legitimate expert decision makers. At the same time, the right to freedom of expression and access to information provides a basis for rejecting many other demands to deviate from the script of search. Thus, the business and human rights framework provides a moral and legal basis for rescripting search and for cabining that rescription.

Mulligan and Griffin, 2018

Google now returns different results when asked “Did the holocaust happen?”. The first hit is the Wikipedia page for “Holocaust denial”, which states clearly that the views of Holocaust deniers are false. The moral case on this issue has been won.

Is it realistic to think that the moral case will be won when the moral case directly contradicts the core business model of these companies? That is perhaps akin to believing that medical insurance companies in the U.S. will cave to moral pressure and change the health care system in recognition of the human right to health.

These are the extreme views available at the moment:

• Privacy is a human right, and our rights are being trod on by Google and Facebook. The ideology that has enabled this has been propagated by non-profit advocacy groups and educational institutions funded by those companies. The human rights of consumers suffer under unchecked corporate control.
• Privacy, as imagined by Amnesty International, is not a human right. They have overstated their moral case. Google and Facebook are intelligent consumer services that operate unproblematically in a broad commercial marketplace for web services. There’s nothing to see here, or worry about.

I’m inclined towards the latter view, if only because the “business model as a human rights violation” angle seems to ignore how services like Google and Facebook add value for users. They do this by lowering search costs, which requires personalized search and data collection. There seem to be some necessary trade-offs between lowering search costs broadly–especially search costs when what’s being searched for is people–and autonomy. But unless these complex trade-offs are untangled, the normative case will be unclear and business will proceed simply as usual.

References

Mulligan, D. K., & Griffin, D. (2018). Rescripting Search to Respect the Right to Truth.

#### Autonomy as link between privacy and cybersecurity

A key aspect of the European approach to privacy and data protection regulation is that it’s rooted in the idea of an individual’s autonomy. Unlike an American view of privacy which suggests that privacy is important only because it implies some kind of substantive harm—such as reputational loss or discrimination–in European law it’s understood that personal data matters because of its relevance to a person’s self-control.

Autonomy etymologically is “self-law”. It is traditionally associated with the concept of rationality and the ability to commit oneself to duty. My colleague Jake Goldenfein argues that autonomy is the principle that one has the power to express one’s own narrative about oneself, and for that narrative to have power. Uninterpretable and unaccountable surveillance, “nudging”, manipulation, profiling, social sorting, and so on are all in a sense an attack on autonomy. They interfere with the individual’s capacity to self-rule.

It is more rare to connect the idea of autonomy to cybersecurity, though here the etymology of the words also weighs in favor of it. Cyber- has its root in in Greek kybernetes, for steersman, governor, pilot, or rudder. To be secure means to be free from threat. So cybersecurity for a person or organization is the freedom of their (self-control) from external threat. Cybersecurity is the condition of being free to control oneself–to be autonomous.

Understood in this way, privacy is just one kind of cybersecurity: the cybersecurity of the individual person. We can speak additionally of the cybersecurity of a infrastructure, such as a power grid, or of an organization, such as a bank, or of a device, such as a smartphone. What both the privacy and cybersecurity discussions implicate are questions of the ontology of the entities involved and their ability to control themselves and control each other.

## November 18, 2019

Ph.D. 2018

#### Open Source Software (OSS) and regulation by algorithms

It has long been argued that technology, especially built infrastructure, has political implications (Winner, 1980). With the rise of the Internet as the dominating form of technological infrastructure, Lessig (1999), among others, argued that software code is a regulating force parallel to the law. By extension of this argument, we would expect open source software to be a regulating force in society.

This is not the case. There is a lot of open source software and much of it is very important. But there’s no evidence to say that open source software, in and of itself, regulates society except in the narrow sense in that the communities that build and maintain it are necessarily constrained by its technical properties.

On the other hand, there are countless platforms and institutions that deploy open source software as part of their activity, which does have a regulating force on society. The Big Tech companies that are so powerful that they seem to rival some states are largely built on an “open core” of software. Likewise for smaller organizations. OSS is simply part of the the contemporary software production process, and it is ubiquitous.

Most widely used programming languages are open source. Perhaps a good analogy for OSS is that it is a collection of languages, and literatures in those languages. These languages and much of their literature are effectively in the public domain. We might say the same thing about English or Chinese or German or Hindi.

Law, as we know it in the modern state, is a particular expression of language with purposeful meaning. It represents, at its best, a kind of institutional agreement that constraints behavior based on its repetition and appeals to its internal logic. The Rule of Law, as we know it, depends on the social reproduction of this linguistic community. Law Schools are the main means of socializing new lawyers, who are then credentialed to participate in and maintain the system which regulates society. Lawyers are typically good with words, and their practice is in a sense constrained by their language, but only in the widest of Sapir-Whorf senses. Law is constrained more the question of which language is institutionally recognized; indeed, those words and phrases that have been institutionally ratified are the law.

Let’s consider again the generative question of whether law could be written in software code. I will leave aside for a moment whether or not this would be desirable. I will entertain the idea in part because I believe that it is inevitable, because of how the algorithm is the form of modern rationality (Totaro and Ninno, 2014) and the evolutionary power of rationality.

A law written in software would need to be written in a programming language and this would all but entail that it is written on an “open core” of software. Concretely: one might write laws in Python.

The specific code in the software law might or might not be open. There might one day be a secretive authoritarian state with software laws that are not transparent or widely known. Nothing rules that out.

We could imagine a more democratic outcome as well. It would be natural, in a more liberal kind of state, for the software laws to be open on principle. The definitions here become a bit tricky: the designation of “open source software” is one within the schema of copyright and licensing. Could copyright laws and license be written in software? In other words, could the ‘openness’ of the software laws be guaranteed by their own form? This is an interesting puzzle for computer scientists and logicians.

For the sake of argument, suppose that something like this is accomplished. Perhaps it is accomplished merely by tradition: the institution that ratifies software laws publishes these on purpose, in order to facilitate healthy democratic debate about the law.

Even with all this in place, we still don’t have regulation. We have now discussed software legislation, but not software execution and enforcement. If software is only as powerful as the expression of a language. A deployed system, running that software, is an active force in the world. Such a system implicates a great many things beyond the software itself. It requires computers and and networking infrastructure. It requires databases full of data specific to the applications for which its ran.

The software dictates the internal logic by which a system operates. But that logic is only meaningful when coupled with an external societal situation. The membrane between the technical system and the society in which it participates is of fundamental importance to understanding the possibility of technical regulation, just as the membrane between the Rule of Law and society–which we might say includes elections and the courts in the U.S.–is of fundamental importance to understanding the possibility of linguistic regulation.

References

Lessig, L. (1999). Code is law. The Industry Standard18.

Hildebrandt, M. (2015). Smart technologies and the end (s) of law: Novel entanglements of law and technology. Edward Elgar Publishing.

Totaro, P., & Ninno, D. (2014). The concept of algorithm as an interpretative key of modern rationality. Theory, Culture & Society31(4), 29-49.

Winner, L. (1980). Do artifacts have politics?. Daedalus, 121-136.

## November 15, 2019

Center for Technology, Society & Policy

#### Information Session for 2020 Fellow Applications

Interested in applying to be a 2020 CTSP fellow? Come to an information session on Wednesday, Nov. 20th at 4:30 PM in South Hall Room 6 (in the basement).

A reminder about eligibility and the funding model for 2020 (for more details see the Call for Applications):

### 2020 Fellowship Model

•   Individual and Paired teams will typically receive \$2,000/each.
•   Teams with more than 2 members will receive a maximum of \$5,000 total, with funding levels determined by project scope. You may optionally submit a budget detailing how your team would allocate up to \$5,000.

With rare exceptions, projects must include a UC Berkeley student to be eligible. Projects that do not include a UC Berkeley student may be considered but may shift funding allocation.

## November 14, 2019

Ph.D. 2018

#### The diverging philosophical roots of U.S. and E.U. privacy regimes

For those in the privacy scholarship community, there is an awkward truth that European data protection law is going to a different direction from U.S. Federal privacy law. A thorough realpolitical analysis of how the current U.S. regime regarding personal data has been constructed over time to advantage large technology companies can be found in Cohen’s Between Truth and Power (2019). There is, to be sure, a corresponding story to be told about EU data protection law.

Adjacent, somehow, to the operations of political power are the normative arguments leveraged both in the U.S. and in Europe for their respective regimes. Legal scholarship, however remote from actual policy change, remains as a form of moral inquiry. It is possible, still, that through professional training of lawyers and policy-makers, some form of ethical imperative can take root. Democratic interventions into the operations of power, while unlikely, are still in principle possible: but only if education stays true to principle and does not succumb to mere ideology.

This is not easy for educational institutions to accomplish. Higher education certainly is vulnerable to politics. A stark example of this was the purging of Marxist intellectuals from American academic institutions under McCarthyism. Intellectual diversity in the United States has suffered ever since. However, this was only possible because Marxism as a philosophical movement is extraneous to the legal structure of the United States. It was never embedded at a legal level in U.S. institutions.

There is a simply historical reason for this. The U.S. legal system was founded under a different set of philosophical principles; that philosophical lineage still impacts us today. The Founding Fathers were primarily influenced by John Locke. Locke rose to prominence in Britain when the Whigs, a new bourgeois class of Parliamentarian merchant leaders, rose to power, contesting the earlier monarchy. Locke’s political contributions were a treatise pointing out the absurdity of the Divine Right of Kings, the prevailing political ideology of the time, and a second treatise arguing for a natural right to property based on the appropriation of nature. This latter political philosophy was very well aligned with Britain’s new national project of colonialist expansion. With the founding of the United States, it was enshrined into the Constitution. The liberal system of rights that we enjoy in the U.S. are founded in the Lockean tradition.

Intellectual progress in Europe did not halt with Locke. Locke’s ideas were taken up by David Hume, whose introduced arguments that were so agitating that they famously woke Immanuel Kant, in Germany, from his “dogmatic slumber”, leading him to develop a new highly systematic system of morality and epistemology. Among the innovations in this work was the idea that human freedom is grounded in the dignity of being an autonomous person. The source of dignity is not based in a natural process such as the tilling of land. It is rather based in on transcendental facts about what it means to be human. The key to morality is treating people like ends, not means; in other words, not using people as tools to other aims, but as aims in themselves.

If this sound overly lofty to an American audience, it’s because this philosophical tradition has never taken hold in American education. In both the United Kingdom and Britain, Kantian philosophy has always been outside the mainstream. The tradition of Locke, through Hume, has continued on in what philosophers will call “analytic philosophy”. This philosophy has taken on the empiricist view that the only source of knowledge is individual experience. It has transformed over centuries but continues to orbit around the individual and their rights, grounded in pragmatic considerations, and learning normative rules using the case-by-case approach of Common Law.

From Kant, a different “continental philosophy” tradition produced Hegel, who produced Marx. We can trace from Kant’s original arguments about how morality is based on the transcendental dignity of the individual to the moralistic critique that Marx made against capitalism. Capitalism, Marx argued, impugns the dignity of labor because it treats it like a means, not an end. No such argument could take root in a Lockean system, because Lockean ethics has no such prescription against treating others instrumentally.

Germany lost its way at the start of the 20th century. But the post-war regime, funded by the Marshall plan, directed by U.S. constitutional scholars as well as repatriating German intellectuals, had the opportunity to rewrite their system of governance. They did so along Kantian lines: with statutory law, reflecting a priori rational inquiry, instead of empiricist Common Law. They were able to enshrine into their system the Kantian basis of ethics, with its focus on autonomy.

Many of the intellectuals influencing the creation of the new German state were “Marxist” in the loose sense that they were educated in the German continental intellectual tradition which, at that time, included Marx as one of its key figures. By the mid-20th century they had naturally surpassed this ideological view. However, as a consequence, the McCarthyist attack on Marxism had the effect of also purging some of the philosophical connection between German and U.S. legal education. Kantian notions of autonomy are still quite foreign to American jurisprudence. Legal arguments in the United States draw instead on a vast collection of other tools based on a much older and more piecemeal way of establishing rights. But are any of these tools up to the task of protecting human dignity?

The EU is very much influenced by Germany and the German legal system. The EU has the Kantian autonomy ethic at the heart of its conception of human rights. This philosophical commitment has recently expressed itself in the EU’s assertion of data protection law through the GDPR, whose transnational enforcement clauses have brought this centuries-old philosophical fight into contemporary legal debate in legal jurisdictions that predate the neo-Kantian legal innovations of Continental states.

The puzzle facing American legal scholars is this: while industrial advocates and representatives tend to disagree with the strength of the GDPR, arguing that it is unworkable and/or based on poorly defined principle, the data protections that it offer seem so far to be compelling to users, and the shifting expectations around privacy in part induced by it are having effects on democratic outcomes (such as the CCPA). American legal scholars now have to try to make sense of the GDPR’s rules and find a normative basis for them. How can these expansive ideas of data protection, which some have had the audacity to argue is a new right (Hildebrandt, 2015), be grafted onto the the Common Law, empiricist legal system in a way that gives it the legitimacy of being an authentically American project? Is there a way to explain data protection law that does not require the transcendental philosophical apparatus which, if adopted, would force the American mind to reconsider in a fundamental way the relationship between individuals and the collective, labor and capital, and other cornerstones of American ideology?

There may or may not be. Time will tell. My own view is that the corporate powers, which flourished under the Lockean judicial system because of the weaknesses in that philosophical model of the individual and her rights, will instinctively fight what is in fact a threatening conception of the person as autonomous by virtue of their transcendental similarity with other people. American corporate power will not bother to make a philosophical case at all; it will operate in the domain of realpolitic so well documented by Cohen. Even if this is so, it is notable that so much intellectual and economic energy is now being exerted in the friction around a poweful an idea.

References

Cohen, J. E. (2019). Between Truth and Power: The Legal Constructions of Informational Capitalism. Oxford University Press, USA.

Hildebrandt, M. (2015). Smart technologies and the end (s) of law: Novel entanglements of law and technology. Edward Elgar Publishing.

## November 09, 2019

Ph.D. 2018

#### Notes on Krussell & Smith, 1998 and macroeconomic theory

I’m orienting towards a new field through my work on HARK. A key paper in this field is Krusell and Smith, 1998 “Income and wealth heterogeneity in the macroeconomy.” The learning curve here is quite steep. These are, as usual, my notes as I work with this new material.

Krusell and Smith are approaching the problem of macroeconomic modeling on a broad foundation. Within this paradigm, the economy is imagined as a large collection of people/households/consumers/laborers. These exist at a high level of abstraction and are imagined to be intergenerationally linked. A household might be an immortal dynasty.

There is only one good: capital. Capital works in an interesting way in the model. It is produced every time period by a combination of labor and other capital. It is distributed to the households, apportioned as both a return on household capital and as a wage for labor. It is also consumed each period, for the utility of the households. So all the capital that exists does so because it was created by labor in a prior period, but then saved from immediate consumption, then reinvested.

In other words, capital in this case is essentially money. All other “goods” are abstracted way into this single form of capital. The key thing about money is that it can be saved and reinvested, or consumed for immediate utility.

Households also can labor, when they have a job. There is an unemployment rate and in the model households are uniformly likely to be employed or not, no matter how much money they have. The wage return on labor is determined by an aggregate economic productivity function. There are good and bad economic periods. These are determine exogenously and randomly. There are good times and bad times; employment rates are determined accordingly. One major impetus for saving is insurance for bad times.

The problem raised by Krusell and Smith in this, what they call their ‘baseline model’, is that because all households are the same, the equilibrium distribution of wealth is far too even compared with realistic data. It’s more normally distributed than log-normally distributed. This is implicitly a critique at all prior macroeconomics, which had used the “representative agent” assumption. All agents were represented by one agent. So all agents are approximately as wealthy as all others.

Obviously, this is not the case. This work was done in the late 90’s, when the topic of wealth inequality was not nearly as front-and-center as it is in, say, today’s election cycle. It’s interesting that one reason why it might have not been front and center was because prior to 1998, mainstream macroeconomic theory didn’t have an account of how there could be such inequality.

The Krusell-Smith model’s explanation for inequality is, it must be said, a politically conservative one. They introduce minute differences in utility discount factor. The discount factor is how much you discount future utility compared to today’s utility. If you have a big discount factor, you’re going to want to consume more today. If you have a small discount factor, you’re more willing to save for tomorrow.

Krussell and Smith show that teeny tiny differences in discount factor, even if they are subject to a random walk around a mean with some persistence within households, leads to huge wealth disparities. Their conclusion is that “Poor households are poor because they’ve chosen to be poor”, by not saving more for the future.

I’ve heard, like one does, all kinds of critiques of Economics as an ideological discipline. It’s striking to read a landmark paper in the field with this conclusion. It strikes directly against other mainstream political narratives. For example, there is no accounting of “privilege” or inter-generational transfer of social capital in this model. And while they acknowledge that in other papers there is the discussion of whether having larger amounts of household capital leads to larger rates of return, Kruselll and Smith sidestep this and make it about household saving.

The tools and methods in the paper are quite fascinating. I’m looking forward to more work in this domain.

References

Krusell, P., & Smith, Jr, A. A. (1998). Income and wealth heterogeneity in the macroeconomy. Journal of political Economy106(5), 867-896.

## November 07, 2019

Center for Technology, Society & Policy

#### “Loved learning the latest” – A Great Night Celebrating CTSP at Google PAIR

CTSP Co-Directors Elizabeth Resor and Anne Jonas offer introductory remarks

On October 22nd CTSP fellows, alums, past and current Co-Directors, faculty advisors and many members of the Bay Area technology community gathered at Google SF for an evening of talks, posters, and lots of mingling, hosted by Google’s PAIR team. Despite BART delays reminding us the ripple effects of complicated infrastructural breakdowns, we had a magical evening guided by Michelle Carney, CTSP alum and founder of MLUX, in exploring the work CTSP has done, our vision for what’s next, and the broader connections around society, policy, and technology.

CTSP Faculty Advisory Board member Mike Rivera, CTSP Fellow Ayo Animashaun, Faculty Advisory Board member Alex Hughes, and Fellow Mahmoud Hamsho at the reception

We reflected on CTSP’s origins and progress over the last 5 years, exploring how Galen Panger and Nicholas Doty had prescient insight in developing a center that took seriously the politics of sociotechnical systems and recognized the cutting-edge work being done in the I School community and beyond. It has been so energizing to see CTSP fellows go on to become leading faculty, community advocates, policy-makers, and trusted advisors for the tech industry. As Jess Holbrook of PAIR reminded us, we must be wary of the powerful trying to claim expertise in developing solutions and alternatives to the very problems they have enabled, and instead look to those who have long been working collaboratively and carefully on the ground (and in the cloud) for their wisdom and best practices. We are so proud to count CTSP fellows among these experts.

CTSP Co-Founder Galen Panger networks at the poster reception

At this event, we got to hear lightning talks from four CTSP project teams and to learn more about another ten projects in posters displayed around the event space. Fellows were thrilled to make connections across cohort years and with those working on a variety of projects inside academia and out. It was especially inspiring to see how some projects have grown and continued over the years. Check out the fantastic posters above!

CTSP Fellow Franchesca Spektor presents on her team’s project

Thank you to everyone who attended and/or made a poster!

It was a fun night, but don’t just take our word for it:

## November 04, 2019

Ph.D. 2018

#### Explainable AI and computational approaches to macroeconomic theory

I have spent some time working with and around people concerned with the ethical implications of AI. A question that arises frequently in that context is to what extent automated decisions made by computational systems are “explainable” or “scrutable” (e.g. Selbst and Barocs, 2018). An important motivation for this line of inquiry is the idea that for AI systems to be effectively regulated by the Rule of Law, they need to be comprehensible to lawyers and understood within lawyerly discursive pracice (Hildebrandt, 2015). This is all very interesting, but analyses of the problem and its potential solutions rarely transcend the disciplinary silos from which the ‘explainability’ concerns originate. I’ve written my opinions about this quite a bit on this blog and I won’t reiterate them.

Instead, I’ve changed what I’m working now. Now I am contributing to open source software libraries for computational methods in macroeconomics, such as the Heterogeneous Agents Resources and toolKit (HARK). This is challenging and rewarding work. One reason why it is challenging and rewarding is how it bumps up against many key issues in the way computational methods are changing social sciences education. This is in many ways related to the explainable AI problem, though it’s in some sense the opposite side of the coin.

I’ll try to explain. Macroeconomic theory, which deals with such problems as how the economy as a whole reacts to changing trends in saving, consumption, and employment, and how agents within the economy react to those aggregate phenomena, has a long history associated with some major heavyweight economists: Keynes, Mankiw, etc. It is a deeply mathematical field that is taken seriously by central banks around the world and, by extension, private banks as well. Regulating the economy is an important job that requires expertise and is an intrinsically quantitatively understood operation; whatever one may think about the field of economics in general or its specific manifestations in history, it’s undeniable that the world needs economists of one kind or another.

So we have here a form of public policy expertise that is not discursive in the same sense that lawyerly practice is discursive. Economics has always imagined itself to be a science, however hotly contested that claim may be. It is also a field that does not shy away from having specialized disciplinary knowledge that must be accessed through demanding training. So economics would seem to be a good domain for computational methods to take root.

I’m finding that there are still challenges of interpretation in this field, but that they are somewhat different. Consider for now only the class of economic models that are built from a priori assumptions without any fitting to empirical data. Classically, economic models were constrained by their analytic tractability, meaning the ability of the economist to derive the results of the model through symbolic manipulation of the model’s mathematical terms. This led to the adoption of many assumptions of questionable realism, which have arguably led to some of the discrediting of economic theory since. But it also led to models that had closed form solutions, which have the dual advantage of being easy to compute (in terms of computational cost) and being easy to interpret, because the relationship between variables is explicit.

With computational models, the modeler has more flexibility. They can plug in the terms of the model and run a simulation to compute the result. But while the relationships between the input and output of the simulation may be observable in some sense in this case, the relationship is not proven. The simulation is not as good for purposes of exposition, or teaching, or explanation.

This is quite interesting, as it is a case where the explainability of a computational system is problematic but not because of a numeric or technical illiteracy on the part of the model reader, or of any intentional secrecy, but rather because of the complexity of the simulation (Burrell, 2016). For the purposes of this discussion, I’ve been discussing model building only, not model fitting, so the complexity in this case does not come from the noisiness of reality and the data it provides. Rather, the complexity results entirely from the internals of the model.

It is now a true word often spoken in jest that most machine learning today is some form of glorified (generalized) linear regression. The class of models considered by machine learning methods today is infinitely wide but ultimately shallow. Even when a need to understand the underlying phenomenon is abandoned, the available range of algorithms and hardware constraints limits machine-learnt models to those that are tractable by, say, a GPU.

But something else can be known.

References

Burrell, Jenna. “How the machine ‘thinks’: Understanding opacity in machine learning algorithms.” Big Data & Society 3.1 (2016): 2053951715622512.

Hildebrandt, Mireille. Smart technologies and the end (s) of law: Novel entanglements of law and technology. Edward Elgar Publishing, 2015.

Selbst, Andrew D., and Solon Barocas. “The intuitive appeal of explainable machines.” Fordham L. Rev. 87 (2018): 1085.

## October 28, 2019

Ph.D. 2018

#### Herbert Simon and the missing science of interagency

Few have ever written about the transformation of organizations by information technology with the clarity of Herbert Simon. Simon worked at a time when disciplines were being reconstructed and a shift was taking place. Older models of economic actors as profit maximizing agents able to find their optimal action were giving way as both practical experience and the exact sciences told a different story.

The rationality employed by firms today is not the capacity to choose the best action–what Simon calls substantive rationality. It is the capacity to engage in steps to discover better ways of acting–procedural rationality.

So we proceed step by step from the simple caricature of the firm depicted in textbooks to the complexities of real firms in the real world of business. At each step towards realism, the problem gradually changes from choosing the right course of action (substantive rationality) to finding way of calculating, very approximately, where a good course of action lies (procedural rationality). With this shift, the theory of the firm becomes a theory of estimation under uncertainty and a theory of computation.

Simon goes on to briefly describe the fields that he believes are poised to drive the strategic behavior of firms. These are Operations Research (OR) and artificial intelligence (AI). The goal of both these fields is to translate problems into mathematical specifications that can be executed by computers. There is some variation within these fields as to whether they aim at satisficing solutions or perfect answers to combinatorial problems, but for the purposes to this article they are the same–certainly the fields have cross-pollinated much since 1969.

Simon’s analysis was prescient. The impact of OR and AI on organizations simply can’t be understated. My purpose in writing this is to point to the still unsolved analytical problems of this paradigm. Simon notes that the computational techniques he refers to percolate only so far up the corporate ladder.

OR and AI have been applied mainly to business decisions at the middle levels of management. A vast range of top management decisions (e..g. strategic decisions about investment, R&D, specialization and diversification, recruitment, development, and retention of managerial talent) are still mostly handled traditionally, that is, by experienced executives’ exercise of judgment.

Simon’s proposal for how to make these kinds of decisions more scientific is the paradigm of “expert systems”, which did not, as far as I know, take off. However, these were early days, and indeed at large firms AI techniques are used to make these kinds of executive decisions. Though perhaps equally, executives defend their own prerogative for human judgment, for better or for worse.

The unsolved scientific problem that I find very motivating is based on a subtle divergence of how the intellectual fields have proceeded. Surely economic value and consequences of business activities are wrapped up not in the behavior of an individual firm, but of many firms. Even a single firm contains many agents. While in the past the need for mathematical tractability led to assumptions of perfect rationality for these agents, we are now far past that and “the theory of the firm becomes a theory of estimation under uncertainty and a theory of computation.” But the theory of decision-making under uncertainty and the theory of computation are largely poised to address problems of the solving a single agent’s specific task. The OR or AI system fulfills a specific function of middle management; it does not, by and large, oversee the interactions between departments, and so on. The complexity of what is widely called “politics” is not captured yet within the paradigms of AI, though anybody with an ounce of practical experience would note that politics is part of almost any organizational life.

How can these kinds of problems be addressed scientifically? What’s needed is a formal, computational framework for modeling the interaction of heterogeneous agents, and a systematic method of comparing the validity of these models. Interagential activity is necessarily quite complex; this is complexity that does not fit well into any available machine learning paradigm.

References

Simon, H. A. (1969). The sciences of the artiﬁcial. Cambridge, MA.

## October 22, 2019

Ph.D. 2018

#### “Private Companies and Scholarly Infrastructure”

I’m proud to link to this blog post on the Cornell Tech Digital Life Initiative blog by Jake Goldenfein, Daniel Griffin, and Eran Toch, and myself.

The academic funding scandals plaguing 2019 have highlighted some of the more problematic dynamics between tech industry money and academia (see e.g. Williams 2019, Orlowski 2017). But the tech industry’s deeper impacts on academia and knowledge production actually stem from the entirely non-scandalous relationships between technology firms and academic institutions. Industry support heavily subsidizes academic work. That support comes in the form of direct funding for departments, centers, scholars, and events, but also through the provision of academic infrastructures like communications platforms, computational resources, and research tools. In light of the reality that infrastructures are themselves political, it is imperative to unpack the political dimensions of scholarly infrastructures provided by big technology firms, and question whether they might problematically impact knowledge production and the academic field more broadly.

Goldenfein, Benthall, Griffin, and Toch, “Private Companies and Scholarly Infrastructure – Google Scholar and Academic Autonomy”, 2019

Among other topics, the post is about how the reorientation of academia onto commercial platforms possibly threatens the autonomy that is a necessary condition of the objectivity of science (Bourdieu, 2004).

This is perhaps a cheeky argument. Questioning whether Big Tech companies have an undue influence on academic work is not a popular move because so much great academic work is funded by Big Tech companies.

On the other hand, calling into question the ethics of Big Tech companies is now so mainstream that it is actively debated in the Democratic 2020 primary by front-running candidates. So we are well within the Overton window here.

On a philosophical level (which is not the primary orientation of the joint work), I wonder how much these concerns are about the relationship between capitalist modes of production and ideology with academic scholarship in general, and how much this specific manifestation (Google Scholar’s becoming the site of a disciplinary collapse (Benthall, 2015) in scholarly metrics is significant. Like many contemporary problems in society and technology, the “problem” may be that a technical intervention that might have at one point seemed like a desirable intervention by challengers (in the Fligstein (1997) field theory sense) is now having the political impact that is questioned and resisted by incumbents. I.e., while there has always been a critique of the system, the system has changed and so the critique comes from a different social source.

References

Benthall, S. (2015). Designing networked publics for communicative action. Interface, 1(1), 3.

Bourdieu, Pierre. Science of science and reflexivity. Polity, 2004.

Fligstein, Neil. “Social skill and institutional theory.” American behavioral scientist 40.4 (1997): 397-405.

Orlowski, A. (2017). Academics “funded” by Google tend not to mention it in their work. The Register, 13 July 2017.

Williams, O. (2019). How Big Tech funds the debate on AI Ethics. New Statesman America, 6 June 2019 < https://www.newstatesman.com/science-tech/technology/2019/06/how-big-tech-funds-debate-ai-ethics>.

MIMS 2011

#### BBC Click on Wikipedia interventions

BBC Click interviewed me for a segment on possible manipulation of Wikipedia by the Chinese state (below). Manipulation of Wikipedia by states is not new. What does seem to be new here, though, is the way in which strategies for intervening in Wikipedia (both through the election of administrators and at individual articles) are so explicitly outlined.

Remember, though, that we can never know who is editing these articles. Even wikiedits bots only pick up edits within government IP address ranges. We have no way of knowing whether the person represented by that IP address in that sitting is employed by the government. The point is that there is a lot to be gained from influencing Wikipedia’s representation of people, places, events and things given Wikipedia’s prioritised role as data source for digital assistants and search engines.

It makes sense, then, that institutions (including governments, corporations and other organisations) will try to give weight to their version of the truth by taking advantage of the weak points of the peer produced encyclopedia. Guarding against that kind of manipulation is critical but not a problem that can be easily solved. More thoughts on that soon…

## October 08, 2019

Center for Technology, Society & Policy

## Come celebrate our 5 Year Anniversary!

To celebrate this fifth year of CTSP we are hosting an event with the Google PAIR team to bring together CTSP alums and current fellows with the local technology community. Come learn about CTSP projects with lightning talks and posters addressing our core focus areas of Health + Sensors, Sustaining Democracy and Building Community, Integrating Safety & Privacy, and Just Algorithms: Fairness, Transparency, and Justice, spark connections with others committed to public interest technology, and help shape the future of CTSP!

When: Tuesday, October 22nd, 2019, 5 – 8:30pm

Where: Google SF, 345 Spear Street, SF, CA 94105

RSVP

### Agenda

5:00 – doors open, social hour

6:00 – 6:15 – CTSP + PAIR Welcome!

6:15 – 6:45 – CTSP 5 Year Recap & 5 Project Talks

6:45 – 8:30 – social hour & poster session

While you learn and mingle enjoy light snacks and refreshments provided by Google!

## Hear Lightning Talks About 5 CTSP Projects

1. Increasing Transparency into the Capabilities of Surveillance and Policing Technologies: A Resource for Citizens and Cities
As cities undergo these often imperceptible changes to enhance policing capabilities and introduce ‘smart city’ initiatives that may infringe upon civil liberties, it is imperative for citizens to have a reliable resource to turn to for understanding the functions and implications of these technologies.
2. Assessing Race and Income Disparities in Crowdsourced Safety Data Collection
How can crowdsourced data be biased based on the required access to smartphones and the knowledge and desire to participate in data-gathering activities?
3. Menstrual Biosensing Survival Guide
A zine about menstrual biosensing technologies to help users protect their intimate data privacy and rethink assumptions of how these apps configure their users.
4. Coordinated Entry System Research and Development for a Continuum of Care in Northern California
A needs assessment and system recommendation around “matching” unhoused people to appropriate services. How can such systems handle values such as privacy, security, autonomy, dignity, safety, and resiliency?
5. Affect & Facial Recognition in Hiring
What happens when an automated understanding of human affect enters the real world, in the form of systems that have life-altering consequences?

## Registration & Cancellation Policy

Due to security purposes, we must close registration/RSVP for our events 48 hours before the event. Please make sure to RSVP before that.

If your plans change, we kindly ask that you cancel your ticket on Eventbrite at least 72 hours before the event so we can release spare tickets to our waitlist. Thanks so much!

PAIR is devoted to advancing the research and design of people-centric AI systems. They’re interested in the full spectrum of human interaction with machine intelligence, from supporting engineers to understanding everyday experiences with AI.

Their goal is to do fundamental research, invent new technology, and create frameworks for design in order to drive a human-centered approach to artificial intelligence. And they want to be as open as possible: they’re building open source tools that everyone can use, hosting public events, and supporting academics in advancing the state of the art.

Check their past projects here, as well as their People + AI Guidebook.

## September 22, 2019

Ph.D. 2018

#### Ashby’s Law and AI control

I’ve recently discovered Ashby’s Law, also know as the First Law of Cybernetics, by reading Stafford Beer’s “Designing Freedom” lectures. Ashby’s Law is a powerful idea, one I’ve been grasping at intuitively for some time. For example, here I was looking for something like it and thought I could get it from the Data Processing Inequality in information theory. I have not yet grokked the mathematical definition of Ashby’s Law, which I gather is in Ross Ashby’s An Introduction to Cybernetics. Though I am not sure yet, I expect the formulation there can use an update. But if I am right about its main claims, I think the argument of this post will stand.

Ashby’s Law is framed in terms of ‘variety’, which is the number of states that it is possible for a system to be in. A six-sided die has six possible states (if you’re just looking at the top of it). A laptop has many more. A brain has many more even than that. A complex organization with many people in it, all with laptops, has even more. And so on.

The law can be stated in many ways. One of them is that:

When the variety or complexity of the environment exceeds the capacity of a system (natural or artificial) the environment will dominate and ultimately destroy that system.

The law is about the relationship between a system and its environment. Or, in another sense, it is about a system to be controlled and a different system that tries to control that system. The claim is that the control unit needs to have at least as much variety as the system to be controlled for it to be effective.

This reminds me of an argument I had with a superintelligence theorist back when I was thinking about such things. The Superintelligence people, recall, worry about an AI getting the ability to improve itself recursively and causing an “intelligence explosion”. Its own intelligence, so to speak, explodes, surpassing all other intelligent life and giving it total domination over the fate of humanity.

Here is the argument that I posed a few years ago, reframed in terms of Ashby’s Law:

• The AI in question is a control unit, C, and the world it would control is the system, S.
• For the AI to have effective domination over S, C would need at least as much variety as S.
• But S includes C within it. The control unit is part of the larger world.
• Hence, no C can perfectly control S.

Superintelligence people will no doubt be unsatisfied by this argument. The AI need not be effective in the sense dictated by Ashby’s Law. It need only be capable of outmaneuvering humans. And so on.

However, I believe the argument gets at why it is difficult for complex control systems to ever truly master the world around them. It is very difficult for a control system to have effective control over itself, let alone itself in a larger systemic context, without some kind of order constraining the behavior of the total system (the system including the control unit) imposed from without. The idea that it is possible to gain total mastery or domination through an AI or better data systems is a fantasy because the technical controls adds their own complexity to the world that is to be controlled.

This is a bit of a paradox, as it raises the question of how any control unites work at all. I’ll leave this for another day.

## September 16, 2019

Ph.D. 2018

#### Bridging between transaction cost and traditional economics

Some time ago I was trying to get my head around transaction cost economics (TCE) because of its implications for the digital economy and cybersecurity. (1, 2, 3, 4, 5). I felt like I had a good grasp of the relevant theoretical claim of TCE which is the interaction between asset specificity and the make-or-buy decision. But I didn’t have a good sense of the mechanism that drove that claim.

I worked it out yesterday.

Recall that in the make or buy decision, a firm is determining whether or not to make some product in-house or to buy it from the market. This is a critical decision made by software and data companies, as often these businesses operate by assembling components and data streams into a new kind of service; these services often are the components and data streams used in other firms. And so on.

The most robust claim of TCE is that if the asset (component, service, data stream) is very specific to the application of the firm, then the firm will be more likely to make it. If the asset is more general-purpose, then it buy it as a commodity on the market.

Why is this? TCE does not attempt to describe this phenomenon in a mathematical model, at least as far as I have found. Nevertheless, this can be worked out with a much more general model of the economy.

Assume that for some technical component there are fix costs $f$ and marginal costs \$c\$. Consider two extreme cases: in case A, the asset is so specific that only one firm will want to buy it. In case B, the asset is very general so there’s many firms that want to purchase it.

In case A, a vendor will have costs of $f + c$ and so will only make the good if the buyer can compensate them at least that much. At the point where the buyer is paying for both the fixed and marginal costs of the product, they might as well own it! If there are other discovered downstream uses for the technology, that’s a revenue stream. Meanwhile, since the vendor in this case will have lock-in power over the buyer (because switching will mean paying the fixed cost to ramp up a new vendor), that gives the vendor market power. So, better to make the asset.

In case B, there’s broader market demand. It’s likely that there’s already multiple vendors in place who have made the fixed cost investment. The price to the buying firm is going to be closer to $c$, the market price that converges over time to the fixed cost, as opposed to $c =+ f$, which includes the fixed costs. Because there are multiple vendors, lock-in is not such an issue. Hence the good becomes a commodity.

A few notes on the implications of this for the informational economy:

• Software libraries have high fixed cost and low marginal cost. The tendency of companies to tilt to open source cores with their products built on top is a natural result of the market. The modularity of open source software is in part explained by the ways “asset specificity” is shaped exogenously by the kinds of problems that need to be solved. The more general the problem, the more likely the solution has been made available open source. Note that there is still an important transaction cost at work here, the search cost. There’s just so many software libraries.
• Data streams can vary a great deal as to whether and how they are asset specific. When data streams are highly customized to the downstream buyer, they are specific; the customization is both costly to the vendor and adding value to the buyer. However, it’s rarely possible to just “make” data: it needs to be sourced from somewhere. When firms buy data, it is normally in a subscription model that takes into account industrial organization issues (such as lock in) within the pricing.
• Engineering talent, and related labor costs, are interesting in that for a proprietary system, engineering human capital gains tend to be asset specific, while for open technologies engineering skill is a commodity. The structure of the ‘tech business’, which requires mastery of open technology in order to build upon it a proprietary system, is a key dynamic that drives the software engineering practice.

There are a number of subtleties I’m missing in this account. I mentioned search costs in software libraries. There’s similar costs and concerns about the inherent riskiness of a data product: by definition, a data product is resolving some uncertainty with respect to some other goal or values. It must always be a kind of credence good. The engineering labor market is quite complex in no small part because it is exposed to the complexities of its products.

## September 15, 2019

Ph.D. alumna

#### Facing the Great Reckoning Head-On

I was recently honored by the Electronic Frontier Foundation. Alongside Oakland Privacy and William Gibson, I received a 2019 Barlow/Pioneer Award. I was asked to give a speech. As I reflected on what got me to this place, I realized I needed to reckon with how I have benefited from men whose actions have helped uphold a patriarchal system that has hurt so many people. I needed to face my past in order to find a way to create space to move forward.

This is the speech I gave in accepting the award. I hope sharing it can help others who are struggling to make sense of current events. And those who want to make the tech industry to do better.

— —

I cannot begin to express how honored I am to receive this award. My awe of the Electronic Frontier Foundation dates back to my teenage years. EFF has always inspired me to think deeply about what values should shape the internet. And so I want to talk about values tonight, and what happens when those values are lost, or violated, as we have seen recently in our industry and institutions.

But before I begin, I would like to ask you to join me in a moment of silence out of respect to all of those who have been raped, trafficked, harassed, and abused. For those of you who have been there, take this moment to breathe. For those who haven’t, take a moment to reflect on how the work that you do has enabled the harm of others, even when you never meant to.

<silence>

The story of how I got to be standing here is rife with pain and I need to expose part of my story in order to make visible why we need to have a Great Reckoning in the tech industry. This award may be about me, but it’s also not. It should be about all of the women and other minorities who have been excluded from tech by people who thought they were helping.

The first blog post I ever wrote was about my own sexual assault. It was 1997 and my audience was two people. I didn’t even know what I was doing would be called blogging. Years later, when many more people started reading my blog, I erased many of those early blog posts because I didn’t want strangers to have to respond to those vulnerable posts. I obfuscated my history to make others more comfortable.

I was at the MIT Media Lab from 1999–2002. At the incoming student orientation dinner, an older faculty member sat down next to me. He looked at me and asked if love existed. I raised my eyebrow as he talked about how love was a mirage, but that sex and pleasure were real. That was my introduction to Marvin Minsky and to my new institutional home.

My time at the Media Lab was full of contradictions. I have so many positive memories of people and conversations. I can close my eyes and flash back to laughter and late night conversations. But my time there was also excruciating. I couldn’t afford my rent and did some things that still bother me in order to make it all work. I grew numb to the worst parts of the Demo or Die culture. I witnessed so much harassment, so much bullying that it all started to feel normal. Senior leaders told me that “students need to learn their place” and that “we don’t pay you to read, we don’t pay you to think, we pay you to do.” The final straw for me was when I was pressured to work with the Department of Defense to track terrorists in 2002.

After leaving the Lab, I channeled my energy into V-Day, an organization best known for producing “The Vagina Monologues,” but whose daily work is focused on ending violence against women and girls. I found solace in helping build online networks of feminists who were trying to help combat sexual assault and a culture of abuse. To this day, I work on issues like trafficking and combating the distribution of images depicting the commercial sexual abuse of minors on social media.

By 2003, I was in San Francisco, where I started meeting tech luminaries, people I had admired so deeply from afar. One told me that I was “kinda smart for a chick.” Others propositioned me. But some were really kind and supportive. Joi Ito became a dear friend and mentor. He was that guy who made sure I got home OK. He was also that guy who took being called-in seriously, changing his behavior in profound ways when I challenged him to reflect on the cost of his actions. That made me deeply respect him.

I also met John Perry Barlow around the same time. We became good friends and spent lots of time together. Here was another tech luminary who had my back when I needed him to. A few years later, he asked me to forgive a friend of his, a friend whose sexual predation I had witnessed first hand. He told me it was in the past and he wanted everyone to get along. I refused, unable to convey to him just how much his ask hurt me. Our relationship frayed and we only talked a few times in the last few years of his life.

So here we are… I’m receiving this award, named after Barlow less than a week after Joi resigned from an institution that nearly destroyed me after he socialized with and took money from a known pedophile. Let me be clear — this is deeply destabilizing for me. I am here today in-no-small-part because I benefited from the generosity of men who tolerated and, in effect, enabled unethical, immoral, and criminal men. And because of that privilege, I managed to keep moving forward even as the collateral damage of patriarchy stifled the voices of so many others around me. I am angry and sad, horrified and disturbed because I know all too well that this world is not meritocratic. I am also complicit in helping uphold these systems.

What’s happening at the Media Lab right now is emblematic of a broader set of issues plaguing the tech industry and society more generally. Tech prides itself in being better than other sectors. But often it’s not. As an employee of Google in 2004, I watched my male colleagues ogle women coming to the cafeteria in our building from the second floor, making lewd comments. When I first visited TheFacebook in Palo Alto, I was greeted by a hyper-sexualized mural and a knowing look from the admin, one of the only women around. So many small moments seared into my brain, building up to a story of normalized misogyny. Fast forward fifteen years and there are countless stories of executive misconduct and purposeful suppression of the voices of women and sooooo many others whose bodies and experiences exclude them from the powerful elite. These are the toxic logics that have infested the tech industry. And, as an industry obsessed with scale, these are the toxic logics that the tech industry has amplified and normalized. The human costs of these logics continue to grow. Why are we tolerating sexual predators and sexual harassers in our industry? That’s not what inclusion means.

I am here today because I learned how to survive and thrive in a man’s world, to use my tongue wisely, watch my back, and dodge bullets. I am being honored because I figured out how to remove a few bricks in those fortified walls so that others could look in. But this isn’t enough.

I am grateful to EFF for this honor, but there are so many underrepresented and under-acknowledged voices out there trying to be heard who have been silenced. And they need to be here tonight and they need to be at tech’s tables. Around the world, they are asking for those in Silicon Valley to take their moral responsibilities seriously. They are asking everyone in the tech sector to take stock of their own complicity in what is unfolding and actively invite others in.

And so, if my recognition means anything, I need it to be a call to arms. We need to all stand up together and challenge the status quo. The tech industry must start to face The Great Reckoning head-on. My experiences are all-too common for women and other marginalized peoples in tech. And it it also all too common for well-meaning guys to do shitty things that make it worse for those that they believe they’re trying to support.

If change is going to happen, values and ethics need to have a seat in the boardroom. Corporate governance goes beyond protecting the interests of capitalism. Change also means that the ideas and concerns of all people need to be a part of the design phase and the auditing of systems, even if this slows down the process. We need to bring back and reinvigorate the profession of quality assurance so that products are not launched without systematic consideration of the harms that might occur. Call it security or call it safety, but it requires focusing on inclusion. After all, whether we like it or not, the tech industry is now in the business of global governance.

“Move fast and break things” is an abomination if your goal is to create a healthy society. Taking short-cuts may be financially profitable in the short-term, but the cost to society is too great to be justified. In a healthy society, we accommodate differently abled people through accessibility standards, not because it’s financially prudent but because it’s the right thing to do. In a healthy society, we make certain that the vulnerable amongst us are not harassed into silence because that is not the value behind free speech. In a healthy society, we strategically design to increase social cohesion because binaries are machine logic not human logic.

The Great Reckoning is in front of us. How we respond to the calls for justice will shape the future of technology and society. We must hold accountable all who perpetuate, amplify, and enable hate, harm, and cruelty. But accountability without transformation is simply spectacle. We owe it to ourselves and to all of those who have been hurt to focus on the root of the problem. We also owe it to them to actively seek to not build certain technologies because the human cost is too great.

My ask of you is to honor me and my story by stepping back and reckoning with your own contributions to the current state of affairs. No one in tech — not you, not me — is an innocent bystander. We have all enabled this current state of affairs in one way or another. Thus, it is our responsibility to take action. How can you personally amplify underrepresented voices? How can you intentionally take time to listen to those who have been injured and understand their perspective? How can you personally stand up to injustice so that structural inequities aren’t further calcified? The goal shouldn’t be to avoid being evil; it should be to actively do good. But it’s not enough to say that we’re going to do good; we need to collectively define — and hold each other to — shared values and standards.

People can change. Institutions can change. But doing so requires all who harmed — and all who benefited from harm — to come forward, admit their mistakes, and actively take steps to change the power dynamics. It requires everyone to hold each other accountable, but also to aim for reconciliation not simply retribution. So as we leave here tonight, let’s stop designing the technologies envisioned in dystopian novels. We need to heed the warnings of artists, not race head-on into their nightmares. Let’s focus on hearing the voices and experiences of those who have been harmed because of the technologies that made this industry so powerful. And let’s collaborate with and design alongside those communities to fix these wrongs, to build just and empowering technologies rather than those that reify the status quo.

Many of us are aghast to learn that a pedophile had this much influence in tech, science, and academia, but so many more people face the personal and professional harm of exclusion, the emotional burden of never-ending subtle misogyny, the exhaustion from dodging daggers, and the nagging feeling that you’re going crazy as you try to get through each day. Let’s change the norms. Please help me.

Thank you.

we’re all taught how to justify history as it passes by
and it’s your world that comes crashing down
when the big boys decide to throw their weight around
but he said just roll with it baby make it your career
keep the home fires burning till america is in the clear

i think my body is as restless as my mind
and i’m not gonna roll with it this time
no, i’m not gonna roll with it this time
— Ani Difranco

## September 14, 2019

MIMS 2012

#### User Research Interview Tip: Shut. Up.

If you were to look at Robert Caro’s notebook, you would see lots of “SU”s, short for “Shut Up!”, scattered throughout his interview notes. The Pulitzer-prize winning author of The Power Broker, among other mammoth books, uses this trick to keep himself silent while interviewing subjects. About this, he writes:

In interviews, silence is the weapon, silence and people’s need to fill it—as long as the person isn’t you, the interviewer. Two of fiction’s greatest interviewers—Georges Simenon’s Inspector Maigret and John le Carré’s George Smiley—have little devices they use to keep themselves from talking and to let silence do its work. Maigret cleans his ever-present pipe, tapping it gently on his desk and then scraping it out until the witness breaks down and talks. Smiley takes off his eyeglasses and polishes them with the thick end of his necktie. As for me, I have less class. When I’m waiting for the person I’m interviewing to break a silence by giving me a piece of information I want, I write “SU” (for Shut Up!) in my notebook. If anyone were ever to look through my notebooks, he would find a lot of “SU”s.

This is a fantastic trick to use during user research interviews, too. Remaining silent after a person’s initial response often leads them to elaborate more, revealing an additional nuance, or an exception to the “typical” use case they just described.

I take longhand notes on paper for this reason. My note taking never keeps pace with the speaker, so I’m always catching up after they stop talking, which forces me to shut up.

When you’re interviewing users, find your own eyeglass polishing or “SU.”

## August 25, 2019

Ph.D. 2018

#### The ontology of software, revisited

I’m now a software engineer again after many years doing and studying other things. My first-person experience, my phenomenological relationship with this practice, is different this time around. I’ve been meaning to jot down some notes based on that fresh experience. Happily, there’s resonance with topics of my academic focus as well. I’m trying to tease out these connections.

To briefly recap: There’s a recurring academic discourse around technology ethics. Roughly speaking, it starts with a concern about a newish technology that has media or funding agency interest. Articles then get written capitalizing on this hot topic; these articles are fractured according to the disciplinary background of their authors.

• Engineers try to come up with an improved version of the technology.
• Lawyers try to come up with ways to regulate the production and use of the technology broadly speaking.
• Organizational sociologists come up with institutional practices (‘ethics boards’, ‘contestability’) which would prevent the technology from being misused.
• Critical theorists argue that the technology would be less worrisome if representational desiderata within the field of technology production were better.
• … and so on.

This is a very active and interesting discourse, but from my (limited) perspective, is rarely impacts industry practice. This isn’t because people in industry don’t care about the ethical implications of their work. It’s because people in industry are engaged full-time in a different discourse. This is the discourse of industry practitioners.

My industrial background is in software development and data science. Obviously there are other kinds of industrial work–hardware, biotech, etc. But it’s fair to say that a great deal of the production of “technology” in the 21st century is, specifically, software development. And my point here is that software development has its own field of discourse that is rich and vivid and a full-time job to keep up with. Here’s some examples of what I’m getting at:

• There is always-already a huge world of communication between engineers about what technologies are interesting, how to use them effectively, how they compare with prior technologies, the implications of these trends for technical careers, and so on. Browse Hacker News. Look at industry software conferences.
• There’s also a huge world of industrial discussion about the social practices of software development. A lot of my knowledge of this is a bit dated. But as I come back to industry, I find myself looking back to now Classic sources on how-to-work-effectively-on-software. I’m linking to articles from Joel Spolsky’s blog. I’m ordering a copy of Fred Brooks’s classic The Mythical Man-Month.
• I’m reading documentation, endlessly, about how to configure and use the various SaaS, IaaS, PaaS, etc. tools that are now necessary parts of full-stack development. When the documentation is limited, I’m engaging with customer service people of technical products, who have their own advice, practices, etc.

This is a complex world of literature and practice. Part of what makes it complex is that it is always-already densely documented and self-referential, enacted by smart and literate people, most of whom are quite socially skilled. It’s people working full-time jobs in a field that is now over 40 years old.

I’ve argued in other posts that if we want to solve the ‘technology ethics’ problem, we should see it as an economic problem. At a high level, I still believe that’s true. I want to qualify that point though, and say: now that I’m back in a more engage position with respect to the field of technical production, I believe there are institutional/organizational ways to address broader social concerns through interventions on engineering practice.

What is missing, in my view, is a sincere engagement with the nitty-gritty of engineering practice itself. I know there are anthropologists who think they do this. I haven’t read anybody who really does it, in their writing, and I believe the reason for that is: anthropologists writing for other academic anthropologists are not going to write what would be actually useful here, which is a guide for product and project management that would likely recapitulate a lot of conventional (but too often ignored) wisdom about software engineering “best practices”–documentation, testing, articulation of use cases, etc. These are the kinds of things that improve technical quality in a real way.

Now that I write this, I recall that the big ethics research teams at, say, Google, do stuff like this. It’s great.

I was going to say something about the ontology of software.

Recall: I have a position on the ontology of data, which I’ve called Situated Information Flow Theory (SIFT). I worked hard on it. According to SIFT, an information flow is a causal flow situated in a network of other causal relations. The meaning of the information depends on that causally defined situation.

What then is software?

“Software” refers to sets of instructions written by people in a specialized “programming” language as text data, which is then interpreted and compiled by a machine. In paradigmatic industrial practice (I’m simplifying, bear with me), ultimately these instructions will be used to control the behavior of a machine that interfaces with the world in a real-time, consequential way. This latter machine is referred to, internally, as being “in production”.

When you’re programming a technical product, first you write software “in development”. You are writing drafts of code. You get your colleagues to review it. You link up the code you wrote to the code the other team wrote and you see if it works together. There is a long and laborious process of building tests for new requirements and fixing the code so that it meets those requirements. There are designs, and redesigns, of internal and external facing features. The complexity of the total task is divided up into modules; the boundaries of those modules shifts over time. The social structure of the team adapts as new modules become necessary.

There is an isomorphism, a well documented phenomenon in organizational social theory, between the technology being created and the social structure that creates it. The team structure mirrors the software architecture.

When the pieces are in place adequately enough–and when the investors/management has grown impatient enough–the software is finally “deployed to production”. It “goes live”. What was an internal exercise is now a process with reputational consequences for the business, as well as possibly real consequences for the users of the technology.

Inevitably, the version of the product “in production” is not complete. There are errors. There are new features requested. So the technology firm now organizes itself around several “cycles” running at different frequencies in parallel. There’s a “development cycle” of writing new software code. There’s a “release cycle” of packaging new improvements into bundles that are documented and tested for quality. The releases are deployed to production on a schedule. Different components may have different development and release cycles. The impedance match or mismatch between these cycles becomes its own source of robustness or risk. (I’ve done some empirical research work on this.)

What does this mean for the ontology of software?

The first thing it means is that the notion that software is a static artifact, something like either a physical object (like a bicycle) or a publication (like a book) is mostly irrelevant to what’s happening. The software production process depends on the fluidity of source code. When software is deployed “as a service”, it’s dubious for it to qualify as a “creative work”, subject to copyright law, except by virtue of legal inertia. Something totally different is going on.

The second thing it means is that the live technical product is an ongoing institutional accomplishment. It’s absurd to ever say that humans are not “in the loop”. This is one of the big insights of the critical/anthro reaction to “Big Tech” in the past five years or so. But it has also been common knowledge within the industry for fifteen years or so.

The third thing it means is that software is the structuring of a system of causal relations. Software, when it’s deployed, determines what causes what. See above for a definition of the the nature of information: it’s a causal flow situated in other causal relations. The link between software and information then is quite clear and direct. Software (as far as it goes) is a definition of a causal situation.

The fourth thing it means is that software products are the result of agreement between people. Software only makes it into production if it has gotten there through agreed-upon processes by the team that deploys it. The strength of software is in the collective input that went into it. In a sense, software is much more like a contract, in legal terms, than it is like a creative work. In the extended network of human and machine actors, software is the result of, the expression of, self-regulation first. Only secondarily does it, in Lessig’s terms, become a regulatory force more broadly.

What is software? Software is a form of social structure.

## August 23, 2019

Center for Technology, Society & Policy

#### Social Impact Un-Pitch Day 2019

On Thursday, September 26th at 6pm the Center for Technology, Society & Policy (CTSP) and the School of Information’s Information Management Student Association (IMSA) are co-hosting the fourth annual Social Impact Un-Pitch Day!

Join CTSP and IMSA to brainstorm ideas for projects that address the challenges of technology, society, and policy. We welcome students, community organizations, local municipal partners, faculty, and campus initiatives to discuss discrete problems that project teams can take on over the course of this academic year. Teams will be encouraged to apply to CTSP to fund their projects.

Location: Room 202, in South Hall.

RSVP here!

Livestream

## Agenda

• 6:10 Introductions from IMSA and CTSP
• 6:15 Example Projects
• 6:20 Sharing Un-Pitches (3 minutes per Un-Pitch)
• 7:10 Mixer (with snacks and refreshments)

## Un-Pitches

Un-Pitches are meant to be informal and brief introductions of yourself, your idea, or your organization’s problem situation. Un-pitches can include designing technology, research, policy recommendations, and more. Students and social impact representatives will be given 3 minutes to present their Un-Pitch. In order to un-pitch, please share 1-3 slides, as PDF and/or a less than 500-word description—at this email: ctsp@nullberkeley.edu. You can share slides and/or description of your ideas even if you aren’t able to attend. Deadline to share materials: midnight September 25th, 2019.

## Funding Opportunities

The next application round for fellows will open in November. CTSP’s fellowship program will provide small grants to individuals and small teams of fellows for 2019. CTSP also has a recurring offer of small project support.

## Prior Projects & Collaborations

Here are several examples of projects that members of the I School community have pursued as MIMS final projects or CTSP Fellow projects (see more projects from 2016, 2017, 2018, and 2019).

## Skills & Interests of Students

The above projects demonstrate a range of interests and skills of the I School community. Students here and more broadly on the UC Berkeley campus are interested and skilled in all aspects of where information and technology meets people—from design and data science, to user research and information policy.

## August 16, 2019

Ph.D. 2016

#### Monome to Fabric

MONOME TO FABRIC
As a follow up to my work with Ebb, I decided to work more closely with the handweaving process to create a fabric that would have multiple regions that change color in response to touch. A friend had an old Monome he wasn’t using anymore so we decided to use it as a control interface for the fabric. As such, selecting regions on the monome changes the color of regions on the fabric. The monome also lets you dynamically set power values to trigger the color change traditions.

#### Fluxus

FLUXUS INSPIRED PROTOTYPING
What began as a playful mail-exchange by myself and Kristina Andersen emerged, two years later, as a broader reflection on our research field of human-computer interaction. Specifically, we began to turn to Fluxus, an avant-garde movement of the late 50′s and 60′s as a model for how we might productively question and critique the values of our field from within. This consisted of a reflection on our own Fluxus inspired correspondence, a workshop we hosted to understand how such approaches are taken up within HCI, and a collaborative activity by myself, Kristina Andersen, Daniela Rosner, Ron Wakkary and James Pierce where we wrote and exchanged “event” scores for attending to our relationships with technology.

From HCI to HCI Amusement: Strategies for Engaging what New Technology Makes Old – CHI 2019 Full Paper

#### Exoskeletons for Caregivers

EXOSKELETONS FOR CAREGIVERS
Just as the spacesuits allow a human body to inhabit “harsh” environments, these exoskeletons are an exercise in creating hyperbolic and storied responses to the physical and emotional frictions I experience as a mother. They represent the pressures to buck up, do more, and sustain one’s body. Their flimsy and “soft” handcrafted nature sits in tension with these pressures, representing my continual failure to guard myself, of giving too much, and seeing my “worth” so-to-speak in the capacity my body affords to provide comfort and kindness. Framed as forms of protection, adaptation, and making due, the crafted nature of the forms brings a quaint and unassuming overtone to expressions of anger and discomfort. They are a reflection on how tasks can take over and occupy the self. The forms they take are often funny, not to make light of the struggles, but simply to make due and remove the taboo of seriousness and the social threat of being perceived as the hysterical women while, indeed, making forms that represent a kind of repetitive mania in their construction.

Sedimentation Garment
The exoskeleton for sedimentation emerged from an observation that related mixed up colors of play-doh with the patterns found in sedimentary rock formations. With thirteen embedded force sensors routed through the fabric, I designed it to be materially sensitive to and remembering of the forces upon my body (and my body upon other bodies). In thinking about how sediment becomes deposited and collected while also eroded by wind and rain inspired an observation of the ways that bodies form in relationship to one and other. Thinking specifically of her daily routines, which involve carrying a three year old who very much enjoys “uppy,” I envisioned a way that my garment might be able to capture and playback the forces exerted by my children upon my body, but also by the other objects that support and shape me (partners, objects, natural forces). As such, this garment is as much about providing support as it is being supported, using the embedded force sensors to “capture” such forces that we experience as fleeting. I designed and hand-wove the garment on a TC2 digital jacquard loom. The color pallet was inspired by images of sedimentary rock as well as the color of play-doh when all of the colors are mixed together—another insignificant but no less unsettling frustration I experience. I integrated thirteen force sensors into the structure of the garment by way of adding a conductive yarn into a “waffle weave” stitch, which compresses upon force and thus, creates fairly stable changes in resistance corresponding to the pressure exerted. I routed the thirteen force sensors through the fabric to attach to a yet-to-be-made PCB situated at the neck. It is envisioned as a garment that becomes sedimented with interaction, a smart surface layer that becomes dirty, moulded, and worn though the meetings of bodies.

The Nipple Poncho
The exoskeleton for sucking is a hand-crocheted garment that covers the chest, back, and upper arms in pacifier nipples.The object is conceptualized as one that allows people to safely suck on you while protecting your body. The sucking here, comes both from the feeling of nursing
children, the role of the pacifier in satisfying the innate and primal need for humans (and mammals) to suck, and the expectations that women serve the role as caregivers and nurturers. I created the garment by repeatedly hand-crocheting hexagons around pacifier nipples and stitching them together into the shape of a poncho.

#### Generative Brioche Knitting

Generative Brioche KnittingDuring a one-week workshop with the Carnegie Mellon University Textiles Lab, I worked closely with Lea Albaugh to write code for a Shima Seiki knitting machine. An initial interest in the idea of skin folks, or the ability of knits to produce “gross” textures, led to an algorithm that could create brioche patterns based on use specifications and chance.

## July 05, 2019

Ph.D. student

#### CHI 2019 Annotated Bibliography (Part 1)

After the 2019 CHI conference (technically the ACM CHI Conference on Human Factors in Computing Systems) and blogging about our own paper on design approaches to privacy, I wanted to highlight other work that I found interesting or thought provoking in a sort of annotated bibliography. Listed in no particular order, though most relate to one or more themes that I’m interested in (privacy, design research, values in design practice, critical approaches, and speculative design).

(I’m still working through the stack of CHI papers that I downloaded to read, so hopefully this is part 1 of two or three posts).

• James Pierce. 2019. Smart Home Security Cameras and Shifting Lines of Creepiness: A Design-Led Inquiry. Paper 45, 14 pages. https://doi.org/10.1145/3290605.3300275 — Pierce uses a design-led inquiry to illustrate and investigate three data practices of IoT products and services (digital leakage, hole-and-corner applications, and foot-in-the-door devices), providing some conceptual scaffolding for thinking about how privacy emerges differently in relation to varying technical (and social) configurations. Importantly, I like that Pierce is pushing design researchers to go beyond conceptualizing privacy as “creepiness”, through his exploration of three tropes of data practices.
• Renee Noortman, Britta F. Schulte, Paul Marshall, Saskia Bakker, and Anna L. Cox. 2019. HawkEye – Deploying a Design Fiction Probe. Paper 422, 14 pages. https://doi.org/10.1145/3290605.3300652 — Building on Shulte’s concept of a “design probe,” Noortman et al. participants interact with a (beautifully designed!) control panel in the home over 3 weeks to act in the role of a caregiver in a design fiction about dementia care. The paper furthers the use of design fiction as a participatory and embodied experience, and as a data collection tool for research. The authors provide some useful reflections on the ways participants imagined and helped build out the fictional world in which they were participating.
• Yaxing Yao, Justin Reed Basdeo, Smirity Kaushik, and Yang Wang. 2019. Defending My Castle: A Co-Design Study of Privacy Mechanisms for Smart Homes. Paper 198, 12 pages. https://doi.org/10.1145/3290605.3300428 — Yao et al. use co-design techniques to explore privacy concerns and potential privacy mechanisms with a range of participants (including diversity in age). Some interesting ideas arise from participants, such as creating an IoT “incognito mode,” as well as raising concerns about accessibility for these systems. Sometimes tensions arise, with participants wanting to trust IoT agents like Alexa as a ‘true friend’ who won’t spy on them, yet harboring some distrust of the companies creating these systems. I like that the authors point to a range of modalities for where we might place responsibility for IoT privacy – in the hardware, apps, platform policy, or operating modes. It’s a nice tie into questions others have asked about how responsibility for privacy is distributed, or what happens when we “handoff” responsibility for protecting values from one part of a sociotechnical system to another part.
• Kristina Andersen and Ron Wakkary. 2019. The Magic Machine Workshops: Making Personal Design Knowledge. Paper 112, 13 pages. https://doi.org/10.1145/3290605.3300342 — Andersen and Wakkary outline a set of workshop techniques to help participants generate personal materials. I appreciate the commitments made in the paper, such as framing workshops as something that should benefit participants themselves, as well as researchers, in part by centering the workshop on the experience of individual participants. They propose a set of workshop elements; it’s nice to see these explicated here, as they help convey a lot of tacit knowledge about running workshops (the details of which are often abbreviated in most papers’ methods sections). I particularly like the “prompt” element to help provide a quick initial goal for participants to engage in while situating the workshop. While the example workshops used in the paper focus on making things out of materials, I’m curious if some of the outlined workshop elements might be useful in other types of workshop-like activities.
• Laura Devendorf, Kristina Andersen, Daniela K. Rosner, Ron Wakkary, and James Pierce. 2019. From HCI to HCI-Amusement: Strategies for Engaging what New Technology Makes Old. Paper 35, 12 pages. https://doi.org/10.1145/3290605.3300265 – Devendorf et al. start by (somewhat provocatively) asking what it might be like to explore a “non-contribution” in HCI. The paper walks through a set of projects and works its way to a set of reflections about the norms of HCI research focusing on the “technological new,” asking what it might mean instead to take the present or the banal more seriously. The paper also starts to ask what types of epistemologies are seen as legitimate in HCI. The paper calls for “para-research” within HCI as a way to focus attention on what is left out or unseen through dominant HCI practices.
• Colin M. Gray and Shruthi Sai Chivukula. 2019. Ethical Mediation in UX Practice. Paper 178, 11 pages. https://doi.org/10.1145/3290605.3300408 – Through a set of case study observations and interview, Gray and Chivukula study how ethics are conducted in practice by UX designers. The paper provides a lot of good detail about ways UX designers bring ethics to the forefront and some of the challenges they face. The authors contribute a set of relationships or mediators, connecting individual designers’ practices to organizational practices to applied ethics.
• Sarah E. Fox, Kiley Sobel, and Daniela K. Rosner. 2019. Managerial Visions: Stories of Upgrading and Maintaining the Public Restroom with IoT. Paper 493, 15 pages. https://doi.org/10.1145/3290605.3300723 – Through interviews, participant observations, and analysis of media materials, Fox et al. investigate managerial labor in regulating access to public bathroom resources. They craft a story of regulation (in a broad sense), about how the bathroom’s management is entangled among local politics and on-the-ground moral beliefs, corporate values, imagined future efficiencies through technology, and strategic uses of interior and technological design. This entanglement allows for particular types of control, allowing some access to resources and making it harder for others.
• William Gaver, Andy Boucher, Michail Vanis, Andy Sheen, Dean Brown, Liliana Ovalle, Naho Matsuda, Amina Abbas-Nazari, and Robert Phillips. 2019. My Naturewatch Camera: Disseminating Practice Research with a Cheap and Easy DIY Design. Paper 302, 13 pages. https://doi.org/10.1145/3290605.3300532 – Gaver et al. detail a DIY nature camera, shown in partnership with a BBC television series and built by over 1000 people. Interestingly, while similar tools could be used for citizen science efforts, the authors are clear that they are instead trying to create a type of public engagement with research that focuses on creating more intimate types of encounters, and engaging people with less technical expertise in making. The cameras help create intimate “encounters” with local wildlife (plus the paper includes some cute animal photos!).
• Sandjar Kozubaev, Fernando Rochaix, Carl DiSalvo, and Christopher A. Le Dantec. 2019. Spaces and Traces: Implications of Smart Technology in Public Housing. Paper 439, 13 pages. https://doi.org/10.1145/3290605.3300669 — Kozubaev et al.’s work adds to a growing body of work questioning and reframing what the “home” means in relation to smart home technology. The authors conduct design workshops with residents (and some managers) in US public housing, providing insight into housing situations where (1) the “home” is not a single-family middle class grouping, and (2) the potential end users of smart home technologies may not have control or consent over the technologies used, and are already subject to various forms of state surveillance.
• Shruthi Sai Chivukula, Chris Watkins, Lucca McKay, and Colin M. Gray. 2019. “Nothing Comes Before Profit”: Asshole Design In the Wild. Paper LBW1314, 6 pages. https://doi.org/10.1145/3290607.3312863 — This late breaking work by Chivukala et al investigates the /r/assholedesign subreddit to explore the concept of “asshole design,” particularly in comparison to the concept of “dark patterns.” They find that asshole design uses some dark pattern strategies, but that dark patterns tend to trick users into doing certain things, while asshole design often restricts uses of products and more often include non-digital artifacts. I think there may be an interesting future regulatory discussion about asshole design (and dark patterns). On one hand, one might consider whether dark pattern or asshole design practices might fit under the FTC’s definition of “unfair and deceptive practices” for possible enforcement action against companies. On the other, as some legislators are introducing bills to ban the use of dark patterns – it becomes very important to think carefully about how dark patterns are defined, and what might get included and excluded in those definitions; the way that this work suggests a set of practices related to, but distinct from, dark patterns could help inform future policy discussions.

## June 29, 2019

Ph.D. student

#### Life update: new AI job

I started working at a new job this month. It is at an Artificial Intelligence startup. I go to an office, use GitHub and Slack, and write software, manipulate data, and manage cloud computing instances for a living. As at this point I am relatively senior as an employee, I’m also involved in meetings of a managerial nature. There are lots of questions about how we organize ourselves and how we interact with other companies that I get to weigh in on.

This is a change from being primarily a postdoctoral researcher or graduate student. That change is apparent even though during my time as a latter I was doing similar industrial work on a part-time basis. Now, at the startup, the purpose of my work is more clearly oriented towards our company’s success.

There is something very natural about this environment for me. It is normal. I am struck by this normality because I have for years been interacting with academics who claim to be studying the very thing that I’m now doing.

I have written a fair bit here about “AI Ethics”. Much of this has been written with frustration at the way the topic is “studied”. In retrospect, a great deal of “AI Ethics” literature is about how people (the authors) don’t like the direction “the conversation” is going. My somewhat glib attitude towards it is that the problem is that most people talking about “AI Ethics” don’t know what they are talking about, and don’t feel like they have to know what they are talking about to have a good point of view on the subject. “AI Ethics” is often an expression of the point of view that while those that are “doing” AI are being somehow inscrutable and maybe dangerous, they should be tamed into accountability towards those who are not doing it, and therefore don’t really know about it. In other words, AI Ethics, as a field, is a way of articulating the interest of one class of people with one relationship to capital to another class of people with a different relationship to capital.

Perhaps I am getting ahead of myself. Artificial Intelligence is capital. I mean that in an economic sense. The very conceit that it is possible to join an “AI Startup”, whose purpose is to build an AI and thereby increase the productivity of its workers and its value to shareholders, makes the conclusion–“AI is capital”–a tautological one. Somehow, this insight rarely makes it into the “AI Ethics” literature.

I have not “left academia” entirely. I have some academic projects that I’m working on. One of these, in collaboration with Bruce Haynes, is a Bourdieusian take on Contextual Integrity. I’m glad to be able to do this kind of work.

However, one source of struggle for me in maintaining an academic voice in my new role, aside from the primary and daunting one of time management, is that many of the insights I would bring to bear on the discussion are drawn from experience. The irony of a training in qualitative and “ethnographic” research into use of technology, with all of its questions of how to provide an emic account based on the testimony of informants, is that I am now acutely aware of how my ability to communicate is limited, transforming me from a “subject” of observation into, in some sense, an “object”.

I enjoy and respect my new job and role. I appreciate that, being a real company trying to accomplish something and not a straw man used to drive a scholarly conversation, “AI” means in our context a wide array of techniques–NLP, convex optimization, simulation, to name a few–smartly deployed in order to best complement the human labor that’s driving things forward. We are not just slapping a linear regression on a problem and calling it “AI”.

I also appreciate, having done work on privacy for a few years, that we are not handling personal data. We are using AI technologies to solve problems that aren’t about individuals. A whole host of “AI Ethics” issues which have grown to, in some corners, change the very meaning of “AI” into something inherently nefarious, are irrelevant to the business I’m now a part of.

Those are the “Pros”. If there were any “Cons”, I wouldn’t be able to tell you about them. I am now contractually obliged not to. I expect this will cut down on my “critical” writing some, which to be honest I don’t miss. That this is part of my contract is, I believe, totally normal, though I’ve often worked in abnormal environments without this obligation.

Joining a startup has made me think hard about what it means to be part of a private organization, as opposed to a public one. Ironically, this public/private institutional divide rarely makes its way into academic conversations about personal privacy and the public sphere. That’s because, I’ll wager, academic conversations themselves are always in a sense public. The question motivating that discourse is “How do we, as a public, deal with privacy?”.

Working at a private organization, the institutional analogue of privacy is paramount. Our company’s DNA is its intellectual property. Our company’s face is its reputation. The spectrum of individual human interests and the complexity of their ordering has its analogs in the domain of larger sociotechnical organisms: corporations and the like.

Paradoxically, there is no way to capture these organizational dynamics through “thick description”. It is also difficult to capture them through scientific modes of visualization. Indeed, one economic reason to form an AI startup is to build computational tools for understanding the nature of private ordering among institutions. These tools allow for comprehension of a phenomenon that cannot be easily reduced to the modalities of sight or speech.

I’m very pleased to be working in this new way. It is in many ways a more honest line of work than academia has been for me. I am allowed now to use my full existence as a knowing subject: to treat technology as an instrument for understanding, to communicate not just in writing but through action. It is also quieter work.

## June 01, 2019

MIMS 2012

#### Interface Lovers Interview

Last week I was featured on Interface Lovers, a site that “put[s] the spotlight on designers who are creating the future and touching the lives of many.” Read my response to their first question about what led me into design.

## What led you into design?

In some ways, I feel like I’ve always been designing, and in other ways, I feel like I stumbled into it without realizing it. I’ve been into art and drawing since I could hold a pencil, taking art classes and doodling throughout my childhood. Then in high school, I signed up for a web design class. The summer before the class even started I was so excited that I bought a book on web development — “Learn HTML in 24 hours” — and taught myself how to build web pages. By the time the school year started, I had already put a website online. Being able to create something that anyone, anywhere in the world could immediately see was completely intoxicating to me.

From there, I went down a rabbit hole of learning Photoshop, Illustrator, 3D modeling, Flash, and any creative technologies even vaguely related to web design. That led me to get a degree in Graphic Communication at Cal Poly, San Luis Obispo, with a concentration in new media. Back then (early 2000s), there weren’t many web design programs, and the ones that existed were shoe-horned into graphic design and art programs. Cal Poly’s graphic communication program was the most technical of the bunch.

As part of my degree at Cal Poly, I took a computer science class and learned C and Java. I found programming to be super fun, too and went deeper down the stack into backend technologies and database development. Basically, anything tangentially related to web development interested me, so I took every class I could.

After college, I went down the programming path and got a job as a data warehouse developer. I went technical because the analytical nature of it meant you know if your work is good — it either works, or it doesn’t. I found design to be very subjective and didn’t feel confident that my web design work was “good” (however that might be measured).

I joined a small company, so I was doing database design, ETLs, backend programming, frontend programming, and UI design. Over time I discovered that updating the interface, even minor updates, elicited strong positive reactions and gratitude from customers, whereas re-factoring a database to cut query times in half rarely did. I realized I wanted to work closer to the customer.

I started spending more time designing user interfaces and studying usability testing. I discovered it married the analytical, scientific part of my brain (which drew me to programming in the first place) to the subjective, intuitive part. This was the tool I needed to “prove” my designs were “right” (which I now know isn’t exactly true, but it felt this way back then).

This made me want to formally study design, so I got my master’s degree at UC Berkeley’s School of Information. The program is the study of technology, broadly speaking — how technology impacts society, how it changes people and their lives, and how to build technology with the needs of people at the center of it. The program was great. It only had a few required classes, then you could take basically whatever you wanted. So I took classes that sounded the most fun and interesting — design, programming, psychology, research, product development, business, and more. I learned a ton about product development and user-centered design while I was there.

One of my favorite classes was behavioral economics for the web class, in which we explored how to apply behavioral economics principles to web sites and use A/B testing to measure their impact. That led me to join Optimizely after grad school, which at the time (2012) was just a simple A/B testing product for the web. I started out doing UI engineering, then switched into product design as the company grew. When I officially became a product designer I felt like I fell into it by accident. It was a result of what the company needed as it grew, not my specific career goal. But when I looked back over what led me there I realized I had always been designing in one way or another.

The company was growing fast, so I was presented the opportunity to move into management. I was resistant at first, but when I realized I could have a bigger impact in that position, I jumped on it. Eventually, my boss left, and I became the Head of Design, leading a team of product designers, user researchers, and UI engineers.

After 5 and a half years at Optimizely, I was ready for a break and new challenges, so I left and took some time off. I realized I wanted to be hands-on again and ended up joining Casetext as a product designer. They’re building legal research tools for lawyers, which pushed me to be a better designer because I was designing for people with expertise I don’t have and can’t acquire.

After a few months, it wasn’t the right fit, so now I’m at Gladly managing their product design team. It feels great to be in management again, working cross-functionally to deliver great experiences to our customers, while growing and nurturing the talents of my team.

Read the full interview on Interface Lovers.

## May 27, 2019

MIMS 2014

#### My Take: WordPress vs. Shopify

WordPress (plus WooCommerce) and Shopify are two of the most popular out-of-the-box e-commerce website solutions out there. Both platforms let you to sell products on the internet without needing to design your own website. But no matter which route you go, good analytics will always be necessary to spot trouble spots in user traffic, and to identify areas for potential growth. With their widespread use, I wanted to re-do my old analytics implementations (this post and this post) in both WordPress and Shopify and give my take on the two platforms.

Set-up/Installation

Winner: Shopify

In terms of getting off the ground with an e-commerce store, Shopify is the more straightforward of the two. It is specifically designed for e-commerce whereas WordPress is a more wide-ranging platform that has been adapted for e-commerce through plug-ins, the most popular among them being WooCommerce. To get set-up with WordPress + WooCommerce, you have to find somewhere to host your site, install WordPress on the server, and configure all the additional necessary plug-ins. Since I was just playing around, I ran everything through MAMP on localhost. Since I’ve set up my own server quite a few times at this point, I didn’t find this too difficult, but there’s no doubt that Shopify makes the process more simple by hosting the site for you.

Winner: Close, but WordPress

Both WordPress and Shopify offer pretty easy integration of the Google Analytics Suite into your e-commerce site. The awesome plug-in, GTM4WP, is great for getting all three of the products I wanted (Tag Manager, Analytics, and Optimize) up and running on WordPress without having to touch any code. Shopify lets you add GA through a configuration on their preference panel. For GTM, you have to directly edit your site’s `theme.liquid` file. But even once you do this, GTM won’t fire on your checkout page unless you’re a Shopify Plus customer who is allowed to edit your checkout.liquid file as well.

Enter the workaround found by the very knowledgable Julian Jeunemann (founder of the fantastic resource Measureschool) for getting GTM to fire properly on all Shopify pages (including the checkout page). The workaround involves adding the GTM container script as custom javascript to run as part of Shopify’s existing GA integration. The solution seems to work; the GTM tag fires on the checkout page (though you need to remove it from the head tag in the `theme.liquid` file so that GTM doesn’t fire twice on all pages). Meanwhile, Google Optimize can be implemented through GTM—though I found that the Optimize tag mysteriously wouldn’t fire on the checkout page. So using the workaround is fine as long as an A/B test doesn’t depend on the checkout page.

Given the somewhat hack-y workaround required to get the Google Analytics Suite going in Shopify, I’m gonna have to go with WordPress for more seamless Google integration with GTM4WP.

Flexibility/Customization

Winner: WordPress

I wanted to try a different A/B test this time round in place of the red button/green button experiment from my prior post. This time I imagined an A/B test where an e-commerce site would test the presence of a coupon code on its checkout page to see whether it would boost their revenue. Like before, I used selenium to simulate traffic to my site and make sure everything was being tracked properly in GA.

To make things a bit more challenging, I wanted to generate a random coupon code for each user rather than just use the same code for everyone. In WordPress, coupon codes are stored in a backend database; in order for a code to be properly applied, it must be written to the database when it’s generated. Luckily, I could use a plug-in, PHP Code Widget, to 1) randomly generate the coupon code when the checkout page loads, 2) write it to the database, and 3) display it to the user.

An extra step is required so that when a user enters the coupon code, the event can be properly tracked by Google Analytics and propagate forward to Google Optimize for the A/B test. Basically, each user’s coupon code must be passed to GTM so that GTM can verify the code was entered correctly. I pass the coupon code to GTM by pushing to the `dataLayer` via javascript. Fortunately, with the PHP code widget, I can execute javascript in PHP with a simple echo call:

`echo "dataLayer.push({'coupon_code': '\$coupon_code' });";`

When WordPress renders the page, this script gets written into the HTML page source and is run in javascript by the browser. Now GTM knows the user’s unique coupon code. From there, a trigger is set in GTM that fires when a user submits the coupon code. A custom javascript variable configured in GTM verifies whether the code was entered correctly and the resulting value (either true/false) is passed onto GA as a Goal that is then used as the target metric in a Google Optimize A/B test.

Trying to replicate this same experiment in Shopify is a challenge. Displaying widgets in Shopify is an add-on feature you have to pay for (in WordPress it’s free). And trying to generate coupon codes individually for each user at page-load time seems to involve wading far too deep into Shopify’s closed source theme files (which they can change on you at any time). This level of customization/flexibility just isn’t what Shopify is built for. Personally, I prefer WordPress for the extra visibility it gives you under the hood. You might lose some convenience that way, but you gain much more control.

Cost

Winner: WordPress

This might be the most important consideration for most—and the area in which I think WordPress really comes out ahead. With WordPress, you still have to pay someone to host your site, but use of the platform itself is free. GoDaddy has introductory hosting plans as low as \$7/month and yearly SSL certificates for \$38/yr (SSL certificates are crucial for a secure store). Shopify’s most basic plan, by contrast, is \$29/month—but then you’ll have to pay for any plug-ins you need to get the functionality you want. So if you want to keep your website administration costs slim, WordPress is the way to go.

## May 19, 2019

MIMS 2012

#### What to Expect if I'm Your Manager

This past January I started my new gig at Gladly, managing the product design team. Unlike at Optimizely, where I transitioned into managing people I already worked with, at Gladly I inherited a team who didn’t know me at all. Inspired by my new boss who did the same thing, I wrote a document to describe what my new team could expect from me as their manager. I decided to re-publish that document here. Enjoy.

This doc is an accelerator in building our relationship. It will take a little while for us to find our rhythm, but we can try to short-circuit the storming phase and get some things on the table from the get go. I look forward to learning similar things about you — at your time.

Some of these bullet points are aspirational. They are standards of behavior that I’m trying to hold myself accountable to. If you ever believe I’m falling short, please tell me.

• My goal is to provide an environment where you can do your best work.
• I will support and encourage you in doing your best work, not tell you what to do. I want each of you to be autonomous and to make your own decisions. This means you may occasionally make mistakes, which is perfectly fine. I’ll be there to help you pick up the pieces.
• In supporting you doing your best work, I will help remove roadblocks that are getting in your way.
• I like to listen and gather context, data, and understanding before making decisions or passing judgement. This means I may ask you a lot of questions to build my knowledge. It also means I will sometimes stay quiet and hold the space for you to keep speaking. It doesn’t mean I’m questioning you, your abilities, or your choices.
• I don’t like to waste my time, and I don’t want to waste yours. If you ever feel like a meeting, project, etc., isn’t a good use of your time, please tell me.
• I take a lot of notes, and write a lot of documents to codify conversations.
• I’m biased towards action and shipping over perfection and analysis paralysis. There’s no better test of a product or feature than getting it in the hands of real users, then iterating and refining.
• I will try to give you small, frequent feedback (positive and negative) in the moment, when it’s fresh, and in person. I don’t like batching up feedback for 1:1s or performance reviews, which turns those into dreadful affairs that no one enjoys, and leads to stale, ineffective feedback. If you have a preferred way of receiving feedback, please tell me.
• My goal is to give you more positive feedback than critical feedback. There’s always positive feedback to give. And positive feedback helps tell you what you’re doing well, and to keep doing it. If you feel like I haven’t given you positive feedback recently, please tell me.
• I like feedback to be a 2-way street, so if there’s anything I’m doing that you don’t like, annoys you, etc., please let me know. If there’s things that I’m doing well that you want me to keep doing, also let me know! Feel free to pull me aside, or tell me in our 1:1s.
• 1:1s are your meetings. You own the agenda. They can be as structured or unstructured as you want. I will occasionally have topics to discuss, but most of the time your items come first.
• You own your career growth. I am there to support and encourage you and to help you find opportunities for growth, but ultimately you’re in control of your career.
• I trust you to make good decisions, get your work done, and use your time wisely. I trust you to not abuse our unlimited vacation policy and to take time off when you need it. If you haven’t taken any time off in awhile, I’ll probably encourage you to take a vacation :) I’m not particularly worried about what hours you work, or where you work, as long as you’re getting your work done (and aren’t working too much).
• Finally, here’s a list of my beliefs about design.

## May 10, 2019

Ph.D. student

#### Where’s the Rest of Design? Or, Bringing Design to the Privacy Table: Broadening “Design” in “Privacy by Design” Through HCI [Paper Talk]

This post is based on a talk given at the 2019 ACM CHI Conference on Human Factors in Computing Systems (CHI 2019), in Glasgow, UK. The full research paper by Richmond Wong and Deirdre Mulligan that the talk is based on, “Bringing Design to the Privacy Table: Broadening “Design” in “Privacy by Design” Through the Lens of HCI” can be found here: [Official ACM Version] [Open Access Pre-Print Version]

In our paper “Bringing Design to the Privacy Table: Broadening Design in Privacy by Design,” we conduct a curated literature review to make two conceptual argument arguments:

1. There is a broad range of design practices used in human computer interaction (HCI) research which have been underutilized in Privacy By Design efforts.
2. Broadening privacy by design’s notion of what “design” can do can help us more fully address privacy, particularly in situations where we don’t yet know what concepts or definitions of privacy are at stake.

But let me start with some background and motivation. I’m both a privacy researcher—studying studying how to develop technologies that respect privacy—and I’m a design researcher, who designs things to learn about the world.

I was excited several years ago to hear about a growing movement called “Privacy By Design,” the idea that privacy protections should be embedded into products and organizational practice during the design of products, rather than trying to address privacy retroactively. Privacy By Design has been put forward in regulatory guidance from the US and other countries, and more recently by the EU’s General Data Protection Regulation. Yet these regulations don’t provide a lot of guidance about what Privacy By Design means in practice.

In interactions with and field observations of the interdisciplinary Privacy By Design community—including lawyers, regulators, academics, practitioners, and technical folks—I’ve  found that there is a lot of recognition of the complexity of privacy: that it’s an essentially contested concept, there are many conceptualizations of privacy; privacy from companies is different than privacy from governments; there are different privacy harms, and so forth.

Privacy by Design conceptualizes “design” in a relatively narrow way

But the discussion of “design” seems much less complex. I had assumed Privacy By Design would have meant applying HCI’s rich breadth of design approaches toward privacy initiatives – user centered design, participatory design, value sensitive design, speculative design, and so on.

Instead, design seemed to be used narrowly, as either a way to implement the law via compliance engineering, or to solve specific privacy problems. Design was largely framed as a deductive way to solve a problem, using approaches such as encryption techniques or building systems to comply with fair information practices. While these are all important and necessary privacy initiatives, but I kept finding myself asking, “where’s the rest of design?” Not just the deductive problem solving aspects of design, but also its the inductive, exploratory, and forward looking aspects.

There’s an opportunity for Privacy By Design to make greater use of the breadth of design approaches used in HCI

There’s a gap here between the way the Privacy By Design views design and the way the HCI community views design. Since HCI researchers and practitioners are in a position to help support or implement privacy by design initiatives, it’s important to try to help broaden the notion of design in Privacy By Design to more fully bridge this gap.

So our paper aims to fulfill 2 goals:

1. Design in HCI is more than just solving problems. We as HCI privacy researchers can more broadly engage the breadth of design approaches in HCI writ large. And there are opportunities to build connections among the HCI privacy research community and HCI design research community & research through design community to use design in relation to privacy in multiple ways.
2. Privacy By Design efforts risk missing out on the full benefits that design can offer if it sticks with a narrower solution and compliance orientation to design. From HCI, we can help build bridges with interdisciplinary Privacy By Design community, and engage them in understanding a broader view of design.

So how might we characterize the breadth of ways that HCI uses design in relation to privacy? In the paper, we conduct a curated review of HCI research to explore and breadth and richness of how design practices are used in relation to privacy. We searched for HCI papers that use both the terms “privacy” and “design,” curating a corpus of 64 papers. Reading through each paper, we openly coded each one by asking a set of questions including: Why is design used; who is design done by; and for whom is design done? Using affinity diagramming on the open codes, we came up with a set of categories, or dimensions, which we used to re-code the corpus. In this post I’m going to focus on the dimensions that emerged when we looked at the “why design?” question, which we call the purposes of design.

We use 4 purposes to discuss the breadth of reasons why design might be used in relation to privacy

We describe 4 purposes of design. They are:

• Design to solve a privacy problem;
• Design to inform or support privacy;
• Design to explore people and situations; and
• Design to critique, speculate, and present critical alternatives.

Note that we use these to talk about how design has been used in privacy research specifically, not about all design writ large (that would be quite a different and broader endeavor!). In practice these categories are not mutually exclusive, and are not the only way to look at the space, but looking at them separately helps give some analytical clarity.  Let’s briefly walk through each of these design purposes.

## To Solve a Privacy Problem

First, design is seen as a way to solve a privacy problem – which occurred most often in the papers we looked at. And I think this is often how we think about design colloquially, as a set of practices to solve problems. This is often how design is discussed in Privacy By Design discussions as well.

When viewing design in this way, privacy is presented a problem that has already been well-defined at the before the design process, and a solution is designed to address that definition of the problem. A lot of responsibility for protecting privacy here is thus placed in the technical system.

For instance, if a problem of privacy is defined as the harms that result from long term data processing and aggregation, we might design a system that limits data retention. If a problem of privacy is defined as not being identified, we might design a system to be anonymous.

## To Inform or Support Privacy

Second, design is seen as a way to inform or support actors who must make privacy-relevant choices, rather than solving a privacy problem outright. This was also common in our set of papers.  Design to inform or support privacy views problems posed by privacy as an information or tools problem. If users receive information in better ways, or have better tools, then they can make more informed choices about how to act in privacy-preserving ways.

A lot of research has been done on how to design usable privacy policies or privacy notices – but it’s still up to the user to read the notice and make a privacy relevant decision. Other types of design work in this vein includes designing privacy icons, controls, dashboards, visualizations, as well as educational materials and activities.

In these approaches, a lot of responsibility for protecting privacy is placed in the choices that people make, informed by a design artifact. The protection of privacy doesn’t arise from the design of the system itself, but rather by how a person chooses to use the system. This orientation towards privacy fits well with US regulations around privacy that make individuals manage and control their own data.

## To Explore People and Situations (Related to Privacy)

Third is using design to explore people and situations. Design is used as a mode of inquiry, to better understand what privacy or the experience of privacy means to certain people, in certain situations. Design here is not necessarily about solving an immediate problem.

Techniques like design probes or collaborative design workshops are some approaches here. For example, a project I presented at CSCW 2018 involved presenting booklets with conceptual designs of potentially invasive products to technology practitioners in training. We weren’t looking to gather feedback in order to develop these conceptual ideas into usable products. Instead, the goal was to use these conceptual design ideas as provocations to better understand the participants’ worldviews. How are they conceptualizing privacy when they see these designs? How do their reactions help us understand where they place responsibility for addressing privacy?

Here, privacy is understood as a situated experience, which emerges from practices from particular groups in specific contexts or situations. The goal is less about solving a privacy problem, and more about understanding how privacy gets enacted and experienced.

## To Critique, Speculate, or Present Critical Alternatives About Privacy

Fourth is design to critique, speculate, or present critical alternatives. (By critical I don’t mean bad or mean, but instead I mean critical like reflexive reflection or careful analysis).  Design here is not about exploring the world as it is, but focuses on how the world could be. Often this consists of creating create conceptual designs that provoke, to create a space to surface and discuss social values. These help us discuss worlds we might strive to achieve or ones we want to avoid. Privacy in this case is situated in different possible sociotechnical configurations of the world, thinking about privacy’s social, legal, and technical aspects together.

For example, in a project I presented at DIS 2017, we created advertisements for fictional sensing products, like a bodily implant for workplace employees. This helped us raise questions beyond basic data collection and use ones. The designs helped us ask questions about how is privacy implicated in the workplace, or through employment law? Can consent really occur with these power dynamics? It also helped us ask normative questions, such as: Who gets to have privacy and who doesn’t? Who or what should be responsible for protecting privacy? Might we look to technical design, to regulations, to market mechanisms, or to individual choice to protect privacy?

## Design Is a Political, Values-Laden Choice

So in summary these are the 4 purposes of design that we identified in this paper: using design to solve, to inform and support, to explore, and to critique and speculate. Again, in practice, they’re not discrete categories. Many design approaches, like user centered design, or participatory design, use design for multiple design purposes.

But this variety of purposes for how design relates to privacy is also a reminder that design isn’t a neutral process, but is itself political and values-laden. (Not political in terms of liberal and conservative, but political in the sense that there is power and social implications in the choices we make about how to use design). Each design purpose suggests a different starting place for how we orient ourselves towards conceptualizing and operationalizing privacy. We might think about privacy as:

• a technical property;
• as situated experiences;
• as privacy as sociotechnically situated.

Privacy can be many and all of these things at once, but the design methods we choose, and the reasons why we choose to use design helps to suggest or foreclose different orientations toward privacy. These choices also suggest that responsibility for privacy might be placed in different places — such as in a technical system, in a person’s choices, in a platform’s policies, in the law, in the market, and so forth.

Research using design to solve and design to inform and support appeared more often in the papers that we looked at

Now I’ve been discussing these 4 design purposes equally, but they weren’t equal in our corpus. Allowing each paper to be coded for multiple categories, a little over half the papers we looked at used design to solve a privacy problem and a little over half used design to inform or support. Less than a quarter used design to explore; even fewer used design to critique and speculate. We don’t claim that the exact percentages are representative of all the privacy literature, but there’s a qualitative difference here, where most of the work we reviewed uses design to solve privacy problems or support and inform privacy.

We are arguing for a big tent approach in privacy by design: using design in all of these ways helps us address a broader set of conceptions of privacy.

This suggests that there’s an opportunity for us to build bridges between the HCI privacy research community, which has rich domain expertise; and the HCI design research & research through design communities, which have rich design methods expertise, particularly using design in ways to explore, and to critique and speculate.

So that’s Argument 1, that we have the opportunity to build new bridges among HCI communities to more fully make use of each others’ expertise, and a broader range of design methods and purposes.

Argument 2 is that Privacy By Design has largely (with some exceptions) thought about design as a problem solving process.  Privacy By Design research and practice could expand on that thinking of design to make more use of a fuller breadth of uses of design that are reflected in HCI.

## Implications for Design Collaboration

So what might some of these collaborations within and across fields look like, if we want to make use of more of design’s breadth? For example if we as privacy researchers, develop a set of usable privacy tools to inform and support most people’s privacy decision making; that might be complemented with design to explore so that we can better understand the often marginalized populations for whom those tools don’t work. For instance Diana Freed et al.’s work shows that social media privacy and security tools can be used against victims of intimate partner violence, violating their privacy and safety. Or, an emerging set of problems we face is thinking about privacy in physically instrumented spaces: how does consent work, what conceptions of privacy and privacy risk are at play? We can complement design to solve and design to support efforts with design to critique and speculate; to craft future scenarios that try to understand what concepts of privacy might be at play, and how privacy can surface differently when technical, social, or legal aspects of the world change.

From a design research perspective, I think there’s growing interest in the design research community to create provocative artifacts to try to surface discussions about privacy, particularly in relation to new and emerging technologies. Critically reflecting on my own design research work, I think it can be tempting to just speak to other designers and resort to conceptions of privacy that say “surveillance is creepy” and not dig deeper into other approaches to privacy. But by collaborating with privacy researchers, we can bring more domain expertise and theoretical depth to these design explorations and speculations, and engage a broader set of privacy stakeholders.

Industry privacy practitioners working on privacy by design initiatives might consider incorporating more UX researchers and designers form their organizations, as privacy allies and as design experts.  Approaches that use design to critique and speculate may also align well with privacy practitioners’ stated desire to find contextual and anticipatory privacy tools to help “think around corners”, as reported by Ken Bamberger and Deirdre Mulligan.

Privacy By Design regulators could incorporate more designers (in addition to engineers and computer scientists) in regulatory discussions about privacy by design, so that this richness of design practice isn’t lost when the words “by design” are written in the law.

Moreover, there’s an opportunity here for us an HCI community to bring HCI’s rich notions of what design can mean to Privacy By Design, so that beyond being a problem solving process, it is also seen as a process that also makes use of the multi-faceted, inductive, and exploratory uses of design that this community engages in.

Paper Citation: Richmond Y. Wong and Deirdre K. Mulligan. 2019. Bringing Design to the Privacy Table: Broadening “Design” in “Privacy by Design” Through the Lens of HCI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). ACM, New York, NY, USA, Paper 262, 17 pages. DOI: https://doi.org/10.1145/3290605.3300492

## April 29, 2019

Ph.D. student

#### Utilizing Design’s Richness in “Privacy by Design”

This post summarizes a research paper, Bringing Design to the Privacy Table, written by Richmond Wong and Deirdre Mulligan. The paper will be presented at the 2019 ACM Conference on Human Factors in Computing Systems (CHI 2019) on Wednesday, May 8 at the 4pm “Help Me, I’m Only Human” paper session.

How might the richness and variety in human computer interaction (HCI) design practices and approaches be utilized in addressing privacy during the development of technologies?

U.S. policy recommendations and the E.U.’s General Data Protection have helped concept of privacy by design (PBD)—embedding privacy protections into products during the initial design phase, rather than retroactively—gain traction. Yet while championing “privacy by design,” these regulatory discussions offer little in the way of concrete guidance about what “by design” means in technical and design practice. Engineering communities have begun developing privacy engineering techniques, to use design as a way to find privacy solutions. Many privacy engineering tools focus on design solutions that translate high level principles into implementable engineering requirements.  However, design in HCI has a much richer concept of what “design” might entail: it also includes thinking about design as a way to explore the world and to critique and speculate about the world. Embracing this richness of design approaches can help privacy by design more fully approach the privacy puzzle.

To better understand the richness of ways design practices can related to privacy, we conducted a curated review of 64 HCI research papers that discuss both privacy and design. One thing we looked at was how each paper viewed the purpose of design in relation to privacy. (Papers could be classified into multiple categories, so percentages add up to over 100). We found four main design purposes:

• To Solve a Privacy Problem (56% of papers) – This aligns with the common perception of design, that design is used to solve problems. This includes creating system architectures and data management systems in ways that collect and use data in privacy-preserving ways. The problems posed by privacy are generally well-defined before the design process; a solution is then designed to address that problem.
• To Inform or Support Privacy (52%) – Design is also used to inform or support people who must make privacy-relevant choices, rather than solving a privacy problem outright. A lot of these papers use design to increase the usability of privacy notices and controls to allow end users to more easily make choices about their privacy. These approaches generally assume that if people have the “right” types of tools and information, then they will choose to act in more privacy-preserving ways.
• To Explore People and Situations (22%) – Design can be used as a form of inquiry to understand people and situations. Design activities, probes, or conceptual design artifacts might be shared with users and stakeholders to understand their experiences and concerns about privacy. Privacy is thus viewed here as relating to different social and cultural contexts and practices; design is used as a way to explore what privacy means in these different situations.
• To Critique, Speculate, or Present Critical Alternatives (11%) – Design can be used to create spaces in which people can discuss values, ethics, and morals—including privacy. Rather than creating immediately deployable design solutions, design here works like good science fiction: creating conceptual designs that try to provoke people into think about relationships among technical, social, and legal aspects of privacy and ask questions such as who gets (or doesn’t get) to have privacy, or who should be responsible for providing privacy.

One thing we found interesting is how some design purposes tend to narrowly define what privacy means or define privacy before the design process, whereas others view privacy as more socially situated and use the process of design itself to help define privacy.

For those looking towards how these dimensions might be useful in privacy by design practice, we mapped our dimensions onto a range of design approaches and methodologies common in HCI, in the table below.

 Design Approach(es) Dimensions of Design Purposes How does design relate to privacy? Software Engineering Solve a problem; Inform and support Conceptions and the problem of privacy solved are defined in advance. Lends itself well to problems related data privacy, or privacy issues to be addressed at a system architecture level. User-Centered Design Solve a problem; Inform and support; Explore Could have conception of privacy defined in advance, or it might surface from users. Lends itself well to individual-based conceptions of privacy Participatory Design; Value Centered Design Solve a problem; Inform and support; Explore; Surface stakeholder conceptions of privacy, involve stakeholders in the design process Resistance, Re-Design, Re-Appropriation Practices Solve a problem; Critique Shows breakdown or contestation in current conceptions of privacy Speculative and Critical Design Explore; Critique Critique current conceptions of privacy, explores and shows potential ways privacy might emerge in new situations

These findings can be of use to several communities:

• HCI privacy researchers and PBD researchers might use this work to reflect on dominant ways in which design has been used thus far (to solve privacy problems, and to inform or support privacy), and begin to explore a broader range of design purposes and approaches in privacy work.
• HCI design researchers might use this work to see how expertise in research through design methods could be married with privacy domain expertise, suggesting potential new collaborations and engagements.
• Industry Privacy Practitioners can begin reaching out to UX researchers and designers in their own organizations both as design experts and as allies in privacy by design initiatives. In particularly, the forward-looking aspects of speculative and critical design approaches may also align well with privacy practitioners’ desire to find contextual and anticipatory privacy tools to help “think around corners”.
• Policymakers should include designers (in addition to engineers and computer scientists) in regulatory discussions about privacy by design (or other “governance by design” initiatives). Many regulators seem to view “design” in “privacy by design” as a way to implement decisions made in law, or as a relatively straightforward way to solve privacy problems. However, this narrow view risks hiding the politics of design; what is left unexamined in these discussions is that different design approaches also suggest different orientations and conceptualizations of privacy. HCI design practices, which have already been used in relation to privacy, suggest a broader set of ways to approach privacy by design.

Our work aims to bridge privacy by design research and practice with HCI’s rich variety of design research. By doing so, we can help encourage more holistic discussions about privacy, drawing connections among privacy’s social, legal, and technical aspects.

Paper Citation:
Richmond Y. Wong and Deirdre K. Mulligan. 2019. Bringing Design to the Privacy Table: Broadening “Design” in “Privacy by Design” Through the Lens of HCI. In CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland UK. ACM, New York, NY, USA, 17 pages. https://doi.org/10.1145/3290605.3300492

This post is crossposted on Medium

## April 19, 2019

Ph.D. 2018

#### ethnography is not the only social science tool for algorithmic impact assessment

Quickly responding to Selbst, Elish, and Latonero’s “Accountable Algorithmic Futures“, Data and Society’s response to the Algorithmic Accountability Act of 2019…

The bill would empower the FTC to do “automated decision systems impact assessment” (ADSIA) of automated decision-making systems. The article argues that the devil is in the details and that the way the FTC goes about these assessments will determine their effectiveness.

The point of their article, which I found notable, is to assert the appropriate intellectual discipline for these impact assessments.

This is where social science comes in. To effectively implement the regulations, we believe that engagement with empirical inquiry is critical. But unlike the environmental model, we argue that social sciences should be the primary source of information about impact. Ethnographic methods are key to getting the kind of contextual detail, also known as “thick description,” necessary to understand these dimensions of effective regulation.

I want to flag this as weird.

There is an elision here between “the social sciences” and “ethnographic methods” here, as if there were no social sciences that were not ethnographic. And then “thick description” is implied to be the only source of contextual detail that might be relevant to impact assessments.

This is a familiar mantra, but it’s also plainly wrong. There’s many disciplines and methods within “the social sciences” that aren’t ethnographic, and many ways to get at contextual detail that does not involve “thick description”. There is a worthwhile and interesting intellectual question: what are the appropriate methods for algorithmic impact assessment. The authors of this piece assume an answer to that question without argument.

## March 26, 2019

Ph.D. 2018

#### Neutral, Autonomous, and Pluralistic conceptions of law and technology (Hildebrandt, Smart Technologies, sections 8.1-8.2)

Continuing notes and review of Part III of Hildebrandt’s Smart Technologies and the End(s) of Law, we begin chapter 8, “Intricate entanglements of law and technology”. This chapter culminates in some very interesting claims about the relationship between law and the printing press/text, which I anticipate provide some very substantive conclusions.

But the chapter warms up by a review of philosophical/theoretical positions on law and technology more broadly. Section 8.2. is structured as a survey of these positions, and in an interesting way: Hildebrandt lays out Neutral, Autonomous, and Pluralistic conceptions of both technology and law in parallel. This approach is dialectical. The Neutral and Autonomous conceptions are, Hildebrandt argues, narrow and naive; the Pluralistic conception captures nuances necessary to understand not only what technology and law are, but how they relate to each other.

#### The Neutral Conception

This is the conception of law and technology as mere instruments. A particular technology is not good or bad, it all depends on how it’s used. Laws are enacted to reach policy aims.

Technologies are judged by their affordances. The goals for which they are used can be judged, separately, using deontology or some other basis for the evaluation of values. Hildebrandt has little sympathy for this view: “I believe that understanding technologies as mere means amounts to taking a naive and even dangerous position”. That’s because, for example, technology can impact the “in-between” of groups and individuals, thereby impacting privacy by its mere usage. This echoes the often cited theme of how artifacts have politics (Winner, 1980): by shaping the social environment by means of their affordances.

Law can also be thought of as neutral instrument. In this case, it is seen as a tool of social engineering, evaluated for its effects. Hildebrandt says this view of law fits “the so-called regulatory paradigm”, which “reigns in policy circles, and also in policy science, which is a social science inclined to take an exclusively external perspective on the law”. The law regulates behavior externally, rather than the actions of citizens internally.

Hildebrandt argues that when law is viewed instrumentally, it is tempting to then propose that the same instrumental effects could be achieved by technical infrastructure. “Techno-regulation is a prime example of what rule by law ends up with; replacing legal regulation with technical regulation may be more efficient and effective, and as long as the default settings are a part of the hidden complexity people simply lack the means to contest their manipulation.” This view is aligned with Lessig’s (2009), which Hildebrandt says is “deeply disturbing”; as it is aligned with “the classical law and economics approach of the Chicago School”, it falls short…somehow. This argument will be explicated in later sections.

Comment

Hildebrandt’s criticism of the neutral conception of technology is that it does not register how technology (especially infrastructure) can have a regulatory effect on social life and so have consequences that can be normatively evaluated without bracketing out the good or bad uses of it by individuals. This narrow view of technology is precisely that which has been triumphed over by scholars like Lessig.

Hildebrandt’s criticism of the neutral conception of law is different. It is that by understanding law primarily by its external effects (“rule by law”) diminishes the true normative force of a more robust legality that sees law as necessarily enacted and performed by people (“Rule of Law”). But nobody would seriously think that “rule by law” is not “neutral” in the same sense that some people think technology is neutral.

The misalignment of these two positions, which are presented as if they are equivalent, obscures a few alternative positions in the logical space of possibilities. There are actually two different views of the neutrality of technology: the naive one that Hildebrandt takes time to dismiss, and the more sophisticated view that technology should be judged by its social effects just as an externally introduced policy ought to be.

Hildebrandt shoots past this view, as developed by Lessig and others, in order to get to a more robust defense of Rule of Law. But it has to be noted that this argument for the equivalence of technology and law within the paradigm of regulation has beneficial implications if taken to its conclusion. For example, in Deirdre Mulligan’s FAT* 2019 keynote, she argued that public sector use of technology, if recognizes as a form of policy, would be subject to transparency and accountability rules under laws like the Administrative Procedure Act.

#### The Autonomous Conception

In the autonomous conception of technology and law, there is no agent using technology or law for particular ends. Rather, Technology and Law (capitalized) act with their own abstract agency on society.

There are both optimistic and pessimistic views of Autonomous Technology. There is hyped up Big Data Solutionism (BDS), and dystopian views of Technology as the enframing, surveilling, overpowering danger (as in, Heidegger). Hildebrandt argues that these are both naive and dangerous views that prevent us from taking seriously the differences between particular technologies. Hildebrant maintains that particular design decisions in technology matter. We just have to think about the implications of those decisions in a way that doesn’t deny the continued agency involved the continuous improvement, operation, and maintenance of the technology.

Hildebrant associates the autonomous conception of law with legal positivism, the view of law as a valid, existing rule-set that is strictly demarcated from either (a) social or moral norms, or (b) politics. The law is viewed as legal conditions for legal effects, enforced by a sovereign with a monopoly on violence. Law, in this sense, legitimizes the power of the state. It also creates a class of lawyers whose job it is to interpret, but not make, the law.

Hildebrandt’s critique of the autonomous conception of law is that it gives the law too many blind spots. If Law is autonomous, it does not need to concern itself with morality, or with politics, or with sociology, and especially not with the specific technology of Information-Communications Infrastructure (ICI). She does not come out and say this outright, but the implication is that this view of Law is fragile given the way changes in the ICI are rocking the world right now. A more robust view of law would give better tools for dealing with the funk we’re in right now.

#### The Pluralistic Conception

The third view of technology and law, the one that Hildebrandt endorses, is the “pluralistic” or “relational” view of law. It does not come as a surprise after the exploration of the “neutral” and “autonomous” conceptions.

The way I like to think about this, the pluralistic conception of technology/law, is: imagine that you had to think about technology and law in a realistic way, unburdened by academic argument of any kind. Imagine, for example, a room in an apartment. Somebody built the room. As a consequence of the dimensions of the room, you can fit a certain amount of furniture in it. The furniture has affordances; you can sit at chairs and eat at tables. You might rearrange the furniture sometimes if you want a different lifestyle for yourself, and so on.

In the academic environment, there are branches of scholarship that like to pretend they discovered this totally obvious view of technology for the first time in, like, the 70’s or 80’s. But that’s obviously wrong. As Winner (1980) points out, when Ancient Greeks were building ships, they obviously had to think about how people would work together to row and command the ship, and built it to be functional. Civil engineering, transportation engineering, and architecture are fields that deal with socially impactful infrastructure, and they have to deal with the ways people react, collectively, to what was built. I can say from experience doing agile development of software infrastructure that software engineers, as well, think about their users when they build products.

So, we might call this the “realistic” view–the view that engineers, who are the best situated to understand the processes of producing and maintaining technology, since that’s their life, have.

I’ve never been a lawyer, but I believe one gets to the pluralistic, or relational, view of law in pretty much the same way. You look at how law has actually evolved, historically, and how it has always been wrapped up in politics and morality and ICI’s.

So, in these sections, Hildebrandt drives home in a responsible, scholarly way the fact that neither law nor technology (especially technological infrastructure, and especially ICI) are autonomous–they are historically situated creates of society–and nor are they instrumentally neutral–they do have a form of agency in their own right.As my comment above notes, to me the most interesting part of this chapter was the gaps and misalignment in the section on the Neutral Conception section. This conception seems most aligned with an analytically clear, normative conception of what law and technology are supposed to be doing, which is what makes this perspective enduringly attractive to those who make them. The messiness or the pluralistic view, while more nuanced, does not provide a guide for design.

By sweeping away the Neutral conception of law as instrumental, Hildebrandt preempts arguments that the law might fail to attain its instrumental goals, or that the goals of law might sometimes be attained through infrastructure. In other words, Hildebrandt is trying to avoid a narrow instrumental comparison between law and technology, and highlights instead that they are relationally tied to each other in a way that prevents either from being a substitute for the other.

References

Hildebrandt, Mireille. Smart technologies and the end (s) of law: novel entanglements of law and technology. Edward Elgar Publishing, 2015.

Lessig, Lawrence. Code: And other laws of cyberspace. ReadHowYouWant. com, 2009.

Winner, Langdon. “Do artifacts have politics?.” Daedalus(1980): 121-136.

## March 25, 2019

Center for Technology, Society & Policy

#### Backstage Decisions, Front-stage Experts: Non-technical Experiences and the Political Engagement of Scientists

by Santiago Molina and Gordon PherriboCTSP Fellows

This is the second in a series of posts on the project “Democratizing” Technology: Expertise and Innovation in Genetic Engineering.

See the first post in the series: Backstage Decisions, Front-stage Experts: Interviewing Genome-Editing Scientists.

Since 2015, scientists, ethicists, and regulators have attempted address the ethical, moral, and social concerns involving genetic modifications to the human germline. Discourse involving these concerns focused on advancing a culture of responsibility and precaution within the scientific community, rather than the creation of new institutional policies and laws. Confidence in scientist’s ability to self-regulate has become increasingly tenuous with the recent news of the birth of genome-edited twins on November 26th, 2018, despite the scientific consensus that such experiments are medically and ethically unwarranted. In response, journalists, social scientists and critical researchers in the life sciences have posed the question: Who should be involved in deciding how genome-editing technologies should be used and for what aims?

In this post, we complicate the idea that technical expertise, which is usually narrowly defined on the basis of professional experience or knowledge, should be the main criteria for having a seat at the table during scientific decision-making. Drawing from eight interviews with scientists who participated in a small meeting held in Napa Valley in 2015, we highlight the role of non-technical experiences in shaping scientists’ views of decision-making about genome editing.

We identify three experiences that have influenced scientists’ views and deliberations about the application and potential consequences and benefits of genetic engineering technologies: 1) reading and group discussions outside of their academic disciplines, 2) direct engagement with patient communities, and 3) involvement in social movements. To wrap up, we make some modest suggestions for what these might mean in the context of STEM education.

## 1. Reading Outside of the Discipline and Group Discussions.

During our interviews we asked scientists how they shaped their viewpoints about biotechnology and its relationship to society. Respondents described their exposure to new viewpoints and reflected on the effect this exposure had on their decision-making. One of the sources of these exposures was reading outside of their academic discipline. We were surprised to hear about how the work of philosophers of science and sociologists of science did inform the decision making of one senior scientist at the Napa Valley meeting. This faculty member discussed their interest in finding opportunities to supplement their laboratory training with philosophical discussions about issues tangential to the science they were working on. With other graduate students, they created a small group that met regularly to discuss concepts and theories in philosophy of science, ethics and sociology of science.

We met- I don’t remember whether it was once a month or once every two weeks to discuss issues around the philosophy and societal issues of science. So we would find books, read books, um from you know – from Bertrand Russell, the philosopher, to Jacob Bronowski to Alfred Lord Whitehead, you know books on the philosophy and the applications of science,

The scientist described that this work added additional layers to their understanding of societal issues related to science. Even though this reading group was instrumental in developing his own awareness of the relationship between science and broader social, political and cultural issues, this respondent also lamented how the opportunity to delve into topics outside of a graduate student’s normal routine, “was not encouraged by any of [their] mentors.” This theme came up in several of our interviews, reinforcing the importance of mentors in shaping how scientists make meaning of their discipline in relation to society, and what educational and professional development opportunities graduate students feel comfortable pursuing outside of their formal training.

## 2. Direct engagement through service.

The most distinctly communicated experiences our interviewees engaged in outside of their formal training were service-related learning experiences that involved direct interaction with communities that would medically benefit from the technology. These experiences appeared to give individuals a greater sense of civic responsibility, and afforded them a more expansive understanding of the relationship between their work and broader communities. For genome-editing researchers, this crucially meant being aware of the social and medical realities of patients that might be research subjects in clinical trials for CRISPR-based therapies.

In our interviews, scientists had direct engagement with people outside of their discipline within national scientific boards, federal organizations, health clinics, and the biotech and pharmaceutical industry. These types of experiences provide an opportunity to collaborate with stakeholders on pressing issues, learn and benefit from industry and market knowledge, and ensure that the outcome of decisions are both relevant and meaningful to community stakeholders outside of the lab.

One of our respondents reflected on how they learned important skills, such as active listening, through professional experiences with indigenous patient communities–which helped this respondent better serve the community’s needs.

I’ve learned a whole lot from the patients I’ve taken care of and the people I’ve met. I certainly learned a great deal from going to the Navajo reservation. I’m – just to be able to sit down in a very different culture and listen and I think it’s very important for doctors to listen to their patients.

This interviewee was additionally committed to modeling the listening behavior of physicians and teaching these listening skills to others. When we further asked “What what do you think was specific about the way that [your mentors] spoke with patients and interacted with them?” the interviewee responded with clarity:

Sitting back and not speaking and letting them talk about what’s important to them.

The interviewee conveyed that if you listen, people will tell you what is most important to them. They further argued that as decision-makers guiding the usage of far-reaching technologies, it is important to not make assumptions about what a particular community needs.

Similarly, in another interview, a molecular biologist described their experience setting up clinical trials and discussing the risks and benefits of an experimental treatment. This experience not only gave them a more concrete sense of what was at stake in the discussions held at the Napa Meeting, but also helped sensitize them towards the lived experiences of the patient communities that may be affected (for better or worse) by genome editing technology. When asked if experiences during their doctoral program, postdoc or work at a biotech firm, had prepared them for discussing genome editing and its implications, the molecular biologist responded:

Having been involved in therapeutic programs in which you’re discussing the pluses and minuses of therapies that can have side effects can prepare you for that. […] To me that was very helpful because it was a very concrete discussion. That conversation was not a like, “oh, I’m an academic and I wanna write a paper and someone’s going to read it and then enough.” […] [In a therapeutic program] the conversation was like, “we have a molecule, are we going to put it in people?” And if the answer is “yes,” like there is a living person on the other end that is going to take that molecule and [they are] going to have to live with the consequences positive and negative. […]

The distinction being drawn here between scientific work with concrete outcomes for people and work with solely academic outcomes, suggests that there are practical experiences that researchers at Universities may only have indirect knowledge of that are important for understanding how the products of science may affect others. As the interviewee further explained, the stakes of being unfamiliar with patient’s experiences are particularly high,

[My work at a biotech firm] has sort of prepared me at least a little bit for some of the discussion around therapeutic editing because different patient populations have wildly different ideas about gene editing. There are certain forms of inherited blindness where people are frankly insulted that you would call it a genetic disease, right? And I think rightly so. That’s their experience. It’s their disease. Why should we call this something that should be quote-unquote “corrected,” right?

In this case, prior experience with clinical trials alerted the researcher towards the heterogeneity of experiences of different patient populations. They further described how, in other interactions with patient advocates through public engagement, they were able to learn a great deal about the uniqueness of each patient group and their different views about genome editing. Here, the researcher additionally conveyed concern over the ableism that is often implicit in medical views of difference. They recounted how listening to perspectives from different patient communities led them to reflect on how procedurally safe genome editing can still cause harm in other ways.

## 3. Involvement in social movements.

The third non-technical form of expertise came from researchers’ political participation. While the recent fervor against the GOP’s “war on science” may give us ample evidence that politics and science don’t mix well, the role of social movements in the creation of scientific knowledge has been extensively documented by sociologists. For example, post World War II environmental movements changed the content, form and meaning of ecological research (Jamison 2006) and Gay Rights and AIDS activists helped steer the direction of biomedical research (Epstein 1996). What is less emphasized in these studies though, is how participation in social movements by scientists can impact their worldview and decision-making. When asked what personal experiences shaped how they thought of the process of decision-making around new biotech, one interviewee mentioned their engagement with political movements in the late 1960’s during anti-Vietnam War protests :

So I was in Berkeley in the late 60s…This is a time of a lot of social activity. Protests that went on against the Vietnam War in favor of civil rights. There was a lot of protest activity going on and I was involved in that to some extent, you know, I went on marches. I went door-to-door one summer in opposition to the Vietnam War…Um, so I had to you know- I had sort of a social equity outlook on life. All the way from my upbringing from college- and then at Berkeley you really couldn’t avoid being involved in some of these social issues.

This respondent went on to discuss how their commitments towards social equity shaped their decision-making around emerging technologies. In another interview, a respondent described how taking time off of their graduate program to work on a local election campaign motivated them to participate in science policy forums later in their career.

However, these example also suggests that how a scientist chooses to engage with social movements can have lasting effects on how they think of themselves as being a part of a larger community. If scientists participate unreflexively, social movements can fail to challenge individual’s to consider how the network building and activism they are doing affects themselves and may be excluding others from different communities.

To give a contemporary example, the March for Science (MfS) movement in January of 2017 protested against the Trump administration’s anti-science policies and actions. While the issues about science funding were urgent, MfS organizers failed to address language issues in MfS that were dismissive of the experience of marginalized communities in science. Whether or not a participant in MfS chose to critically engage in the movement, will influence how this individual sees the world and whether they intentionally or unintentionally reproduce inequities in science. By asking scientists to think about both their role in society and about the community of science itself, social movements provide a large quantity of knowledge and creativity that scientists can contribute to and use as a resource when making decisions and reflecting on the implications of emerging technologies.

## The Value of Non-technical Expertise in Training

Many of the experiences that shaped our interviewees decision-making occurred during their early graduate and professional training. Despite the personal and professional value they found in these experiences, our interviewees noted the lack of support from their graduate mentors in their exploration of non-technical interests and a lack of incentives to participate more broadly in political endeavors during their training. While this may be changing for newer generations of scientists, this raises questions about how scientists in the natural and physical sciences are socialized into the broader scientific community, and the impact of that socialization on what they think of their political responsibilities are.

For example, a consensus study of the National Academies of Sciences, Engineering, and Medicine (2018) found that there is a lack of social and institutional support for activities located outside of the traditional realm of an individual’s discipline and argued for the creation of novel training pathways that could lead to holistic STEM research training. One way of creating more holistic STEM training programs noted by the study that our findings support would be to provide resources and structures to facilitate the connection between graduate training in the life sciences and fields, such as STS, sociology and philosophy. Exposure to these disciplines can help aspiring researchers grapple with the social interactions of their discipline and serve as additional tools for constructive debates around scientific issues. Promoting interdisciplinary collaboration may also help reduce stigma associated with non-traditional pathways to scientific training and provide easier channels to integrate professional development and internship opportunities into the curriculum.

The urgency of this current gap in training is apparent if you look at who is currently at the the decision making table. The committees and meetings for deliberation about the social and ethical issues of genome editing are almost exclusively constituted by senior scientists. These researchers are mainly conscripted into these roles because of their technical expertise and status in disciplinary networks. Historically, the academic institutions these scientists were trained in were not built to prepare scientists for making political decisions or for appreciating the social complexity and nuance that comes with the introduction of emergent technologies into society. In our third blog post we will explore the political stakes of this form of science governance, which are surprisingly high.

References:

Epstein, S. (1996). Impure science: AIDS, activism, and the politics of knowledge (Vol. 7). Univ of California Press.

Jamison, A. (2006). Social movements and science: Cultural appropriations of cognitive praxis. Science as Culture, 15(01), 45-59.

National Academies of Sciences, Engineering, and Medicine (2018) Graduate STEM Education for the 21st Century. Washington, DC: The National Academies Press. doi: https://doi.org/10.17226/25038.

## March 18, 2019

MIMS 2012

#### Adding clarity by removing information

“Where’s the clipboard?”

A customer wrote this in to our support team after using our Copy with cite feature. This feature allows customers to copy snippets of text from a court case, and paste them into the document they’re writing with the case’s citation appended to the end. It’s a huge time saver for lawyers when writing legal documents, and is Casetext’s most heavily used feature.

When I first saw this feedback, I assumed it was an anomaly. The success message says “Copied to clipboard,” but who doesn’t know what the clipboard is? How much clearer could we make the success message?

But then it came in again, and again, and again, until eventually the pattern was undeniable. It was only a small percentage of users overall who were writing in, but it was happening regularly enough that we knew we had to fix it.

The original, confusing toast message that pops up after the user clicks, “Copy with cite.”

To debug this issue, I opened Fullstory (a service that lets you watch back user sessions, among other things) so I could observe the people who wrote in actually use the product. After watching a few sessions, a pattern emerged. People would click “Copy with cite,” zig-zag their mouse around the screen, opening and closing menus, then write in to support to ask, “Where’s the clipboard?” (or some variation thereof).

At first I didn’t understand why they were frantically throwing their cursor around the screen. What were they looking for? After watching many sessions, I finally realized what they were doing: they were looking for the clipboard in Casetext! They thought the “clipboard” was a feature in our product, as opposed to the clipboard on their OS.

Now that I understood the problem, the next challenge was how to fix it. How do we communicate that the clipboard we’re referring to is the system’s clipboard? I started rewriting the text, with options like, “Copied to clipboard. Press ctrl + V to paste.” “Copied to your system’s clipboard.” “Copied to your clipboard” [emphasis added].

But none of these options felt right. They were either too wordy, too technical, or just more confusing. I took a step back and re-examined the problem. The word “clipboard” is what was tripping people up. What if we just removed that word altogether? Could we get away with just saying “Copied” instead? For the people having trouble with this feature it may prevent them from thinking the clipboard is a feature we offer. For people who aren’t getting confused in the first place, this is should be just as clear as saying “Copied to clipboard.”

The refined toast message.

The change felt a little risky, but at the same time it felt right. To validate this would work, I decided to just make the change and see what happens. An A/B test and usability testing would both be correct methods to test my hypothesis, but in this case neither tool was the right fit. An A/B test would have taken too long to get statistically valid results, since the conversion event of “failing” to use the feature was very low. It’s also a difficult conversion event to measure. And a usability test would have been more time-consuming and costly than just shipping the change.

Since the solution was easy to implement (the code change was just removing 13 characters), and the impact if I was wrong was low (the change was unlikely to be worse, and it was easy to reverse course if it was), learning by making a change to the working product was the fastest, cheapest way to go.

After shipping the fix I kept an eye on the support messages coming in. In the following 2 months, only one person was confused about what to do after clicking “Copy with cite,” as compared to the 8 people who had written in in the previous 2 months. Not a perfect record, but an improvement nonetheless!

In this case, the best way to improve the clarity of the UI was to provide less information.

## March 15, 2019

Center for Technology, Society & Policy

#### Symposium: “Governing Machines – Defining and Enforcing Public Policy Values in AI Systems”

CTSP is proud to be a co-sponsor of  the 23rd Annual BCLT/BTLJ Symposium: Governing Machines: Defining and Enforcing Public Policy Values in AI Systems

Algorithms that analyze data, predict outcomes, suggest solutions, and make decisions are increasingly embedded into everyday life. Machines automate content filtering, drive cars and fly planes, trade stocks, evaluate resumes, assist with medical diagnostics, and contribute to government decision-making. Given the growing role of artificial intelligence and machine learning in society, how should we define and enforce traditional legal obligations of privacy, non-discrimination, due process, liability, professional responsibility, and reasonable care?

This symposium will convene scholars and practitioners from law, policy, ethics, computer science, medicine, and social science to consider what roles we should allow machines to play and how to govern them in support of public policy goals.

Co-sponsored by: CTSP, the Center for Long-Term Cybersecurity, and the Algorithmic Fairness and Opacity Working Group (AFOG) at UC Berkeley.

### Bonus!

Two 2017 CTSP fellows will be panelists:

• Amit Elazari on “Trust but Verify – Validating and Defending Against Machine Decisions”
• Uri Hacohen on “Machines of Manipulation”

## March 14, 2019

Ph.D. 2018

#### Antinomianism and purposes as reasons against computational law (Notes on Hildebrandt, Smart Technologies, Sections 7.3-7.4)

Many thanks to Jake Goldenfein for discussing this reading with me and coaching me through interpreting it in preparation for writing this post.

Following up on the discussion of sections 7.1-7.2 of Hildebrandt’s Smart Technologies an the End(s) of Law (2015), this post discusses the next two sections. The main questions left from the last section are:

• How strong is Hildebrandt’s defense of the Rule of Law, as she explicates it, as worth preserving despite the threats to it that she acknowledges from smart technologies?
• Is the instrumental power of smart technology (i.e, its predictive function, which for the sake of argument we will accept is more powerful than unassisted human prognostication) somehow a substitute for Law, as in its pragmatist conception?

In sections 7.3-7.4, Hildbrandt discusses the eponymous ends of law. These are not its functions as could be externally and sociologically validated, but rather its internally recognized goals or purposes. And these are not particular goals, such as environmental justice, that we might want particular laws to achieve. Rather, these are abstract goals that the law as an entire ‘regime of veridiction’ aims for. (“Veridiction” means “A statement that is true according to the worldview of a particular subject, rather than objectively true.” The idea is that the law has a coherent worldview of its own.

Hildebrandt’s description of law is robust and interesting. Law “articulates legal conditions for legal effect.” Legal personhood (a condition) entails certain rights under the law (an effect). These causes-and-effects are articulated in language, and this language does real work. In Austin’s terminology, legal language is performative–it performs things at an institutional and social level. Relatedly, the law is experienced as a lifeworld, or Welt, but not a monolithic lifeworld that encompasses all experience, but one of many worlds that we use to navigate reality, a ‘mode of existence’ that ‘affords specific roles, actors and actions while constraining others’. [She uses Latour to make this point, which in my opinion does not help.] It is interesting to compare this view of society with Nissenbaum’s ((2009) view of society differentiated into spheres, constituted by actor roles and norms.

In section 7.3.2, Hildebrandt draws on Gustav Radbruch for his theory of law. Consistent with her preceding arguments, she emphasizes that for Radbruch, law is antinomian, (a strange term) meaning that it is internally contradictory and unruly, with respect to its aims. And there are three such aims that are in tension:

• Justice. Here, justice is used rather narrowly to mean that equal cases should be treated equally. In other words, the law must be applied justly/fairly across cases. To use her earlier framing, justice/equality implied that legal conditions cause legal effects in a consistent way. In my gloss, I would say this is equivalent to the formality of law, in the sense that the condition-effect rules must address the form of a case, and not treat particular cases differently. More substantively, Hildebrandt argues that Justice breaks down into more specific values: distributive justice, concerning the fair distribution of resources across society, and corrective justice, concerning the righting of wrongs through, e.g., torts.
• Legal certainty. Legal rules must be binding and consistent, whether or not they achieve justice or purpose. “The certainty of the law requires its positivity; if it cannot be determined what is just, it must be decided what is lawful, and this from a position that is capable of enforcing the decision.” (Radbruch). Certainty about how the law will be applied, whether or not the application of the law is just (which may well be debated), is a good in itself. [A good example of this is law in business, which is famously one of the conditions for the rise of capitalism.]
• Purpose. Beyond just/equal application of the law across cases and its predictable positivity, the law aims at other purposes such as social welfare, redistribution of income, guarding individual and public security, and so on. None of these purposes is inherent in the law, for Radbruch; but in his conception of law, by its nature it is directed by democratically determined purposes and is instrumental to them. These purposes may flesh out the normative detail that’s missing in a more abstract view of law.

Two moves by Hildebrandt in this section seem particularly substantial to her broader argument and corpus of work.

The first is the emphasis on the contrast between the antinomian conflict between justice, certainty, and purpose with the principle of legal certainty itself. Law, at any particular point in time, may fall short of justice or purpose, and must nevertheless be predictably applied. It also needs to be able to evolve towards its higher ends. This, for Hildebrandt, reinforces the essential ambiguous and linguistic character of law.

[Radbruch] makes it clear that a law that is only focused on legal certainty could not qualify as law. Neither can we expect the law to achieve legal certainty to the full, precisely because it must attend to justice and to purpose. If the attribution of legal effect could be automated, for instance by using a computer program capable of calculating all the relevant circumstances, legal certainty might be achieved. But this can only be done by eliminating the ambiguity that inheres in human language: it would reduce interpretation to mindless application. From Radbruch’s point of view this would fly in the face of the cultural, value-laden mode of existence of the law. It would refute the performative nature of law as an artificial construction that depends on the reiterant attribution of meaning and decision-making by mindful agents.

Hildebrandt, Smart Technologies, p. 149

The other move that seems particular to Hildebrandt is the connection she draws between purpose as one of the three primary ends of law and purpose-binding a feature of governance. The latter has particular relevance to technology law through its use in data protection, such as in the GDPR (which she addresses elsewhere in work like Hildebrandt, 2014). The idea here is that purposes do not just imply a positive direction of action; they also restrict activity to only those actions that support the purpose. This allows for separate institutions to exist in tension with each other and with a balance of power that’s necessary to support diverse and complex functions. Hildebrandt uses a very nice classical mythology reference here

The wisdom of the principle of purpose binding relates to Odysseus’s encounter with the Sirens. As the story goes, the Sirens lured passing sailors with the enchantment of their seductive voices, causing their ships to crash on the rocky coast. Odysseus wished to hear their song without causing a shipwreck; he wanted to have his cake and eat it too. While he has himself tied to the mast, his men have their ears plugged with beeswax. They are ordered to keep him tied tight, and to refuse any orders he gives to the contrary, while being under the spell of the Sirens as they pass their island. And indeed, though he is lured and would have caused death and destruction if his men had not been so instructed, the ship sails on. This is called self-binding. But it is more than that. There is a division of tasks that prevents him from untying himself. He is forced by others to live by his own rules. This is what purpose binding does for a constitutional democracy.

Hildebrandt, Smart Technologies, p. 156

I think what’s going on here is that Hildebrandt understands that actually getting the GDPR enforced over the whole digital environment is going to require a huge extension of the powers of law over business, organization, and individual practice. From some corners, there’s pessimism about the viability of the European data protection approach (Koops, 2014), arguing that it can’t really be understood or implemented well. Hildebrandt is making a big bet here, essentially saying: purpose-binding on data use is just a natural part of the power of law in general, as a socially performed practice. There’s nothing contingent about purpose-binding in the GDPR; it’s just the most recent manifestation of purpose as an end of law.

Commentary

It’s pretty clear what the agenda of this work is. Hildebrandt is defending the Rule of Law as a social practice of lawyers using admittedly ambiguous natural language over the ‘smart technologies’ that threaten it. This involves both a defense of law as being intrinsically about lawyers using ambiguous natural language, and the power of that law over businesses, etc. For the former, Hildebrandt invokes Radbruch’s view that law is antinomian. For the second point, she connects purpose-binding to purpose as an end of law.

I will continue to play the skeptic here. As is suggested in the quoted package, if one takes legal certainty seriously, then one could easily argue that software code leads to more certain outcomes than natural language based rulings. Moreover, to the extent that justice is a matter of legal formality–attention to the form of cases, and excluding from consideration irrelevant content–then that too weighs in favor of articulation of law in formal logic, which is relatively easy to translate into computer code.

Hildebrandt seems to think that there is something immutable about computer code, in a way that natural language is not. That’s wrong. Software is not built like bridges; software today is written by teams working rapidly to adapt it to many demands (Gürses and Hoboken, 2017). Recognizing this removes one of the major planks of Hildebrandt’s objection to computational law.

It could be argued that “legal certainty” implies a form of algorithmic interpretability: the key question is “certain for whom”. An algorithm that is opaque due to its operational complexity (Burrell, 2016) could, as an implementation of a legal decision, be less predictable to non-specialists than a simpler algorithm. So the tension in a lot of ‘algorithmic accountability’ literature between performance and interpretability would then play directly into the tension, within law, between purpose/instrumentality and certainty-to-citizens.

Overall, the argument here is not compelling yet as a refutation of the idea of law implemented as software code.

As for purpose-binding and the law, I think this may well be the true crux. I wonder if Hildebrandt develops it later in the book. There are not a lot of good computer science models of purpose binding. Tschantz, Datta, and Wing (2012) do a great job mapping out the problem but that research program has not resulted in robust technology for implementation. There may be deep philosophical/mathematical reasons why that is so. This is an angle I’ll be looking out for in further reading.

References

Burrell, Jenna. “How the machine ‘thinks’: Understanding opacity in machine learning algorithms.” Big Data & Society3.1 (2016): 2053951715622512.

Gürses, Seda, and Joris Van Hoboken. “Privacy after the agile turn.” The Cambridge Handbook of Consumer Privacy. Cambridge Univ. Press, 2017. 1-29.

Hildebrandt, Mireille. “Location Data, Purpose Binding and Contextual Integrity: What’s the Message?.” Protection of Information and the Right to Privacy-A New Equilibrium?. Springer, Cham, 2014. 31-62.

Hildebrandt, Mireille. Smart technologies and the end (s) of law: novel entanglements of law and technology. Edward Elgar Publishing, 2015.

Koops, Bert-Jaap. “The trouble with European data protection law.” International Data Privacy Law 4.4 (2014): 250-261.

Nissenbaum, Helen. Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press, 2009.

Tschantz, Michael Carl, Anupam Datta, and Jeannette M. Wing. “Formalizing and enforcing purpose restrictions in privacy policies.” 2012 IEEE Symposium on Security and Privacy. IEEE, 2012.

## March 08, 2019

I filed the following comments today on the CCPA to the CA AG.

March 8, 2019

VIA Email

California Department of Justice
ATTN: Privacy Regulations Coordinator
300 S. Spring St.
Los Angeles, CA 90013

Re: Comments on Assembly Bill 375, the California Consumer Privacy Act of 2018

Dear Attorney General Becerra,

I helped conceive of the high-level policy goals of the privacy initiative that was withdrawn from the ballot with passage of AB 375. Here I provide comment to give context and explain the high-level policy goals of the initiative, in hopes that it helps your office in contemplating regulations for the CCPA.

Strong policy support for the initiative

As you interpret the CCPA, please bear in mind that the initiative would have passed because Americans care about privacy. In multiple surveys, Americans have indicated support for stronger privacy law and dramatic enforcement. Americans have rarely been able to vote directly on privacy, but when they do, they overwhelmingly support greater protections. One example comes from a 2002 voter referendum in North Dakota where 73% of citizens voted in favor of establishing opt-in consent protections for the sale of financial records.[1]

A series of surveys performed at Berkeley found that Americans wanted strong penalties for privacy transgressions. When given options for possible privacy fines, 69% chose the largest option offered, “more than \$2,500,” when “a company purchases or uses someone’s personal information illegally.” When probed for nonfinancial penalties, 38% wanted companies to fund efforts to help consumers protect their privacy, while 35% wanted executives to face prison terms for privacy violations.

Information is different

The CCPA is unusually stringent compared to other regulatory law because information is different from other kinds of services and products. When a seller makes an automobile or a refrigerator, the buyer can inspect it, test it, and so on. It is difficult for the seller to change a physical product. Information-intensive services however are changeable, they are abstract, and since we have no physical experience with information, consumers cannot easily see the flaws and hazards of them in the way one could see an imperfection in a car’s hood.

Because information services can be changed, privacy laws tend to become stringent. Information companies have a long history of changing digital processes to trick consumers and to evade privacy laws in ways that physical product sellers simply could not.[2]

Some of the CCPA’s most derided provisions (e.g. application to household level data) are in response to specific evasions of industries made possible because information is different than product regulation. Here are common examples:

• Sellers claim not to sell personal data with third parties, but then go on to say we “may share information that our clients provide with specially chosen marketing partners.”[3] For this reason, the initiative tightened definitions and required more absolute statements about data selling. Companies shouldn’t use the word “partner” or “service provider” to describe third party marketers.
• Companies have evaded privacy rules by mislabeling data “household-level information.” For instance, the DMA long argued that phone numbers were not personal data because they were associated with a household.
• Many companies use misleading, subtle techniques to identify people. For instance, retailers asked consumers their zip code and used this in combination with their name from credit card swipes to do reverse lookups at data brokers.[4]
• Information companies use technologies such as hash-matching to identify people using “non personal” data.[5]

Careful study of information-industry tricks informed the initiative and resulted in a definitional landscape that attempts to prevent guile. Those complaining about it need only look to the industry’s own actions to understand why these definitions are in place. For your office, this means that regulations must anticipate guile and opportunistic limitations of Californians’ rights.

Creating markets for privacy services was a major goal of the initiative. The ability to delegate opt out rights, for instance, was designed so that Californians could pay a for profit company (or even donate to a non-profit such as EFF) in order to obtain privacy services.

There are important implications of this: first, the market-establishing approach means that more affluent people will have more privacy. This sounds objectionable at first, but it is a pragmatic and ultimately democratizing pro-privacy strategy. A market for privacy cannot emerge without privacy regulation to set a floor for standards and to make choices enforceable. Once privacy services emerge, because they are information services and because they can scale, privacy services will become inexpensive very quickly. For instance, credit monitoring and fraud alert services are only available because of rights given to consumers in the Fair Credit Reporting Act that can be easily invoked by third party privacy services. These services have become very inexpensive and are used by tens of millions of Americans.

Some will argue that the CCPA will kill “free” business models and this will be iniquitous. This reasoning underestimates the power of markets and presents free as the only solution to news. The reality is much more complex. Digital advertising supported services do democratize news access, however, they also degrade quality. One cost of the no-privacy, digital advertising model is fake news. Enabling privacy will improve quality and this could have knock-on effects.

Second, the market strategy relieves pressure on your office. The market strategy means that the AG does not have to solve all privacy problems. (That is an impossible standard to meet and perfection has become a standard preventing us from having any privacy.)

Instead, the AG need only set ground rules that allow pro-privacy services to function effectively. A key ground rule that you should promote is a minimally burdensome verification procedure, so that pro-privacy services can scale and can easily deliver opt out requests. For instance, in the telemarketing context, the FTC made enrolling in the Do-Not-Call Registry simple because it understood that complexifying the process would result in lower enrollment.

There is almost no verification to enroll in the Do-Not-Call Registry and this is a deliberate policy choice. One can enroll by simply calling from the phone number to be enrolled, or by visiting a website and getting a round-trip email. What this means is that online, a consumer can enroll any phone number, even one that is not theirs, so long as they provide an email address. The FTC does not run email/phone number verification.

The low level of verification in the Do-Not-Call Registry is a reflection of two important policy issues: first, excessive verification imposes transaction costs on consumers, and these costs are substantial. Second, the harm of false registrations is so minimal that it is outweighed by the interest in lowering consumer transaction costs. Most people are honest and there is no evidence of systematic false registrations in the Do-Not-Call Registry. More than 200 million numbers are now enrolled.

The AG should look to the FTC’s approach and choose a minimally invasive verification procedure for opt out requests that assumes 1) that most Californians are honest people and will not submit opt out requests without authority, and 2) that verification stringency imposes a real, quantifiable cost on consumers. That cost to consumers is likely to outweigh the interest of sellers to prevent false registrations. In fact, excessive verification could kill the market for privacy services and deny consumers the benefit of the right to opt out. A reasonable opt out method would be one where a privacy service delivers a list of identifiable consumers to a business, for instance through an automated system, or simply a spreadsheet of names and email addresses.

The AG should look to Catalog Choice as a model for opt outs. Catalog Choice has carefully collected all the opt out mechanisms for paper mail marketing catalogs. A consumer can sign up on the site, identify catalogs to opt out from (9,000 of them!), and Catalog Choice sends either an automated email or a structured list of consumers to sellers to effectuate the opt out. This service is free. Data feeds from Catalog Choice are even recognized by data brokers as a legitimate way for consumers to stop unwanted advertising mail. Catalog choice performs no verification of consumer identity. Again, this is acceptable, because the harm of a false opt-out is negligible, and because deterring that harm would make it impossible for anyone to opt out efficiently.

I served on the board of directors of Catalog Choice for years and recall no incidents of fraudulent opt outs. The bigger problem was with sellers who simply would not accept opt outs. A few would summarily deny them for no reason other than that allowing people to opt out harmed their business model, or they would claim that Catalog Choice needed a power of attorney to communicate a user’s opt out. The AG should make a specific finding that a power of attorney or any other burdensome procedure is not necessary for delivering verified opt out requests.

The AG should assume that sellers will use guile to impose costs on opt out requests and to deter them. Recall that when consumer reporting agencies were required to create a free credit report website, CRAs used technical measures to block people from linking to it, so that the consumer had to enter the URL to the website manually. CRAs also set up confusing, competing sites to draw consumers away from the free one. The FTC actually had to amend its rule to require this disclosure on all “free” report sites.

The definition of sell

The definition of sell in the CCPA reflects the initiative’s broad policy goal of stopping guile in data “sharing.”

From a consumer perspective, any transfer of personal information to a third party for consideration is a sale (subject to exceptions for transactional necessity, etc). But the information industry has interpreted “sale” to only mean transfers for money consideration. That is an unfounded, ahistorical interpretation.

The initiative sought to reestablish the intuitive contract law rule that any transfer for value is the “consideration” that makes a data exchange a sale. In the information industry’s case, that valuable consideration is often a barter exchange. For instance, in data cooperatives, sellers input their own customer list into a database in exchange for other retailers’ data.[6] Under the stilted definition of “sale” promoted by the information industry, that is not data selling. But from a consumer perspective, such cooperative ”sharing” has the same effect as a “sale.”

Recent reporting about Facebook makes these dynamics clearer in the online platform context.[7] Properly understood, Facebook sold user data to application developers. If application developers enabled “reciprocity” or if developers caused “engagement” on the Facebook platform, Facebook would give developers access to personal data. From a consumer perspective, users gave their data to Facebook, and Facebook transferred user data to third parties, in exchange for activity that gave economic benefit to Facebook. That’s a sale. The AG should view transfers of personal information for value, including barter and other exchange, as “valuable consideration” under the CCPA. Doing so will make the marketplace more honest and transparent.

Disclosures that consumers understand

Over 60% of Americans believes that if a website has a privacy policy, it cannot sell data to third parties.[8]

I have come to the conclusion, based on a series of 6 large scale consumer surveys and the extensive survey work of Alan Westin, that the term “privacy policy” is inherently misleading. Consumers do not read privacy policies. They see a link to the privacy policy, and they conclude “this website must have privacy.” My work is consonant with Alan Westin’s, who over decades of surveys, repeatedly found that most consumers think businesses handle personal data in a “confidential way.” Westin’s findings imply that consumers falsely believe that there is a broad norm against data selling.

In writing consumer law, one can’t take a lawyer’s perspective. Consumers do not act nor do they think like lawyers. Lawyers think the issue is as simple as reading a disclosure. But to the average person, the mere presence “privacy policy” means something substantive. It looks more like a quality seal (e.g. “organic”) rather than an invitation to read.

This is why the initiative and the CCPA go to such extraordinary measures to inform consumers with “Do not sell my personal information” disclosures. Absent such a clear and dramatic disclosure, consumers falsely assume that sellers have confidentiality obligations.

The CCPA is trying to thread a needle between not violating commercial speech interests and disabusing consumers of data selling misconceptions. These competing interests explain why the CCPA is opt-out for data selling. CCPA attempts to minimize impingement on commercial free speech (in the form of data selling) while also informing consumers of businesses’ actual practices.

Let me state this again: the government interest in commanding the specific representation “Do not sell my personal information,” is necessary to both 1) disabuse consumers of the false belief that services are prohibited from selling their data, and 2) to directly tell consumers that they have to take action and exercise the opt out under CCPA. It would indeed make more sense from a consumer perspective for the CCPA to require affirmative consent. But since that may be constitutionally problematic, the CCPA has taken an opt out approach, along with a strong statement to help consumers understand their need to take action. Without a visceral, dramatic disclosure, consumers will not know that they need to act to protect their privacy. Your regulatory findings should recite these value conflicts, and the need for compelled speech in order to correct a widespread consumer misconception.

Data brokers and opting out

Vermont law now requires data brokers to register, and its registry should help Californians locate opt out opportunities. However, the AG can further assist in this effort by requiring a standardized textual disclosure that is easy to find using search engines. Standardized is important because businesses tend to develop arbitrary terminology that has no meaning outside the industry. Text is important because it is easier to search for words than images, and because logo-based “buttons” carry arbitrary or even conflicting semiotic meaning.

Non-discrimination norms

Section §125 of the CCPA is the most perplexing, yet it is harmonious with the overall intent of the initiative to create markets. My understanding of §125 is that it seeks to 1) prevent platforms such as Facebook from offering a price that is widely divergent from costs. For instance, Facebook’s claims its average revenue per user (ARPU) is about \$100/year in North America. The CCPA seeks to prevent Facebook from charging fees that would be greatly in excess of \$10/month. Thus, the AG could look to ARPU as a peg for defining unreasonable incentive practices. 2) CCPA was attempting to prevent the spread of surveillance capitalism business models into area where information usually is not at play, for instance, at bricks and mortar businesses.

One area to consider under §125 are the growing number of businesses that reject cash payment. These businesses are portrayed as progressive but actually the practice is regressive (consumers spend more when they use plastic, the practice is exclusionary for the unbanked, it subjects consumers to more security breaches, and it imposes a ~3% fee on all transactions).  Consumers probably do not understand that modern payment systems can reidentify them and build marketing lists. The privacy implications of digital payments are not disclosed nor mitigated, and as such, bricks and mortar businesses that demand digital payment may be coercive under CCPA.

Pro-privacy incentives

Privacy laws present a paradox: schemes like the GDPR can induce companies to use data more rather than less. This is because the GDPR’s extensive data mapping and procedural rules may end up highlighting unrealized information uses. The CCPA can avoid this by creating carrots for privacy-friendly business models, something that the GDPR does not do.

The most attractive carrot for companies is an exception that broadly relieves them of CCPA duties. The AG should make the short term transient use exemption the most attractive and usable one. That exception should be interpreted broadly and be readily usable by those acting in good faith. For instance, short-term uses should be interpreted to include retention up to 13 months so long as the data are not repurposed. The broad policy goals of the CCPA are met where an exception gives companies strong pro-privacy incentives. There’s no better one than encouraging companies to only collect data it needs for transactions, and to only keep it for the time needed to ensure anti-fraud, seasonal sales trend analysis, and other service-related reasons. For many businesses, this period is just in excess of one year.

Respectfully submitted,

/Chris Hoofnagle

Chris Jay Hoofnagle*
Adjunct full professor of information and of law
UC Berkeley
*Affiliation provided for identification purposes only

[1] North Dakota Secretary of State, Statewide Election Results, June 11, 2002.

[2] Hoofnagle et al., Behavioral Advertising: The Offer You Can’t Refuse, 6 Harv. L. & Pol’y Rev. 273 (2012).

[3] Jan Whittington & Chris Hoofnagle, Unpacking Privacy’s Price, 90 N.C. L. Rev. 1327 (2011).

[4] Pineda v. Williams Sonoma, 51 Cal.4th 524, 2011 WL 446921.

[6] From Nextmark.com: “co-operative (co-op) database

a prospecting database that is sourced from many mailing lists from many different sources. These lists are combined, de-duplicated, and sometimes enhanced to create a database that can then be used to select prospects. Many co-op operators require that you put your customers into the database before you can receive prospects from the database.

[8] Chris Jay Hoofnagle and Jennifer M. Urban, Alan Westin’s Privacy Homo Economicus, 49 Wake Forest Law Review 261 (2014).

## February 26, 2019

Ph.D. student

#### Response to Abdurahman

Abdurahman has responded to my response to her tweet about my paper with Bruce Haynes, and invited me to write a rebuttal. While I’m happy to do so–arguing with intellectuals on the internet is probably one of my favorite things to do–it is not easy to rebut somebody with whom you have so little disagreement.

Abdurahman makes a number of points:

1. Our paper, “Racial categories in machine learning”, omits the social context in which algorithms are enacted.
2. The paper ignores whether computational thinking “acolytes like [me]” should be in the position of determining civic decisions.
3. That the ontological contributions of African American Vernacular English (AAVE) are not present in the FAT* conference and that constitutes a hermeneutic injustice. (I may well have misstated this point).
4. The positive reception to our paper may be due to its appeal to people with a disingenuous, lazy, or uncommitted racial politics.
5. “Participatory design” does not capture Abdurahman’s challenge of “peer” design. She has a different and more broadly encompassing set of concerns: “whose language is used, whose viewpoint and values are privileged, whose agency is extended, and who has the right to frame the “problem”.”
6. That our paper misses the point about predictive policing, from the perspective of people most affected by disparities in policing. Machine learning classification is not the right frame of the problem. The problem is an unjust prison system and, more broadly the unequal distribution of power that is manifested in the academic discourse itself. “[T]he problem is framed wrongly — it is not just that classification systems are inaccurate or biased, it is who has the power to classify, to determine the repercussions / policies associated thereof and their relation to historical and accumulated injustice?”

I have to say that I am not a stranger most of this line of thought and have great sympathy for the radical position expressed.

I will continue to defend our paper. Re: point 1, a major contribution of our paper was that it shed light on the political construction of race, especially race in the United States, which is absolutely part of “the social context in which algorithmic decision making is enacted”. Abdurahman must be referring to some other aspect of the social context. One problem we face as academic researchers is that the entire “social context” of algorithmic decision-making is the whole frickin’ world, and conference papers are about 12 pages or so. I thought we did a pretty good job of focusing on one, important and neglected aspect of that social context, the political formation of race, which as far as I know has never previously been addressed in a computer science paper. (I’ve written more about this point here).

Re: point 2, it’s true we omit a discussion of the relevance of computational thinking to civic decision-making. That is because this is a safe assumption to make in a publication to that venue. I happen to agree with that assumption, which is why I worked hard to submit a paper to that conference. If I didn’t think computational thinking was relevant, I probably would be doing something else with my time. That said, I think it’s wildly flattering and inaccurate to say that I, personally, have any control over “civic decision-making”. I really don’t, and I’m not sure why you’d think that, except for the erroneous myth that computer science research is, in itself, political power. It isn’t; that’s a lie that the tech companies have told the world.

I am quite aware (re: point 3) that my embodied and social “location” is quite different from Abdurahman’s. For example, unlike Abdurahman, it would be utterly pretentious for me to posture or “front” with AAVE. I simply have no access to its ontological wisdom, and could not be the conduit of it into any discourse, academic or informal. I have and use different resources; I am also limited by my positionality like anybody else. Sorry.

“Woke” white liberals potentially liking our argument? (Re: point 4) Fair. I don’t think that means our argument is bad or that the points aren’t worth making.

Re: point 5: I must be forgiven for not understanding the full depth of Abdurahman’s methodological commitments on the basis of a single tweet. There are a lot of different design methodologies and their boundaries are disputed. I see now that the label of “participatory design” is not sufficiently critical or radical enough to capture what she has in mind. I’m pleased to see she is working with Tap Parikh on this, who has a lot of experience with critical/radical HCI methods. I’m personally not an expert on any of this stuff. I do different work.

Re: point 6: My personal opinions about the criminal justice system did not make it into our paper, which again was a focused scientific article trying to make a different point. Our paper was about how racial categories are formed, how they are unfair, and how a computational system designed for fairness might address that problem. I agree that this approach is unlikely to have much meaningful impact on the injustices of the cradle-to-prison system in the United States, the prison-industrial complex, or the like. Based on what I’ve heard so far, the problems there would be best solved by changing the ways judges are trained. I don’t have any say in that, though–I don’t have a law degree.

In general, while I see Abdurahman’s frustrations as valid (of course!), I think it’s ironic and frustrating that she targets our paper as an emblem of the problems with the FAT* conference, with computer science, and with the world at large. First, our paper was not a “typical” FAT* paper; it was a very unusual one, positioned to broaden the scope of what’s discussed there, motivated in part by my own criticisms of the conference the year before. It was also just one paper: there’s tons of other good work at that conference, and the conversation is quite broad. I expect the best solution to the problem is to write and submit different papers. But it may also be that other venues are better for addressing the problems raised.

I’ll conclude that many of the difficulties and misunderstandings that underlie our conversation are a result of a disciplinary collapse that is happening because of academia’s relationship with social media. Language’s meaning depends on its social context, and social media is notoriously a place where contexts collapse. It is totally unreasonable to argue that everybody in the world should be focused on what you think is most important. In general, I think battles over “framing” on the Internet are stupid, and that the fact that these kinds of battles have become so politically prominent is a big part of why our society’s politics are so stupid. The current political emphasis on the symbolic sphere is a distraction from more consequential problems of economic and social structure.

As I’ve noted elsewhere, one reason why I think Haynes’s view of race is refreshing (as opposed to a lot of what passes for “critical race theory” in popular discussion) is that it locates the source of racial inequality in structure–spatial and social segregation–and institutional power–especially, the power of law. In my view, this politically substantive view of race is, if taken seriously, more radical than one based on mere “discourse” or “fairness” and demands a more thorough response. Codifying that response, in computational thinking, was the goal of our paper.

This is a more concrete and specific way of dealing with the power disparities that are at the heart of Abdurahman’s critique. Vague discourse and intimations about “privilege”, “agency”, and “power”, without an account of the specific mechanisms of that power, are weak.

## February 23, 2019

Ph.D. student

#### Beginning to read “Smart Technologies and the End(s) of Law” (Notes on: Hildebrandt, Smart Technologies, Sections 7.1-7.2)

I’m starting to read Mireille Hildebrandt‘s Smart Technologies and the End(s) of Law (2015) at the recommendation of several friends with shared interests in privacy and the tensions between artificial intelligence and the law. As has been my habit with other substantive books, I intend to blog my notes from reading as I get to it, in sections, in a perhaps too stream-of-consciousness, opinionated, and personally inflected way.

For reasons I will get to later, Hildebrandt’s book is a must-read for me. I’ve decided to start by jumping in on Chapter 7, because (a) I’m familiar enough with technology ethics, AI, and privacy scholarship to think I can skip that and come back as needed, and (b) I’m mainly reading because I’m interested in what a scholar of Hildebrandt’s stature says when she tackles the tricky problem of law’s response to AI head on.

I expect to disagree with Hildebrant in the end. We occupy different social positions and, as I’ve argued before, people’s position on various issues of technology policy appears to have a great deal to do with their social position or habitus. However, I know I have a good deal to learn about legal theory while having enough background in philosophy and social theory to parse through what Hildebrandt has to offer. And based on what I’ve read so far, I expect the contours of the possible positions that she draws out to be totally groundbreaking.

#### Notes on: Hildebrandt, Smart Technologies, §7.1-7.2

“The third part of this book inquires into the implications of smart technologies and data-driven agency for the law.”

– Hildebrandt, Smart Technologies,p.133

Lots of people write about how artificial intelligence presents an existential threat. Normally, they are talking about how a superintelligence is posing an existential threat to humanity. Hildebrandt is arguing something else: she is arguing that smart technologies may pose an existential threat to the law, or the Rule of Law. That is because the law’s “mode of existence” depends on written text, which is a different technical modality, with different affordances, than smart technology.

My take is that the mode of existence of modern law is deeply dependent upon the printing press and the way it has shaped our world. Especially the binary character of legal rules, the complexity of the legal system and the finality of legal decisions are affordances of — amongst things — the ICI [information and communication infrastructure] of the printing press.

– Hildebrandt, Smart Technologies, p.133

This is just so on point, it’s hard to know what to say. I mean, this is obviously on to something. But what?

To make her argument, Hildebrandt provides a crash course in philosophy of law and legal theory, distinguishing a number of perspectives that braid together into an argument. She discusses several different positions:

• 7.2.1 Law as an essentially contested concept (Gallie). The concept of “law” [1] denotes something valuable, [2] covers intricate complexities, that makes it [3] inherently ambiguous and [4] necessarily vague. This [5] leads interested parties into contest over conceptions. The contest is [6] anchored in past, agreed upon exemplars of the concept, and [7] the contest itself sustains and develops the concept going forward. This is the seven-point framework of an “essentially contested concept”.
• 7.2.2 Formal legal positivism. Law as a set of legal rules dictated by a sovereign (as opposed to law as a natural moral order) (Austin). Law as a coherent set of rules, defined by its unity (Kelsen). A distinction between substantive rules and rules about rule-making (Hart).
• 7.2.3 Hermeneutic conceptions. The practice of law is about the creative interpretation of (e.g.) texts (case law, statutes, etc.) to application of new cases. The integrity of law (Dworkin) constrains this interpretation, but the projection of legal meaning into the future is part of the activity of legal practice. Judges “do things with words”–make performative utterances through their actions. Law is not just a system of rules, but a system of meaningful activity.
• 7.2.3 Pragmatist conceptions (Realism legal positivism). As opposed to the formal legal positivism discusses earlier that sees law as rules, realist legal positivism sees law as a sociological phenomenon. Law is “prophecies of what the courts will do in fact, and nothing more pretentious” (Holmes). Pragmatism, as an epistemology, argues that the meaning of something is its practical effect; this approach could be seen as a constrained version of the hermeneutic concept of law.

To summarize Hildebrandt’s gloss on this material so far: Gallie’s “essentially contested concept” theory is doing the work of setting the stage for Hildebrant’s self-aware intervention into the legal debate. Hildebrandt is going to propose a specific concept of the law, and of the Rule of Law. She is doing this well-aware that this act of scholarship is engaging in contest.

#### Punchline

I detect in Hildebrandt’s writing a sympathy or preference for hermeneutic approaches to law. Indeed, by opening with Gallie, she sets up the contest about the concept of law as something internal to the hermeneutic processes of the law. These processes, and this contest, are about texts; the proliferation of texts is due to the role of the printing press in modern law. There is a coherent “integrity” to this concept of law.

The most interesting discussion, in my view, is loaded in to what reads like an afterthought: the pragmatist conception of law. Indeed, even at the level of formatting, pragmatism is buried: hermeneutic and pragmatist conceptions of law are combined into one section (7.2.3), where as Gallie and the formal positivists each get their own section (7.2.1 and 7.2.2).

This is odd, because the resonances between pragmatism and ‘smart technology’ are, in Hildebrandt’s admission, quite deep:

Basically, Holmes argued that law is, in fact, what we expect it to be, because it is this expectation that regulates our actions. Such expectations are grounded in past decisions, but if these were entirely deterministic of future decisions we would not need the law — we could settle for logic and simply calculate the outcome of future decisions. No need for interpretation. Holmes claimed, however, that ‘the life of law has not been logic. It has been experience.’ This correlates with a specific conception of intelligence. As we have seen in Chapter 2 and 3, rule-based artificial intelligence, which tried to solve problems by means of deductive logic, has been superseded by machine learning (ML), based on experience.

– Hildebrandt, Smart Technologies, p.142

Hildebrandt considers this connection between pragmatist legal interpretation and machine learning only to reject it summarily in a single paragraph at the end of the section.

If we translate [a maxim of classical pragmatist epistemology] into statistical forecasts we arrive at judgments resulting from ML. However, neither logic nor statistics can attribute meaning. ML-based court decisions would remove the fundamental ambiguity of human language from the centre stage of the law. As noted above, this ambiguity is connected with the value-laden aspect of the concept of law. It is not a drawback of natural language, but what saves us from acting like mindless agents. My take is that an approach based on statistics would reduce judicial and legislative decisions to administration, and thus collapse the Rule of Law. This is not to say that a number of administrative decisions could not be taken by smart computing systems. It is to confirm that such decisions should be brought under the Rule of Law, notably by making them contestable in a court of law.

– Hildebrandt, Smart Technologies, p.143

This is a clear articulation of Hildebrandt’s agenda (“My take is that…”). It is also clearly an aligning the practice of law with contest, ambiguity, and interpretation as opposed to “mindless” activity. Natural language’s ambiguity is a feature, not a bug. Narrow pragmatism, which is aligned with machine learning, is a threat to the Rule of Law

#### Some reflections

Before diving into the argument, I have to write a bit about my urgent interest in the book. Though I only heard about it recently, my interests have tracked the subject matter for some time.

For some time I have been interested in the connection between philosophical pragmatism and the concerns about AI, which I believe can be traced back to Horkheimer. But I thought nobody was giving the positive case for pragmatism its due. At the end of 2015, totally unaware of “Smart Technologies” (my professors didn’t seem aware of it either…), I decided that I would write my doctoral dissertation thesis defending the bold thesis that yes, we should have AI replace the government. A constitution written in source code. I was going to back the argument up with, among other things, pragmatist legal theory.

I had to drop the argument because I could not find faculty willing to be on the committee for such a dissertation! I have been convinced ever since that this is a line of argument that is actually rather suppressed. I was able to articulate the perspective in a philosophy journal in 2016, but had to abandon the topic.

This was probably good in the long run, since it meant I wrote a dissertation on privacy which addressed many of the themes I was interested in, but in greater depth. In particular, working with Helen Nissenbaum I learned about Hildebrandt’s articles comparing contextual integrity with purpose binding in the GDPR (Hildebrandt, 2013; Hildebrandt, 2014), which at the time my mentors at Berkeley seemed unaware of. I am still working on puzzles having to do with algorithmic implementation or response to the law, and likely will for some time.

Recently, been working at a Law School and have reengaged the interdisciplinary research community at venues like FAT*. This has led me, seemingly unavoidably, back to what I believe to be the crux of disciplinary tension today: the rising epistemic dominance of pragmatist computational statistics–“data science”and its threat to humanistic legal authority, which is manifested in the clash of institutions that are based on each, e.g., iconically, “Silicon Valley” (or Seattle) and the European Union. Because of the explicitly normative aspects of humanistic legal authority, it asserts itself again and again as an “ethical” alternative to pragmatist technocratic power. This is the latest manifestation of a very old debate.

Hildebrandt is the first respectable scholar (a category from which I exclude myself) that I’ve encountered to articulate this point. I have to see where she takes the argument.

So far, however, I think here argument begs the question. Implicitly, the “essentially contested” character of law is due to the ambiguity of natural language and the way in which that necessitates contest over the meaning of words. And so we have a professional class of lawyers and scholars that debate the meaning of words. I believe the the regulatory power of this class is what Hildebrandt refers to as “the Rule of Law”.

While it’s true that an alternative regulatory mechanism based on statistical prediction would be quite different from this sense of “Rule of Law”, it is not clear from Hildebrandt’s argument, yet, why her version of “Rule of Law” is better. The only hint of an argument is the problem of “mindless agents”. Is she worried about the deskilling of the legal profession, or the reduced need for elite contest over meaning? What is hermeneutics offering society, outside of the bounds of its own discourse?

References

Benthall, S. (2016). Philosophy of computational social science. Cosmos and History: The Journal of Natural and Social Philosophy12(2), 13-30.

Sebastian Benthall. Context, Causality, and Information Flow: Implications for Privacy Engineering, Security, and Data Economics. Ph.D. dissertation. Advisors: John Chuang and Deirdre Mulligan. University of California, Berkeley. 2018.

Hildebrandt, Mireille. “Slaves to big data. Or are we?.” (2013).

Hildebrandt, Mireille. “Location Data, Purpose Binding and Contextual Integrity: What’s the Message?.” Protection of Information and the Right to Privacy-A New Equilibrium?. Springer, Cham, 2014. 31-62.

Hildebrandt, Mireille. Smart technologies and the end (s) of law: novel entanglements of law and technology. Edward Elgar Publishing, 2015.

## February 17, 2019

Ph.D. student

#### A few brief notes towards “Procuring Cybersecurity”

I’m shifting research focus a bit and wanted to jot down a few notes. The context for the shift is that I have the pleasure of organizing a roundtable discussion for NYU’s Center for Cybersecurity and Information Law Institute, working closely with Thomas Streinz of NYU’s Guarini Global Law and Tech.

The context for the workshop is the steady feed of news about global technology supply chains and how they are not just relevant to “cybersecurity”, but in some respects are constitutive of cyberinfrastructure and hence the field of its security.

I’m using “global technology supply chains” rather loosely here, but this includes:

• Transborder personal data flows as used in e-commerce
• Software- (and Infrastructure-)-as-a-Service being marketing internationally (including Google used abroad, for example)
• Enterprise software import/export
• Electronics manufacturing and distribution.

Many concerns about cybersecurity as a global phenomenon circulate around the imagined or actual supply chain. These are sometimes national security concerns that result in real policy, as when Australia recently banned Hauwei and ZTE from supplying 5G network equipment for fear that it would provide a vector of interference from the Chinese government.

But the nationalist framing is certainly not the whole story. I’ve heard anecdotally that after the Snowden revelations, Microsoft’s internally began to see the U.S. government as a cybersecurity “adversary“. Corporate tech vendors naturally don’t want to be known as being vectors for national surveillance, as this cuts down on their global market share.

Governments and corporations have different cybersecurity incentives and threat models. These models intersect and themselves create the dynamic cybersecurity field. For example, these Chinese government has viewed foreign software vendors as cybersecurity threats, and has responded by mandating source code disclosure. But as this is a vector of potential IP theft, foreign vendors have balked, seeing this mandate as a threat. (Ahmed and Weber, 2018).Complicating things further, a defensive “cybersecurity” measure can also serve the goal of protecting domestic technology innovation–which can be framed as providing a nationalist “cybersecurity” edge in the long run.

What, if anything, prevents a total cyberwar of all against all? One answer is trade agreements that level the playing field, or at least establish rules for the game. Another is open technology and standards, which provide an alternative field driven by the benefits of interoperability rather than proprietary interest and secrecy. Is it possible to capture any of this in accurate model or theory?

I love having the opportunity to explore these questions, as they are at the intersection of my empirical work on software supply chains (Benthall et al., 2016; Benthall, 2017) and also theoretical work on data economics in my dissertation. My hunch for some time has been that there’s a dearth of solid economics theory for the contemporary digital economy, and this is one way of getting at that.

References

Ahmed, S., & Weber, S. (2018). China’s long game in techno-nationalism. First Monday, 23(5).

Benthall, S., Pinney, T., Herz, J. C., Plummer, K., Benthall, S., & Rostrup, S. (2016). An ecological approach to software supply chain risk management. In 15th Python in Science Conference.

Benthall, S. (2017, September). Assessing software supply chain risk using public data. In 2017 IEEE 28th Annual Software Technology Conference (STC) (pp. 1-5). IEEE.

## February 09, 2019

Ph.D. student

#### Why STS is not the solution to “tech ethics”

“Tech ethics” are in (1) (2) (3) and a popular refrain at FAT* this year was that sensitivity to social and political context is the solution to the problems of unethical technology. How do we bring this sensitivity to technical design? Using the techniques of Science and Technology Studies (STS), argue variously Dobbe and Ames, as well as Selbst et al. (2019). Value Sensitive Design (VSD) (Friedman and Bainbridge, 2004) is one typical STS-branded technique for bringing this political awareness into the design process. In general, there is broad agreement that computer scientists should be working with social scientists when developing socially impactful technologies.

In this blog post, I argue that STS is not the solution to “tech ethics” that it tries to be.

Encouraging computer scientists to collaborate with social science domain experts is a great idea. My paper with Bruce Haynes (1) (2) (3) is an example of this kind of work. In it, we drew from sociology of race to inform a technical design that addressed the unfairness of racial categories. Significantly, in my view, we did not use STS in our work. Because the social injustices we were addressing were due to broad reaching social structures and politically constructed categories, we used sociology to elucidate what was at stake and what sorts of interventions would be a good idea.

It is important to recognize that there are many different social sciences dealing with “social and political context”, and that STS, despite its interdisciplinarity, is only one of them. This is easily missed in an interdisciplinary venue in which STS is active, because STS is somewhat activist in asserting its own importance in these venues. STS frequently positions itself as a reminder to blindered technologists that there is a social world out there. “Let me tell you about what you’re missing!” That’s it’s shtick. Because of this positioning, STS scholars frequently get a seat at the table with scientists and technologists. It’s a powerful position, in sense.

What STS scholars tend to ignore is how and when other forms of social scientists involve themselves in the process of technical design. For example, at FAT* this year there were two full tracks of Economic Models. Economic Models. Economics is a well-established social scientific discipline that has tools for understanding how a particular mechanism can have unintended effects when put into a social context. In economics, this is called “mechanism design”. It addresses what Selbst et al. might call the “Ripple Effect Trap”–the fact that a system in context may have effects that are different from the intention of designers. I’ve argued before that wiser economics are something we need to better address technology ethics, especially if we are talking about technology deployed by industry, which is most of it! But despite deep and systematic social scientific analysis of secondary and equilibrium effects at the conference, these peer-reviewed works are not acknowledged by STS interventionists. Why is that?

As usual, quantitative social scientists are completely ignored by STS-inspired critiques of technologists and their ethics. That is too bad, because at the scale at which these technologies are operating (mainly, we are discussing civic- or web-scale automated decision making systems that are inherently about large numbers of people), fuzzier debates about “values” and contextualized impact would surely benefit from quantitative operationalization.

The problem is that STS is, at its heart, a humanistic discipline, a subfield of anthropology. If and when STS does not deny the utility or truth or value of mathematization or quantification entirely, as a field of research it is methodologically skeptical about such things. In the self-conception of STS, this methodological relativism is part of its ethnographic rigor. This ethnographic relativism is more or less entirely incompatible with formal reasoning, which aspires to universal internal validity. At a moralistic level, it is this aspiration of universal internal validity that is so bedeviling to the STS scholar: the mathematics are inherently distinct from an awareness of the social context, because social context can only be understood in its ethnographic particularity.

This is a false dichotomy. There are other social sciences that address social and political context that do not have the same restrictive assumptions of STS. Some of these are quantitative, but not all of them are. There are qualitative sociologists and political scientists with great insights into social context that are not disciplinarily allergic to the standard practices of engineering. In many ways, these kinds of social sciences are far more compatible with the process of designing technology than STS! For example, the sociology we draw on in our “Racial categories in machine learning” paper is variously: Gramscian racial hegemony theory, structuralist sociology, Bourdieusian theories of social capital, and so on. Significantly, these theories are not based exclusively on ethnographic method. They are based on disciplines that happily mix historical and qualitative scholarship with quantitative research. The object of study is the social world, and part of the purpose of the research is to develop politically useful abstractions from it that generalize and can be measured. This is the form of social sciences that is compatible with quantitative policy evaluation, the sort of thing you would want to use if, for example, understanding the impact of an affirmative action policy.

Given the widely acknowledge truism that public sector technology design often encodes and enacts real policy changes (a point made in Deirdre Mulligan’s keynote), it would make sense to understand the effects of these technologies using the methodologies of policy impact evaluation. That would involve enlisting the kinds of social scientific expertise relevant to understand society at large!

But that is absolutely not what STS has to offer. STS is, at best, offering a humanistic evaluation of the social processes of technology design. The ontology of STS is flat, and its epistemology and ethics are immediate: the design decision comes down to a calculus of “values” of different “stakeholders”. Ironically, this is a picture of social context that often seems to neglect the political and economic context of that context. It is not an escape from empty abstraction. Rather, it insists on moving from clear abstractions to more nebulous ones, “values” like “fairness”, maintaining that if the conversation never ends and the design never gets formalized, ethics has been accomplished.

This has proven, again and again, to be a rhetorically effective position for research scholarship. It is quite popular among “ethics” researchers that are backed by corporate technology companies. That is quite possibly because the form of “ethics” that STS offers, for all of its calls for political sensitivity, is devoid of political substance. An apples-to-apples comparison of “values”, without considering the social origins of those values and the way those values are grounded in political interests that are not merely about “what we think is important in life”, but real contests over resource allocation. The observation by Ames et al. (2011) that people’s values with respect to technology varies with socio-economic class is terribly relevant, Bourdieusian lesson in how the standpoint of “values sensitivity” may, when taken seriously, run up against the hard realities of political agonism. I don’t believe STS researchers are truly naive about these points; however, in their rhetoric of design intervention, conducted in labs but isolated from the real conditions of technology firms, there is an idealism that can only survive under the self-imposed severity of STS’s own methodological restrictions.

Independent scholars can take up this position and publish daring pieces, winning the moral high ground. But that is not a serious position to take in an industrial setting, or when pursuing generalizable knowledge about the downstream impact of a design on a complex social system. Those empirical questions require different tools, albeit far more unwieldy ones. Complex survey instruments, skilled data analysis, and substantive social theory are needed to arrive at solid conclusions about the ethical impact of technology.

References

Ames, M. G., Go, J., Kaye, J. J., & Spasojevic, M. (2011, March). Understanding technology choices and values through social class. In Proceedings of the ACM 2011 conference on Computer supported cooperative work (pp. 55-64). ACM.

Friedman, B., & Bainbridge, W. S. (2004). Value sensitive design.

Selbst, A. D., Friedler, S., Venkatasubramanian, S., & Vertesi, J. (2018, August). Fairness and Abstraction in Sociotechnical Systems. In ACM Conference on Fairness, Accountability, and Transparency (FAT*).

## February 03, 2019

Ph.D. 2016

#### A Machine for Being Frustrated

A MACHINE FOR NECCESSARY FRUSTRATION
An exploration into new mechanisms for DIY jacquard weaving, as well as an ongoing interest in asking how non-human materials or forces can be engaged as collaborators resulted in the prototype of the wind loom—-a modified tapestry loom that with every 4th warp connected to a sail that moves the warp position in and out. The fabrication of the loom was led by Jen Mah and Rachel Bork, who iterated between several prototypes for laser-cut heddle/hooks that can be attached to the yarn, arms are connected to umbrellas that can move when the wind blows, easily attachable and detachable components to support easy travel, and so on. Nearly everything about this design process has been frustrating, from the difficulty of waiting for windy days to test to the stress and anticipation that such a wind loom produces. As I considered a redesign, I began to think about this experience of frustrating and my almost reflexive response to design it away. It has made me wonder if collaborating with the wind ought to be frustrating and if we just stuck through frustration a bit more, then maybe we wouldn’t have some of the negative effects we see emergent from innovation. Rather than seeing this as a “wind loom” I began to think of it as a kind of tool for becoming frustrated and learning how to deal with that emotion. In consolation, you will learn a great deal about the wind patterns in your local region.

## February 02, 2019

Ph.D. student

#### All the problems with our paper, “Racial categories in machine learning”

Bruce Haynes and I were blown away by the reception to our paper, “Racial categories in machine learning“. This was a huge experiment in interdisciplinary collaboration for us. We are excited about the next steps in this line of research.

That includes engaging with criticism. One of our goals was to fuel a conversation in the research community about the operationalization of race. That isn’t a question that can be addressed by any one paper or team of researchers. So one thing we got out of the conference was great critical feedback on potential problems with the approach we proposed.

This post is an attempt to capture those critiques.

#### Need for participatory design

One striking challenge, raised by Khadijah Abdurahman on Twitter, is that we should be developing peer relationships with the communities we research. I read this as a call for participatory design. It’s true this was not part of the process of the paper. In particular, Ms. Abdurahman points to a part of our abstract that uses jargon from computer science.

There are a lot of ways to respond to this comment. The first is to accept the challenge. I would personally love it if Bruce and I could present our research to folks on Roosevelt Island and get feedback from them.

There are other ways to respond that address the tensions of this comment. One is to point out that in addition to being an accomplished scholar of the sociology of race and how it forms, especially in urban settings, Bruce is a black man who is originally from Harlem. Indeed, Bruce’s family memoir shows his deep and well-researched familiarity with the life of marginalized people of the hood. So a “peer relationship” between an algorithm designer (me) and a member of an affected community (Bruce) is really part of the origin of our work.

Another is to point out that we did not research a particular community. Our paper was not human subjects research; it was about the racial categories that are maintained by the Federal U.S. government and which pervade society in a very general way. Indeed, everybody is affected by these categories. When I and others who looks like me are ascribed “white”, that is an example of these categories at work. Bruce and I were very aware of how different kinds of people at the conference responded to our work, and how it was an intervention in our own community, which is of course affected by these racial categories.

The last point is that computer science jargon is alienating to basically everybody who is not trained in computer science, whether they live in the hood or not. And the fact is we presented our work at a computer science venue. Personally, I’m in favor of universal education in computational statistics, but that is a tall order. If our work becomes successful, I could see it becoming part of, for example, a statistical demography curriculum that could be of popular interest. But this is early days.

#### The Quasi-Racial (QR) Categories are Not Interpretable

In our presentation, we introduced some terminology that did not make it into the paper. We named the vectors of segregation derived by our procedure “quasi-racial” (QR) vectors, to denote that we were trying to capture dimensions that were race-like, in that they captured the patterns of historic and ongoing racial injustice, without being the racial categories themselves, which we argued are inherently unfair categories of inequality.

First, we are not wedded to the name “quasi-racial” and are very open to different terminology if anybody has an idea for something better to call them.

More importantly, somebody pointed out that these QR vectors may not be interpretable. Given that the conference is not only about Fairness, but also Accountability and Transparency, this critique is certainly on point.

To be honest, I have not yet done the work of surveying the extensive literature on algorithm interpretability to get a nuanced response. I can give two informal responses. The first is that one assumption of our proposal is that there is something wrong with how race and racial categories are intuitive understood. Normal people’s understanding of race is, of course, ridden with stereotypes, implicit biases, false causal models, and so on. If we proposed an algorithm that was fully “interpretable” according to most people’s understanding of what race is, that algorithm would likely have racist or racially unequal outcomes. That’s precisely the problem that we are trying to get at with our work. In other words, when categories are inherently unfair, interpretability and fairness may be at odds.

The second response is that educating people about how the procedure works and why its motivated is part of what makes its outcomes interpretable. Teaching people about the history of racial categories, and how those categories are both the cause and effect of segregation in space and society, makes the algorithm interpretable. Teaching people about Principal Component Analysis, the algorithm we employ, is part of what makes the system interpretable. We are trying to drop knowledge; I don’t think we are offering any shortcuts.

#### Principal Component Analysis (PCA) may not be the right technique

An objection from the computer science end of the spectrum was that our proposed use of Principal Component Analysis (PCA) was not well-motivated enough. PCA is just one of many dimensionality reduction techniques–why did we choose it in particular? PCA has many assumptions about the input embedded within it, including the component vectors of interest are linear combinations of the inputs. What if the best QR representation is a non-linear combination of the input variables? And our use of unsupervised learning, as a general criticism, is perhaps lazy, since in order to validate its usefulness we will need to test it with labeled data anyway. We might be better off with a more carefully calibrated and better motivated alternative technique.

These are all fair criticisms. I am personally not satisfied with the technical component of the paper and presentation. I know the rigor of the analysis is not of the standard that would impress a machine learning scholar and can take full responsibility for that. I hope to do better in a future iteration of the work, and welcome any advice on how to do that from colleagues. I’d also be interested to see how more technically skilled computer scientists and formal modelers address the problem of unfair racial categories that we raised in the paper.

I see our main contribution as the raising of this problem of unfair categories, not our particular technical solution to it. As a potential solution, I hope that it’s better than nothing, a step in the right direction, and provocative. I subscribe to the belief that science is an iterative process and look forward to the next cycle of work.

Please feel free to reach out if you have a critique of our work that we’ve missed. We do appreciate all the feedback!

## January 16, 2019

Ph.D. student

#### Notes on O’Neil, Chapter 2, “Bomb Parts”

Continuing with O’Neil’s Weapons of Math Destruction on to Chapter 2, “Bomb Parts”. This is a popular book and these are quick chapters. But that’s no reason to underestimate them! This is some of the most lucid work I’ve read on algorithmic fairness.

This chapter talks about three kinds of “models” used in prediction and decision making, with three examples. O’Neil speak highly of the kinds of models used in baseball to predict the trajectory of hits and determine the optimal placement of people in the field. (Ok, I’m not so good at baseball terms). These are good, O’Neil says, because they are transparent, they are consistently adjusted with new data, and the goals are well defined.

O’Neil then very charmingly writes about the model she uses mentally to determine how to feed her family. She juggles a lot of variables: the preferences of her kids, the nutrition and cost of ingredients, and time. This is all hugely relatable–everybody does something like this. Her point, it seems, is that this form of “model” encodes a lot of opinions or “ideology” because it reflects her values.

O’Neil then discusses recidivism prediction, specifically the LSI-R (Level of Service Inventory–Revised) tool. It asks questions like “How many previous convictions have you had?” and uses that to predict likelihood of future prediction. The problem is that (a) this is sensitive to overpolicing in neighborhoods, which has little to do with actual recidivism rates (as opposed to rearrest rates), and (b) e.g. black neighborhoods are more likely to be overpoliced, meaning that the tool, which is not very good at predicting recidivism, has disparate impact. This is an example of what O’Neil calls an (eponymous) weapon of math destruction.(WMD)

She argues that the three qualities of a WMD are Scale, Opacity, and Damage. Which makes sense.

As I’ve said, I think this is a better take on algorithmic ethics than almost anything I’ve read on the subject before. Why?

First, it doesn’t use the word “algorithm” at all. That is huge, because 95% of the time the use of the word “algorithmic” in the technology-and-society literature is stupid. People use “algorithm” when they really mean “software”. Now, they use “AI System” to mean “a company”. It’s ridiculous.

O’Neil makes it clear in this chapter that what she’s talking about are different kinds of models. Models can be in ones head (as in her plan for feeding her family) or in a computer, and both kinds of models can be racist. That’s a helpful, sane view. It’s been the consensus of computer scientists, cognitive scientists, and AI types for decades.

The problem with WMDs, as opposed to other, better models, is that the WMDS models are unhinged from reality. O’Neil’s complaint is not with use of models, but rather that models are being used without being properly trained using sound sampling on data and statistics. WMDs are not artificially intelligences; they are artificial stupidities.

In more technical terms, it seems like the problem with WMDs is not that they don’t properly trade off predictive accuracy with fairness, as some computer science literature would suggest is necessary. It’s that the systems have high error rates in the first place because the training and calibration systems are poorly designed. What’s worse, this avoidable error is disparately distributed, causing more harm to some groups than others.

This is a wonderful and eye-opening account of unfairness in the models used by automated decision-making systems (note the language). Why? Because it shows that there is a connection between statistical bias, the kind of bias that creates distortions in a quantitative predictive process, and social bias, the kind of bias people worry about politically, which consistently uses the term in both ways. If there is statistical bias that is weighing against some social group, then that’s definitely, 100% a form of bias.

Importantly, this kind of bias–statistical bias–is not something that every model must have. Only badly made models have it. It’s something that can be mitigated using scientific rigor and sound design. If we see the problem the way O’Neil sees it, then we can see clearly how better science, applied more rigorously, is also good for social justice.

As a scientist and technologist, it’s been terribly discouraging in the past years to be so consistently confronted with a false dichotomy between sound engineering and justice. At last, here’s a book that clearly outlines how the opposite is the case!

## January 15, 2019

Ph.D. 2015

#### Researchers receive grant to study the invisible work of maintaining open-source software

Researchers at the UC Berkeley Institute for Data Science (BIDS), the University of California, San Diego, and the University of Connecticut have been awarded a grant of \$138,055 from the Sloan Foundation and the Ford Foundation as part of a broad initiative to investigate the sustainability of digital infrastructures. The grant funds research into the maintenance of open-source software (OSS) projects, particularly focusing on the visible and invisible work that project maintainers do to support their projects and communities, as well as issues of burnout and maintainer sustainability. The research project will be led by BIDS staff ethnographer and principal investigator Stuart Geiger and will be conducted in collaboration with Lilly Irani and Dorothy Howard at UC San Diego, Alexandra Paxton at the University of Connecticut, and Nelle Varoquaux and Chris Holdgraf at UC Berkeley.

Many open-source software projects have become foundational components for many stakeholders and are now widely used behind-the-scenes to support activities across academia, the tech industry, government, journalism, and activism. OSS projects are often initially created by volunteers and provide immense benefits for society, but their maintainers can struggle with how to sustain and support their projects, particularly when widely used in increasingly critical contexts. Most OSS projects are maintained by only a handful of individuals, and community members often talk about how their projects might collapse if only one or two key individuals leave the project. Project leaders and maintainers must do far more than just write code to ensure a project’s long-term success: They resolve conflicts, perform community outreach, write documentation, review others’ code, mentor newcomers, coordinate with other projects, and more. However, many OSS project leaders and maintainers have publicly discussed the effects of burnout as they find themselves doing unexpected and sometimes thankless work.

The one-year research project — The Visible and Invisible Work of Maintaining Open-Source Digital Infrastructure — will study these issues in various software projects, including software libraries, collaboration platforms, and discussion platforms that have come to be used as critical digital infrastructure. The researchers will conduct interviews with project maintainers and contributors from a wide variety of projects, as well as analyze projects’ code repositories and communication platforms. The goal of the research is to better understand what project maintainers do, the challenges they face, and how their work can be better supported and sustained. This research on the invisible work of maintenance will help maintainers, contributors, users, and funders better understand the complexities within such projects, helping set expectations, develop training programs, and formulate evaluations.

## January 12, 2019

Ph.D. student

#### Reading O’Neil’s Weapons of Math Destruction

I probably should have already read Cathy O’Neil’s Weapons of Math Destruction. It was a blockbuster of the tech/algorithmic ethics discussion. It’s written by an accomplished mathematician, which I admire. I’ve also now seen O’Neil perform bluegrass music twice in New York City and think her band is great. At last I’ve found a copy and have started to dig in.

On the other hand, as is probably clear from other blog posts, I have a hard time swallowing a lot of the gloomy political work that puts the role of algorithms in society in such a negative light. I encounter is very frequently, and every time feel that some misunderstanding must have happened; something seems off.

It’s very clear that O’Neil can’t be accused of mathophobia or not understanding the complexity of the algorithms at play, which is an easy way to throw doubt on the arguments of some technology critics. Yet perhaps because it’s a popular book and not an academic work of Science and Technology Studies, I haven’t it’s arguments parsed through and analyzed in much depth.

This is a start. These are my notes on the introduction.

O’Neil describes the turning point in her career where she soured on math. After being an academic mathematician for some time, O’Neil went to work as a quantitative analyst for D.E. Shaw. She saw it as an opportunity to work in a global laboratory. But then the 2008 financial crisis made her see things differently.

The crash made it all too clear that mathematics, once my refuge, was not only deeply entangled in the world’s problems but also fueling many of them. The housing crisis, the collapse of major financial institutions, the rise of unemployment–all had been aided and abetted by mathematicians wielding magic formulas. What’s more, thanks to the extraordinary powers that I loved so much, math was able to combine with technology to multiply the chaos and misfortune, adding efficiency and scale to systems I now recognized as flawed.

O’Neil, Weapons of Math Destruction, p.2

As an independent reference on the causes of the 2008 financial crisis, which of course has been a hotly debated and disputed topic, I point to Sassen’s 2017 “Predatory Formations” article. Indeed, the systems that developed the sub-prime mortgage market were complex, opaque, and hard to regulate. Something went seriously wrong there.

But was it mathematics that was the problem? This is where I get hung up. I don’t understand the mindset that would attribute a crisis in the financial system to the use of abstract, logical, rigorous thinking. Consider the fact that there would not have been a financial crisis if there had not been a functional financial services system in the first place. Getting a mortgage and paying them off, and the systems that allow this to happen, all require mathematics to function. When these systems operate normally, they are taken for granted. When they suffer a crisis, when the system fails, the mathematics takes the blame. But a system can’t suffer a crisis if it didn’t start working rather well in the first place–otherwise, nobody would depend on it. Meanwhile, the regulatory reaction to the 2008 financial crisis required, of course, more mathematicians working to prevent the same thing from happening again.

So in this case (and I believe others) the question can’t be, whether mathematics, but rather which mathematics. It is so sad to me that these two questions get conflated.

O’Neil goes on to describe a case where an algorithm results in a teacher losing her job for not adding enough value to her students one year. An analysis makes a good case that the cause of her students’ scores not going up is that in the previous year, the students’ scores were inflated by teachers cheating the system. This argument was not consider conclusive enough to change the administrative decision.

Do you see the paradox? An algorithm processes a slew of statistics and comes up with a probability that a certain person might be a bad hire, a risky borrower, a terrorist, or a miserable teacher. That probability is distilled into a score, which can turn someone’s life upside down. And yet when the person fights back, “suggestive” countervailing evidence simply won’t cut it. The case must be ironclad. The human victims of WMDs, we’ll see time and again, are held to a far higher standard of evidence than the algorithms themselves.

O’Neil, WMD, p.10

Now this is a fascinating point, and one that I don’t think has been taken up enough in the critical algorithms literature. It resonates with a point that came up earlier, that traditional collective human decision making is often driven by agreement on narratives, whereas automated decisions can be a qualitatively different kind of collective action because they can make judgments based on probabilistic judgments.

I have to wonder what O’Neil would argue the solution to this problem is. From her rhetoric, it seems like her recommendation must be prevent automated decisions from making probabilistic judgments. In other words, one could raise the evidenciary standard for algorithms so that they we equal to the standards that people use with each other.

That’s an interesting proposal. I’m not sure what the effects of it would be. I expect that the result would be lower expected values of whatever target was being optimized for, since the system would not be able to “take bets” below a certain level of confidence. One wonders if this would be a more or less arbitrary system.

Sadly, in order to evaluate this proposal seriously, one would have to employ mathematics. Which is, in O’Neil’s rhetoric, a form of evil magic. So, perhaps it’s best not to try.

O’Neil attributes the problems of WMD’s to the incentives of the data scientists building the systems. Maybe they know that their work effects people, especially the poor, in negative ways. But they don’t care.

But as a rule, the people running the WMD’s don’t dwell on these errors. Their feedback is money, which is also their incentive. Their systems are engineered to gobble up more data fine-tune their analytics so that more money will pour in. Investors, of course, feast on these returns and shower WMD companies with more money.

O’Neil, WMD, p.13

Calling out greed as the problem is effective and true in a lot of cases. I’ve argued myself that the real root of the technology ethics problem is capitalism: the way investors drive what products get made and deployed. This is a worthwhile point to make and one that doesn’t get made enough.

But the logical implications of this argument are off. Suppose it is true that “as a rule”, the makers of algorithms that do harm are made by people responding to the incentives of private capital. (IF harmful algorithm, THEN private capital created it.) That does not mean that there can’t be good algorithms as well, such as those created in the public sector. In other words, there are algorithms that are not WMDs.

So the insight here has to be that private capital investment corrupts the process of designing algorithms, making them harmful. One could easily make the case that private capital investment corrupts and makes harmful many things that are not algorithmic as well. For example, the historic trans-Atlantic slave trade was a terribly evil manifestation of capitalism. It did not, as far as I know, depend on modern day computer science.

Capitalism here looks to be the root of all evil. The fact that companies are using mathematics is merely incidental. And O’Neil should know that!

Here’s what I find so frustrating about this line of argument. Mathematical literacy is critical for understanding what’s going on with these systems and how to improve society. O’Neil certainly has this literacy. But there are many people who don’t have it. There is a power disparity there which is uncomfortable for everybody. But while O’Neil is admirably raising awareness about how these kinds of technical systems can and do go wrong, the single-minded focus and framing risks giving people the wrong idea that these intellectual tools are always bad or dangerous. That is not a solution to anything, in my view. Ignorance is never more ethical than education. But there is an enormous appetite among ignorant people for being told that it is so.

References

O’Neil, Cathy. Weapons of math destruction: How big data increases inequality and threatens democracy. Broadway Books, 2017.

Sassen, Saskia. “Predatory Formations Dressed in Wall Street Suits and Algorithmic Math.” Science, Technology and Society22.1 (2017): 6-20.

## January 11, 2019

Ph.D. student

We've spent a lot of collective time and effort on design and policy to support the privacy of the user of a piece of software, whether it's the Web or a mobile app or a device. But more current and more challenging is the privacy of the non-user of the app, the privacy of the bystander. With the ubiquity of sensors, we are increasingly observed, not just by giant corporations or government agencies, but by, as they say, little brothers.

Consider the smartphone camera. Taking digital photos is free, quick and easy; resolution and quality increase; metadata (like precise geolocation) is attached; sharing those photos is easy via online services. As facial recognition has improved, it has become easier to automatically identify the people depicted in a photo, whether they're the subject of a portrait or just in the background. If you don't want to share records of your precise geolocation and what you're doing in public places, with friends, family, strangers and law enforcement, it's no longer enough to be careful with the technology you choose to use, you'd also have to be constantly vigilant about the technology that everyone around you is using.

While it may be tempting to draw a "throw your hands up" conclusion from this -- privacy is dead, get over it, there's nothing we can easily do about it -- we actually have widespread experience with this kind of value and various norms to protect it. At conferences and public events, it's not uncommon to have a system of stickers on nametags to either opt-in or opt-out of photos. This is a help (not a hindrance) for event photographers: rather than asking everyone to pose in your photo, or asking everyone after the fact if they're alright with your posting a public photo, or being afraid of posting a photo and facing the anger of your attendees, you can just keep an eye on the red and green dots on those plastic nametags and feel confident that you're respecting the attendees at your event.

There are similar norms in other settings. Taking video in the movie theater violates legal protections, but there are also widespread and reasonably well-enforced norms against capturing video of live theater productions or comedians who test out new material in clubs, on grounds that may not be copyright. Art museums will often tell you whether photos are welcome or prohibited. In some settings the privacy of the people present is so essential that unwritten or written rules prohibit cameras altogether: at nude hot springs, for example, you just can't use a camera at all. You wouldn't take a photo in the waiting room of your doctor's office and you'll invite anger and social confrontation if you're taking photos of other people's children at your local playground.

And even in "public" or in contexts with friends, there are spoken or unspoken expectations. "Don't post that photo of me drinking, please." "Let me see how I look in that before you post it on Facebook." "Everyone knows that John doesn't like to have his photo taken."

As cameras become small and more widely used, and encompass depictions of more people, and are shared more widely and easily, and identifications of depicted people can also be shared, our social norms and spoken discussions don't easily keep up. Checking with people before you post a photo of them is absolutely a good practice and I encourage you to follow it. But why not also use technology to facilitate this checking others' preferences?

We have all the tools we need to make "no photos please" nametag stickers into unobtrusive and efficiently communicated messages. If you're attending a conference or party and don't want people to take your photo, just tap the "no photos please" setting on your smartphone before you walk in. And if you're taking photos at an event, your camera will show a warning when it knows that someone in the room doesn't want their photo taken, so that you can doublecheck with the people in your photo and make sure you're not inadvertently capturing someone in the background. And the venue can remind you that way too, in case you don't know the local norm that pictures shouldn't be taken in the church or museum.

As a technical matter, I think we're looking at Bluetooth broadcast beacons, from smartphones or stationary devices. That could be a small Arduino-based widget on the wall of a commercial venue, or one day you might have a poker-chip-sized device in your pocket that you can click into private mode. When you're using a compatible camera app on your phone or a compatible handheld camera, your device regularly scans for nearby Bluetooth beacons and if it sees a "no photos please" message, it shows a (dismissable) warning.

The discretionary communication of preferences is ideal in part because it isn't self-enforcing. For example, if the police show up at the political protest you're attending and broadcast a "no photos please" beacon, you can (and should) override your camera warning to take photos of their official activity, as a safeguard for public safety and accountability. An automatically-enforcing DRM-style system would be both infeasible to construct and, if it were constructed, inappropriately inviting to government censorship or aggressive copyright maximalism. Technological hints are also less likely to confusingly over-promise a protection: we can explain to people that the "no photos please" beacon doesn't prevent impolite or malicious people from surreptitiously taking your photo, just as people are extremely familiar with the fact that placards, polite requests and even laws are sometimes ignored.

Making preferences technically available could also help with legal compliance. If you're taking a photo at an event and get a "no photos" warning, your device UI can help you log why you might be taking the photo anyway. Tap "I got consent" and your camera can embed metadata in the file that you gathered consent from the depicted people. Tap "Important public purpose" at the protest and you'll have a machine-readable affirmation in place of what you're doing, and your Internet-connected phone can also use that signal to make sure photos in this area are promptly backed up securely in case your device is confiscated.

People's preferences are of course more complicated than just "no photos please" or "sure, take my photo". While I, like many, have imagined that sticky policies could facilitate rules of how data is subsequently shared and used, there are good reasons to start with the simple capture-time question. For one, it's familiar, from these existing social and legal norms. For another, it can be a prompt for real-time in-person conversation. Rather than assuming an error-free technical-only system of preference satisfaction, this can be a quick reminder to check with the people right there in front of you for those nuances, and to do so prior to making a digital record.

The Internet is amazing for letting us communicate with people around the world around shared interests. We should see the opportunity for networking technology to also facilitate communications, including conversations about privacy, with those nearby.

Some end notes that my head wants to let go of: There is some prior art here that I don't want to dismiss or pass over, I just think we should push it further. A couple examples:

• Google folks have developed broadcast URLs that they call The Physical Web so that real-life places can share a Web page about them (over mDNS or Bluetooth Low Energy) and I hope one day we can get a link to the presenter's current slide using networking rather than everyone taking a picture of a projected URL and awkwardly typing it into our laptops later.
• The Occupy movement showed an interest in geographically-located Web services, including forums and chatrooms that operate over WiFi but not connected to the Internet. Occupy Here:
Anyone within range of an Occupy.here wifi router, with a web-capable smartphone or laptop, can join the network “OCCUPY.HERE,” load the locally-hosted website http://occupy.here, and use the message board to connect with other users nearby.

Getting a little further afield but still related, it would be helpful if the network provider could communicate directly with the subscriber using the expressive capability of the Web. Lacking this capability, we've seen frustrating abuses of interception: captive portals redirect and impersonate Web traffic; ISPs insert bandwidth warnings as JavaScript insecurely transplanted into HTTP pages. Why not instead provide a way for the network to push a message to the client, not by pretending to be a server you happen to connect to around that same time, but just as a clearly separate message? ICMP control messages are an existing but underused technology.

## January 09, 2019

Ph.D. student

#### computational institutions as non-narrative collective action

Nils Gilman recently pointed to a book chapter that confirms the need for “official futures” in capitalist institutions.

Nils indulged me in a brief exchange that helped me better grasp at a bothersome puzzle.

There is a certain class of intellectuals that insist on the primacy of narratives as a mode of human experience. These tend to be, not too surprisingly, writers and other forms of storytellers.

There is a different class of intellectuals that insists on the primacy of statistics. Statistics does not make it easy to tell stories because it is largely about the complexity of hypotheses and our lack of confidence in them.

The narrative/statistic divide could be seen as a divide between academic disciplines. It has often been taken to be, I believe wrongly, the crux of the “technology ethics” debate.

I questioned Nils as to whether his generalization stood up to statistically driven allocation of resources; i.e., those decisions made explicitly on probabilistic judgments. He argued that in the end, management and collective action require consensus around narrative.

In other words, what keeps narratives at the center of human activity is that (a) humans are in the loop, and (b) humans are collectively in the loop.

The idea that communication is necessary for collective action is one I used to put great stock in when studying Habermas. For Habermas, consensus, and especially linguistic consensus, is how humanity moves together. Habermas contrasted this mode of knowledge aimed at consensus and collective action with technical knowledge, which is aimed at efficiency. Habermas envisioned a society ruled by communicative rationality, deliberative democracy; following this line of reasoning, this communicative rationality would need to be a narrative rationality. Even if this rationality is not universal, it might, in Habermas’s later conception of governance, be shared by a responsible elite. Lawyers and a judiciary, for example.

The puzzle that recurs again and again in my work has been the challenge of communicating how technology has become an alternative form of collective action. The claim made by some that technologists are a social “other” makes more sense if one sees them (us) as organizing around non-narrative principles of collective behavior.

It is I believe beyond serious dispute that well-constructed, statistically based collective decision-making processes perform better than many alternatives. In the field of future predictions, Phillip Tetlock’s work on superforecasting teams and prior work on expert political judgment has long stood as an empirical challenge to the supposed primacy of narrative-based forecasting. This challenge has not been taken up; it seems rather one-sided. One reason for this may be because the rationale for the effectiveness of these techniques rests ultimately in the science of statistics.

It is now common to insist that Artificial Intelligence should be seen as a sociotechnical system and not as a technological artifact. I wholeheartedly agree with this position. However, it is sometimes implied that to understand AI as a social+ system, one must understand it one narrative terms. This is an error; it would imply that the collective actions made to build an AI system and the technology itself are held together by narrative communication.

But if the whole purpose of building an AI system is to collectively act in a way that is more effective because of its facility with the nuances of probability, then the narrative lens will miss the point. The promise and threat of AI is that is delivers a different, often more effective form of collective or institution. I’ve suggested that computational institution might be the best way to refer to such a thing.

## January 08, 2019

MIMS 2012

#### My Yardstick for Empathy

A different perspective. Photo by Jamie Street on Unsplash

How do you know if you’re being empathetic? It’s easy to throw the term around, but difficult to actually apply. This is important to understand in my chosen field of design, but can also help anyone improve their interactions with other people.

My yardstick for being empathetic is imagining myself make the same decisions, in the same situation, that another person made.

If I look at someone’s behavior and think, “That doesn’t make sense,” or “Why did they do that?!” then I’m not being empathetic. I’m missing a piece of context — about their knowledge, experiences, skills, emotional state, environment, etc. — that led them to do what they did. When I feel that way, I push myself to keep searching for the missing piece that will make their actions become the only rational ones to take.

Is this always possible? No. Even armed with the same knowledge, operating in the same environment, and possessing the same skills as another person, I will occasionally make different decisions than them. Every individual is unique, and interpret and act on stimuli differently.

Even so, imagining myself behave the same as another person is what I strive for. That’s my yardstick for empathy.

If you want to learn more about empathy and how to apply it to your work and personal life, I highly recommend Practical Empathy by Indi Young.

## January 04, 2019

MIMS 2012

In 2018 I read 23 books, which is a solid 9 more than last year’s paltry 14, and 1 more than 2016). I credit the improvement to the 4-month sabbatical I took in the spring. Not working really frees up time 😄

For the last 2 years I said I needed to read more fiction since I only read 3 in 2016 and 2 in 2017. So how did I do? I’m proud to say I managed to read 7 fiction books this year (if you can count My Dad Wrote a Porno as “fiction”…). My reading still skews heavily to non-fiction, and specifically design, but that’s what I’m passionate about and it helps me professionally, so I’m ok with it.

I also apparently didn’t finish any books in January or February. I thought this might have been a mistake at first, but when I looked back on that time I realized it’s because I was wrapping things up at Optimizely, and reading both Quicksilver by Neal Stephenson and Story by Robert McKee at the same time, which are long books that took awhile to work through.

## Highlights

### Story: Substance, Structure, Style, and the Principles of Screenwriting

by Robert McKee

I’ve read next to nothing about writing stories before, but Robert McKee’s primer on the subject is excellent. Even though I’m not a fiction author, I found his principles for writing compelling narratives valuable beyond just the domain of screenwriting.

### Handstyle Lettering

Published and edited by Victionary

There wasn’t much to “read” in this book, but it was full of beautiful hand-lettered pieces that continue to inspire me to be a better letterer.

### The Baroque Cycle

by Neal Stephenson

Neal Stephenson’s Baroque Cycle is a broad, staggering, 3-volume and 2,500+ page opus of historical science fiction, making it no small feat to complete (I read the first 2 this year, and am almost done with the 3rd volume). It takes place during the scientific revolution of the 17th and 18th centuries when the world transitioned out of feudal rule towards a more rational and merit-based society that we would recognize as modern. It weaves together a story between fictional and non-fictional characters, including Newton, Leibniz, Hooke, Wren, royalty, and other persons-of-quality. Although the series can be slow and byzantine at times, Stephenson makes up for it with his attention to detail and the sheer amount of research and effort he put into accurately capturing the time period and bringing the story to life. Even just having the audacity to put yourself in Newton’s head to speak from his perspective, much less to do so convincingly, makes the series worth the effort.

by Richard P. Rumelt

Strategy is a fuzzy concept, but Rumelt makes it concrete and approachable with many examples of good and bad strategy. Read my full notes here. Highly recommended.

### Bird by Bird: Some Instructions on Writing and Life

by Anne Lamott

A great little meditation on the writing process (and life!), sprinkled with useful tips and tricks throughout.

### Creative Selection: Inside Apple’s design process

by Ken Kocienda

Ken Kocienda was a software engineer during the “golden age of Steve Jobs,” and provides a fascinating insight into the company’s design process. I’m still chewing on what I read (and hope to publish more thoughts soon), but it’s striking how different it is from any process I’ve ever seen at any company, and different from best practices written about in books. It’s basically all built around Steve Jobs’ exacting taste, with designers and developers demoing their work to Steve with the hope of earning his approval. Very difficult to replicate, but the results speak for themselves.

by David Ogilvy

## Full List of Books Read

• Story: Substance, Structure, Style, and the Principles of Screenwriting by Robert McKee (3/7/18)
• The Color of Pixar by Tia Kratter (3/18/18)
• Conversational Design by Erika Hall (3/27/18)
• Quicksilver by Neal Stephenson (4/3/18)
• Handstyle Lettering published and edited by Victionary (4/24/18)
• Bimimicry: Innovation Inspired by Nature by Janine M. Benyus (5/4/18)
• Design is Storytelling by Ellen Lupton (5/11/18)
• Trip by Tao Lin (5/20/18)
• Good Strategy, Bad Strategy: The Difference and Why it Matters by Richard P. Rumelt (5/27/18)
• Bird by Bird: Some Instructions on Writing and Life by Anne Lamott (6/10/18)
• The Inmates are Running the Asylum by Alan Cooper (6/13/18)
• It Chooses You by Miranda July (6/13/18)
• String Theory by David Foster Wallace (6/22/18)
• Invisible Cities by Italo Calvino (6/28/18)
• My Dad Wrote a Porno by Jamie Morton, James Cooper, Alice Levine, and Rocky Flintstone (7/1/18)
• The User Experience Team of One by Leah Buley (7/8/18)
• Change by Design by Tim Brown (9/3/18)
• Darkness at Noon by Arthur Koestler (9/16/2018)
• Creative Selection: Inside Apple’s design process during the golden age of Steve Jobs by Ken Kocienda (9/20/18)
• The Confusion by Neal Stephenson (9/26/18)
• How to Change Your Mind by Michael Pollan (10/27/18)
• Ogilvy on Advertising by David Ogilvy (11/11/18)
• Draft No. 4. On the writing process by John McPhee (11/14/18)

## December 30, 2018

Ph.D. student

#### State regulation and/or corporate self-regulation

The dust from the recent debates about whether regulation or industrial self-regulation in the data/tech/AI industry appears to be settling. The smart money is on regulation and self-regulation being complementary for attaining the goal of an industry dominated by responsible actors. This trajectory leads to centralized corporate power that is lead from the top; it is a Hamiltonian not Jeffersonian solution, in Pasquale’s terms.

I am personally not inclined towards this solution. But I have been convinced to see it differently after a conversation today about environmentally sustainable supply chains in food manufacturing. Nestle, for example, has been internally changing its sourcing practices to more sustainable chocolate. It’s able to finance this change from its profits, and when it does change its internal policy, it operates on a scale that’s meaningful. It is able to make this transition in part because non-profits, NGO’s, and farmers cooperatives lay through groundwork for sustainable sourcing external to the company. This lowers the barriers to having Nestle switch over to new sources–they have already been subsidized through philanthropy and international aid investments.

Supply chain decisions, ‘make-or-buy’ decisions, are the heart of transaction cost economics (TCE) and critical to the constitution of institutions in general. What this story about sustainable sourcing tells us is that the configuration of private, public, and civil society institutions is complex, and that there are prospects for agency and change in the reconfiguration of those relationships. This is no different in the ‘tech sector’.

However, this theory of economic and political change is not popular; it does not have broad intellectual or media appeal. Why?

One reason may be because while it is a critical part of social structure, much of the supply chain is in the private sector, and hence is opaque. This is not a matter of transparency or interpretability of algorithms. This is about the fact that private institutions, by virtue of being ‘private’, do not have to report everything that they do and, probably, shouldn’t. But since so much of what is done by the massive private sector is of public import, there’s a danger of the privatization of public functions.

Another reason why this view of political change through the internal policy-making of enormous private corporations is unpopular is because it leaves decision-making up to a very small number of people–the elite managers of those corporations. The real disparity of power involved in private corporate governance means that the popular attitude towards that governance is, more often than not, irrelevant. Even less so that political elites, corporate elites are not accountable to a constituency. They are accountable, I suppose, to their shareholders, which have material interests disconnected from political will.

This disconnected shareholder will is one of the main reasons why I’m skeptical about the idea that large corporations and their internal policies are where we should place our hopes for moral leadership. But perhaps what I’m missing is the appropriate intellectual framework for how this will is shaped and what drives these kinds of corporate decisions. I still think TCE might provide insights that I’ve been missing. But I am on the lookout for other sources.

## December 24, 2018

Ph.D. student

#### Ordoliberalism and industrial organization

There’s a nice op-ed by Wolfgang Münchau in FT, “The crisis of modern liberalism is down to market forces”.

Among other things, it reintroduces the term “ordoliberalism“, a particular Germanic kind of enlightened liberalism designed to prevent the kind of political collapse that had precipitated the war.

In Münchau’s account, the key insight of ordoliberalism is its attention to questions of social equality, but not through the mechanism of redistribution. Rather, ordoliberal interventions primarily effect industrial organization, favoring small to mid- sized companies.

As Germany’s economy remains robust and so far relatively politically stable, it’s interesting that ordoliberalism isn’t discussed more.

Another question that must be asked is to what extent the rise of computational institutions challenges the kind of industrial organization recommended by ordoliberalism. If computation induces corporate concentration, and there are not good policies for addressing that, then that’s due to a deficiency in our understanding of what ‘market forces’ are.

## December 22, 2018

Ph.D. student

#### When *shouldn’t* you build a machine learning system?

Luke Stark raises an interesting question, directed at “ML practitioner”:

As an “ML practitioner” in on this discussion, I’ll have a go at it.

In short, one should not build an ML system for making a class of decisions if there is already a better system for making that decision that does not use ML.

An example of a comparable system that does not use ML would be a team of human beings with spreadsheets, or a team of people employed to judge for themselves.

There are a few reasons why a non-ML system could be superior in performance to an ML system:

• The people involved could have access to more data, in the course of their lives, in more dimensions of variation, than is accessible by the machine learning system.
• The people might have more sensitized ability to make semantic distinctions, such as in words or images, than an ML system
• The problem to be solved could be a “wicked problem” that is itself over a very high-dimensional space of options, with very irregular outcomes, such that they are not amenable to various forms of, e.g., linear approximations
• The people might be judging an aspect of their own social environment, such that the outcome’s validity is socially procedural (as in the outcome of a vote, or of an auction)

These are all fine reasons not to use an ML system. On the other hand, the term “ML” has been extended, as with “AI”, to include many hybrid human-computer systems, which has led to some confusion. So, for example. crowdsourced labels of images provide useful input data to ML systems. This hybrid system might perform semantic judgments over a large scale of data, at a high speed, at a tolerable rate of accuracy. Does this system count as an ML system? Or is it a form of computational institution that rivals other ways of solving the problem, and just so happens to have a machine learning algorithm as part of its process?

Meanwhile, the research frontier of machine learning is all about trying to solve problems that previously haven’t been solved, or solved as well, as alternative kinds of systems. This means there will always be a disconnect between machine learning research, which is trying to expand what it is possible to do with machine learning, and what machine learning research should, today, be deployed. Sometimes, research is done to develop technology that is not mature enough to deploy.

We should expect that a lot of ML research is done on things that should not ultimately be deployed! That’s because until we do the research, we may not understand the problem well enough to know the consequences of deployment. There’s a real sense in which ML research is about understanding the computational contours of a problem, whereas ML industry practice is about addressing the problems customers have with an efficient solution. Often this solution is a hybrid system in which ML only plays a small part; the use of ML here is really about a change in the institutional structure, not so much a part of what service is being delivered.

On the other hand, there have been a lot of cases–search engines and social media being important ones–where the scale of data and the use of ML for processing has allowed for a qualitatively different form of product or service. These are now the big deal companies we are constantly talking about. These are pretty clearly cases of successful ML.

#### computational institutions

As the “AI ethics” debate metastasizes in my newsfeed and scholarly circles, I’m struck by the frustrations of technologists and ethicists who seem to be speaking past each other.

While these tensions play out along disciplinary fault-lines, for example, between technologists and science and technology studies (STS), the economic motivations are more often than not below the surface.

I believe this is to some extent a problem of the nomenclature, which is again the function of the disciplinary rifts involved.

Computer scientists work, generally speaking, on the design and analysis of computational systems. Many see their work as bounded by the demands of the portability and formalizability of technology (see Selbst et al., 2019). That’s their job.

This is endlessly unsatisfying to critics of the social impact of technology. STS scholars will insist on changing the subject to “sociotechnical systems”, a term that means something very general: the assemblage of people and artifacts that are not people. This, fairly, removes focus from the computational system and embeds it in a social environment.

A goal of this kind of work seems to be to hold computational systems, as they are deployed and used socially, accountable. It must be said that once this happens, we are no longer talking about the specialized domain of computer science per se. It is a wonder why STS scholars are so often picking fights with computer scientists, when their true beef seems to be with businesses that use and deploy technology.

The AI Now Institute has attempted to rebrand the problem by discussing “AI Systems” as, roughly, those sociotechnical systems that use AI. This is one the one hand more specific–AI is a particular kind of technology, and perhaps it has particular political consequences. But their analysis of AI systems quickly overflows into sweeping claims about “the technology industry”, and it’s clear that most of their recommendations have little to do with AI, and indeed are trying, once again, to change the subject from discussion of AI as a technology (a computer science research domain) to a broader set of social and political issues that do, in fact, have their own disciplines where they have been researched for years.

The problem, really, is not that any particular conversation is not happening, or is being excluded, or is being shut down. The problem is that the engineering focused conversation about AI-as-a-technology has grown very large and become an awkward synecdoche for the rise of major corporations like Google, Apple, Amazon, Facebook, and Netflix. As these corporations fund and motivate a lot of research, there’s a question of who is going to get pieces of the big pie of opportunity these companies represent, either in terms of research grants or impact due to regulation, education, etc.

But there are so many aspects of these corporations that are neither addressed by the terms “sociotechnical system”, which is just so broad, and “AI System”, which is as broad and rarely means what you’d think it does (that the system uses AI is incidental if not unnecessary; what matters is that it’s a company operating in a core social domain via primarily technological user interfaces). Neither of these gets at the unit of analysis that’s really of interest.

An alternative: “computational institution”. Computational, in the sense of computational cognitive science and computational social science: it denotes the essential role of theory of computation and statistics in explaining the behavior of the phenomenon being studied. “Institution”, in the sense of institutional economics: the unit is a firm, which is comprised of people, their equipment, and their economic relations, to their suppliers and customers. An economic lens would immediately bring into focus “the data heist” and the “role of machines” that Nissenbaum is concerned are being left to the side.

## December 20, 2018

Ph.D. student

#### Tensions of a Digitally-Connected World in Cricket Wireless’ Holiday Ad Campaign

In the spirit of taking a break over the holidays, this is more of a fun post with some very rough thoughts (though inspired by some of my prior work on paying attention to and critiquing narratives and futures portrayed by tech advertising). The basic version is that the Cricket Wireless 2018 Holiday AdFour the Holidays (made by ad company Psyop), portrays a narrative that makes a slight critique of an always-connected world and suggests that physical face-to-face interaction is a more enjoyable experience for friends than digital sharing. While perhaps a over-simplistic critique of mobile technology use, the twin messages of “buy a wireless phone plan to connect with friends” and “try to disconnect to spend time with friends” highlight important tensions and contradictions present in everyday digital life.

But let’s look at the ad in a little more detail!

Last month, while streaming Canadian curling matches (it’s more fun than you might think, case in point, I’ve blogged about the sport’s own controversy with broom technology) there was a short Cricket ad playing with a holiday jingle. And I’m generally inclined to pay attention to an ad with a good jingle. Looking it up online brought up a 3 minute long short film version expanding upon the 15 second commercial (embedded above), which I’ll describe and analyze below.

It starts with Cricket’s animated characters Ramon (the green blob with hair), Dusty (the orange fuzzy ball), Chip (the blue square), and Rose (the green oblong shape) on a Hollywood set, “filming” the aforementioned commercial, singing their jingle:

The four, the merrier! Cricket keeps us share-ier!

Four lines of unlimited data, for a hundred bucks a month!

After their shoot is over, Dusty wants the group to watch fireworks from the Cricket water tower (which is really the Warner Brothers Studio water tower, though maybe we should call it Chekov’s water tower in this instance) on New Year’s Eve. Alas, the crew has other plans, and everyone flies to their holiday destinations: Ramon to Mexico, Dusty to Canada, Chip to New York, and Rose to Aspen.

The video then shows each character enjoying the holidays in their respective locations with their smartphones. Ramon uses his phone to take pictures of food shared on a family table; Rose uses hers to take selfies on a ski lift.

The first hint that there might be a message critiquing an always-connected world is when the ad shows Dusty in a snowed-in, remote Canadian cabin. Presumably this tells us that he gets a cell signal up there, but in this scene, he is not using his phone. Rather, he’s making cookies with his two (human) nieces (not sure how that works, but I’ll suspend my disbelief), highlighting a face-to-face familial interaction using a traditional holiday group activity.

The second hint that something might not be quite right is the dutch angel establishing shot of New York City in the next scene. The non-horizontal horizon line (which also evokes the off-balance establishing shot of New York from an Avengers: Infinity War trailer) visually puts the scene off balance. But the moment quickly passes, as we see Chip on the streets of New York taking instagram selfies.

Dutch angle of New York from Cricket Wireless’ “Four the Holidays” (left) and Marvel’s Avengers Infinity War (right)

Then comes a rapid montage of photos and smiling selfies that the group is sending and sharing with each other, in a sort of digital self-presentation utopia. But as the short film has been hinting at, this utopia is not reflective of the characters’ lived experience.

The video cuts to Dusty, skating alone on a frozen pond, successfully completing a trick, but then realizes that he has no one to share the moment with. He then sings “The four the merrier, Cricket keeps us share-ier” in a minor key as re-envisions clouds in the sky as the form of the four friends. The minor key and Dusty’s singing show skepticism in the lyrics’ claim that being share-ier is indeed merrier.

The minor key continues, as Ramon sings while envisioning a set of holiday lights as the four friends, and Rose sees a department store window display as the four friends. Chip attends a party where the Cricket commercial (from the start of the video) airs on a TV, but is still lonely. Chip then hails a cab, dramatically stating in a deep voice “Take me home.”

In the last scene, Chip sits atop the Cricket Water Tower (or, Chekov’s Water Tower returns!) at 11:57pm on New Year’s Eve, staring alone at his phone, discontent. This is the clearest signal about the lack of fulfillment he finds from his phone, and by extension, the digitally mediated connection with his friends.

Immediately this is juxtaposed with Ramon singing with his guitar from the other side of the water tower, still in the minor key. Chip hears him and immediately becomes happier, and the music shifts to a major key as Rose and Dusty enter as the tempo picks up, and the drums and orchestra of instruments join in. And the commercial ends with the four of them watching New Year’s fireworks together. It’s worth noting the lyrics at the end:

Ramon: The four the merrier…

Chip [spoken]: Ramon?! You’re here!

Rose: There’s something in the air-ier

All: That helps us connect, all the season through. The four, the merrier

Dusty: One’s a little harrier (So hairy!)

All: The holidays are better, the holidays are better, the holidays are better with your crew.

Nothing here is explicitly about Cricket wireless, or the value of being digitally connected. It’s also worth noting that the phone that Chip was previously staring at is nowhere to be found after he sees Ramon. There is some ambiguous use of the word “connect,” which could refer to both a face-to-face interaction or a digitally mediated one, but the tone of the scene and emotional storyline bringing the four friends physically together seems to suggest that connect refers to the value of face-to-face interaction.

So what might this all mean (beyond the fact that I’ve watched this commercial too many times and have the music stuck in my head)? Perhaps the larger and more important point is that the commercial/short film is emblematic of a series of tensions around connection and disconnection in today’s society. Being digitally connected is seen as a positive that allows for greater opportunity (and greater work output), but at the same time discontent is reflected in culture and media, ranging from articles on tech addiction, to guides on grayscaling iPhones to combat color stimulation, to disconnection camps. There’s also a moralizing force behind these tensions: to be a good employee/student/friend/family member/etc, we are told that we must be digitally connected and always-on, but at the same time, we are told that we must also be dis-connected or interact face-to-face in order to be good subjects.

In many ways, the tensions expressed in this video — an advertisement for a wireless provider trying to encourage customers to sign up for their wireless plans, while presenting a story highlighting the need to digitally disconnect — parallels the tensions that Ellie Harmon and Melissa Mazmanian find in their analysis of media discourse of smartphones: that there is both a push for individuals to integrate the smartphone into everyday life, and to dis-integrate the smartphone from everyday life. What is fascinating to me here is that this video from Cricket exhibits both of those ideas at the same time. As Harmon and Mazmanian write,

The stories that circulate about the smartphone in American culture matter. They matter for how individuals experience the device, the ways that designers envision future technologies, and the ways that researchers frame their questions.

While Four the Holidays doesn’t tell the most complex or nuanced story about connectivity and smartphone use, the narrative that Cricket and Psyop created veers away from a utopian imagining of the world with tech, and instead begins to reflect  some of the inherent tensions and contradictions of smartphone use and mobile connectivity that are experienced as a part of everyday life.

## December 19, 2018

Ph.D. student

#### The politics of AI ethics is a seductive diversion from fixing our broken capitalist system

There is a lot of heat these days in the tech policy and ethics discourse. There is an enormous amount of valuable work being done on all fronts. And yet there is also sometimes bitter disciplinary infighting and political intrigue about who has the moral high ground.

The smartest thing I’ve read on this recently is Irina Raicu’s “False Dilemmas” piece, where she argues:

• “Tech ethics” research, including research explore the space of ethics in algorithm design, is really code for industry self-regulation
• Industry self-regulation and state regulation are complementary
• Any claims that “the field” is dominated by one perspective or agenda or another is overstated

All this sounds very sane but it doesn’t exactly explain why there’s all this heated discussion in the first place. I think Luke Stark gets it right:

But what does it mean to say “the problem is mostly capitalism”? And why is it impolite to say it?

To say “the problem [with technology ethics and policy] is capitalism” is to note that most if not all of the social problems we associate with today’s technology have been problems with technology ever since the industrial revolution. For example, James Beniger‘s The Control Revolution, Horkheimer‘s Eclipse of Reason, and so on all speak to the tight link that there has always been between engineering and the capitalist economy as a whole. The link has persisted through the recent iterations of recognizing first data science, then later artificial intelligence, as disruptive triumphs of engineering with a variety of problematic social effects. These are old problems.

It’s impolite to say this because it cuts down on the urgency that might drive political action. More generally, it’s an embarrassment to anybody in the business of talking as if they just discovered something, which is what journalists and many academics do. The buzz of novelty is what gets people’s attention.

It also suggests that the blame for how technology has gone wrong lies with capitalists, meaning, venture capitalists, financiers, and early stage employees with stock options. But also, since it’s the 21st century, pension funds and university endowments are just as much a part of the capitalist investing system as anybody else. In capitalism, if you are saving, you are investing. Lots of people have a diffuse interest in preserving capitalism in some form.

There’s a lot of interesting work to be done on financial regulation, but it has very little to do with, say, science and technology studies and consumer products. So to acknowledge that the problem with technology is capitalism changes the subject to something remote and far more politically awkward than to say the problem is technology or technologists.

As I’ve argued elsewhere, a lot of what’s happening with technology ethics can be thought of as an extension of what Nancy Fraser called progressive neoliberalism: the alliance of neoliberalism with progressive political movements. It is still hegemonic in the smart, critical, academic and advocacy scene. Neoliberalism, or what is today perhaps better characterized as finance capitalism or surveillance capitalism, is what is causing the money to be invested in projects that design and deploy technology in certain ways. It is a system of economic distribution that is still hegemonic.

Because it’s hegemonic, it’s impolite to say so. So instead a lot of the technology criticism gets framed in terms of the next available moral compass, which is progressivism. Progressivism is a system of distribution of recognition. It calls for patterns of recognizing people for their demographic and, because it’s correlated in a sensitive way, professional identities. Nancy Fraser’s insight is that neoliberalism and progressivism have been closely allied for many years. One way that progressivism is allied with neoliberalism is that progressivism serves as a moral smokescreen for problems that are in part caused by neoliberalism, preventing an effective, actionable critique of the root cause of many technology-related problems.

Progressivism encourages political conflict to be articulated as an ‘us vs. them’ problem of populations and their attitudes, rather than as problem of institutions and their design. This “us versus them” framing is baldly stated than in the 2018 AI Now Report:

The AI accountability gap is growing: The technology scandals of 2018 have shown that the gap between those who develop and profit from AI—and those most likely to suffer the consequences of its negative effects—is growing larger, not smaller. There are several reasons for this, including a lack of government regulation, a highly concentrated AI sector, insufficient governance structures within technology companies, power asymmetries between companies and the people they serve, and a stark cultural divide between the engineering cohort responsible for technical research, and the vastly diverse populations where AI systems are deployed. (Emphasis mine)

There are several institutional reforms called for in the report, but the focus on a particular sector that it constructs as “the technology industry” composed on many “AI systems”, it cannot address broader economic issues such as unfair taxation or gerrymandering. Discussion of the overall economy is absent from the report; it is not the cause of anything. Rather, the root cause is a schism between kinds of people. The moral thrust of this claim hinges on the implied progressivism: the AI/tech people, who are developing and profiting, are a culture apart. The victims are “diverse”, and yet paradoxically unified in their culture as not the developers. This framing depends on the appeal of progressivism as a unifying culture whose moral force is due in large part because of its diversity. The AI developer culture is a threat in part because it is separate from diverse people–code for its being white and male.

This thread continues throughout the report, as various critical perspectives are cited in the report. For example:

A second problem relates to the deeper assumptions and worldviews of the designers of ethical codes in the technology industry. In response to the proliferation of corporate ethics initiatives, Greene et al. undertook a systematic critical review of high-profile “vision statements for ethical AI.” One of their findings was that these statements tend to adopt a technologically deterministic worldview, one where ethical agency and decision making was delegated to experts, “a narrow circle of who can or should adjudicate ethical concerns around AI/ML” on behalf of the rest of us. These statements often assert that AI promises both great benefits and risks to a universal humanity, without acknowledgement of more specific risks to marginalized populations. Rather than asking fundamental ethical and political questions about whether AI systems should be built, these documents implicitly frame technological progress as inevitable, calling for better building.

That systematic critical reviews of corporate policies express self-serving views that ultimately promote the legitimacy of the corporate efforts is a surprise to no one; it is no more a surprise than the fact that critical research institutes staffed by lawyers and soft social scientists write reports recommending that their expertise is vitally important for society and justice. As has been the case in every major technology and ethical scandal for years, the first thing the commentariat does is publish a lot of pieces justifying their own positions and, if they are brave, arguing that other people are getting too much attention or money. But since everybody in either business depends on capitalist finance in one way or another, the economic system is not subject to critique. In other words, once can’t argue that industrial visions of ‘ethical AI’ are favorable to building new AI products because they are written in service to capitalist investors who profit from the sale of new AI products. Rather, one must argue that they are written in this way because the authors have a weird technocratic worldview that isn’t diverse enough. One can’t argue that the commercial AI products neglect marginal populations because these populations have less purchasing power; one has to argue that the marginal populations are not represented or recognized enough.

And yet, the report paradoxically both repeatedly claims that AI developers are culturally and politically out of touch and lauds the internal protests at companies like Google that have exposed wrongdoing within those corporations. The actions of “technology industry” employees belies the idea that problem is mainly cultural; there is a managerial profit-making impulse that is, in large, stable companies in particular, distinct from that the rank-and-file engineer. This can be explained in terms of corporate incentives and so on, and indeed the report does in places call for whistleblower protections and labor organizing. But these calls for change cut against and contradict other politically loaded themes.

There are many different arguments contained in the long report; it is hard to find a reasonable position that has been completely omitted. But as a comprehensive survey of recent work on ethics and regulation in AI, its biases and blind spots are indicative of the larger debate. The report concludes with a call for a change in the intellectual basis for considering AI and its impact:

It is imperative that the balance of power shifts back in the public’s favor. This will require significant structural change that goes well beyond a focus on technical systems, including a willingness to alter the standard operational assumptions that govern the modern AI industry players. The current focus on discrete technical fixes to systems should expand to draw on socially-engaged disciplines, histories, and strategies capable of providing a deeper understanding of the various social contexts that shape the development and use of AI systems.

As more universities turn their focus to the study of AI’s social implications, computer science and engineering can no longer be the unquestioned center, but should collaborate more equally with social and humanistic disciplines, as well as with civil society organizations and affected communities. (Emphasis mine)

The “technology ethics” field is often construed, in this report but also in the broader conversation, as one of tension between computer science on the one hand, and socially engaged and humanistic disciplines on the other. For example, Selbst et al.’s “Fairness and Abstraction in Sociotechnical Systems” presents a thorough account of pitfalls of computer science’s approach to fairness in machine learning, and proposes a Science and Technology Studies. The refrain is that by considering more social context, more nuance, and so on, STS and humanistic disciplines avoids the problems that engineers, who try to provide portable, formal solutions, don’t want to address. As the AI Now report frames it, a benefit of the humanistic approach is that it brings the diverse non-AI populations to the table, shifting the balance of power back to the public. STS and related disciplines claim the status of relevant expertise in matters of technology that is somehow not the kind of expertise that is alienating or inaccessible to the public, unlike engineering, which allegedly dominates the higher education system.

I am personally baffled by these arguments; so often they appear to conflate academic disciplines with business practices in ways that most practitioners I engage with would not endorse. (Try asking an engineer how much they learned in school, versus on the job, about what it’s like to work in a corporate setting.) But beyond the strange extrapolation from academic disciplinary disputes (which are so often about the internal bureaucracies of universities it is, I’d argue after learning the hard way, unwise to take them seriously from either an intellectual or political perspective), there is also a profound absence of some fields from the debate, as framed in these reports.

I’m referring to the quantitative social sciences, such as economics and quantitative sociology, or what might be more be more generally converging on computational social science. These are the disciplines that one would need to use to understand the large-scale, systemic impact of technology on people, including the ways costs and benefits are distributed. These disciplines deal with social systems and include technology–there is a long tradition within economics studying the relationship between people, goods, and capital that never once requires the term “sociotechnical”–in a systematic way that can be used to predict the impact of policy. They can also connect, through applications of business and finance, the ways that capital flows and investment drive technology design decisions and corporate competition.

But these fields are awkwardly placed in technology ethics and politics. They don’t fit into the engineering vs. humanities dichotomy that entrances so many graduate students in this field. They often invoke mathematics, which makes them another form of suspicious, alien, insufficiently diverse expertise. And yet, it may be that these fields are the only ones that can correctly diagnose the problems caused by technology in society. In a sense, the progressive framing of the problems of technology makes technogy’s ills a problem of social context because it is unequipped to address them as a problem of economic context, and it wouldn’t want know that it is an economic problem anyway, for two somewhat opposed reasons: (a) acknowledging the underlying economic problems is taboo under hegemonic neoliberalism, and (b) it upsets the progressive view that more popularly accessible (and, if you think about it quantitatively, therefore as a result of how it is generated and constructed more diverse) humanistic fields need to be recognized as much as fields of narrow expertise. There is no credence given to the idea that narrow and mathematized expertise might actually be especially well-suited to understand what the hell is going on, and that this is precisely why members of these fields are so highly sought after by investors to work at their companies. (Consider, for example, who would be best positioned to analyze the “full stack supply chain” of artificial intelligence systems, as is called for by the AI Now report: sociologists, electrical engineers trained in the power use and design of computer chips, or management science/operations research types whose job is to optimize production given the many inputs and contingencies of chip manufacture?)

At the end of the day, the problem with the “technology ethics” debate is a dialectic cycle whereby (a) basic research is done by engineers, (b) that basic research is developed in a corporate setting as a product funded by capitalists, (c) that product raises political hackles and makes the corporations a lot of money, (d) humanities scholars escalate the political hackles, (e) basic researchers try to invent some new basic research because the politics have created more funding opportunities, (f) corporations do some PR work trying to CYA and engage in self-regulation to avoid litigation, (g) humanities scholars, loathe to cede the moral high ground, insist the scientific research is inadequate and that the corporate PR is bull. But this cycle is not necessarily productive. Rather, it sustains itself as part of a larger capitalist system that is bigger than any of these debates, structures its terms, and controls all sides of the dialog. Meanwhile the experts on how that larger system works are silent or ignored.

References

Fraser, Nancy. “Progressive neoliberalism versus reactionary populism: A choice that feminists should refuse.” NORA-Nordic Journal of Feminist and Gender Research 24.4 (2016): 281-284.

Greene, Daniel, Anna Laura Hoffman, and Luke Stark. “Better, Nicer, Clearer, Fairer: A Critical Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning.” Hawaii International Conference on System Sciences, Maui, forthcoming. Vol. 2019. 2018.

Raicu, Irina. “False Dilemmas”. 2018.

Selbst, Andrew D., et al. “Fairness and Abstraction in Sociotechnical Systems.” ACM Conference on Fairness, Accountability, and Transparency (FAT*). 2018.

Whittaker, Meredith et al. “AI Now Report 2018”. 2018.

## December 14, 2018

Ph.D. student

#### The secret to social forms has been in institutional economics all along?

A long-standing mystery for me has been about the ontology of social forms (1) (2): under what conditions is it right to call a particular assemblage of people a thing, and why? Most people don’t worry about this; in literatures I’m familiar with it’s easy to take a sociotechnical complex or assemblage, or a company, or whatever, as a basic unit of analysis.

A lot of the trickiness comes from thinking about this as a problem of identifying social structure (Sawyer, 200; Cederman, 2005). This implies that people are in some sense together and obeying shared norms, and raises questions about whether those norms exist in their own heads or not, and so on. So far I haven’t seen a lot that really nails it.

But what if the answer has been lurking in institutional economics all along? The “theory of the firm” is essentially a question of why a particular social form–the firm–exists as opposed to a bunch of disorganized transactions. The answers that have come up are quite good.

Take for example Holmstrom (1982), who argues that in a situation where collective outcomes depend on individual efforts, individuals will be tempted to free-ride. That makes it beneficial to have somebody monitor the activities of the other people and have their utility be tied to the net success of the organization. That person becomes the owner of the company, in a capitalist firm.

What’s nice about this example is that it explains social structure based on an efficiency argument; we would expect organizations shaped like this to be bigger and command more resources than others that are less well organized. And indeed, we have many enormous hierarchical organizations in the wild to observe!

Another theory of the firm is Williamson’s transaction cost economics (TCE) theory, which is largely about the make-or-buy decision. If the transaction between a business and its supplier has “asset specificity”, meaning that the asset being traded is specific to the two parties and their transaction, then any investment from either party will induce a kind of ‘lock-in’ or ‘switching cost’ or, in Williamson’s language, a ‘bilateral dependence’. The more of that dependence, the more a free market relationship between the two parties will expose them to opportunistic hazards. Hence, complex contracts, or in the extreme case outright ownership and internalization, tie the firms together.

I’d argue: bilateral dependence and the complex ‘contracts’ the connect entities are very much the stuff of “social forms”. Cooperation between people is valuable; the relation between people who cooperate is valuable as a consequence; and so both parties are ‘structurated’ (to mangle a Giddens term) individually into maintaining the reality of the relation!

References

Cederman, L.E., 2005. Computational models of social forms: Advancing generative process theory 1. American Journal of Sociology, 110(4), pp.864-893.

Holmstrom, Bengt. “Moral hazard in teams.” The Bell Journal of Economics (1982): 324-340.

Sawyer, R. Keith. “Simulating emergence and downward causation in small groups.” Multi-agent-based simulation. Springer Berlin Heidelberg, 2000. 49-67.

Williamson, Oliver E. “Transaction cost economics.” Handbook of new institutional economics. Springer, Berlin, Heidelberg, 2008. 41-65.

## December 09, 2018

Ph.D. student

#### Transaction cost economics and privacy: looking at Hoofnagle and Whittington’s “Free”

As I’ve been reading about transaction cost economics (TCE) and independently scrutinizing the business model of search engines, it stands to reason that I should look to the key paper holding down the connection between TCE and privacy, Hoofnagle and Whittinton’s “Free: Accounting for the Costs of the Internet’s Most Popular Price” (2014).

I want to preface the topic by saying I stand by what I wrote earlier: that at the heart of what’s going on with search engines, you have a trade of attention; it requires imagining the user has have attention-time as a scarce resource. The user has a query and has the option to find material relevant to the query in a variety of ways (like going to a library). Often (!) they will do so in a way that costs them as little attention as possible: they use a search engine, which gives an almost instant and often high-quality response; they are also shown advertisements which consume some small amount of their attention, but less than they would expend searching through other means. Advertisers pay the search engine for this exposure to the user’s attention, which funds the service that is “free”, in dollars (but not in attention) to the users.

Hoofnagle and Whittington make a very different argument about what’s going on with “free” web services, which includes free search engines. They argue that the claim that these web services are “free” is deceptive because the user may incur costs after the transaction on account of potential uses of their personal data. An example:

The freemium business model Anderson refers to is popular among industries online. Among them, online games provide examples of free services with hidden costs. By prefacing play with the disclosure of personal identification, the firms that own and operate games can contact and monitor each person in ways that are difficult for the consumer to realize or foresee. This is the case for many games, including Disney’s “Club Penguin,” an entertainment website for children. After providing personal information to the firm, consumers of Club Penguin receive limited exposure to basic game features and can see numerous opportunities to enrich their play with additional features. In order to enrich the free service, consumers must buy all sort of enhancements, such as an upgraded igloo or pets for one’s penguin. Disney, like others in the industry, places financial value on the number of consumers it identifies, the personal information they provide, and the extent to which Disney can track consumer activity in order to modify the game and thus increase the rate of conversion of consumers from free players to paying customers.

There are a number of claims here. Let’s enumerate them:

1. This is an example of a ‘free’ service with hidden costs to users.
2. The consumer doesn’t know what the game company will do with their personal information.
3. In fact, the game will use the personal information to personalize pitches for in-game purchases that ‘enrich’ the free service.
4. The goal of the company is to convert free players to paying customers.

Working backwards, claim (4) is totally true. The company wants to make money by getting their customers to pay, and they will use personal information to make paying attractive to the customers (3). But this does not mean that the customer is always unwitting. Maybe children don’t understand the business model when they begin playing Penguin Club, but especially today parents certainly do. App Stores, for example, now label apps when they have “in-app purchases”, which is a pretty strong signal. Perhaps this is a recent change due to some saber rattling by the FTC, which to be fair would be attributable as a triumph to the authors if this article had influence on getting that to happen. On the other hand, this is a very simple form of customer notice.

I am not totally confident that even if (2), (3), and (4) are true, that that entails (1), that there are “hidden costs” to free services. Elsewhere, Hoofnagle and Whittington raise more convincing examples of “costs” to release of PII, including being denied a job and resolving identity theft. But being convincingly sold an upgraded igloo for your digital penguin seems so trivial. Even if it’s personalized, how could it be a hidden cost? It’s a separate transaction, no? Do you or do you not buy the igloo?

Parsing this through requires, perhaps, a deeper look at TCE. According to TCE, agents are boundedly rational (they can’t know everything) and opportunistic (they will make an advantageous decision in the moment). Meanwhile, the world is complicated. These conditions imply that there’s a lot of uncertainty about future behavior, as agents will act strategically in ways that they can’t themselves predict. Nevertheless, agents engage in contracts with some kinds of obligations in them in the course of a transaction. TCE’s point is that these contracts are always incomplete, meaning that there are always uncertainties left unresolved in contracts that will need to be negotiated in certain contingent cases. All these costs of drafting, negotiating, and safeguarding the agreement are transaction costs.

Take an example of software contracting, which I happen to know about from personal experience. A software vendor gets a contract from a client to do some customization. The client and the vendor negotiated some sort of scope of work ex ante. But always(!), the client doesn’t actually know what they want, and if the vendor delivers on the specification literally the client doesn’t like it. Then begins the ex post negotiation as the client tries to get the vendor to tweak the system into something more usable.

Software contracting often resolves this by getting off the fixed cost contracting model and onto a cost-and-materials contact that allows billing by hours of developer time. Alternatively, the vendor can internalize the costs into the contract by inflating the cost “estimates” to cover for contingencies. In general, this all amounts to having more contract and a stronger relationship between the client and vendor, a “bilateral dependency” which TCE sees as a natural evolution of the incomplete contract under several common conditions, like “asset specificity”, which means that the asset is specialized to a particular transaction (or the two agents involved in it). Another term for this is lock-in, or the presence of high switching costs, though this way of thinking about it reintroduces the idea of a classical market for essentially comparable goods and services that TCE is designed to mitigate against. This explains how technical dependencies of an organization become baked in more or less constitutionally as part of the organization, leading to the robustness of installed base of a computing platform over time.

This ebb and flow of contract negotiation with software vendors was a bit unsettling to me when I first encountered it on the job, but I think it’s safe to say that most people working in the industry accept this as How Things Work. Perhaps it’s the continued influence of orthodox economics that makes this all seem inefficient somehow, and TCE is the right way to conceptualize things that makes better sense of reality.

But back to the Penguins…

Hoofnagle and Whittington make the case that sharing PII with a service that then personalizes its offerings to you creates a kind of bilateral dependence between service and user. They also argue that loss of privacy, due to the many possible uses of this personal information (some nefarious), is a hidden cost that can be thought of as an ex post transaction cost that is a hazard because it has not been factored into the price ex ante. The fact that this data is valuable to the platform/service for paying their production costs, which is not part of the “free” transaction, is an indication that this data is a lot more valuable than consumers think it is.

I can’t get over the feeling that successfully selling a user a personalized, upgraded digital igloo is such an absurd example of a “hidden cost” that it belies the whole argument that these services have hidden costs.

Splitting hairs perhaps, it seems reasonable to say that Penguin Club has a free version, which is negotiated as one transaction. Then, conditional on the first transaction, it offers personalized igloos for real dollars. This purchase, if engaged in, would be another, different transaction, not an ex post renegotiation of the original contract with the Disney. This small difference changes the cost of the igloo from a hidden transaction cost into a normal, transparent cost. So it’s no big deal!

Does the use of PII create a bilateral dependence between Disney and the users of Penguin Club? Yes, in a sense. Any application of attention to an information service, learning how to use it and getting it to be part of your life, is in a sense a bilateral dependence with a switching cost. But there are so many other free games to play on the internet that these costs seem hardly hidden. They could just be understood as part of the game. Meanwhile, we are basically unconcerned with Disney’s “dependence” on the consumer data, because Disney can get new users easily (unless the user is a “whale”, who actual pays the company). And “dependence” Disney has on particular users is a hidden cost for Disney, not for the user, and who cares about Disney.

The cases of identity theft or job loss are strange cases that seem to have more to do with freaky data reuse than what’s going on with a particular transaction. Purpose binding notices and restrictions, which are being normed on through generalized GDPR compliance, seem adequate to deal with these cases.

So, I have two conclusions:

(1) Maybe TCE is the right lens for making an economic argument for why purpose binding restrictions are a good idea. They make transactions with platforms less incomplete, avoiding the moral hazard of ex post use of data in ways that incurs asymmetrically unknown effects on users.

(2) This TCE analysis of platforms doesn’t address the explanatorily powerful point that attention is part of the trade. In addition to being concretely what the user is “giving up” to the platform and directly explaining monetization in some circumstances, the fact that attention is “sticky” and creates some amount of asset-specific learning is a feature of the information economy more generally. Maybe it needs a closer look.

References

Hoofnagle, Chris Jay, and Jan Whittington. “Free: accounting for the costs of the internet’s most popular price.” UCLA L. Rev. 61 (2013): 606.

## December 06, 2018

Ph.D. student

#### Data isn’t labor because using search engines is really easy

A theme I’ve heard raised in a couple places recently, including Ibarra et al. “Should We Treat Data As Labor?” and the AI Now 2018 Report, is that there is something wrong with how “data”, particularly data “produced” by people on the web, is conceptualized as part of the economy. Creating data, the argument goes, requires labor. And as the product of labor, it should be protected according to the values and practices of labor movements in the past. In particular, the current uses of data in, say, targeted advertising, social media, and search, are exploitative; the idea that consumers ‘pay’ for these services with their data is misleading and ultimately unfair to the consumer. Somehow the value created by the data should be reapportioned back to the user.

This is a sexy and popular argument among a certain subset of intellectuals who care about these things. I believe the core emotional appeal of this proposal is this: It is well known that a few well-known search engine and social media companies, namely Google and Facebook, are rich. If the value added by user data were in part returned to the users, the users, who are compared to Google and Facebook not rich, would get something they otherwise would not get. I.e., the benefits for recognizing the labor involved in creating data is redistribution of surplus to The Rest of Us.

I don’t have a problem personally with that redistributive impulse. However, I don’t think the “data is labor” argument actually makes much sense.

Why not? Well, let’s take the example of a search engine. Here is the transaction between a user and a search engine:

• Alice types a query, “avocado toast recipes”, into the search engine. This submits data to the company computers.
• The company computers use that data to generate a list of results that it deems relevant to that query.
• Alice sees the results, and maybe clicks on one or two of them, if they are good, in the process of navigating to the thing she was looking for in the first place.
• The search engine records that click as well, in order to better calibrate how to respond to others making that query.

We might forget that the search engine is providing Alice a service and isn’t just a ubiquitous part of the infrastructure we should take for granted. The search engine has provided Alice with relevant search results. What this does is (dramatically) reduce Alice’s search costs; had she tried to find the relevant URL by asking her friends, organically surfing the web, or using the library, who knows what she would have found or how long it would take her. But we would assume that Alice is using the search engine because it gets her more relevant results, faster.

It is not clear how Alice could get this thing she wants without going through the motions of typing and clicking and submitting data. These actions all seem like a bare minimum of what is necessary to conduct this kind of transaction. Similarly, when I got to a grocery store and buy vegetables, I have to get out my credit card and swipe it at the machine. This creates data–the data about my credit card transaction. But I would never advocate for recognizing my hidden labor at the credit card machine is necessary to avoid the exploitation of the credit card companies, who then use that information to go about their business. That would be insane.

Indeed, it is a principle of user interface design that the most compelling user interfaces are those that require the least effort from their users. Using search engines is really, really easy because they are designed that way. The fact that oodles of data are collected from a person without that person exerting much effort may be problematic in a lot of ways. But it’s not problematic because it’s laborious for the user; it is designed and compelling precisely because it is labor-saving. The smart home device industry has taken this even further, building voice-activated products for people who would rather not use their hands to input data. That is, if anything, less labor for the user, but more data and more processing on the automated part of the transaction. That the data is work for the company, and less work for the user, indicates that data is not the same thing as user labor.

There is a version of this argument that brings up feminism. Women’s labor, feminists point out, has long been insufficiently recognized and not properly remunerated. For example, domestic labor traditionally performed by women has been taken for granted, and emotional labor (the work of controlling ones emotions on the job), which has often been feminized, has not been taken seriously enough. This is a problem, and the social cause of recognizing women’s labor and rewarding it is, ceteris paribus, a great thing. But, and I know I’m on dicey ground here, so bear with me, this does not mean that everything that women do that they are not paid to do is unrecognized labor in the sense that is relevant for feminist critiques. Case in point, both men and women use credit cards to buy things, and make telephone calls, and drive vehicles through toll booths, and use search engines, and do any number of things that generate “data”, and in most of these cases it is not remunerated directly; but this lack of remuneration isn’t gendered. I would say, perhaps controversially, that the feminist critique does not actually apply to the general case of user generated data much at all! (Though is may apply in specific cases that I haven’t thought of.)

So in conclusion, data isn’t labor, and labor isn’t data. They are different things. We may want a better, more just, political outcome with respect to the distribution of surplus from the technology economy. But trying to get there through an analogy between data and labor is a kind of incoherent way to go about it. We should come up with a better, different way.

So what’s a better alternative? If the revenue streams of search engines are any indication, then it would seem that users “pay” for search engines through being exposed to advertising. So the “resource” that users are giving up in order to use the search engine is attention, or mental time; hence the term, attention economy.

Framing the user cost of search engines in terms of attention does not easily lend itself to an argument for economic reform. Why? Because search engines are already saving people a lot of that attention by making it so easy to look stuff up. Really the transaction looks like:

• Alice pays some attention to Gob (the search engine).
• Gob gives Alice some good search results back in return, and then…
• Gob passes on some of Alice’s attention through to Bob, the advertiser, in return for money.

So Alice gives up attention but gets back search results and the advertisement. Gob gets money. Bob gets attention. The “data” that matters is not the data transmitted from Alice’s computer up to Gob. Rather, the valuable data is the data that Alice receives through her eyes: of this data, the search results are positively valued, the advertisement is negatively valued, but the value of the bundled good is net positive.

If there is something unjust about this economic situation, it has to be due to the way consumer’s attention is being managed by Gob. Interestingly, those who have studied the value of ‘free’ services in attentional terms have chalked up a substantial consumer surplus due to saved attention (Brynjolfsson and Oh, 2012) This appears to be the perspective of management scientists, who tend to be pro-business, and is not a point repeated often by legal scholars, who tend to be more litigious in outlook. For example, legal scholarship has detailed the view of how attention could be abused through digital market manipulation (Calo, 2013).

Ironically for data-as-labor theorists, the search-engine-as-liberator-of-attention argument could be read as the view that what people get from using search engines is more time, or more ability to do other things with their time. In other words, we would use a search engine instead of some other, more laborious discovery mechanism precisely because it would cost us net negative labor. That absolutely throws a wrench in any argument that the users of search engines should be rewarded on dignity of labor grounds. Instead, what’s happened is that search engines are ubiquitous because consumers have undergone a phase transition in their willingness to work to discover things, and now very happily use search engines which, on the whole, seem like a pretty good deal! (The cost of being-advertised-to is small compared to the benefits of the search results.)

If we start seeing search engines as a compelling labor-saving device rather than a exploiter of laborious clickwork, then some of the disregard consumers have for privacy on search engines becomes more understandable. People are willing to give up their data, even if they would rather not, because search engines are saving them so much time. The privacy harms that come as consequence, then, can be seen as externalities to what is essentially a healthy transaction, rather than a perverse matter of a business model that is evil to the bone.

This is, I wager, on the whole a common sense view, one that I’d momentarily forgotten because of my intellectual milieu but now am ashamed to have overlooked. It is, on the whole, far more optimistic than other attempt to characterize the zeitgeist of new technology economy.

Somehow, this rubric for understanding the digital economy appears to have fallen out of fashion. Davenport and Beck (2001) wrote a business book declaring attention to be “the new currency of business”, which if the prior analysis is correct makes more sense than data being the new currency (or oil) of business. The term appears to have originated in an article by Goldhaber (1997). Ironically, the term appears to have had no uptake in the economics literature, despite it being the key to everything! The concept was understood, however, by Herbert Simon, in 1971 (see also Terranova, 2012):

In an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.

(A digitized version of this essay, which amazingly appears to be set by a typewriter and then hand-edited (by Simon himself?) can be found here.)

This is where I bottom out–the discover that the line of thought I’ve been on all day starts with Herbert Simon, that the sciences of the artificial are not new, they are just forgotten (because of the glut of other information), and exhaustingly hyped. The attention economy discovered by Simon explains why each year we are surrounded with new theories about how to organize ourselves with technology, when perhaps the wisest perspectives on these topics are ones that will not hype themselves because their authors cannot tweet from the grave.

References

Arrieta-Ibarra, Imanol, et al. “Should We Treat Data as Labor? Moving beyond” Free”.” AEA Papers and Proceedings. Vol. 108. 2018.

Brynjolfsson, Erik, and JooHee Oh. “The attention economy: measuring the value of free digital services on the Internet.” (2012).

Calo, Ryan. “Digital market manipulation.” Geo. Wash. L. Rev. 82 (2013): 995.

Davenport, Thomas H., and John C. Beck. The attention economy: Understanding the new currency of business. Harvard Business Press, 2001.

Goldhaber, Michael H. “The attention economy and the net.” First Monday 2.4 (1997).

Simon, Herbert A. “Designing organizations for an information-rich world.” (1971): 37-72.

Terranova, Tiziana. “Attention, economy and the brain.” Culture Machine 13 (2012).

## December 04, 2018

Ph.D. student

#### the make or buy decision (TCE) in the software and cybersecurity

The paradigmatic case of transaction cost economics (TCE) is the make-or-buy decision. A firm, F, needs something, C. Do they make it in-house or do they buy it from somewhere else?

If the firm makes it in-house, they will incur some bureaucratic overhead costs in addition to the costs of production. But they will also be able to specialize C for their purposes. They can institute their own internal quality controls. And so on.

If the firm buys it on the open market from some other firm, say, G, they don’t pay the overhead costs. They do lose the benefits of specialization, and the quality controls are only those based on economic competitive pressure on suppliers.

There is an intermediate option, which is a contract between F and G which establishes an ongoing relationship between the two firms. This contract creates a field in which C can be specialized for F, and there can be assurances of quality, while the overhead is distributed efficiently between F and G.

This situation is both extremely common in business practice and not well handled by neoclassical, orthodox economics. It’s the case that TCE is tremendously preoccupied with.

My background and research is in the software industry, which is rife with cases like these.

Developers are constantly faced with a decision to make-or-buy software components. In principle, they can developer any component themselves. In practice, this is rarely cost-effective.

In software, open source software components are a prevalent solution to this problem. This can be thought of as a very strange market where all the prices are zero. The most popular open source libraries are very generic , having little “asset specificity” in TCE terms.

The lack of contract between developers and open source components/communities is sometimes seen as a source of hazard in using open source components. The recent event-stream hack, where an upstream component was injected with malicious code by a developer who had taken over maintaining the package, illustrates the problems of outsourcing technical dependencies without a contract. In this case, the quality problem is manifest as a supply chain cybersecurity problem.

In Williamson’s analysis, these kinds of hazards are what drive firms away from purchasing on spot markets and towards contracting or in-house development. In practice, the role of open source support companies fills the role of being a responsible entity G that firm F can build a relationship with.