Enabling Researchers is a Moral Imperative

Enabling researchers is a moral imperative, even for small companies, because internet platforms are increasingly reverting back to closed systems. As Rachelle Faust and Daniel Arnaudo put it in The World’s Growing Information Black Box: Inequity in Platform Research (2025):

Ultimately, platforms have moved away from independent data access because they view it as a cost center, with little upside to sharing information that they view as potentially damaging, and transparency is a risk. The attacks on Rohingya people in Myanmar in 2015, where posts on Facebook contributed to the violence, spurred criticism and more data transparency efforts to show platforms were responsive and open.

But wait - there’s another upside. Independent research can generate novel customer insights. It can uncover uncomfortable truths you might gloss over otherwise. Large firms would rather hire expensive consultancies than maintain access for independent research. They don’t really have to worry about these problems. And that’s exactly the edge for small and medium-sized organizations and businesses.

The orthogonal bet is more important than ever. As I wrote in Finding Your Place in an Unimaginably Large Space, the subscription marketplace is dominated by a handful of businesses and only becoming more concentrated as infrastructure and outreach costs rise. The way forward is to invest in the network as the incumbents invest in their silos. Networks of organizations make them more resilient. Networks of data make it more valuable.

And so we are opening Yorba up to our first researchers from the Nova School of Business and Economics, Daniela Schmitt and Rodrigo Belo.

The Internet is Closing

The Association of Internet Researchers and the Coalition for Independent Technology Research were deeply critical a few years back when Twitter/X began closing their API and replaced it with an expensive and under-performing alternative. A researcher today can no longer build a complete archive to study as the API returns what X considers “relevant.”

Reddit is also closing its public API by March 2027 as well as the RSS feeds. Researchers must now contact Reddit to request research access and new data retention restrictions could make reproducible results - the cornerstone of research - impossible.

And then there is Meta/Facebook/Instagram. Like all large platforms, they explicitly prohibit automated data collection by scraping their site. They have replaced their old researcher access known as CrowdTangle with a more restrictive Meta Content Library and Content Library API.

Opening Up While Keeping Data Safe

It’s true that designing and maintaining an expressive API that works for researchers is expensive. Therefore we have focused on tagging the data itself using Atlas, a library built for describing systems and information. This ensures that the data can be served in many different contexts with certain guarantees. A person will always need access to their data, but that same data could be anonymized and viewed by researchers. While system performance metrics can rely on anonymized data, system logging is more useful with pseudonymous data and a short retention policy. Yorba is not just investing in research access, we’re investing in data integrity as a concept.

The front door for researchers to explore the data is through our adoption of open vocabularies and MCP tooling. After the researchers get a sense of what they want, we can either build a custom API or query the database directly. Either way we’re still leveraging the benefits of strong data semantics via Atlas.

Data labeling itself can be difficult. Over time names will change, contexts shift, and labels will be forgotten. To avoid another maintenance headache, Yorba entities are referenced with a compound identity. An endpoint that provides an aggregate of subscription costs might be labeled as #{:metrics/counter :domain/fees :domain/metrics}. Labels generally accrete and Atlas ensures no two sets of label identifiers are the same; every identity remains unique. Over time, graph relations emerge so that developers, researchers, and LLMs can see what is missing or codependent in the web of Yorba data.

We can even tag data with something like :todo/pii-protection when the data type emitted needs to be reviewed before it is available to any tooling. Thus a semi-regular review can always surface the known unknowns easily.

The tools that provide these insights are also tagged. Therefore only tools tagged with :scope/research are available to researchers.

Build-time invariants keep research tooling sanitized. A declared but unlabeled field like :personal-information in {:provider-url "example.com" :personal-information "abc-123"} fails the build, even if :provider-url is tagged as public information. Any tool tagged :scope/research that returns data tagged :co.yorba.user/data will also fail to build.

The Pendulum

We have published our own research as our own annual State of Clutter reports. But that’s the easy part. The hard part is opening up and allowing others to come to their own conclusions.

The history of systems is a pendulum swinging between open and closed. The current enclosure of intangible resources both artificially abandons the free market while also ensuring that a social commons cannot grow. In other words, it works for almost no one other than a handful of people.

The pendulum will swing back. The organizations and businesses that survive will be the ones prepared for it.

Image: Camillo Golgi, Nerve cells in a dog’s olfactory bulb (detail), 1885, illustration from Sulla fina anatomia degli organi centrali del sistema nervoso, via The Public Domain Review. Public Domain.