Crawling on the IndieWeb

2026-08-12

What is IndieWeb?

The IndieWeb is a movement centered around using your own website and domain as your identity. Instead of joining a platform, your website is the place where you publish, and technologies such as Webmention allow you to interact with other sites.

It is, in some ways, just like running a blog, but with the ability to participate in a community without giving up ownership of your presence.


Why do I care?

As someone who loves writing random webpages, the IndieWeb is incredibly intriguing to me. It is also a wonderful developer challenge: an excuse to learn about web standards, APIs, data structures, and all the little pieces that make the web work.

Adding interaction makes it even more interesting. A personal website starts to become a minimal social media client of its own. That feels both fun and empowering.

But it is certainly not for the everyday user who just wants to create an account and immediately start publishing and interacting. Depending on how involved you want to be, there can be quite a bit of work involved.

I have read quite a few posts arguing that the IndieWeb's technical barrier is one of its biggest downsides. There is some truth to that, depending on your perspective. I ran into that barrier myself when I tried to set up a minimal blog that could participate in the IndieWeb. There is a surprising lack of options for someone who simply wants to get started quickly.

For me, though, accessibility and popularity are not really the point. I am not much of a social media person. My enjoyment of the IndieWeb and related technologies depends much more on whether I can successfully implement them and achieve useful interoperability with the rest of the ecosystem.

I am also perfectly comfortable with the IndieWeb remaining a small and somewhat secluded community rather than becoming mainstream. This is not about tech elitism. I do not build these things to feel superior to people who did not build their own IndieWeb site. Quite the opposite: the reason I want to participate is to connect with people who are actively building and using this stuff.

People also sometimes ask: why invent and use Webmentions when technologies such as email already exist?

Because they solve different problems.

Imagine someone wants to interact with a blogger. Without the IndieWeb, they can send an email to the address listed on the site, but that interaction is private to the participants of the email conversation. Other readers cannot see it or respond to it in the context of the original post.

Webmentions make that interaction part of the web page itself.

There are plenty of other social protocols too, some standardized by the W3C and some not. But most social networks are account- and profile-centric. The IndieWeb is website-centric: your website is your identity, profile, and publishing platform.

These technologies are therefore not necessarily competitors. They solve different problems. The interesting part is knowing which tool to use for the problem you are trying to solve.


How do I join?

I searched Google and GitHub extensively for ready-to-deploy projects and hosted platforms.

One notable option is Micro.blog, but it is a paid service. That is a non-starter for someone with a zero-cost tinkering philosophy like mine. Another is Idno, formerly known as Known. It is much more feature-rich, so I gave it a try and eventually used it as a reference implementation for my own project.

I also discovered the many small components in the p3k ecosystem. Once I opted into IndieAuth, a surprisingly large number of these tools became available to me.

The first thing I did was pick a domain and put it on a free shared hosting platform with PHP. Those are surprisingly easy to find if you know where to look. I then started adding IndieAuth delegation links to my HTML <head>.

I chose email as the authentication method because it was the lowest-friction option, although it does expose your email address. The alternative was PGP. I tried PGP multiple times and it simply would not work for me. To this day, I still have not figured out why.

Soon, IndieAuth was working. I started exploring various p3k components and found even more links that I could add to my site. One of the most important was Webmention.io, which meant I did not have to implement Webmention handling myself. Thanks, Aaron.

At this point, I could already receive Webmentions simply by adding links to HTML. How cool is that?

And I had no backend at all. The page was essentially static. You could even host something like this on Neocities' free tier.

Then I tried Quill.

I could not use it because I did not have a Micropub endpoint. That sent me looking for a plug-and-play Micropub implementation that I could host myself. I could not find one that fit what I wanted, so I decided to build one.

I wanted something minimal and working, and I wanted to open-source it so that other people curious about the IndieWeb could get started more easily than I did.

Known

You may be wondering: what about Known?

I successfully hosted Known on shared hosting, and the IndieWeb features worked surprisingly well. I had a great time publishing with it, and I very much appreciate that the project exists and that people are still slowly working on it.

There are barriers, though. For example, the Like and Bookmark plugin did not send Webmentions when I expected it to. I ended up using Quill to send likes and bookmarks so that the appropriate endpoint would actually trigger the outgoing Webmentions.

The larger barrier was the PHP codebase itself. Adding another link to the HTML <head> or injecting a hidden link into a post sounds like a tiny modification, but Known has many components wired together. Before long, I found myself reading and searching through the code almost like a contributor.

At this point, I still did not know which approach I wanted to use for my long-term IndieWeb presence.


POSSE: Publish (on your) Own Site, Syndicate Elsewhere

I started reading about POSSE and realized that Known had fairly limited options when it came to POSSE support and flexibility. It had a Bridgy plugin for posting to Twitter, which is obviously dysfunctional now.

I had read about Bridgy in the IndieWeb documentation, but I did not really understand how POSSE worked yet.

So I did what I usually do when I want to understand something: I started messing with it.

I created Bluesky and Mastodon accounts and connected them to Bridgy. Then I used its web interface to connect both my own implementation and my Known instance.

I tested manual publishing, and that was the moment I realized how ridiculously cool Bridgy was. It felt like magic. I could not believe cross-posting could be this easy.

I then read the documentation about automation and discovered two options:

  1. Webmention: insert a hidden <a> link into the HTML content.
  2. Micropub: syndicate the entire post to Bridgy's Micropub endpoint, using a different publishing token for each syndication target.

That gave me an idea: I could build a Bridge Publish plugin for Known that used Bridgy to POSSE to social silos.


Bridgy Publish Plugin for Known

I did not want to modify Known's Micropub endpoint, because then it would not really be a plugin anymore. So I chose the Webmention approach.

Naturally, that meant the plugin could only work with content that supported HTML.

I based the plugin on Known's webhook plugin and started modifying it. After a lot of testing and code reading, I finally managed to insert a hidden link into the HTML, make sure it survived post-processing, and verify that it actually appeared where it needed to.

I even followed Known's i18n procedure and used Node.js to generate a language file.

Looking back, I think more Known users might find the plugin useful. But I am not much of a social media person, and I do not actively promote my open-source projects, so my plugin still has exactly zero stars.

And honestly, I am okay with that.


My IndieWeb Implementation

As I read more documentation and code, I became increasingly comfortable with the idea of writing my own Micropub implementation.

I started by experimenting with the data Quill sends and modeled my endpoint around processing and saving that data as raw JSON files.

Why not a database?

Because fewer dependencies are better.

I do want to learn more technologies, but I appreciate minimalism and lightweight systems even more. So file-based storage it is.

Once Quill could find my Micropub endpoint and publish successfully to my blog, I started polishing the post page that displays the content.

And there we go: a minimal IndieWeb implementation. I could publish to my own site.

Perhaps this blog post shouldn't really be a development journal or a collection of commit messages. I spent a lot of time making the implementation work the way I wanted while following the relevant standards. I have continued to polish and maintain it intermittently ever since, one idea and one feature at a time.

For more details, there is a project README. But the short version is that I built something very easy to deploy and drastically lowered the technical barrier to getting started.

It is still not for everyone. You need to be relatively technical to understand the instructions and put the pieces together.

But that is exactly what I wanted: not to force IndieWeb to become mainstream, but to make it a lot easier for the next curious person to crawl in.