Export all your Kindle highlights and notes

19 Sep 2019 projects reading

Previously, I’ve talked about the current state of annotations on the digital books and documents.

On this post, I want to introduce a tool I developed to help Kindle users handle their highlights, notes, bookmarks, etc.

Getting annotations from your Kindle

So you’ve been reading on your Kindle device. You highlighted some interesting parts. You wrote important notes. And you bookmarked relevant pages. Now how do you access it outside of the Kindle?

The Kindle has a mobile app and a web page where you can see your annotations and also export them. But whether your annotations for a particular book/document will be available there depends on how you acquired it.

I made a table to illustrate how it currently works (2019):

Got from…	Mobile app	Web
Bought on Amazon	✅	✅
Sent via e-mail	✅	❌
Stored via USB	❌	❌

So if you bought the book from Amazon, you have the best of worlds. You can access your annotations from Amazon’s mobile app or the web.

If you sent the content to your Kindle e-mail, Amazon automatically tagged it as “personal document” (even if it is a book). Its annotations will only be available on the mobile app.

Lastly, if you added content to your Kindle via USB, bad luck. No way to access its annotations or the content out of the Kindle.

My Clippings to the rescue

So we get a somewhat fragmented experienced. Depending on how you acquired each content, its annotations are available on one place or another; and thus are easily exportable or not.

Luckily, the Kindle stores a plain text file on its documents folder called My Clippings. For each highlight, note, bookmark and clip you set while on your Kindle, it appends this file with the corresponding information.

My Clippings shortcomes

Now, this file is very handy, no doubt. But yet it is kind of a mess. As I said, it is appended whenever you do one of the above operations. This means:

It is not ordered by document, but by time. So, for instance, if you switch reading between two books, you’ll get some annotations for one, then for the other, then for the first again, etc.
As the entries are ordered by time, if you annotate something in page 100, and later on page 50, the later will be after the former.
If you edit or delete a highlight, note, etc, the file doesn’t reflect it. It adds the newly edition, but keeps the previous one. It doesn’t delete any entry. The result is tons of similar, duplicated, or previously removed entries floating on there.
For every append, it adds info about the correspondent book, the time the append was done, etc. The actual content can be missed with all that noise, making the file difficult to read.

Solutions available

So while having tons of valuable data, this file is not really readable nor easily browseable. The obvious thing to do was to try to parse it. I looked for applications to do that and found a few: web and desktop. However, none satisfied me:

Most of them didn’t work. While I didn’t investigate the causes for each codebase, some failed because locale specific strings were hardcoded.
Web based solutions seemed to add complexity to the problem. Having to periodically upload My Clippings to a platform, manage my data there in a complex interface and then export my annotations again seemed overkill.

I didn’t try all the applications available. I quickly realized that it would be much better to code my own solution from ground up, than testing many until one worked or reasonably satisfied me. Besides, I still would have to adapt them to my needs.

Meet Fyodor

So I created a tool to solve those problems. It’s Fyodor, written in Ruby and libre and open-source.

Fyodor parses all the precious data from My Clippings and makes it readable. It generates well formatted markdown files: one per book or document that you read.

Some of the features:

Your highlights and notes are ordered by page (not time).
Cleans up empty and duplicated entries as possible.
Tries to be locale agnostic, and allows configuration for different Kindle languages.
The output is easily editable and read by virtually any computer (plain text).
And more…

The workflow, pros and cons

To extract your Kindle annotations:

Connect your Kindle via USB.
Copy My Clippings from documents folder to your computer.
Run Fyodor.
Now you have a bunch of markdown files, one for each document. Edit them as you will.

Advantages of this method:

You extract all the annotations from all your Kindle content with one command.
You don’t need to remember how each document got to your Kindle just to export its annotations.
The content you loaded via USB can finally have their annotations set free from the device (remember the table?).
All your annotations are stored in one single folder, conveniently organized per book.
Clean and editable output.

Of course, we still have some limitations that the Kindle imposes us:

We can’t guess all highlights and notes that were deleted.
We don’t have chapter information.
We need to transfer My Clippings via USB.

But yet, this is a much better state of affairs, and I’m fairly satisfied. I hope this app makes your life better. Check it out here.