Category: Mac/iOS Projects

I’m working on a new app

I was inspired recently and decided to write another app. I’m going to write a note-taking app that lets you nest notes in a hierarchical fashion and use markdown for text formatting.

I wanted to understand what challenges it might impose, so I decided to prototype some UI today.

Here’s what I’ve got so far.

After about four hours of coding, the following work:

  • Basic data type to represent a node and its children.
  • Functioning UI to view the nodes. You can select a node and view its children. As you dig deeper into a hierarchy, more and more tables are added and you can scroll left and right to view each level.
  • Editing and adding of nodes works. You can double-click a node and a modal window pops up to let you edit the text. When done, the changes are saved and the UI is updated. You can add a new node with a shortcut key.
  • Markdown formatting. I used SwiftyMarkdown to convert my markdown text into an NSAttributedString that I could display in the table view. Unfortunately, SwiftyMarkdown doesn’t build for macOS, only iOS. I hacked it quickly so that it doesn’t rely on UIFont. The hack is temporary, and I’ll likely re-do this work later and send a pull request to the author.

The big thing that doesn’t work yet is saving and loading a document. I want a working end-to-end scenario soon, so even though I’m still thinking about what I need in the document format, I’ll move forward with a simple JSON dictionary for the time being. (This file format won’t cut it if I want to sync across multiple devices. I’ll probably do something similar to what I did with Lil Todo.)

The app will be made up of nested tables, as you’d see in a UI like macOS’s Finder. Each table will contain a list of notes. Each note will use markdown for formatting. For simplicity, notes don’t have titles. They just have text. When you click on a note, you’ll see a list of additional notes to its right. These are sub-notes. This will allow you to easily create hierarchical notes, like an outline. You’ll be able to add as many sub-notes as you’d like to any note. You can also drag and drop notes around to change the hierarchy or grouping of things.

I will admit the UI is somewhat inspired by gingko, however I don’t plan on showing all the notes at a given level as they do there. (I find that a little confusing.) I’ve always been fascinated by using nesting to organize thoughts. A couple of other apps that I’ve enjoyed using that are somewhat related are MindNode and Tree. Both are simple and beautiful apps.

Some other features that I plan for this app:

  • Images as notes. You’ll be able to drag an image as a note anywhere. To keep the UI simple, I’m not planning on “embedding” images within the text. They’ll simply be treated as other notes.
  • Cloud file sync. I’ll support Dropbox again and possible other cloud file sync engines. I don’t plan on writing a custom service for this, but the app will support single-user editing a document across multiple devices.
  • Views. It should be possible to collect all notes that have to-do items in them and present them all the to-dos in a massive list. It should also be possible to collect all notes that have a hashtag in them and also present them in a massive list. These will be treated as “views” that allow you to look at your data in a different way.

My plan for this app is to write it as quickly as possible, get an MVP working, and ship it to the Mac App Store. It’s March 5, 2017, so let’s see how quickly I can do that. My second goal is also to write up development notes on this blog to share some of the techniques I use to write software.

If you got this far, thanks for reading! Be sure to also follow me on twitter for updates.

How Lil Todo Syncs Tasks Across Multiple Devices Just Using Dropbox

Lil Todo is a to-do app that I wrote for the Mac. (An iOS version of the app is coming soon!) One of the great things about it is you can run it on multiple Macs and use a single Dropbox account to save and sync all of your tasks. Mark a task completed on one device and the change shows up on all the other devices.

Since the app just uses Dropbox and doesn’t use its own sync service, you don’t need to create a new account to sync (assuming you already have a Dropbox account). This is great for me too because it means I don’t have to write and maintain a sync service. 🙂

So how does that work? From the user perspective, it sounds so simple, but behind the scenes it’s not as straightforward as you’d think.

Dropbox == Simple File Store

Dropbox is just a file store and doesn’t really handle multiple edits to the same file very well. If two people edit the same file and both attempt to save it, Dropbox will detect a conflict and save two copies of the file, letting you pick which of the two to keep. If we used a single file to represent our database of tasks in Dropbox and had multiple Macs editing and saving that file, we’d run into conflicts all the time.

If that’s the case, how can we use Dropbox to represent our database of tasks across multiple devices?

The Solution: One Weird Trick

The answer is simple but not necessarily intuitive: have each device running Lil Todo write to its own file in Dropbox. If each instance of the app has its own file to write to, we’d never have any conflicts since each file has a single owner. (Multiple devices are okay to read that same file, of course.)

That’s great, but how do you represent a database of tasks using multiple files, one for each device?

Here’s the novel bit: instead of writing to a database of tasks, have each device write the changes it would like to make to the database. In other words, each file for each device is a transaction log of all edits made from that device. (This is also known as Event Sourcing, although when I came up with my initial solution, I had no idea it was already a thing. I promise!)

Transaction Logs, Baby

When you launch Lil Todo for the first time on a device, the app is assigned a universally unique identifier (UUID). This identifier is guaranteed to be unique across all devices and is used to name the transaction log for that particular device.

When you create your first task, a “Create task” entry is added to the log with details of the new task. When you mark a task done, an “Edit task” entry is added to the log with the ‘done’ property set to true. In fact, all edits to a task (changing the title, due date, or notes) causes a new “Edit task” entry to be added to the log with a list of the properties that have changed. (Even deleting a task simply adds an “Edit task” entry with the ‘deleted’ property to true. Tasks aren’t actually deleted in the file.)

As you can see, we don’t actually write to a traditional database of tasks. Instead, we write a history of edits to all tasks.

This transaction log file grows over time and represents all task creations and edits made from that device. When you sync to Dropbox, we upload the latest version of your device’s transaction log to the Lil Todo folder in Dropbox. We also download all the transaction logs from all other devices. We then take all the transaction logs and merge them in memory to form one large, universal transaction log (which I call Voltron).

Every entry in the transaction logs has a timestamp to ensure we can sort all of the logs across all devices in the order in which they occurred. It’s simply a matter of “playing” back all entries in the universal log and reconstructing the entire database in memory. Once we have the full database reconstructed, we’re ready to show it in the UI and let the user make edits (which, remember, simply cause new entries to be added to the log for that particular device).

Dealing with Conflicts

Note that if you try to edit the same task on two different devices at about the same time, the most recent edit will “win”. In the unlikely case that the edits actually happen at exactly the same time, the device with the UUID that appears later sorted alphabetically will “win”. The system is deterministic. It has to be to ensure consistency across all devices. Once all devices sync with the most recent version of all logs, their database contents will all match. (This is a form of eventual consistency!)

It’s worth noting that the timestamps across all the devices aren’t necessarily perfectly in sync. Each device uses UTC for its timestamps, but it’s possible they could be off by a second or two. For the goals of this app, this is acceptable as the target scenario for this app is a single user jumping from device to device to make edits. I don’t expect the target audience to be editing the same task on two different devices within a second or two of each other. Most users will make a bunch of edits on a device and later move to another device later (at least not seconds) to make more edits.

This app works great for a single user, but if this app required multiple users across multiple devices, it would not work as well. We’d need to have much more active syncing of the files to ensure each user has the most up to date version. At the moment, the app syncs to Dropbox every few minutes as needed.

Performance Concerns

Replaying the transaction log does take longer and longer as it grows. At the moment, a cached version of the database is written to disk and loaded on launch to avoid the perf hit of regenerating the it, but the database does have to be regenerated if the app detects that there are new entries added to any of the transaction logs from other devices during sync.

It should be possible to alleviate some of these performance issues by occasionally creating snapshots of the database given the current state of the logs. When new transaction logs are downloaded, we’d simply re-play the universal transaction log from the most recent database snapshot that did not contain the first new log entry. (This snapshot feature has not yet been implemented in the interest of shipping the product fast! Once the author starts to feel the pain of long load-times, he promises to add this feature. 😁)

Closing Thoughts

Overall, I’m quite happy with the implementation. It means I don’t have to create a new sync service. Creating a new service to host other people’s private tasks is a big deal. I’d have to worry about security, maintenance, and long-term plans (I’d have to keep the service up if people relied on it).

This sync system has been reliable thus far and there are no worries about race conditions or locking issues since each device is solely responsible for its own transaction log. The system can also scale up indefinitely (at the cost of perf) since each device gets a UUID and it’s simply a matter for all devices to “notice” the new transaction log in the shared app folder on Dropbox. There is no central registry of devices, just transaction logs representing each device. No locking of resources is ever needed.

In the future, I’d like to refactor some of the code I’ve written for this so it can be generalized to any file store (cloud or otherwise) and any type of database. If I do that, I can post it on github so others can use it.

If you’re considering writing an app and using a third party file store like Dropbox to sync across devices for a single user, I recommend giving this technique a try.

If you got this far, thanks for reading! I’d love to hear your thoughts below.

One last plug: give Lil Todo a try while you’re here! Please do check out my other projects as well.

Little Phrase Book v1 for Mac released

About a week ago, I released Little Phrase Book for macOS to the Mac App Store.

If you’re learning a new language and want a way to keep track of all the foreign phrases you’re learning, try out Little Phrase Book. You can easily search existing phrases by typing in the search field and seeing results displayed right away. It also has a built-in flash card feature so that you can review all of your phrases and commit them to memory over time.

Download Little Phrase Book today from the Mac App Store.

I’m quite proud of this release because unlike a lot of projects I’ve worked on in the past, I released this one as soon as it was good enough. Essentially, I had in mind a sort of minimum viable product in mind, and when it got to that stage, I cleaned up the UI, fixed as many bugs that prevented the golden scenarios from working, and then sent it off to Apple to review.

In all, it took less than a month to go from the first commit to it appearing in the store. With that in mind, I also didn’t work on it nearly that much. In total number of hours, I’d guess I spent maybe 20 hours working on this app before shipping.

I have more plans ahead for it, but it was important to get the basic features working so that I could dogfood it daily to see what worked and what didn’t. I added a very early flash card implementation. It’s quite naïve at the moment, but it’ll be full-featured in the future. I plan on implementing a spaced-repetition algorithm so that you can truly use this app daily to memorize things.

Another thing I’ve noticed is that the app is actually quite useful for just storing general notes that you need to recall occasionally. For instance, there are some command-line tools at work that I need to use, but their syntax is a little cryptic and hard to remember. I just created a phrase book for commands and whenever I need to “recall” the cryptic command, I just type in the search term and it immediately appears. This is much faster than putting things in a notebook or text file and having to do a cmd-F to find the entry.