![](\"https:\/\/www.ussherpress.com\/liltodo\/screen1.jpg\")
<\/p>\n<\/p>\n
Lil Todo<\/a> is a to-do app that I wrote for the Mac.\u00a0(An iOS version of the app\u00a0is coming soon!) One of the great things about it is you can run it on multiple Macs and use a single Dropbox account to save and sync all of your tasks. Mark a task completed on one device and the change shows up on all the other\u00a0devices.<\/p>\n Since the app just uses Dropbox and doesn’t use its own sync service, you don’t need to create a new account to sync (assuming you already have a Dropbox account). This is great for me too because it means I don’t have to write and maintain a sync service. \ud83d\ude42<\/p>\n So how does that work? From the user perspective, it sounds so simple, but behind the scenes it’s not as straightforward\u00a0as you’d think.<\/p>\n Dropbox is just a file store\u00a0and doesn’t really handle multiple edits to the same file very well. If\u00a0two people edit the same file and both attempt to save it, Dropbox will detect a conflict and save two copies of the file, letting you pick which of the two to keep. If\u00a0we used a single file to\u00a0represent our database of tasks in Dropbox and had multiple Macs editing\u00a0and saving that file, we’d run into conflicts\u00a0all the time.<\/p>\n If that’s the case,\u00a0how\u00a0can we use Dropbox to represent our database of tasks across multiple devices?<\/p>\n The answer is simple but not necessarily intuitive: have each device running Lil Todo\u00a0write to its own file in Dropbox. If each instance of the app has its own file to write to, we’d never have any conflicts since each file has a single owner. (Multiple devices are okay to read that same file, of course.)<\/p>\n That’s great, but how do you represent a\u00a0database of tasks\u00a0using multiple files, one for each device?<\/p>\n Here’s the novel bit: instead of writing to a database of tasks, have each device write the changes it would like to make to the database. In other words, each file for each device is a transaction log<\/a> of all edits made from that device. (This is also known as Event Sourcing<\/a>, although when I came up with my initial solution, I had no idea it was already a thing. I promise!)<\/p>\n When you launch Lil Todo for the first time on a device, the app is assigned a universally unique identifier<\/a> (UUID). This identifier is guaranteed to be unique across all devices and is used to name the transaction log for that\u00a0particular device.<\/p>\n When you create your first task, a “Create task” entry is added to the log with details of the new task. When you mark a task done, an “Edit task” entry is added to the log with the ‘done’ property set to true. In fact, all edits to a task (changing the title, due date, or notes) causes a new “Edit task” entry to be added to the log with a list of the properties that have changed. (Even deleting a task simply adds an “Edit task” entry with the\u00a0‘deleted’ property to true. Tasks aren’t actually deleted in the file.)<\/p>\n As you can see, we don’t actually write to a traditional database of tasks. Instead, we write a history of edits to all tasks.<\/p>\n This transaction log file grows\u00a0over time and represents all task creations and edits made from that device. When you sync to Dropbox, we upload the latest version of your device’s transaction log to the Lil Todo folder in Dropbox. We also download all the transaction logs from all other devices.\u00a0We then take all the transaction logs\u00a0and merge them in memory to form one large, universal transaction log\u00a0(which I call Voltron).<\/p>\n Every\u00a0entry\u00a0in the transaction logs has\u00a0a timestamp\u00a0to ensure we can sort all of the logs across all devices in\u00a0the order in which they occurred. It’s simply a matter of\u00a0“playing”\u00a0back all entries in\u00a0the universal log and reconstructing the entire database in memory. Once we have\u00a0the full database reconstructed, we’re ready to show it in the UI and let the user make edits (which, remember, simply cause new entries to be added to the log for that particular device).<\/p>\n Note that\u00a0if you try to edit the same task on two different devices at about the same time, the most recent edit will “win”. In the unlikely\u00a0case that the edits actually happen at\u00a0exactly the same time, the device with the UUID that appears\u00a0later sorted alphabetically will “win”. The system is deterministic. It has to be to ensure consistency across all devices. Once all devices sync with the most recent version of all logs, their database contents will all match. (This is a form of\u00a0eventual consistency<\/a>!)<\/p>\n It’s worth noting that the timestamps across all the devices aren’t\u00a0necessarily perfectly in sync. Each device uses UTC<\/a>\u00a0for its timestamps, but\u00a0it’s possible they could be off by a\u00a0second or two. For the goals of this app, this is acceptable as the target scenario for this app is a single user jumping from device to device to make edits. I don’t expect the target audience to be editing the same task on two different devices within a second or two of each other. Most users will make a bunch of edits on a device and later move to another device later (at least not seconds) to make more edits.<\/p>\n This app works great for a single user, but if this app required multiple users<\/em>\u00a0across multiple devices, it would not work as well. We’d need to have much more active syncing of the files to ensure each user has the most up to date version. At the moment, the app syncs to Dropbox every few minutes as\u00a0needed.<\/p>\n Replaying the transaction log does take longer and longer as it grows. At the moment, a cached version of the database is written to disk and loaded on launch to avoid the perf hit of regenerating the it, but the database does have to be regenerated if the app\u00a0detects that there are new entries added to any of the transaction logs from other devices during sync.<\/p>\n It should be possible to alleviate some of these performance issues by occasionally creating snapshots of the database given the current state of the logs. When new transaction logs are downloaded, we’d simply re-play the universal transaction log\u00a0from the most recent database snapshot that did not contain the first new log entry. (This snapshot\u00a0feature has not yet been implemented in the interest of shipping the product fast! Once the author starts to feel the pain of long load-times, he promises to add this feature. 😁)<\/p>\n Overall, I’m quite happy with\u00a0the implementation. It means I don’t have to create a new sync service. Creating a new service to host other people’s private tasks is a big deal. I’d have to worry about security, maintenance, and long-term plans (I’d have to keep the service up if people relied on it).<\/p>\n This sync system has been reliable thus far and there are no worries about race conditions or locking issues since each device is solely responsible for its own transaction log. The system can also scale up indefinitely (at the cost of perf) since each device gets a UUID and it’s simply a matter for all devices to “notice” the new transaction log in the shared app folder on Dropbox. There is no central registry of devices, just transaction logs representing each device. No locking of resources is ever needed.<\/p>\n In the future, I’d like to refactor some of the code I’ve written for this so it can be generalized to any file store (cloud or otherwise) and any type of database.\u00a0If I do that, I can post it on github so others can use it.<\/p>\n If you’re considering writing an app and using a third party file store like Dropbox to sync across devices for a single user, I recommend giving this technique a try.<\/p>\n If you got this far, thanks for reading! I’d love to hear your\u00a0thoughts below.<\/p>\n One last plug: give Lil Todo<\/a> a try while you’re here! Please\u00a0do\u00a0check out my other projects<\/a> as well.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":" Lil Todo is a to-do app that I wrote for the Mac.\u00a0(An iOS version of the app\u00a0is coming soon!) One of the great things about it is you can run it on multiple Macs and use a single Dropbox account to save and sync all of your tasks. Mark a task completed on one device … Continue reading How Lil Todo Syncs Tasks Across Multiple Devices Just Using Dropbox<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[131,120],"tags":[122,121],"_links":{"self":[{"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1262"}],"collection":[{"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1262"}],"version-history":[{"count":27,"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1262\/revisions"}],"predecessor-version":[{"id":1410,"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/1262\/revisions\/1410"}],"wp:attachment":[{"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1262"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1262"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ussherpress.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1262"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Dropbox == Simple File Store<\/h1>\n
The Solution: One Weird Trick<\/h1>\n
Transaction Logs, Baby<\/h1>\n
Dealing with Conflicts<\/h1>\n
Performance Concerns<\/h1>\n
Closing Thoughts<\/h1>\n