This is the sixth in an eight-part series on implementing data sharing in Shopping UK using CloudKit.
Shopping UK is a smart shopping list for UK shoppers. It knows almost every product in the supermarket and will arrange them by aisle. Lists can be shared with family or friends.
Last week, we looked at what happens when another device accepts an invitation to collaborate on a shared list. Today, we’ll look at how data is synchronised between devices and some of the challenges to consider.
In Shopping UK, we need to synchronise data from two Record Types:
Here’s how they are related:
ShoppingList represents the list itself but from a synchronisation perspective, it is very dull. cloudkit.share and JournalEntry are where the action happens.
However, the app is interested in tracking when a participant joins or leaves the list so it can update the Activity view:
Whenever a change is detected in a cloudkit.share CKRecord, the app compares the share’s participants list with the previously cached list to identify what has changed:
In this example, Charlie and Diane joined the list and Bob left.
This list of participant changes is used to populate the Activity list.
For sharing purposes, every list is represented as a set of JournalEntry records, as discussed in Part 2. JournalEntry can be used to represent every type of change made to a list, and the full list of JournalEntry records is enough to reconstruct the entire list from scratch.
For example, this list:
Is represented as seven JournalEntry CKRecords (newest entry at the top):
|3||Rename List||Weekly Shopping|
In eight steps, we can see how these seven JournalEntry records are applied to a blank list to recreate the full shopping list:
Once a JournalEntry has been added to iCloud, it will never be modified — it is a read-only log of changes. The only way JournalEntry records are ever removed is when the app compresses the list, which will happen regularly to prevent the list becoming large and causing performance issues. List compression will be discussed in detail in part 7.
Every newly created JournalEntry record must be synchronised between each device and iCloud.
We also looked at three ways data can be read from iCloud:
These operations form the basis of synchronisation. Now let’s look at the full cycle and how each device knows when a change has been made.
When a change is made to a list on one device, a record representing that change must be uploaded to iCloud as soon as possible.
The device that made the change will send a CKModifyRecordsOperation request to iCloud using CloudKit:
After CloudKit has updated the iCloud database with the new entry, it sends a CKDatabaseNotification message to all other devices running the app.
iCloud sends a list of changes made since the last request. This list will include the newly created “Add potatoes” record:
Shopping UK uses CloudKit Subscriptions to inform the app when another user makes a change to a shared list.
After the app has registered a subscription with CloudKit, the app will be notified when the watched data changes.
Subscriptions are managed separately for each user. They are not global — they live in the user’s iCloud account.
There are three types of subscription:
During app start up (willFinishLaunchingWithOptions), Shopping UK will attempt to register four subscriptions, one for each Record Type (cloudkit.share and JournalEntry) and one for each CKDatabase (Private and Shared)
CKModifySubscriptionsOperation is used to register the four subscriptions:
Once a subscription has been registered, iOS will call the AppDelegate’s didReceiveRemoteNotification method to tell the app when a change happens.
This notification can be received while the app is active, in the background or even when it is not running. (source)
the system calls this method when your app is running in the foreground or background. In addition, if you enabled the remote notifications background mode, the system launches your app (or wakes it from the suspended state) and puts it in the background state when a remote notification arrives
Failed. Be consistent with how you use these.
Important: A notification will not always be received
A notification is not guaranteed to be sent from iCloud. iOS uses heuristics for deciding when to send a notification.
Apple doesn’t document the heuristics. Presumably because it doesn’t want developers to game the system. The best official document I found was Apple TechNote TN2265 from 2016:
your app will receive the notification if iOS or OS X determines it is energy-efficient to do so. If the energy or data budget for the device has been exceeded, your app will not receive any more notifications with the content-available key until the budget has been reset. This occurs once a day and cannot be changed by user or developer action.
iOS tracks how long your app takes to process each notification and what result was returned by your app.
|Notification||Time Taken (ms)||Result|
iOS uses this data to decide how to handle the next notification. Should it be sent to the app at all? And, if judged worthy, how soon should it be sent?
My guess, based on a lot of testing, is iOS groups these results by type, then calculates stats on each group:
iOS appears to use this information, and other data (battery charge, battery decrease rate, number of notifications received) to decide how much budget to give your app for future notifications.
While testing, I learnt a couple of important lessons.
Lesson 1: Always return a result
I found a bug in my app, which meant a result was not returned when the app was in the background. When this happened, CloudKit would refuse to send notifications for a few minutes. My app wasn’t playing by the rules so it was being punished.
Lesson 2: Classify results correctly
Another bug meant my app would sometimes return
nodata instead of
newdata, despite actually fetching and processing genuine data.
This would happen about 50% of the time.
I’d add an item to a list on one device but the notification wouldn’t appear on the second device.
I’d add a second item and this time a notification would be received on the second device.
Because the app was incorrectly reporting work as
nodata, iOS reduced the number of notifications it sent by half.
From iOS’s perspective this made sense — why waste resources on notifications that take the same time to process
but don’t produce useful work.
It’s all about saving battery. If an app abuses the time given to process remote notifications, iOS will grant it less time in the future. If the app fails to respond at all, iOS will penalise the app. If the app says it hasn’t done anything useful with the time it was given, iOS will grant it less time in future. This can result in notifications being skipped, combined or delayed.
The best strategy is to be efficient with resources — don’t do more than you need to when a notification arrives — and be
truthful with what you did. If your app used its time to fetch new data, return
newdata. If your app didn’t need to do anything, return
And if your app could not complete processing for whatever reason, return
failed. This will allow iOS to classify the time taken in the
correct dataset so it can draw the correct conclusions.
One final tip, when testing, if your notifications arrive on some devices but not others, try:
After receiving the notification, Shopping UK fetches the list of changes using CKFetchDatabaseChangesOperation and CKFetchRecordZoneChangesOperation as described in Step 3 above, and Part 3 of this series.
It may be tempting to skip the “fetch” step and use the information provided in the notification message, but this is unreliable. iOS may decide to combine or drop notifications for individual changes so an explicit fetch is required to ensure no changes are missed.
After the app has made the request, CloudKit will send several responses, which may include:
See Part 3 for full details.
When the app receives changes, it stores them locally until it is the right time to apply them to the user interface.
The “right time” depends on which list is active, and on what the user is doing.
When the app is in the “Planning” screen, the JournalEntry operations can be applied to the current list straight away. But, if the app is in the “Shopping” screen, a different approach is needed. If items were added to the visible list immediately, it would disrupt the user’s flow (imagine if a set of new items were added to your list just as you were about to mark it off. It would be annoying to accidentally tap the wrong item because the app had decided to shuffle the list items around just at that moment).
To provide a better user experience, Shopping UK displays each “Add” as a visible notification at the top of the list. Then, when the user taps the notification, the queued items will move to the right category in the list.
When an item is “Marked-off” or “Deleted”, the app handles things differently. Since the item is already in the list, there is no need to display a separate notification. Instead, the item’s table cell is updated to reflect the state change.
This behaviour makes it a delight to use Shopping UK when family and friends are shopping together. Some people like to split up in the supermarket to save time. To make this work, it is vital that changes appear quickly and are visually obvious, so the same item isn’t purchased twice.
Most of the time, synchronisation happens automatically. If a user makes a change to the list, a JournalEntry record representing the change will be immediately uploaded to iCloud. If a CKDatabaseNotification is received, which indicates someone else has made a change, the app will immediately fetch changes from iCloud.
In addition, the app will fetch when the app is launched (in case any changes were made by other participants while the app was in the background).
But this is not always enough.
To handle unforeseen situations, such as CloudKit subscriptions not arriving on time, I also added a way to do invoke the full upload-and-fetch synchronisation process manually.
This can be triggered by pulling down on the “planning” screen (shown below) or the “shopping” screen (the one with checkboxes)
Another way to invoke manual sync is the “Synchronise Now” button on the Activity sheet:
I don’t normally like adding multiple ways of doing the same thing — I’d rather ensure there is a single reliable way to accomplish something — but when there are a lot of moving parts, some of which out of my control (i.e. iCloud, database subscriptions, and the network), it is reassuring to know users have a manual alternative available if needed.
Next week, we’ll look at how a share is changed or stopped, what happens when a list is deleted, and background maintenance.