This is the first in an eight-part series on implementing data sharing in Shopping UK using CloudKit.

Part 1: Introduction

The idea is simple: My wife and I share a shopping list. When I empty the last jar of honey, I add “honey” to our shared list. When my wife finishes the coffee, she does the same. We do this throughout the week. When I’m shopping, I look at the list to remind me what to buy, and I’ll mark-off items as I place them in the trolley. The list on my wife’s phone will automatically update to reflect what I’ve bought so she doesn’t buy duplicates when she’s next at the shops.

I’ve been building and maintaining Shopping UK for 10 years, and it’s been modestly successful. It has a 4.8 rating on the App Store, over 45,000 people use the app regularly, and it brings in a monthly revenue of between £1,000 and £1,500 per month — from a mixture of advertising and the premium subscription (mostly subscriptions).

There are many apps like this, but, for UK shoppers, Shopping UK is probably the easiest to use. It knows almost every product you’re likely to buy from a UK supermarket, it will automatically arrange your items by supermarket aisle, and, since 2018, it has offered support for sharing a list with family and friends.

Sharing a shopping list between two devices is such an easy concept to understand — many tutorials use it as an example when building a demo app. But things soon become a little more complex. Getting to 80% done is easy. It’s the last 20% that takes all the time. In this series of articles, I want to focus on that difficult 20%.

In 2018, when I first introduced the shopping list sharing feature, I built my own cloud-based data synchronisation mechanism. At the time, I foolishly thought my needs were unique and I would need full control over the servers to support future features. Plus, the geeky part of me was attracted to the interesting technical challenge!

I have come to regret this decision.

Although my home-grown list sharing solution worked, it had some serious flaws:

• Cost. The cost of running the server and accompanying SQL database in Azure was around £80 per month. This is not a crazy amount at present, but I was worried if the app became more successful the cost could shoot up. A larger database and additional app servers may be needed and because I offer sharing to all users for free, an increase in app usage would not directly lead to higher revenue.

  Hosting cost breakdown: Azure App Service: £55/month, SQL Server with 20 DTUs: £22/month, plus bandwidth, file and table storage, and analytics. If load increased, the database could be increased to 50 DTUs at £55/month (A DTU is a Database Transaction Unit, a unit of measure representing a blended measure of CPU, memory, reads, and writes)

• Support and Maintenance. The servers are managed by Azure, but I had to support the app services and server-side code myself. Every year, the SSL certificate needed to be renewed. What if the service stopped working in the night or while I was on holiday? The weight of this hung over me. As platforms and frameworks were upgraded or deprecated, I needed to change and retest my code. The server’s deployment schedule was different to the iOS app, and it required a very different process and set of tools.

• Performance. Sharing relied on a polling mechanism. Every 10 seconds while the app was active, a message containing added or changed items would be sent to the server. The server would respond with changes made by other list participants. This meant users had to wait up to 20 seconds after someone made a change before their list refreshed. At peak times, items were not always synchronised on the first attempt because the server was under high load and some requests would time-out. Added to all this, the frequent polling caused battery drain.

• Lack of flexibility. The server-side code was optimised for the initial design. This was fast, but at the expense of flexibility. Adding new features was difficult because the server code would need to change with the client code (and because there are tens of thousands of users, the old and new versions needed to be supported at the same time). This “change tax” limited the features I wanted to add, such as adding images to list items, sharing custom categories, sending notifications to other users when an item is added or marked-off the shopping list.

Serious work was needed, and it would require a large investment of effort. This meant, I had to time it right. In 2020, I was planning to start work on a CloudKit based sharing mechanism when COVID hit. In my day job, I’m a Laboratory Information Management System (LIMS) architect and consultant. During the COVID crisis, things became very busy. I was brought in to design and build solutions to help track the spread of COVID in the UK and this meant the side project had to wait.

Once the Covid crisis subsided, I had a few months of quieter time to learn CloudKit and build a new list sharing solution.

In this eight-part series, I’ll describe the design of the finished solution, which will include:

• Basic CloudKit concepts.
• How data is moved between a device and iCloud.
• How to initiate sharing, and how a sharing invitation is accepted by the app.
• Strategies for synchronising data between devices: when and how to synchronise.
• How to manage sharing participants, what happens when you stop sharing, leave a list or sign out of iCloud.
• Tips for diagnosing issues, handling edge cases, concurrency and logging.

We’ll start next week with an introduction to CloudKit’s basic concepts and design of the data schema.

If you get a chance, please try Shopping UK and let me know what you think at @wheelies