I'm updating a local Core Data-driven app to support iCloud. One challenge I anticipate is around what I'm calling "stock objects" -- seed data the app provides at first launch that 99% of users will customize.
Stock Objects
Item A
Item B
Customized Objects
Tomatoes
Potatoes
If the user then launches the app for the first time on a new device, I think the default behavior would be for the re-creation of the stock objects which would get merged with the customized objects from the iCloud persistent store (Item A, Item B, Tomatoes, Potatoes), resulting in a messy user experience.
One approach might be to check for iCloud data synchronously at first launch, and if it exists, not create the stock objects. But still, the user could be offline on first launch, and then on second launch, the same undesirable merge of local stock objects with iCloud custom objects would happen.
Are there ways to add logic to iCloud merges, so that the arrival of customized objects from the cloud (Tomatoes and Potatoes) can signal me to delete local stock objects (Item A and Item B) before they get beamed up?
Thanks!
Why don't you just avoid this by not providing seed objects? Simply prompt the user through the UI to enter his own objects on first launch. Ask, if she has used the app before and explain that it might pay to wait to sync from iCloud before recreating the objects.
I'm not sure if this is the best approach, but this is what I'm doing:
When the user elects to enable iCloud, I check if their iCloud directory is empty.
If so, no problem; I migrate the database to a new local store that has iCloud options enabled so that all existing data moves to the cloud.
If not, I check if the local database is empty. If it is, I wipe out all the stock objects and take everything from the cloud. If it isn't, I tell the user currently the app can't merge a local database with an iCloud database, and ask them to delete their data (or reinstall), thus sidestepping this problem.
Related
My app uses CoreData with iCloud as backend. Multiple devices can access the iCloud database which is thus .public.
The local CoreData store is synchronized with iCloud using an NSPersistentCloudKitContainer.
I use history tracking according to Apple’s suggestions.
There, Apple suggests to prune history when possible. They say
Because persistent history tracking transactions take up space on
disk, determine a clean-up strategy to remove them when they are no
longer needed. Before pruning history, a single gatekeeper should
ensure that your app and its clients have consumed the history they
need.
Originally this was also suggested in the WWDC 2017 talk starting at 26:10.
My question is: How do I implement this single gatekeeper?
I assume the idea is that a single instance knows at what time every user of the app has last synchronized their device. If so the history of transactions before this date can be pruned.
But what if a user synchronized the local data and then does no longer use the app for a long time? In this instance the history cannot be pruned until this user again synchronizes the local data. So the history data could grow arbitrarily large. This seems to me as a central problem that I don’t know how to solve.
The Apple docs cited above suggest:
Similar to fetching history, you can use deleteHistory(before:) to
delete history older than a token, a transaction, or a date. For
example, you can delete all transactions older than seven days.
But this does not solve the problem to my mind.
Aside of this general problem, my idea is to have an iCloud record type in the public iCloud database that stores for every device directly (i.e. without CoreData) the last date when the local database was updated. Since all devices can read these records it is easy to identify the last time when all local databases have been updated and I could prune the history before this date.
Is this the right way to handle the problem?
EDIT:
The problem has recently been addressed in this post. The author demonstrates with tests with Apple's demo app that there is indeed a problem, if the history is purged too early. My answer there indicates that with the suggested delay of 7 days, an error is probably extremely rare.
UPDATE:
In this post from a WWDC22 Core Data Lab, an Apple Core Data framework engineer answers the question "Do I ever need to purge the persistent history tracking data?" as follows:
No. We don’t recommend it. NSPersistentCloudKitContainer uses the
persistent history token to track what to sync. If you delete history
the cloud sync is reset and has to upload everything from scratch. It
will recover but it’s not a good customer experience. It shouldn’t
normally be necessary to delete history. For example, the Apple Photos
app doesn’t trim its history, so unless you’re generating massive
amounts of history don’t do it.
By now I think my question was partly based on a misunderstanding:
In CoreData, a persistent store is handled by one or more persistent store coordinators.
If there is only one, the coordinator has complete control over the store, and there is no need for history tracking.
If there is more than one coordinator, one coordinator can change the store while another is not aware of the changes.
Thus, persistent history tracking of the store records all transactions in the store.
The store can then notify other users of the store by sending a NSPersistentStoreRemoteChange notification.
Upon receiving this notification, the transaction history can be fetched and processed.
After processing a transaction, it is no longer needed by the user that processed it.
In a CoreData + CloudKit situation, a persistent store is mirrored to iCloud.
This means there is in the simplest situation one persistent store coordinator of the app, and - invisible to the app - one persistent store coordinator that does the mirroring.
Since both coordinators can change the store independently, history tracking is required.
If the app changes the store, I assume that Apple’s mirroring software receives the NSPersistentStoreRemoteChange notifications, processes the transactions and forward them to iCloud. Normally, i.e. if there is an iCloud connection, this takes only seconds, so that the transaction history is only needed short time.
If iCloud changes are mirrored to the store, the app receives the NSPersistentStoreRemoteChange notifications, and has to process the transactions.
After they have been processed, they are no longer needed neither by the app nor by the mirroring software and can be pruned.
This means that ifs there is only one user of the persistent store on the app’s device, pruning can indeed be done short time after processing the notification.
If the device is offline, e.g. in flight mode or switched off, it will not receive NSPersistentStoreRemoteChange notifications, and will not prune the transaction history.
So it is indeed safe to prune the persistent history after say seven days after it has been processed.
The situation is different if there is more than one user of the store on a device, e.g. an additional app extension. In this case one has to ensure that other targets than the app have also processed the transactions before the history is pruned. This can indeed be done by a single gatekeeper. How this can be done is e.g. described in this post.
I have two Azure Websites set up - one that serves the client application with no database, another with a database and WebApi solution that the client gets data from.
I'm about to add a new table to the database and populate it with data using a temporary Seed method that I only plan on running once. I'm not sure what the best way to go about it is though.
Right now I have the database initializer set to MigrateDatabaseToLatestVersion and I've tested this update locally several times. Everything seems good to go but the update / seed method takes about 6 minutes to run. I have some questions about concurrency while migrating:
What happens when someone performs CRUD operations against the database while business logic and tables are being updated in this 6-minute window? I mean - the time between when I hit "publish" from VS, and when the new bits are actually deployed. What if the seed method modifies every entry in another table, and a user adds some data mid-seed that doesn't get hit by this critical update? Should I lock the site while doing it just in case (far from ideal...)?
Any general guidance on this process would be fantastic.
Operations like creating a new table or adding new columns should have only minimal impact on the performance and be transparent, especially if the application applies the recommended pattern of dealing with transient faults (for instance by leveraging the Enterprise Library).
Mass updates or reindexing could cause contention and affect the application's performance or even cause errors. Depending on the case, transient fault handling could work around that as well.
Concurrent modifications to data that is being upgraded could cause problems that would be more difficult to deal with. These are some possible approaches:
Maintenance window
The most simple and safe approach is to take the application offline, backup the database, upgrade the database, update the application, test and bring the application back online.
Read-only mode
This approach avoids making the application completely unavailable, by keeping it online but disabling any feature that changes the database. The users can still query and view data while the application is updated.
Staged upgrade
This approach is based on carefully planned sequences of changes to the database structure and data and to the application code so that at any given stage the application version that is online is compatible with the current database structure.
For example, let's suppose we need to introduce a "date of last purchase" field to a customer record. This sequence could be used:
Add the new field to the customer record in the database (without updating the application). Set the new field default value as NULL.
Update the application so that for each new sale, the date of last purchase field is updated. For old sales the field is left unchanged, and the application at this point does not query or show the new field.
Execute a batch job on the database to update this field for all customers where it is still NULL. A delay could be introduced between updates so that the system is not overloaded.
Update the application to start querying and showing the new information.
There are several variations of this approach, such as the concept of "expansion scripts" and "contraction scripts" described in Zero-Downtime Database Deployment. This could be used along with feature toggles to change the application's behavior dinamically as the upgrade stages are executed.
New columns could be added to records to indicate that they have been converted. The application logic could be adapted to deal with records in the old version and in the new version concurrently.
The Entity Framework may impose some additional limitations in the options, because it generates the SQL statements on behalf of the application, so you would have to take that into consideration when planning the stages.
Staging environment
Changing the production database structure and executing mass data changes is risky business, especially when it must be done in a specific sequence while data is being entered and changed by users. Your options to revert mistakes can be severely limited.
It would be necessary to do extensive testing and simulation in a separate staging environment before executing the upgrade procedures on the production environment.
I agree with the maintenance window idea from Fernando. But here is the approach I would take given your question.
Make sure your database is backed up before doing anything (I am assuming its SQL Azure)
Put up a maintenance page on the Client Application
Run the migration via Visual Studio to your database(I am assuming you are doing this through the console) or a unit test
Publish the website/web api websites
Verify your changes.
The main thing is working with the seed method via Entity Framework is that its easy to get it wrong and without a proper backup while running against Prod you could get yourself in trouble real fast. I would probably run it through your test database/environment first (if you have one) to verify what you want is happening.
I'm currently working on my first Core Data iPhone App and would like your opinion on how to update records/doing general maintenance in a production stage after app has been approved.
All the public scenes will of course be available to users, but a few scenes where "only admins" can access must be private or protected against public access. What the best strategy you find on this? I am thinking for example, inserting a log-in & password scene in between let's say, the main table-view and the edit/add new rows scene to update a new row or label title for example.
How to push those updates so the changes can be re-deployed to the production app(App store) How about a external source like a server based solution to submit updates? How do you update the persistent store in that case? I have
This app is a simple table-view list of workshop training sessions for teachers(i.e, Math 101, Algebra I, Algebra I, Advanced Algebra, etc..) where people can write a feedback note after they attended the session on a detail view scene by clicking on any of the rows.
Thanks so much for your advice.
In general, you should avoid using "back doors" in your apps. Keep all the controllers that are not for the user completely removed from the production target.
As for future changes in your data model, once you get more familiar with the Core Data API you will discover that what you want to do is quite simple.
As for the update of the data model - in all but the most complicated cases it can be completely automatic. Just version your data model, make sure the automatic migration option is enabled when you add the persistent store, and you can simply update your production app with a new version on the app store - all migration will be done for you automatically.
What is the best way to check iCloud for existing data?
I need to check that data doesn't exist on the local device, or iCloud so I can then download it.
Since you included the core-data tag I'm assuming you mean that you're using Core Data rather than iCloud file APIs or the ubiquitous key-value store.
With Core Data's built-in iCloud support, you check on existing data in exactly the same way as if you were not using iCloud. Once you create your Core Data stack, you check what data exists by doing normal Core Data fetches. There's no (exposed) concept of local vs. cloud data, there's just one data store that happens to know how to communicate with iCloud. You don't explicitly initiate downloads-- they happen automatically.
At app launch time when you call addPersistentStoreWithType:configuration:URL:options:error:, Core Data internally initiates the download of any data that's not available locally yet. As a result this method may block for a while. If it returns successfully, then all current downloads can be assumed to be complete.
If new changes appear while your app is running, Core Data will download and import them and, when it's finished, will post NSPersistentStoreDidImportUbiquitousContentChangesNotification to tell you what just happened.
This all describes how Core Data's iCloud is supposed to work. In practice you'll probably find that it doesn't always work as intended.
Thanks to #Tom Harrington for pointing out, this error is nothing to do with the developer/coding - it's purely to do with iCloud/Apple/connection issues.
More on this SO answer I found.
I recently submitted an upgrade of my app which included a lightweight coredata migration (including new fields in existing tables and a couple of new tables). I followed every tip regarding this migration, including some I found on this site.
I thoroughly tested the update on three different devices and it all went ok!!!
However, this update is crashing an all my devices and probably on all my customers. I can't explain why this is happening.
Could you please help me understand this debacle?
To truly test your app and migration, you need to run your original app to create data store according to the original data model. Then you need to run your new app, opening data store that was generated with original app. This can be a real pain and is easier (at least initially) to do in Simulator because you have more control over the file system and can swap in a saved original data store. On iDevice you need to regenerate original data store for each test.
If you are testing on your own development devices then you have already migrated your data store. Is it possible that your test devices created their data stores with new data model - and never actually performed a migration?
I only generally use automatic migration during beta testing, for quick revisions, other than that I always use a mapping model, so that you have control.
the other issue is that if your model shifts far enough between releases, auto migration from v1-v2 could be fine, and v2-v3 could be ok, but v1-v3 could be too drastic to be inferred. by making maps for them, you retain control of the migration.