Grain Science 1.3.2

So, version 1.3.2 of Grain Science is out, and it’s primarily a bug-fix update. I wanted to talk a bit more about this update and its implications.

You see, the main bug that’s addressed in this update, isn’t actually a bug in Grain Science at all — it’s a bug in iOS that we’ve uncovered. Unfortunately, it can cause data to be lost. This is a serious issue which we reported to Apple, who are taking steps to fix it in a future iOS update, and have also provided us with some assistance in working around the problem in the meantime.

Explaining the full details of the problem will require getting quite technical, so here are a few quick notes:

  • If you purchased Grain Science 1.3 or later, you will not be affected by this bug. You can probably ignore the rest of this post!
  • If you purchased an earlier version of Grain Science and have never updated it, you won’t be affected. If you update from an earlier version to version 1.3.2 (or later) you won’t be affected, either.
  • If you purchased an earlier version of Grain Science and then updated, you may be affected. Read on to find out.
  • The bug occurs when updating the instrument library from an old version of Grain Science for use in a new version.
  • The bug can cause the audio files used by an instrument to go missing, if they were recorded or imported. Built-in audio files, and instruments that only use built-in audio, are safe.
  • Due to the design of Grain Science, we may be able to recover any missing audio, 100% automatically, when Grain Science 1.3.2 is launched — but we can’t guarantee it.

Background

Grain Science uses an Apple technology called “Core Data” to store your instruments. This has many advantages — it’s fast, uses less memory, and can use less battery power, than many alternatives. It offers a number of features that we take advantage of, to keep Grain Science responsive and easy to use.

Your instrument library is essentially a database, in Core Data’s “sqlite3” format. This format of database is great for handling instrument settings, because it’s designed to store large quantities of small pieces of data. It’s not ideal for storing the audio files though, because each recording tends to be a single, very large, block of data.

In iOS 5, Apple added a new feature to Core Data that allows it to automatically detect when a lump of data is added to the database and is too large to be sensibly stored there, and keep it in a separate location instead. This feature is called “External Storage”. Naturally, Grain Science enables External Storage and uses it to store the audio recordings used by instruments.

What Happened?

Sometimes it is necessary to update the format of the database when we bring out a new version of Grain Science — for example, when adding a new FX unit, we need to add new data to the database, to store the FX unit’s dial and program settings. One of Core Data’s really useful features is that when you need to change the format of a database, you can simply tell it what the old format was, and what the new format is, and it will “migrate” the data from the old format to the new one quickly and easily.

Unfortunately, a bug in iOS (Apple Bug ID #8895727) means that the External Storage for a database can be destroyed when migrating a database from an old format to a new one. This doesn’t seem to happen in all circumstances (only Apple know the full details at this point), and it never occurred during the extensive testing while we were developing the Grain Science updates. But, it is something that can happen, and has happened to some people. In previous versions of Grain Science this could also cause a crash, when selecting an instrument that used audio, and that audio had gone missing.

In Grain Science 1.3.2 we have taken all the steps we can to prevent data loss, recover user data wherever possible, and ensure reliability if your data has been affected.

Am I Affected?

We haven’t changed the format of the Grain Science instrument library since version 1.3 so, if you purchased v1.3 or later, then this bug will not affect you, as your database will not need to be upgraded. Future versions of Grain Science should also be safe to upgrade to, because from this version 1.3.2 onwards, we make a backup of the External Storage before letting iOS perform the database upgrade, and restore it afterwards.

If you used an earlier version of Grain Science, and only used the built-in instruments, you won’t be affected. If you created your own instruments, but only used the built-in audio (eg Sine Wave, Sawtooth Wave, Prestrung), you won’t be affected. If you used an earlier version, and have never updated it, you won’t be affected — and it will be safe to update to version 1.3.2 due to the backup mentioned above. But do be sure to update directly from your current version, to version 1.3.2.

If you recorded your own audio, or imported some from elsewhere, then you may be affected. When you launch Grain Science 1.3.2, it will check to see if you are affected or not, and if you are, then it will tell you exactly which files are missing (please note that it’s not the upgrade to v1.3.2 that caused the problem — Grain Science is reporting on damage that was already done by iOS some time ago). If you try and use an instrument with missing audio, it will substitute a sine wave as a placeholder until you restore or replace the audio.

Fixing It Automatically

The audio files in your instrument library are kept compressed (in a lossless format) in order to keep the size of your backups (in iTunes or in iCloud) small. Before they can be used, they need to be decompressed — this is what’s happening when you see the “Unpacking Audio” sign flash up when you switch to a new instrument. To save this happening every time you switch instruments, we keep a copy of the uncompressed audio on your device in a separate location (the “cache”).

So the good news is that if you have lost audio from the instrument library, Grain Science 1.3.2 will check to see if the uncompressed version is in your device’s cache, and if so, will add it back in to your instrument library automatically. The cache is kept around for as long as possible, so there’s a good chance your data will have been kept safe this way.

However, iOS may clear the contents of the cache from time to time — particularly if your device runs out of storage space. Normally Grain Science doesn’t care if the cache is deleted, because it will just unpack the audio from the instrument library again. But if the iOS bug has deleted your audio, we will be unable to rescue it for you using this method.

There are, however, some alternatives.

Emergency Rescue

So, if auto-recovery hasn’t rescued all of your audio, what now?

If your instruments used audio you imported from other apps (rather than recording directly into Grain Science), then really the simplest and fastest way to recover them is to simply delete the “empty” audio files from your instrument library, re-import them, and point the grain units for those instruments to the new audio files. This is a hassle, but less of a hassle than any other option.

If you have original recordings you need to recover, things get a bit more complicated.

Before going any further, you should open up iTunes App File Sharing — see this page on Apple’s support site for details if you’re unfamiliar with this — and copy the MyInstruments.grain-science-library file to your desktop. Compress it (for example on Mac, control-click on it and pick Compress "MyInstruments.grain-science-library"). This is a backup copy of the latest version of your instrument library.

At this point, your best bet is probably to contact us. We can arrange a method for you to send the instrument library to us (it’s often too large to simply send by email) and we will see if we can recover any additional audio from it — we’ve discovered that in some cases, Core Data seems to be misplacing rather than deleting the audio, and although this isn’t reliable, or anything we can perform an automatic rescue on, we have engineered some tools that can sometimes identify which audio files belong where.

Thanks for taking the time to read this report all the way to the end. We hope it answers any questions you might have, but if not, please don’t hesitate to get in touch.