Caching Server and iCloud Data

Apple made an important change in Caching Server in Server 5. By default, it will now cache personal iCloud data in addition to the usual apps, books, and updates. While I guess there can be some value here in saving bandwidth, I seriously doubt most sysadmins would consider this worth the potentially massive storage requirements. On my home server, for example, I was already up to 60GB of iCloud data. And that is for two people!

There is no way to turn it off in the GUI, but an Apple Service Provider in the UK found a way to disable it via command line. Check out their post for the details.

Photos Postmortem

Five days remain. That is how long I have to wait until I can attempt to sync photos again. I will probably wait longer. As previously covered on Technecast and here and here, my first attempt at photo syncing through iCloud went very badly. So I finally gave up and told iCloud to delete it all and start over. But to do this I must wait thirty days before Apple is comfortable deleting my library from the cloud.

So what went wrong? Things seemed to be doomed to failure right from the beginning. In retrospect I should not have committed to uploading so quickly. After installing 10.10.3 I figured I would have time to fix up iPhoto before opening the new app and importing it all. But that wasn’t the case. Photos started importing in the background immediately, meaning the work I was currently doing in iPhoto didn’t make it into the new app. In hindsight I should have never told Photos to upload without first getting in and checking the library. What can I say, I was excited.

I was uploading a damaged library and wanted to stop it. I tried to immediately stop the upload and fix things. But it appeared the damage was done. About a thousand photos made it to the web, and a whole bunch of albums. I could not delete anything from the web app even though I wanted to just start over. The only reset option had a thirty day grace period. Not yet.

Next I loaded the cloud library on my iPhone to attempt to delete the excess there. It seemed to work at first, but then out of nowhere all the albums I had removed returned to the web app. And this time they were stuck. The albums only appeared on the web. But fine, if they didn’t appear in a client and weren’t a problem I could live with that until Apple fixed things.

Days later I finally uploaded the library from my Mac. All 300+ GB were there. Then I turned on the phone. The first attempt to download it all got stuck. So I turned it off and started again, and finally it worked. This was the one time when I got to see the promise of this service and it was amazing. But it didn’t last.

Then came the iPad. The iPad was crashing constantly. So I figured it was finally time to let go of five years of iPad data and restore the device from scratch. Then the first thing I did was enable photo syncing. Things seemed to be going well and after a day and a half everything was downloaded.

Then it got weird. Photos started disappearing. The total number reported by the iPad was going down. Then back up. Then down. Then up. This went on for the two days before I finally gave up.

I wish I could say what went wrong, but I have no ideas. The process is completely opaque. There is really nothing you can do to troubleshoot. Just let it go and try again.

There was one final gift. I decided to turn the whole thing off and start over. When I turned off the iPhone syncing it asked whether I wanted to keep the photos locally or delete them. I chose delete since the master library is on the Mac. It then took about 10 hours to slowly delete the photos, stalling out multiple times. Even then it finished with about 1000 photos remaining. I had to delete those manually.

The whole process was so frustrating, especially since I barely got to see it work. So now I wait for two things. The first is for my cloud library to delete entirely. I am hoping that a full reset will give me a chace to start over with a nice clean upload. The second is the new software currently in beta. Both OS X and iOS are likely to get fixed soon. I also removed over 700 album’s created from iPhoto events in the Mac app. The way Photos works I don’t need them.

I still hope this works. Maybe in a month or so I will finally get to enjoy the good photo library experience others seemed to have had. But not yet.

Trial by Cloud: Photos

I was hoping to write a follow up of our podcast discussion on last week’s Technecast concerning iCloud Photo Library. I wanted to wait until all my devices were finally synced. But I can wait no longer for my iPad to get in the game.

If you didn’t listen to the podcast, basically I had a horrible time getting photo library turned on. The original upload was painfully slow, error prone, and required me to completely quit, wipe out what was already loaded, and restart, something that the software does not make easy. The web app was, and remains, horrible. The un-deletable phantom albums remain. All and all it was not a good on-boarding.

Photos improperly spaced on the website.
Unintentional collage art in the iCloud Photos web app. The dates and locations are wrong too.

But then, finally, the Mac uploaded. So I enabled it on my iPhone. The phone had no photos so it was just a matter of downloading the cloud library. This took two days. That seems like a really long time, especially for what are supposed to be space optimized versions of the photos. But when it finally worked, it was magical. Take a photo on the phone and it was on the computer before I even had to open the Photos app. Albums, edits, everything synced beautifully. This is what I had been waiting for. The promise of photos fully realized. So time to get the iPad to the party.

First of all, it is an iPad 3, AKA the worst iPad. Seriously even Apple killed this model in 6 months. But it has been okay enough that I didn’t feel like upgrading. Maybe that has to happen now. Maybe the old iPad can’t handle this software. I wish I could accurately explain what is happening. But I can’t. Because I really don’t know what the f#ck my iPad is doing right now.

First of all the initial attempt at turning on iCloud Photo Library caused my iPad to kernel panic about once per hour. This is not the first time I have had crashing issues so I finally decided to ditch this installation, which goes all the way back to day one of the iPad, and start fresh. I completely wiped the device and started from a bare, clean install of iOS.

So far so good. No crashing. Photos turned on and literally nothing else. No other apps. No email accounts. Nothing. This was over the weekend. It’s still not done.

It has reported it was done. More than once. But then all of a sudden it starts syncing again. Here is the really weird part. The number of photos reported in “All Photos” remains consistent, but the number of photos in the “moments” section rises and falls like a ship in a hurricane. I have now at least three times seen the photos gradually disappear, then reappear, then disappear again. The status at the bottom will go back and forth between preparing and uploading. It shouldn’t be uploading anything. And it doesn’t seem to be.

I have no idea what is going on. I feel like the whole thing is out of control but I don’t want to stop it. The iPhone took a while but eventually settled. But it didn’t take this long. So the iPad is sitting, screen on, for a week. It doesn’t seem to sync when the screen is locked so I am leaving it on all the time in the photos app, hoping every day I will come home and see it completed. I have stopped taking photos completely, not wanting to add any complexity until this process is done.

So what I can say about iCloud Photo Library is that it is amazing when it works, but there are way to many issues to feel comfortable recommending this to anyone. You can’t troubleshoot. You just have to hope it works. It may take hours, days, weeks even. I want to love this feature because I have been asking for it for years. Now I just want it to freaking work already.

Apple has a bad reputation when it comes to web services. This is not helping. It is starting to remind me of the launch of MobileMe. That’s the most damning this I can say.

It should be noted that many users are having a fine time. So my experience is not shared by everyone. I am merely stating my own case. I am hopefully still that the bugs get worked out. That he web app begins working normally. That the initial sync will eventually finally finish and I can get on with using my camera again. But for more than the cost of Dropbox for half the space, I shouldn’t have to hope this hard.

Bad Advice Masked as Security

John Gruber makes an excellent point over on Daring Fireball about the rather extreme overreaction to the iCloud celebrity photo theft.

Don’t trust Apple “with any of your data” isn’t just wrong because it’s a hyperbolic overreaction, it’s wrong because it’s potentially dangerous. What has been mostly overlooked in the reaction to this photo leak scandal, and completely lost in Auerbach’s argument, is that backups are a form of security — in the same sense that life insurance is a form of security for your children and spouse.

Exactly right. I see this happen all the time. iCloud backup has been the single most effective tool against data loss I have ever seen. The chances of losing or breaking your phone are orders of magnitude greater than the chances of someone brute forcing their way into your account, unless you happen to be very high profile.

But I will go a step further. Many articles I have read instruct users to simply turn off iCloud entirely, all in one move. This not only removes the security of the backup, but it potentially can lead to data loss itself. Turning off iCloud can cause a user to lose their address book, calendar, notes, documents, and other important data. It has very far reaching consequences. For the most part those stay in iCloud, and can be added back later. But it is entirely possible (again, I have seen this happen) for things to go wrong. Even if not data loss, it can result in duplicates, conflicts, and other issues if iCloud is then reenabled later. Not to mention other cloud syncing (Google, Exchange, etc) may take over and split the data, which is very confusing for regular users to figure out. And many may not realize that turning off iCloud has any of these effects and will just assume that their apps are suddenly broken.

But this is typical of our media culture. Issues that are really bad but unlikely to happen to you are reported breathlessly, while the real dangers go unmentioned. I guarantee you that more people lost their iPhones this weekend than celebrities had photos released. Then again, more people died driving to work than on roller coasters, but guess what gets reported. FUD is alive and well.

What to do about Aperture

Apple announced last week, through bloggers[1], that they would be discontinuing development of Aperture and that both it and iPhoto would be discontinued in favor of the new Photos for OS X when it ships next year. This has, of course, led to a lot of worry about what to do now.

My advice…nothing. Do nothing. If your photo workflow currently revolves around either of these apps keep it that way. At least until we have a better idea what the transition away from these apps will be. Apple has stated that there will be tools to migrate to their new app, as well as to migrate to competitors like Lightroom. Migrating now is likely to be a more difficult process than it will be in the coming months as the tools become available.

Also it’s entirely possible that moving away from them now will make it extremely difficult to move to Photos when it ships. And if Photos works as promised, it looks like you will want to be using it. The idea of a perfect cloud sync for every photo you’ve taken is very intriguing. Perhaps it won’t work for you, but there is no way to know that yet.

So change nothing for now. Stick with what you are doing, and migrate to the new apps once the tools are fully tested and the process is easy. Your current setup will continue to work at least until then.

  1. An interesting tactic for them in the more open Tim Cook era.  ↩

iCloud Strikes Again

One of the apps I use on a daily basis is Ember by Realmac Software. These are the same folks who are responsible for website builder RapidWeaver and minimalist to-do app Clear. They are among the most well known and best Apple platform developers in the business. You know their software is quality because Ember costs $49. Even in the App Store economy of $0.99 to free[1], Realmac has a $49 app that sells. I happen to be a person who takes and manages a lot of screenshots, both for my day job and for blog posts such as this one. So naturally I gave Ember a try and instantly fell in love with its features and ease of use. Take my money please!

A screenshot of Ember managing my screenshots of the draft of this article. Take that Inception!
A screenshot of Ember managing my screenshots of the draft of this article. Take that Inception!

But even quality apps can have their problems, especially when they rely on one of Apple’s more notorious features – iCloud, specifically Core Data sync. Core Data sync has been the bane of many a developer’s existence. One of my favorite podcast apps, Instacast, was among the first to abandon iCloud’s sync for its own custom solution. This paid off as Instacast has been far more reliable since leaving iCloud behind.

Given that I have experienced this first hand, in every case where I am offered the choice to sync with either iCloud or an alternative such as Dropbox, I choose the alternative.

But Ember syncs with iCloud and iCloud alone. Up until about a week ago it was mostly fine. Yes, it would sometimes take a minute or two for the sync to engage, and god forbid iCloud ever allow for user control over the sync process, but otherwise it was pretty solid. Then came the 1.5 update on the Mac. This update added the ability to organize your collections into folders. Yay, I like folders, so I updated right then and there on my work computer.


After the app relaunched, my library of hundreds of images was blown to bits. The app now reported about 40 images, and most were blank thumbnails that lead to nothing. And this being iCloud, the deleted data doesn’t go to the trash. Oh no, it’s a one way trip to oblivion.


And I was not alone. Lots of users on Twitter were frantically @ing Ember’s account with tales similar to mine. And thanks to iOS 7’s auto-update feature, my iPhone and iPad were also hit before I could react in time. Thankfully my home computer was shut down at the time. I actually unplugged my router when I got home so that I could open Ember and backup the library[2] before Mavericks’ auto-update feature did the same thing.

Realmac later confirmed that iCloud Core Data sync was indeed to blame. Like many a developer before them, they appear to be looking at ways to abandon Core Data sync entirely.

It goes without saying that we want you to have confidence in saving your inspiration with Ember. We’re currently hard at work on an update to Ember that removes our reliance on the opaque iCloud integration that Apple provides, and adopts an iCloud sync solution that is not only far less fragile and tempestous (sic), but something that allows us to consider further sync services in the future. This new solution also gives us the ability to fix things without waiting for the yearly OS X / iOS release cycle.

We know that there’s also a lot of interest in non-iCloud sync options for a variety of reasons (iCloud storage limits, personal preference, teams). This forthcoming update remains focused on iCloud, however we appreciate all your feedback about sync in Ember, and hope that our future plans will interest those of you who’d prefer other options.

Ultimately I was able to restore my library and nothing was lost. But I was lucky. If you are an Ember user I highly recommend backing up your library regularly (Ember>Backup>Create Library Backup), especially if you have not yet upgraded to version 1.5. And for other developers, it appears Core Data sync remains too unreliable to use.[3]

Moral of the story, sync is hard. And if you are going to offer a sync solution, for the love of Odin make regular, offline backups an automatic feature.

I should state that Realmac was extremely responsive to my support inquiry, and I would continue to recommend Ember to anyone who needs to work with a lot of screenshots and images on a regular basis. Just remember to backup your library, especially before any future app updates.

  1. By free I mean loaded to the gills with in-app purchases of course. Ember on iOS does use IAP to be fair, but this was planned from the beginning, limited to a few advanced features, and given that the iOS app is still meant as an extension of the Mac version rather than a standalone product, I consider their IAP usage to be legitimate.  ↩
  2. Yes I have Time Machine running, but this often does not help with iCloud related issues as the server will end up ruining the restored backup as soon as it engages anyway. If you are a developer using iCloud, please consider an auto backup feature that stores locally. Ember doesn’t have this. Were it not for my home computer being off, I would have lost my library permanently.  ↩
  3. Keep in mind that there are other ways to sync with iCloud that do appear to be reasonably stable, for example, document sync. In fact, I usually prefer iCloud sync for documents over solutions such as Dropbox for their simplicity. Calendar sync generally works well too, although it is pretty much just CalDAV so one would expect this.  ↩