Mistitled

Rich Siegel's Tumblr

WWDC 2014 Prognostications

Things that I think will change as of WWDC 2014 that nobody but a very few developers are likely to care about, although the implications may be fairly wide ranging:

  • 64-bit Mac App Store submissions

    Mac App Store submissions for new apps and upgrades other than maintenance updates will have to be 64-bit only. Any executables containing i386 code will be rejected automatically during submission. Maintenance updates to already shipping applications will be grandfathered in.

  • 64-bit only Mac OS SDK

    The 10.10 SDK will no longer include any 32-bit frameworks or libraries. This includes not just old frameworks such as HIToolbox that were never upgraded to x86_64, but also i386 architectures in all other frameworks. Building maintenance updates or modernizing legacy code will require using an older SDK. (10.8/10.9 SDKs will probably continue to support i386, but we will probably see the 10.7 SDK dropped at the next Xcode revision).

  • "Visual Deprecation"

    Assuming that the rumored sweeping changes to the overall look of the OS (and, more importantly, to the metrics of various UI widgets) come to pass, applications which are built to deploy on 10.8 or later and which use autolayout will get the new appearance. Applications built against older SDKs or which do not use autolayout will get the “legacy” (that is, 10.9) appearance.

    This will work against a consistent look and feel across all running applications, but may be effective as leverage to move developers toward the current best practices.

(This does not mean that 32-bit applications will suddenly stop working, because that isn’t the case. Existing applications will run on 10.10 as well as they did on 10.9, with the usual caveats that come with a major OS upgrade.)

The Gathering Storm: Our Travails with iCloud Sync

Preface

Recently, and with increasing frequency, developers have been speaking out about the difficulties they’ve had trying to implement support for iCloud in their products, and journalists have been reporting the news. I told our story to Ars Technica, and they’ve made their own report. In this post, I provide the background information behind our contributions to the Ars article.

Introduction

iCloud, the service provided by Apple, Inc. for sharing data among multiple Macs and iOS devices, brings with it the promise of power and convenience. However, many developers are having problems trying to fulfill the iCloud promise for their customers:

And we at Bare Bones Software have had difficulties with our digital junk drawer product, Yojimbo, as well.

iCloud Syncing 101

iCloud is the Apple-sanctioned sync service; support for it is built in to current operating systems, and it is available to all Mac and iOS users. As such, using iCloud is a natural and rational decision for a developer to make when deploying Mac and/or iOS applications that need to share data across multiple Apple devices for a single user.

In concept, the service is pretty simple. A central iCloud server holds the truth: the canonical version of the user’s data for an app. As the user manipulates an app’s data, iCloud tracks and reconciles the changes into the central truth, and makes sure that all copies of the data, on each computer, are brought up to date. In order for this to work, though, a lot has to happen behind the scenes. What we casually refer to as iCloud is many parts, each with a role to play.

  • The Ubiquity system runs locally on your Mac, manages data in your home/Library/Mobile Documents/ folder, identifies changed files, and submits them to the iCloud service. In reverse, the Ubiquity system receives data from the iCloud service and stores it locally.

  • The iCloud service, which runs on Apple’s servers and receives data from each Mac logged in to your iCloud account; and supplies data to each Mac in response to requests made by the Ubiquity system.

  • Code in the OS and application frameworks which communicate with the Ubiquity system to move data back and forth.

Applications that wish to support iCloud may do so in one of three basic ways:

  • Key-value storage (KVS): This is the simplest form of support, in which keys (item names, loosely speaking) are paired with values (item content). KVS is very limited; Apple has established both quantity and size limits on KVS usage in iCloud clients. So while it is an excellent solution for preferences and simple configuration data, it is not appropriate for document storage, and is unsuited for applications (such as Yojimbo) which must store multiple types of indexed items of various sizes, ranging from a few words of text to dozens of megabytes of PDF data.

  • Documents in the Cloud: This is another simple form of iCloud support, in which applications using the system’s standard document model may store discrete documents in such a way that the iCloud system synchronizes them between multiple devices (including iOS devices). Documents in the Cloud are supported fairly widely by applications on the Mac, and this method works well for document-oriented tools such as TextEdit or the iWork products. However, because the synchronization is done only at the level of a single document, the Documents in the Cloud model lacks the ability to discretely sync changes from multiple clients, or take advantage of other database functions.

  • Core Data syncing: This is where the rubber meets the road for database-backed applications. Core Data is the application-level database framework supplied by OS X and iOS that provides the means for applications to store items, and data about those items, in a single database. Yojimbo, our product, was one of the very first to ship using Core Data storage — we’ve used it since 2006, and it works great for storing data locally in the way that the product needs it to. Syncing database changes with iCloud, however, is a very complicated and difficult job for Core Data.

So, what’s so hard?

"Don’t you just turn on iCloud syncing, and it all Just Works?”

"Just works" might be the customer experience; but developers must do more to support iCloud (or any new OS technology) than simply flipping a switch. Things can go wrong, and they often do.

Opaque failure modes

In general, when iCloud data doesn’t synchronize correctly (and this happens, in practice, often), neither the developer nor the user has any idea why.

Sometimes, iCloud simply fails to move data from one computer to another. If you know the secret incantation to maximize the amount of logging done by the Ubiquity daemon, there are helpful folks at Apple who may be able to tell you what the entrails mean after the fact. Sometimes you’ll get errors such as, “The file upload timed out.” Apart from not being semantically relevant (Yojimbo doesn’t request any file uploads, because syncing is done at the Core Data object level), the message is also unhelpful because it doesn’t provide any information that either the user or developer can apply to diagnose and resolve the problem.

Corrupted baselines are another common obstacle. While attempting to deploy iCloud sync on Mac OS X 10.7, we ran into a situation in which the baseline (a reference copy of the synchronization data) would become corrupted in various easily encountered situations. There is no recovery from a corrupted baseline, short of digging in to the innards of your local iCloud storage and scraping everything out, and there is no visible indication that corruption has occurred — syncing simply stops.

Finally, one of the most maddening issues: the iCloud-just-says-no problem. Sometimes, when initializing the iCloud application subsystem, it will simply return an opaque internal error. When it fails, there’s no option to recover — all you can do is try again (and again…) until it finally works. And when it does initialize successfully, it can take an extremely long time. In some cases, we’ve seen delays of up to 25 minutes while waiting for the iCloud stack to initialize. There’s no discernable consistency or rationale for when it says no and when it finally says yes…you can just keep trying, and eventually it might just work, or not.

User Experience Shortfalls

Did you know that when you turn off the “Documents & Data” syncing option in the iCloud system preferences, the iCloud system deletes all of your locally stored iCloud data? Or when you sign out of iCloud, the system moves your iCloud data to a location outside of your application’s sandbox container, and the application can no longer use it?

This is part of iCloud’s central design philosophy: if you are using iCloud for data storage, the assumption appears to be that you are using iCloud exclusively for data storage. If you stop using iCloud on a given device, the system assumes that you are no longer interested in your stored data, and it gets removed from your device. Or if you sign out of iCloud (whether or not you intend to log in again), the system stashes your iCloud data in an undisclosed location. Applications which are sandboxed (as iCloud clients must be) are not able to see into this location, so when you sign out of iCloud, your iCloud data becomes inaccessible.

Consider the implications for a Core Data shoebox application such as Yojimbo: You dutifully make your Core Data storage ubiquitous, and then you sign out of iCloud. Poof, there goes your Yojimbo database.

There are ways of coping with this, at least in theory. The recovery from iCloud signout involves taking the opportunity to migrate all of your Core Data storage from Mobile Documents to the private sandbox container on your Mac. We found, to our dismay, that the practical reality didn’t hold up to theory — part of the problem is that you don’t get notified until after the data has been made inaccessible, and once in that state, there’s no choice but to use Core Data to make a copy of the data that’s just been sequestered. And of course, given a database of sufficient size, the process of using Core Data to relocate the database ties up the application in an unresponsive state, without visible progress, for as long as it takes. (And woe betide you if something goes wrong in the middle of it.)

Bugs

Of course, virtually every piece of software has bugs. The Core Data/iCloud sync interface is not immune, and unfortunately the bugs there are the ones that have caused us the most difficulty. There are bugs in Ubiquity on 10.7.x, which is why we had to (reluctantly) make the decision to require 10.8.2 as the minimum OS for supporting iCloud. Even on the latest 10.8 OS update there are bugs. Yes, we’ve reported them — in fact, we’ve gotten hands-on, in-person assistance from the Core Data and Ubiquity engineers at Apple. And even so, some bugs persist or are intractable. Here’s a sampling of what we’ve run into:

Object class confusion

In some circumstances (and we haven’t been able to figure out which, yet), iCloud actually changes the object class of an item when synchronizing it. Loosely described, the object class determines the type of the object in the database — in Yojimbo’s case, this would be a Note, or a Password, or a Web Archive, and so forth. When this bug occurs, iCloud synchronization sometimes causes PDFs to be changed to Web Archives; and sometimes causes Web Archives to be changed to PDFs. Of course, the object’s data or properties aren’t changed; only its object class; and when this happens, things fall apart when trying to view an object of the wrong class.

Broken relationships, lost containers

In many Core Data clients, objects which appear as single items in the UI actually have multiple parts that are held together by relationships. For example, a Web Archive item consists of a WebArchiveBlob item which contains the actual data, and a WebArchive item which maintains other relevant information about the item. (Splitting items up in this way makes it possible to change the properties of an item and not require the entire item to be resynchronized.) The WebArchive item contains relationships that identify its associated WebArchiveBlob; and the WebArchiveBlob contains relationships that identify the particular WebArchive as its owner.

This is all fine, except when iCloud fumbles. In some cases (again, not all the time), iCloud may do one of the following:

  1. Owner relationships in an item’s data will point to the wrong owner;

  2. Owner items get lost in synchronization and never appear on computers other than the one on which they were created. (This leads to the item never appearing in the UI on any other machine.) When this happens, bogus relationships get created between blob items and an arbitrary unrelated owner.

In some cases, we can repair bad relationships; it’s a process that requires significant amounts of time and memory, but can be done. Far more serious, though, is the case in which the owner item doesn’t get synced. It’s simply lost, and will never show up on a synced computer. There’s no way for us to know what it was, or reconstruct it; and no way to fix the incorrect relationships that got created for the blobs.

Lost blobs

Sometimes (without any apparent consistency or repeatability), the associated data for an object (for example, the PDF data for a PDF item, or the web archive data for a Web Archive item) would simply fail to show up on the destination machine. Sometimes it would arrive later (much later — minutes or hours). When the blob data doesn’t show up, we have no choice but to wait — the application can’t display what isn’t there, and there are no mechanics by which Core Data can make a request for immediate delivery of the data from iCloud.

Moreover, there’s no way to detect when the data finally arrives; all the application can do is repeat the request to Core Data ad infinitum until the data is available. (The folks at Apple did provide a helpful workaround in which we can peek in an undocumented location to discover when the data is finally available, though.)

It’s not just us

As mentioned earlier, quite a few Mac and iOS developers have been discovering, the hard way, what we’ve also found. It’s been more than a year since we undertook the job of converting Yojimbo to use iCloud for synchronization, and despite our best efforts, we have been unable to get it working reliably to the point that we can ship a new version to customers. And we find it noteworthy that, to the best of our knowledge, none of Apple’s currently shipping products on Mac OS X use iCloud to sync Core Data. (One might think that Address Book or Calendars would, but in fact they use CardDAV and CalDAV, respectively. These are both HTTP-based protocols for communicating directly with a server, and do not rely on Core Data or the Ubiquity system.)

Alternatives

We, and other developers similarly affected by these challenges, have considered using other services for syncing. Another possibility would be for developers to implement their own sync services.

Syncing is an enormously complicated problem to solve in a reliable way. It’s not an impossible problem to solve, but we’re a small, independent software shop, and we have to be judicious about how we spend our time and resources. In addition, providing a custom sync service also requires a significant capital expense: the master copy of the data has to live somewhere, and that in turn requires investment in data-center-grade server hardware, software, applications, staff, and infrastructure services to keep it all running.

Dropbox is an excellent service for sharing files between multiple computers. However, Dropbox is not suitable for synchronizing between Core Data applications; you can’t just dump your database in a Dropbox and have it work as reliably as we’d like. (We do provide a technical note which describes the process for anyone willing to try to sync Yojimbo with Dropbox; the caveats and the steps required to make it work between desktop installations should shed some light on the issues involved.)

As for other alternatives, a couple of motivated developers have undertaken to engineer their own sync services for use by folks who are writing Core Data applications that require synchronization. Unfortunately, none of the alternatives we have investigated so far have been suitable “out of the box”, but there are some good candidates. We’re continuing along this line of inquiry, because a robust sync service that supports our needs is something that our software was conceived to use.

It is very important to note, however, that using a third-party sync service has implications beyond the simple one of whether syncing works or not. It is vitally important that whatever service we choose is robust and reliable on the back end (where sync state is maintained and the data is stored); it won’t do anyone any good if the sync service doesn’t work reliably.

In addition, third-party services are not free. Data synchronization requires storage and bandwidth; those come at an ongoing cost, and the provider must either pass the cost along to the customer, or go out of business. So, developers who use third-party sync services have to make some difficult decisions about how their software is priced and sold, if they wish to have a product with any longevity.

"What’s next?"

Yojimbo was designed with synchronization as a core feature; so as far as we’re concerned, sync needs to work, and uphold our standards for performance and reliability. We and other affected developers are continuing to iterate with Apple regarding the technical problems we’ve run into. However, if iCloud sync can’t be made to work, perhaps another service will do the job.

We don’t yet know how circumstances will unfold, nor can we tell how long it’ll take to overcome the challenges. But we will continue to work on providing synchronization in Yojimbo, in whatever form it may take.

fin

Andrew Singer

Last night I learned (via tweet from my friend and former colleague Philip Borenstein) that Andrew Singer had passed away recently.

Andrew was one of the founders of THINK Technologies, and nearly thirty years after the fact, those are names recognized by very few. Andrew and his colleagues at THINK conceived and created (among other things) Macintosh Pascal, LightspeedC and Lightspeed Pascal (later rebranded as THINK C and THINK Pascal). I was privileged to work for him at the very beginning of my professional career.

The THINK products helped define the Mac software ecosystem in the 1980s, and I don’t think it’s an overstatement to say that the THINK products, and Andrew’s influence, made the Mac software industry successful at a crucial time. There’s a great writeup here (thanks to Philip for the link) about Macintosh Pascal and THINK Pascal and their effect on how Mac software was developed.

I can say without exaggeration that were it not for Andrew’s decision to hire me, and to put me next to some of the brightest minds in existence, and to start me on the path of learning not only the proper craft of software engineering, but many other valuable professional and life lessons along the way, that I wouldn’t have followed the road that has brought me to where I am now; and that my life would be very different.

So if you use a Mac, thank Andrew Singer and the smart people he hired, and raise a glass to his memory.

BAR SOPA

Speaking as someone who’s made a living for the past 20 years producing content that is often pirated, I am strongly opposed to SOPA/PIPA. They are the wrong answer to the wrong question. They are an attempt to finish the work started by the repugnant DMCA.

The lobbyists of SOPA/PIPA are making a statement: “We’re not making enough money by overcharging for movie tickets. We’re not making enough money by weakening and circumventing the First Sale Doctrine. We’re not making enough money by making you pay for the same content multiple times. The DMCA didn’t increase our profits by a large enough amount. Our business model is based on avarice and greed, and not on integrity and creativity. So now we’re going to lobby to enact legislation which allows law enforcement to act in our business and financial interests.”

As the producers of intellectual property it is up to us to make sure that our businesses can survive the inevitable piracy of our product. A good way to do this is to make a product for which more people want to pay a fair price than who are willing to steal it. Another important step is to realize that the people who are stealing your product would, in all likelihood, not have paid for it in any event. (Obviously this applies to end users; and not to the criminals who are reselling pirated material. There are already laws in place that apply to criminal piracy.)

Take the money you would have spent on fighting piracy and use it to make a better product and/or service. Making more laws is not the answer.

Deployment Report: Humanscale M8 Monitor Arm

I have a 30-inch Apple Cinema Display, which has given me good service for many years; however, I’ve long desired greater flexibility of placement, more free space on my desktop, and generally a workspace that is a better ergonomic fit.

After some investigation, I settled on the Humanscale M8. One of the problems I ran into with monitor arms in general is that there’s such a dizzying array of models and options that “paralysis of choice” quickly sets in.) The M8 is rated for monitors up to 40lb, which easily covers the ACD-30 as well as current-generation iMacs and monitors. Also, it uses a spring-loaded tension system, instead of gas cylinders; this is marketed as being more reliable in the long term. Time will tell.

Installation was complicated by two factors: first, my desk is a granite table, which rests on a steel frame which is itself inlaid into a wooden leg structure. An edge clamp is not usable. So, I called a stone cutter to come and drill a suitable hole in the table to accommodate the M8’s through-hole mounting base.

The other problem I had is that the M8 (like every other arm that’s currently available) uses the VESA mount plate standard. Apple has a VESA adapter kit for its cinema displays — in fact, there are two SKUs. One of them fits the current-generation LED displays and iMacs, and the other fits the older aluminum-case displays. At the time I purchased the M8, the VESA kit for the older displays was not available in the US; oddly, the UK Apple Store did have it, so I enlisted the aid of a friend in the UK to purchase and ship the kit. Checking today, both versions are actually available in the US Apple Store: VESA kit for aluminum cinema displays and VESA kit for newer cinema displays, the Thunderbolt display, and iMacs.

With the desk drilled and parts acquired, installation was reasonably straightforward. If you’re using the bolt-through mount, I recommend the use of blue Loctite on the bolt so that you won’t have to worry about it loosening up. I also used blue Loctite on the bolts attaching the VESA adapter plate to the monitor. It’s also a good idea to have a helper to steady the display while you place the arm and adjust the pivots.

Once in place, fine tuning is pretty easy, and I’m very happy with the results. I can set the display a little higher and a little closer than the factory foot allows, and although it’s a small difference in placement, it’s a big difference in readability and comfort. Also, I can now swing the arm and display entirely off to the side (without really disrupting the basic adjustments). This is great for having conversations with the rest of the room, since the display no longer functions as a room divider.

Humanscale’s web site shows a list price of US$485. I got mine from ErgoDirectUSA for $285; you can probably find similarly discounted pricing through other outlets.

Deployment report: CarMD “Vehicle Health System”

Deployment report: CarMD “Vehicle Health System”

The CarMD product is a combination of a hardware product (an OBD-II code reader) and a web service, with a little bit of desktop software to tie it together. The hardware product is roughly the size of a fat TV remote; it fits reasonably well into one’s hand. On one end there’s a connector that plugs into the OBD-II connector on your car (present on all cars sold in the US from model year 1996-on); on the other end there’s a USB type-B socket to receive a USB cable (which is included in the package).

To use the CarMD, you plug it in to your car’s OBD-II port. Then you turn the car’s ignition to the “ON” position (without starting). In about a minute or so, the CarMD reads the data from the car’s onboard diagnostics, and beeps to indicate completion.

Given the typical variation in the locations of cars’ OBD-II data ports, and the relative inconvenience of those locations, the optional extender cable should be considered mandatory. (It’s available as a separate accessory, or as part of an “essentials” bundle which includes the reader and the extension cable for a slightly discounted price.)

After gathering data from the car, you start the CarMD application (that you have installed at some prior point), and plug the unit into your computer using the included USB cable. The desktop application connects you to the CarMD web site. Each CarMD unit has a unique identifier; if the web site doesn’t recognize yours, you’ll be asked to fill in some information: your name and contact info; the car’s VIN (from which the web site will determine the year/make/model) and the car’s mileage. Thereafter, the web site will recognize you from the unit’s unique ID, log you in, and display the information gleaned from the unit.

There are some rough edges on the software side. There were typos in the quick-start reference guide, as well as in the software UI itself. The CD included version 3.1.0 of the Mac application; but the web site offers up version 3.0.5, and nothing newer. (This could be a problem for users with MacBook Airs and other computers which lack a working optical drive.)

The UI of the Mac application makes only minimal use of standard OS look and feel; it looks pretty rough and lacks useful feedback such as progress indicators when loading data from the hardware and reading the diagnostic data (both of which took long enough for me to wonder if something was wrong).

A CarMD account is free, which is good — the hardware is absolutely useless without it. The unit does not display any human-readable information; nor does the desktop application. The web site itself displays only minimal technical information; either your car is working correctly (no codes stored in the OBD-II memory); or it needs attention (there are codes); or there’s a must-fix-before-emissions-inspection condition. This lack of detail is unfortunate as it makes the CarMD much less attractive to enthusiasts and home mechanics.

Although the basic CarMD service is free, the web site offers “premium” content for a fee. For example, any factory-issued Technical Service Bulletins are listed; but you cannot read them without first paying a fee for each one (currently US$2.99). A free account allows you to keep up to three cars; you can generate up to six reports per month (and you only need to generate a report if there’s a code stored in your car’s OBD-II memory). This is probably enough for most households.

Even so, CarMD offers a “Premium Vehicle Health Membership”, which for about $30 offers you access to several services not available for free accounts: “unlimited access to TSBs”, “unlimited access to warranty and scheduled maintenance intervals for all your registered vehicles”, “special private content”, “special discounts from partners”, “subcription to the CarMD newsletter”. Speaking as an enthusiast, the value proposition isn’t compelling to me; discovering the maintenance schedule is pretty easy for anyone willing to crack open the owner’s manual, and most enthusiast clubs (such as the BMW CCA) provide a great central resource for TSBs and other make-specific technical info.

One big shortcoming of the CarMD hardware is that it’s a static reader, and not a continuous monitor (such as the ScanGauge, which I intend to try out and write up). That USB port would be lovely for connecting to a MacBook Air or iPad and gathering all kinds of data as you drive. Again, that doesn’t make much difference to the target audience for the CarMD; but it limits its appeal to advanced users, enthusiasts, and tinkerers.

Even with these caveats, the CarMD remains useful as a way for you to know what’s wrong with your car, in more detail than you might get from a dealer service writer or independent mechanic.