Flying with TestFlight

TestFlight
Apple’s TestFlight

Apple launched their new TestFlight beta service in October of 2014. I’ve been a long time user of the original TestFlight service by Burstly which offered a pretty amazing way to get your builds out to beta testers pretty quickly.

Since Apple bought TestFlight it seems to have lost some very handy features like crash reporting (supposedly coming soon) and the ability to test on iOS 7 or below, but it’s also gained some major enhancements like being able to ship without UDID’s and no more SDK.

Over the last few weeks there have been a few questions pop up on Twitter about how certain parts of TestFlight work, so hopefully I can cover them here for you.

Invitations/Users

TestFlight invitation email
TestFlight invitation email

First of all your going to need some people who are actually willing to test out your app. In iTunes Connect, under the Prerelease tab of your app you need to add a list of External Testers. Apple allows you to have up to 1,000 testers which is great.

The first myth I’m going to quash is that you can actually invite beta testers using any email address. It does not need to be a users AppleID or App Store ready email address. Apple will simply send them an invitation email and once they’ve installed the TestFlight app from the App Store and accepted the invite, it will simply link the 2 together. I’ve accepted beta requests to multiple email addresses and they simply all list together in the TestFlight app.

Beta versions of an app appear on the user’s home screen with a small orange dot next to the app title to remind the user that they are running a beta. Beta’s are only valid for 30 days.

So what happens when they expire?

I personally think Apple totally overlooked this part of the process. When a TestFlight beta expires, it simply fails to launchTestFlight Beta. No warning message, no alert, it simply gives the impression that the app has crashed.

The first time this happened to me I was about to email the developer to let them know about the launch crash when it dawned on me to check the TestFlight app where it did in fact tell me it had expired.

The second time I experienced this was as the developer receiving a support request as to why my app would no longer load. Luckily they mentioned the orange dot and it suddenly became clear.

Internal Testers

If you’d like to get builds out to certain testers more quickly than waiting for the Apple approval process then you can use 25 Internal Testers. Internal Testers are added through the iTunes Connect Users and Roles section. The minimum role the user needs is Technical. This brings its own problems as any user with the technical role has the ability to change anything to do with your apps within iTunes Connect, from pricing to app description to deletion. They will also receive constant iTC emails whenever an app is approved/rejected/Ready for sale. I think Apple should allow us 25 users that have a specific Tester role, but while this isn’t the case, please choose your internal testers wisely.

Developers/Uploading

This is where things get interesting and confusing. First of all, with the new iTunes Connect system you can now upload an app build at any point without having to set iTunes Connect waiting to accept the upload. Every time you upload a new build, you are required to increment the build number in the archive. iTunes Connect will not accept a build with the same or lower build number than what has already been uploaded.

Developers still using the old Burstly TestFlight SDK will have noticed that during upload of a binary, you get a warning about the SDK being present. Currently this doesn’t seem to be a problem but maybe in the future Apple may start rejecting binaries using the old SDK.

Apple approval process

Apps made available to external testers require a Beta App Review and must comply with the full App Store Review Guidelines before testing can begin. A review is required for new versions of your app that contain significant changes.

Okay, so your uploaded app needs to go through the standard App Review process. But doesn’t that usually take longer than a week?

No.

From experience, beta reviews tend to happen within 24 hours, unless it’s the weekend were I’ve encountered no activity until Monday, however I’ve spoken with other developers that have had beta reviews occur at the weekend. Maybe Apple just has a skeleton crew working at the weekend.

Rejection

Yes, I have actually managed to have a beta app rejected. The reviewer couldn’t test the app as it requires specific hardware, so they requested a video of the app in use. This is the equivalent of a Meta data rejected status, however with TestFlight every rejection requires a new build to be uploaded and resubmitted. So you need to go back to Xcode, increment your build number and start again.

Significant changes

So you’ve had your first beta app approved and users are testing. What happens when you want to send out an update with some bug fixes. How does Apple know you’ve made significant changes to the app? How long does a re-review take?

The key is with the version number.

So long as you leave the app’s version number the same as the one that was approved, you can upload a new build and use it straight away. You do need to go through the process of submitting it for review, but ticking the box to say you’ve made no significant changes allows it to be instantly approved. As soon as you increment the app’s version number you are forced to submit for another full beta review. Also, once you’ve had a version approved for the App Store, you can not upload another build with the same version number or lower.

Maximum 2 builds per day

That’s right, Apple are limiting you to 2 reviews per day for the same app. That includes the no significant changes reviews that are automatic. So use them wisely. It’s tempting to want to quickly send a user a workaround to a bug to see if it works, then another, then another, but as soon as you get to your 3rd you are stopped in your tracks.

Processing

There seems to be a bug in the iTC system regarding Processing of uploaded builds. I’ve seen many people mention that they’ve waited hours or days for a build to finish processing. As silly as it might sound, I think refreshing the iTunes Connect Prerelease page too soon could be the culprit here.

Take a look at this screen capture:

iTunes Connect beta processing

I uploaded build 1, refreshed iTC to see that it had uploaded, after a couple of refreshes it moved from Uploaded to Processing. A day later I emailed iTunes Connect support. In the mean time I thought why not try uploading a new build.

Build 2 was uploaded and 10 minutes later I checked iTC and it was already processed and ready to submit or use internally even though build 1 was still processing.

In the mean time I received a reply from iTunes Connect support saying that Processing can take 24 hours and they are pleased to see that the build was now successfully processed. Erm… No it isn’t. Still to this day it is sat there Processing.

After my first app submission was rejected I thought i’d test out my theory. I uploaded build number 3, refreshed iTC to see when it moved to Processing then bam… a couple of hours later it’s still Processing. I uploaded build 4, left iTC for 10 minutes, came back and build 4 is ready for use whereas again build 3 is still Processing today.

Edit: I’ve heard from others that builds are stuck Processing regardless of visiting iTC. So it’s still a mystery as to why some uploads process quicker than others. And what does Processing actually mean?

3rd party code

With each beta review submission you must complete the same tedious fields in iTC’s form. Mike Piontek has created a safari plugin to quickly and automatically fill these in for you.

Brian Donohue has written a post Hacking TestFlight Invites with details on how to automatically add new users to the TestFlight service using a web form. I haven’t got around to doing this myself yet, and I’m not sure my web skills are up to the challenge, so if anyone has a complete working sample available I’d love to take a look.

 

Thanks for reading. I hope my experiences have helped you, and if you think of anything that might need adding to this post, please drop me a line.

  • Gary F

    Darren, that’s useful info. It’s frustrating how long it takes to get beta apps reviewed. Even minor changes with bug fixes take a painful amount of time. I haven’t seen an option to indicate there’s no significant changes? Apple should trust developers a bit more as these are only beta versions for a small, closed audience. My first beta submission to Google Play took 2 hours to approve and an update (version number bumped up by 1) took 30 mins to approve. Apple take note.

    • Darren

      Yes it’s frustrating. The “No significant changes” option is only available once you have had a BETA approved for a version, upload a new build but don’t change the version number.