Skip to content

Question: Does bluepill support running Xcode UI testing? #16

New issue
New issue
Closed
Closed
Question: Does bluepill support running Xcode UI testing?#16
Assignees
oliverhu
Labels
enhancementpr coming
Milestone
 
XCTUITest support
@dyang

Description

@dyang
dyang
opened on Jan 19, 2017 · edited by dyang

I managed to run bluepill against one of my apps with a bunch of UI tests based on Xcode UI testing. However I'm getting the following error. A bit googling suggests that this might be due to UI testing having different test template/config than "regular" XCTests. Is Xcode UI testing supported by bluepill? :)

Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'No target application path specified via test configuration: <XCTestConfiguration: 0x60800011a9d0>

Thanks!

Activity

oliverhu

oliverhu commented on Jan 19, 2017

@oliverhu
oliverhu
on Jan 19, 2017
Member

hey @dyang , we haven't actually tested this with Xcode UI testing. Our UI tests are all KIF tests.

We collect the test target by searching SampleApp.app/Plugin/*.xctest. You can test by adding a additional-xctests flag in the config file to the UI testing target and try from there.

It would be great if you can tweak the code base a bit and let us know the result.

oliverhu
mentioned this on Jan 19, 2017
krze

krze commented on Jan 19, 2017

@krze
krze
on Jan 19, 2017 · edited by krze

When you compile an XCUITest there are no .xctest files generated in the target app's plugin folder. Instead it's placed inside the runner, named TEST_TARGET_NAME-runner.app/Plugin/TEST_TARGET_NAME.xctest.

I gave what you said a whirl and all 5 sims exited with the same error:

[  INFO  ] Started Simulator 5 (PID 42860).
bp: invalid configuration
	iOS 10.1 is not a valid runtime.

When checking valid runtimes:

== Runtimes ==
iOS 9.3 (9.3 - 13E233) (com.apple.CoreSimulator.SimRuntime.iOS-9-3)
iOS 10.2 (10.2 - 14C89) (com.apple.CoreSimulator.SimRuntime.iOS-10-2)
tvOS 10.1 (10.1 - 14U591) (com.apple.CoreSimulator.SimRuntime.tvOS-10-1)
watchOS 3.1 (3.1 - 14S471a) (com.apple.CoreSimulator.SimRuntime.watchOS-3-1)

I'll install a 10.1 sim and give it another shot

oliverhu

oliverhu commented on Jan 19, 2017

@oliverhu
oliverhu
on Jan 19, 2017
Member

@krze that's another issue mentioned in #8 , you need to download iOS 10.1 simulator Or alternatively, search for BP_DEFAULT_RUNTIME and change the value to 10.2

krze

krze commented on Jan 19, 2017

@krze
krze
on Jan 19, 2017

Looks like the target application isn't being pulled from the XCUITest .xctest file:

2017-01-19 17:47:04.802 ApplicationNameX[44527:3337770] *** Assertion failure in -[XCUIApplication init], /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-11753/XCTestFramework/UI Testing/XCUIApplication.m:72
/Users/krze/develop/ApplicationName/UITests/TestHelpers.swift:0: error: -[UITests.AppRatingPromptTests testShowsAfterTestSetup] : failed: caught "NSInternalInconsistencyException", "No target application path specified via test configuration: &lt;XCTestConfiguration: 0x60000011df40&gt;
	                  testBundleURL:file:///Users/krze/develop/ApplicationName/TestBuild/Build/Products/UITest-iphonesimulator/UITests-Runner.app/PlugIns/UITests.xctest/
	         testBundleRelativePath:(null)
	              productModuleName:UITests
	                    testsToSkip:
	                     testsToRun:(null)
	             reportResultsToIDE:no
	              sessionIdentifier:18C96A25-3055-407C-8CEF-EFDAA68F8BFC
	     pathToXcodeReportingSocket:(null)
	      disablePerformanceMetrics:no
	treatMissingBaselinesAsFailures:no
	                baselineFileURL:(null)
	       baselineFileRelativePath:(null)
	          targetApplicationPath:(null)
	      targetApplicationBundleID:(null)
	               reportActivities:no
	       testsMustRunOnMainThread:no
	         initializeForUITesting:no
	 aggregateStatisticsBeforeCrash:(null)
"
(
	0   CoreFoundation                      0x000000011ab7834b __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x000000011a5e021e objc_exception_throw + 48
	2   CoreFoundation                      0x000000011ab7c442 +[NSException raise:format:arguments:] + 98
	3   Foundation                          0x00000001146dde4d -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 195
	4   XCTest                              0x000000012ba022a6 -[XCUIApplication init] + 195
	5   UITests                             0x000000012c409f1f _TTSf4n_g_n___TFE7UITestsCSo10XCTestCase11setUpTumblrfT8loggedInSb8withUserVS_11TumblrCreds_T_ + 207
	6   UITests                             0x000000012c3d23e0 _TFC7UITests20AppRatingPromptTests5setUpfT_T_ + 640
	7   UITests                             0x000000012c3d2726 _TToFC7UITests20AppRatingPromptTests5setUpfT_T_ + 22
	8   XCTest                              0x000000012b9e0e43 __24-[XCTestCase invokeTest]_block_invoke_2 + 173
	9   XCTest                              0x000000012ba197df -[XCTestContext performInScope:] + 190
	10  XCTest                              0x000000012b9e0d83 -[XCTestCase invokeTest] + 255
	11  XCTest                              0x000000012b9e159c -[XCTestCase performTest:] + 457
	12  XCTest                              0x000000012b9de664 -[XCTestSuite performTest:] + 491
	13  XCTest                              0x000000012b9de664 -[XCTestSuite performTest:] + 491
	14  XCTest                              0x000000012b9de664 -[XCTestSuite performTest:] + 491
	15  XCTest                              0x000000012b9ebd2b -[XCTestObservationCenter _observeTestExecutionForBlock:] + 602
	16  XCTest                              0x000000012ba1af0d _XCTestMain + 1050
	17  CoreFoundation                      0x000000011ab1d25c __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
	18  CoreFoundation                      0x000000011ab02304 __CFRunLoopDoBlocks + 356
	19  CoreFoundation                      0x000000011ab01a75 __CFRunLoopRun + 901
	20  CoreFoundation                      0x000000011ab01494 CFRunLoopRunSpecific + 420
	21  GraphicsServices                    0x000000011c739a6f GSEventRunModal + 161
	22  UIKit                               0x000000011877a964 UIApplicationMain + 159
	23  ApplicationName                     0x000000010effccdf main + 111
	24  libdyld.dylib                       0x000000011868e68d start + 1
)
oliverhu

oliverhu commented on Jan 19, 2017

@oliverhu
oliverhu
on Jan 19, 2017
Member

Hmm, this should be doable, we just need to fill some extra fields in the XCTestConfiguration file.

Check XCTestConfiguration.h and SimulatorHelper.m

In SimulatorHelper.m:
Add [xctConfig setTargetApplicationPath:PATH_TO_APP]; to + (NSString *)testEnvironmentWithConfiguration:(BPConfiguration *)config; should do the trick.

Let me know if you that works :D

oliverhu
added
enhancement
on Jan 19, 2017
oliverhu
added this to the XCTUITest support milestone on Jan 19, 2017
casshill

casshill commented on Jan 20, 2017

@casshill
casshill
on Jan 20, 2017

I tried the above solution and also added the target app bundle identifier like this
[xctConfig setTargetApplicationBundleID:BUNDLE_ID];

But I still can't get my tests to run. I don't think the test application is launching properly
<testcase classname="..." name="test..." time="61.096000"> <failure type="Failure" message="-[...Tests test...] : UI Testing Failure - Unable to update application state promptly.">

plu

plu commented on Jan 20, 2017

@plu
plu
on Jan 20, 2017

To make the UI Tests work, you need to set a few properties on XCTestConfiguration:

  • testsMustRunOnMainThread
  • initializeForUITesting
  • targetApplicationPath
  • targetApplicationBundleID

You can peek a bit here how Facebook is doing it: https://github.com/facebook/FBSimulatorControl/blob/master/XCTestBootstrap/Bundles/FBTestConfiguration.m#L85-L95

oliverhu

oliverhu commented on Jan 20, 2017

@oliverhu
oliverhu
on Jan 20, 2017
Member

@plu love it, thanks!

oliverhu
self-assigned this
on Jan 20, 2017
lochness42
mentioned this on Jan 20, 2017
oliverhu

oliverhu commented on Jan 20, 2017

@oliverhu
oliverhu
on Jan 20, 2017 · edited by oliverhu
Member

@plu it turns out to be more complicated than I thought. I managed to set the environment same as the one Xcode is using to run UI test (arguments, environment and XCTestConfiguration)

// XCTUITest configurations
[xctConfig setPathToXcodeReportingSocket:nil];

[xctConfig setTargetApplicationPath:config.appBundlePath];
[xctConfig setTargetApplicationPath:@"PATHTO_BPSampleAppUITests-Runner.app"];
[xctConfig setTargetApplicationBundleID:@"LI.BPSampleApp"];
[xctConfig setTestsToRun:[NSSet setWithObjects:@"BPSampleAppUITests/testExample", nil]];
[xctConfig setTestsMustRunOnMainThread:YES];
[xctConfig setInitializeForUITesting:YES];
[xctConfig setTreatMissingBaselinesAsFailures:NO];
[xctConfig setReportResultsToIDE:YES];
[xctConfig setReportActivities:YES];

And get:

  Jan 20 01:34:23 khu-mn1 CoreSimulatorBridge[15027]: Launch successful for 'com.apple.test.BPSampleAppUITests-Runner'
  Jan 20 01:34:23 khu-mn1 assertiond[15022]: Could not create or rename power assertion for <BKNewProcess: 0x7ff009c10a90; com.apple.mobilecal; pid: 15102; hostpid: -1>: 0xe00002c9
  Jan 20 01:34:23 khu-mn1 logd[15012]: metadata shared cached uuid is null (using logd's shared cache info) XCTRunner (15103)
  Jan 20 01:34:23 khu-mn1 logd[15012]: Failed to harvest strings for pathless uuid '00000000-0000-0000-0000-000000000000'
  Jan 20 01:34:23 khu-mn1 assertiond[15022]: Could not create or rename power assertion for <BKNewProcess: 0x7ff009c10a90; com.apple.mobilecal; pid: 15102; hostpid: -1>: 0xe00002c9
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: assertion failed: 15G1217 14C89: libxpc.dylib + 62677 [0339A402-6531-30E9-8BC2-589C463CD6D6]: 0x7d
  Jan 20 01:34:23 khu-mn1 watchlistd[15090]: Now playing app did change to '(null)' (playing: 0) from '(null)'
  Jan 20 01:34:23 khu-mn1 watchlistd[15090]: WLKPlaybackSummary - Parameter failed validation bundleID. It is nil
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: Running tests...
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: a
  Jan 20 01:34:23 khu-mn1 XCTRunner[15103]: c  // Before the launching app statement

And it got stuck at the launching app statement. Searched some old threads from Xctool, there is extra testmanagerdr to deal with to run XCTUITest?

Does FBSimualatorControl support running XCTUITest?

oliverhu
added
help wanted
on Jan 20, 2017

32 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementpr coming

Type

No type

Projects

No projects

Milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @plu@dyang@nickmshelley@jimmyeisenhauer@oliverhu

      Issue actions
        Question: Does bluepill support running Xcode UI testing? · Issue #16 · MobileNativeFoundation/bluepill