XCUITest + SauceLabs + GitHub Actions = Incredible Mobile iOS CI/CD Automation!

As a Software Development Engineer in Test (SDET), ensuring the quality of iOS applications is paramount. One effective way to achieve this is by integrating automated testing into the development workflow. In this blog post, we’ll explore how to set up Sauce Labs with XCUITest in GitHub Workflows to streamline iOS app testing within your CI/CD pipeline.

Why Sauce Labs?

Sauce Labs provides a cloud-based platform for automated testing of web and mobile applications. With its extensive device, simulators & emulators coverage, SauceLabs empowers teams to run tests across various configurations, ensuring broad test coverage and faster feedback loops. While SauceLabs offers a good solution for mobile iOS automation, it’s worth noting that there are other cloud-based testing platforms available in the market such as:

• AWS Device Farm
• MacStadium
• Perfecto
• LambdaTest
• Firebase Test Lab
• Experitest

Integrating Sauce Labs with XCUITest in GitHub Workflows

I have complete source code available for you for the demo project (link below)

1.) Sign up for SauceLabs. Begin by registering for Sauce Labs, leveraging their enticing 28-day free trial option via Sauce Labs Sign-Up.

2.) Grab SauceLab’s Credentials (we will need it later steps):
Navigate to the Sauce Labs user settings page here, and copy both the “User Name” and “Access Key” for subsequent steps.

3.) Preparing the Build for Testing:
Our initial task is to prepare the Test App and UIRunner files for SauceLabs. To begin, let’s craft a new file named “prepareTestBuild.sh” in your root directory.

Here are the contents of prepareTestBuild.sh file to prepare build to run on Simulators:

xcodebuild \
  clean build-for-testing \
  -project DemoApp1.xcodeproj \
  -scheme DemoApp1 \
  -derivedDataPath './customFolder' \
  -sdk iphonesimulator \
  -arch x86_64 \
  -configuration Debug \
  CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO

# Check if AppsToTest folder exists, create one if not, and ensure it's empty
if [ ! -d "AppsToTest" ]; then
    mkdir -p AppsToTest
else
    rm -rf AppsToTest/*
fi

# Zip the app if it exists
if [ -d "customFolder/Build/Products/Debug-iphonesimulator/DemoApp1.app" ]; then
    zip -r AppsToTest/DemoApp1.zip customFolder/Build/Products/Debug-iphonesimulator/DemoApp1.app
else
    echo "Error: DemoApp1.app not found"
fi

# Zip the app if it exists
if [ -d "customFolder/Build/Products/Debug-iphonesimulator/DemoApp1UITests-Runner.app" ]; then
    zip -r AppsToTest/DemoApp1UITests-Runner.zip customFolder/Build/Products/Debug-iphonesimulator/DemoApp1UITests-Runner.app
else
    echo "Error: DemoApp1UITests-Runner.app not found"
fi

And here are the contents of prepareTestBuild-rdc.sh file to prepare build to run on Real Devices:

xcodebuild \
  CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO \
  clean build-for-testing \
  -project DemoApp1.xcodeproj \
  -scheme "DemoApp1" \
  -sdk iphoneos \
  -configuration Debug \
  -derivedDataPath './customFolder-rdc'


cd customFolder-rdc/Build/Products/Debug-iphoneos
mkdir Payload
cp -r DemoApp1.app Payload/
zip --symlinks -r DemoApp1.ipa Payload
rm -rf Payload; mkdir Payload
cp -r DemoApp1UITests-Runner.app Payload/
zip --symlinks -r DemoApp1UITests-Runner.ipa Payload

cd ../../../..

# Check if AppsToTest folder exists, create one if not, and ensure it's empty
if [ ! -d "AppsToTest-rdc" ]; then
    mkdir -p AppsToTest-rdc
else
    rm -rf AppsToTest-rdc/*
fi


# Copy .ipa to AppToTest-rdc folder
cp customFolder-rdc/Build/Products/Debug-iphoneos/DemoApp1.ipa AppsToTest-rdc/
cp customFolder-rdc/Build/Products/Debug-iphoneos/DemoApp1UITests-Runner.ipa AppsToTest-rdc/

Execute the script “prepareTestBuild.sh” to generate “DemoApp1.zip” and “DemoApp1UITests-runner.zip” within the designated “AppsToTest” folder.

#For Simulator
sh prepareTestBuild.sh

#OR

#For Real Devices
sh repareTestBuild-rdc.sh

For Simulators, this will generate “DemoApp1.zip”, and “DemoApp1UITests-runner.zip” in AppsToTest folder in your root directory.

And Real Devices, it will generate “DemoApp1.ipa”, and “DemoApp1UITests-runner.ipa” in AppsToTest-rdc folder in root directory.

4.) Install SauceCtl

Leverage SauceCtl, a command-line tool tailored by Sauce Labs, facilitating the efficient management and execution of automation scripts on their cloud-based testing platform.

• Install saucectl via npm with the command

npm install -g saucectl

Next we will use saucectl to configure the project:

saucectl init xcuitest

Follow the prompts to select desired configurations. I selected below configurations:

Select region: us-west-1
Select target: Virtual Devices
Select simulator: iPhone 15
Select platform version: 17.0
Application to test: AppsToTest/DemoApp1.app
Hint: Enter folder name AppsToTest/ and press TAB
Test Application: AppsToTest/DemoApp1UITests-Runner.app
Download artifacts: always

This will create a .sauce folder in your root folder, and add config.yml inside it.

5.) Customizing Configurations:

We can either modify the default config.yml in .sauce directory as per our needs or we can create distinct config files for different test suites or target devices.

I duplicated config.yml and named the new file config-sim.yml. This dedicated config file will be utilized for all tests intended to run on the Simulator.

Here are contents of the new config-sim.yml file (For Simulators):

apiVersion: v1alpha
kind: xcuitest
showConsoleLog: false
sauce:
  region: us-west-1
  # Set concurrency to 3 if let's say you want 3 parallel simulators  
  concurrency: 1
  # Additional properties to provide project information for distinguishing environments
  # (e.g., project name, build number)
  metadata:
    build: RC 1.0.1
    tags:
      - Simulator
      - e2e 
  # Number of times to retry a failed suite
  retries: 3
xcuitest:
  # Details specific to XCUITest project
  app: AppsToTest/DemoApp1.zip
  testApp: AppsToTest/DemoApp1UITests-Runner.zip
suites:
- name: xcuitest - iPhone 11 Simulator
  # Time in seconds to wait for a suite to finish
  timeout: 5m
  # Concurrency setting to split tests into groups for faster execution
  shard: concurrency
  testListFile: testListFile.txt
  # Retry strategy
  smartRetry:
    failedOnly: true
  # Specify simulator name and OS version(s) (e.g., iPhone 12 iOS 15.0)
  simulators:
  - name: "iPhone 11 Simulator"
    platformVersions:
    - "16.2"
artifacts:
  # Specify how to manage test output (logs, video, screenshots, junit.xml)
  cleanup: true # Clear download directory before adding new artifacts
  download:
    match:
    - '*'
    when: fail # Download artifacts only for failed test suites
    directory: ./artifacts

Here are contents of the new config-rdc.yml file (For Real Devices):

apiVersion: v1alpha
kind: xcuitest
showConsoleLog: false
sauce:
  region: us-west-1
  #Set concurrency to 3 if let's say you want 3 parallel real devices  
  concurrency: 1  
  metadata:
    build: RC 1.0.1
    tags:
      - Simulator
      - e2e 
  retries: 3
xcuitest:
  app: AppsToTest-rdc/DemoApp1.ipa
  testApp: AppsToTest-rdc/DemoApp1UITests-Runner.ipa
suites:
- name: xcuitest - Real Device Testing
  timeout: 5m
  shard: concurrency
  testListFile: testListFile.txt
  smartRetry:
    failedOnly: true
  devices:
  - name: "iPhone.*"
    options:
      deviceType: ANY
artifacts:
  cleanup: true
  download:
    match:
    - '*'
    when: always
    directory: ./artifacts

Few things to note, that I added in addition to what we had in our default config file:

metaData : To provide properties that allows me to send additional info about my project that will help distinguish in various environments.

retries : To provide number of times to retry a failed suite.

xcuitest : To provide details specific to XCUITest project like location of app, and test-runner files.

timeout : Instructs saucectl on how long to wait for suite to finish.

shard : Setting shard to concurrency will automatically split the tests in several groups to speed up execution time.

smartRetry : To specify the retry strategy. When failedOnly is true, saucectl will run only failed tests on retry.

simulators : To pass the simulator name, and OS version(s). Checkout SauceLabs supported virtual devices

artifacts : To specify how to manage test output such as logs, video, screenshots, junit.xml files. Pass cleanup as true ensure, all contents of the download directory is cleared before new artifacts are added. when specifies when to download artifacts. It’s best to download artifacts only for failed test suites, so simply pass when as fail

6.) Executing Tests:
Now, it’s time to perform a test run to verify the setup we’ve completed thus far.

# For Simulator
sh prepareTestBuild.sh

# OR

# For Real Devices
sh prepareTestBuild-rdc.sh

The steps mentioned above will prepare the necessary builds for saucectl. Let’s proceed to run the tests now.

# For Simulator
saucectl run -c .sauce/config-sim.yml

# OR

# For Real Devices
saucectl run -c .sauce/config-rdc.yml

Here comes the magic of saucectl: it will upload the test app and UIRunner file in background, and kickstart the test execution on Sauce Lab infrastructure. You can access the SauceLabs portal online by following the link provided by saucectl in the terminal.

Below are screenshots of my results:

7.) Integration with GitHub CI/CD Pipeline:
Next, let’s integrate this with the GitHub CI/CD pipeline. I opted for GitHub due to its cost-free nature. However, it’s worth noting that SauceLabs seamlessly integrates with a diverse array of CI/CD tools, including GitLab, Azure, Bitbucket, Jenkins, CircleCI, TeamCity, and more.

• Add SAUCE_USERNAME and SAUCE_ACCESS_KEY in GitHub repository secret variables. How to add secrets to GitHub repo
• Add github_workflow.yml file in directory .github/workflows

Contents of my github_workflow.yml file are below:

name: Sauce Labs Tests

on:
  pull_request:
    branches:
      - main

jobs:
  test:
    runs-on: macos-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '14'

      - name: Prepare test build
        run: |
          chmod +x prepareTestBuild.sh
          ./prepareTestBuild.sh

      - name: Install dependencies
        run: npm install

      - name: List generated files
        run: |
          ls -R

      - name: Run Sauce Labs tests
        run: |
          npm install -g saucectl
#         For Simulators
          saucectl run -c .sauce/config-sim.yml
#         OR
#         For Real Devices
#         saucectl run -c .sauce/config-rdc.yml
        env:
          SAUCE_USERNAME: ${{secrets.SAUCE_USERNAME}}
          SAUCE_ACCESS_KEY: ${{secrets.SAUCE_ACCESS_KEY}}

That’s it. Sauce Test Labs will execute on each pull requests where the destination branch is main

Screenshot of Github workflow action:

Don’t worry, I’ve got the complete source code available for you in a demo project:

GitHub - ssharma0312/DemoApp1: SwiftUI demo project for SauceLabs