Mobile Test Automation: LambdaTest using XCUITest and GitHub Actions

In the field of software quality assurance, ensuring the reliability of applications is paramount. Automated testing plays a crucial role in maintaining this standard throughout the development cycle. This blog post will explore how to set up real device testing using LambdaTest and XCUITest within GitHub Workflows, thereby improving the effectiveness of iOS app testing in your CI/CD pipeline.

What is LambdaTest?

LambdaTest provides a vast array of real devices and browsers for automated testing across multiple platforms such as Web, Android, and iOS. It supports a broad spectrum of testing frameworks, including Selenium, Cypress, Playwright, and Puppeteer for web applications, as well as Appium, Espresso, and XCUITest for mobile apps on real devices. This blog will specifically focus on native iOS app testing using XCUItest. LambdaTest’s extensive device library allows teams to perform tests across numerous configurations, enhancing test coverage and accelerating feedback loops. While LambdaTest stands out for mobile iOS automation, it is also essential to consider other cloud-based testing platforms available in the industry.

• SauceLabs
• BrowserStack
• Perfecto
• Firebase Test Lab
• AWS Device Farm
• MacStadium
• Perfecto
• Digital.ai
• Marathon Labs

Some of the pros and cons of LambdaTest, in my opinion

Pros:

• Extensive selection of devices available in both public and private cloud environments.
• VPN/Tunnel capability allowing connection to a company’s private network.
• HyperExecute CLI simplifies test configuration and execution.
• Built-in retry feature for re-running failed tests.
• Video playback option for reviewing tests.
• Capability to re-sign applications.
• Excellent documentation and robust support.
• Covers all or most industry-standard testing frameworks, to my knowledge.

Cons:

• The primary limitation I’ve noted with LambdaTest is the current absence of simulators for iOS XCUITest. However, this feature may be added to LambdaTest in the near future.

Step-by-Step guide to Integrate LambdaTest with XCUITest in GitHub Workflows

I have complete source code available for you for the demo project (link below)

1.) Sign up for LambdaTest. Begin by registering for LambdaTest, leveraging their free trial via LambdaTest Sign-Up.

2.) Grab LambdaTest’s Credentials (we will need it later steps):
Navigate to the LambdaTest Account Settings > Password & Security, and copy both the “Username” and “Access Key” for subsequent steps.

3.) Preparing the Build for Testing:
Our initial task is to prepare the Test App and UIRunner files for LambdaTest. To begin, let’s craft a new file named “prepareTestBuild-rdc.sh” in your root directory.

Here are the contents of prepareTestBuild-rdc.sh file to prepare build to run on Real Devices:

#!/bin/bash

# Define variables for common paths and settings
PROJECT_NAME="LambdaTestiOSDemo" # Change to your project name
SCHEME_NAME="LambdaTestiOSDemo" # Change to your schmeme name
CONFIGURATION="Debug"
SDK="iphoneos"
DERIVED_DATA_PATH="./customFolder-rdc"
BUILD_PRODUCTS_PATH="$DERIVED_DATA_PATH/Build/Products/$CONFIGURATION-$SDK"
APPS_TO_TEST_DIR="AppsToTest-rdc"

# Clean and build the project for testing
xcodebuild clean build-for-testing \
  CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO CODE_SIGNING_ALLOWED=NO \
  -project "$PROJECT_NAME.xcodeproj" \
  -scheme "$SCHEME_NAME" \
  -sdk "$SDK" \
  -configuration "$CONFIGURATION" \
  -derivedDataPath "$DERIVED_DATA_PATH"

# Navigate to the directory containing build products
cd "$BUILD_PRODUCTS_PATH"

# Package the main app into an IPA
mkdir Payload
cp -r "$PROJECT_NAME.app" Payload/
zip --symlinks -r "$PROJECT_NAME.ipa" Payload
rm -rf Payload

# Package the test runner app into an IPA
mkdir Payload
cp -r "${PROJECT_NAME}UITests-Runner.app" Payload/
zip --symlinks -r "${PROJECT_NAME}UITests-Runner.ipa" Payload
rm -rf Payload

# Navigate back to the project root directory
cd -

# Ensure the AppsToTest directory exists and is empty
if [ ! -d "$APPS_TO_TEST_DIR" ]; then
    mkdir -p "$APPS_TO_TEST_DIR"
else
    rm -rf "$APPS_TO_TEST_DIR/*"
fi

# Copy the IPAs to the AppsToTest directory
cp "$BUILD_PRODUCTS_PATH/$PROJECT_NAME.ipa" "$APPS_TO_TEST_DIR/"
cp "$BUILD_PRODUCTS_PATH/${PROJECT_NAME}UITests-Runner.ipa" "$APPS_TO_TEST_DIR/"

Execute the script “prepareTestBuild-rdc.sh” to generate the app and UITests-runner ipas within the designated “AppsToTest-rdc” folder in root directory. No need to worry for signing the app, as LambdaTest re-signs the app on their end anyways.

# For Real Devices
sh repareTestBuild-rdc.sh

It will generate “LambdaTestiOSDemo.ipa”, and “LambdaTestiOSDemoUITests-Runner.ipa” in AppsToTest-rdc folder in root directory.

4.) Download HyperExecute CLI

HyperExecute CLI is a command line interface that is used to run tests on LambdaTest HyperExecute. Download HyperExecute CLI and let’s add it to a new folder “LambdaTestFiles” in your root directory.

5.) Customizing Configurations

Add hyperexecute-xcui.yaml in “LambdaTestFiles” folder in your root directory.

Here are contents of the new hyperexecute-xcui.yaml file:

version: "0.2"
# number of concurrent devices
concurrency: 2
# Target OS for the tests
runson: ios
# Distributes tests among the concurrent number of tasks.
autosplit: true
# When retryOnFailure set to true, then maxRetries key indicates the number of retries that can be done if test failed
retryOnFailure: true
maxRetries: 2
framework:
  # Target test framework
  name: "ios/xcui"
  args:
    # File location of the application under test
    appPath: AppsToTest-rdc/LambdaTestiOSDemo.ipa
    # File location of the uitests-runner file
    testSuitePath: AppsToTest-rdc/LambdaTestiOSDemoUITests-Runner.ipa
    # Enables video playback of the tests
    video: true
     # Only run tests matching criteria    
    filters:
      attributes:
      - type: className
        values: ["ClassA"]
    # When set to `any`, it picks any available devices
    deviceSelectionStrategy: any
    # Select any iPhones running on iOS 16 major
    devices: ["iPhone.*-16"]

# Job label help you to identify the execution
jobLabel: ['HyperExecute', 'XCUI', 'Real Device']

You can customize this yaml file based on needs or we can create distinct config files for different test suites or target devices.

Refer LambdaTest HyperExecute YAML Support Page for more details.

6.) Executing Tests:
Now, it’s time to perform a test run to verify the setup we’ve completed thus far.

# Prepares the app and uitests-runner files
sh prepareTestBuild-rdc.sh

The steps mentioned above will prepare the necessary builds for LambdaTest. Let’s proceed to run the tests now using HyperExecute CLI.

export LT_USERNAME={YourLambdaTestUsername)
export LT_ACCESS_KEY={YourLambdaTestAccessKey)

# Allow permissions
chmod u+x ./LambdaTestFiles/hyperexecute

# Execute test suite
./LambdaTestFiles/hyperexecute --config ./LambdaTestFiles/hyperexecute-xcui.yaml

Replace username, and access key with your credentials (step 2)

Here comes the magic of HyperExecute CLI: it will upload the test app and UIRunner file in background, and kickstart the test execution on LambdaTest infrastructure. You can access the LambdaTest portal online by following the link provided by HyperExecute CLI in the terminal.

Below are screenshots of my results:

7.) Integration with GitHub CI/CD Pipeline:
Next, let’s integrate this with the GitHub CI/CD pipeline. I opted for GitHub Actions due to its cost-free nature. However, it’s worth noting that LambdaTest seamlessly integrates with a diverse array of CI/CD tools, including AWS CodePipeline, Azure DevOps, GitHub Actions, GitLab, CircleCI, Jenkins, BitBucket, Travis CI, TeamCity, GoCD, Bamboo CI, and Semaphore.

• Add LT_ACCESS_KEY and LT_USERNAME (refer step 2) in GitHub repository secret variables. How to add secrets to GitHub repo
• Add github_workflow.yml file in directory .github/workflows

Contents of my github_workflow.yml file are below:

name: LambdaTest Tests

on:
  pull_request:
    branches:
      - main

jobs:
  test:
    runs-on: macos-14

    steps:
      - uses: maxim-lobanov/setup-xcode@v1
        with:
          xcode-version: 15.2

      - name: Checkout code
        uses: actions/checkout@v4

      - name: Prepare test build
        run: |
          chmod +x prepareTestBuild-rdc.sh
          ./prepareTestBuild-rdc.sh

      - name: List generated files
        run: |
          ls -R

      - name: Run LambdaTest tests
        run: |
          ./runLambdaTest.sh
        env:
          LT_USERNAME: ${{secrets.LT_USERNAME}}
          LT_ACCESS_KEY: ${{secrets.LT_ACCESS_KEY}}

That’s it. Tests will now execute on each pull requests where the destination branch is main on LambdaTest

Screenshot of Github workflow action:

Don’t worry, I’ve got the complete source code available for you in a demo project:

GitHub - ssharma0312/LambdaTestXCUITestDemo

Sources:

XCUI Testing on LambdaTest | LambdaTest