Flip a Discerning Eye to Your Apps with SUITcase

Introduction

Have you ever ever puzzled the best way to add look verifications to your automated checks? The Devexperts staff is blissful to introduce its secret weapon – SUITCase. That is our inner software for snapshot testing on iOS, iPadOS, and tvOS apps, and we’ve shared it on GitHub.¹

On this article, we’ll let you know all about SUITCase and its options. It doesn’t matter whether or not you check iOS, iPadOS, or tvOS apps – this text is beneficial for all High quality Assurance engineers.

Downside

We use sturdy frameworks like XCTest and EarlGrey to check iOS apps.² Why would we need to improve their capabilities?

These two screenshots symbolize the ultimate step in a check case that verifies the cropping function in Apple’s Images app. If we press the SQUARE button on the underside toolbar, we get a cropped picture. 

The screenshots of Apple’s Images app earlier than and after we crop the picture.

Which assertions are essential to confirm this habits? Effectively, we undoubtedly need to test the ensuing picture. The issue is, there’s no option to test the photographs with XCTest and EarlGrey APIs. 

What can we confirm, then? The subsequent photographs symbolize the info accessible utilizing the built-in XCTest APIs. We name these photographs “T-800 Imaginative and prescient”. We will confirm the weather’ existence, sort, label, location, measurement, hittability, and in the event that they’re chosen.³ For instance, we will confirm that the RESET button on the highest has appeared and the crop handles have moved. However we can’t confirm what’s occurred to the picture.

Visualization of parts and their properties.

After we check our iPadOS and tvOS apps, we face the identical downside. For the sake of concision, we’ll solely focus on iOS app testing; however as we talked about, SUITCase helps iOS, iPadOS, and tvOS apps.

Snapshot testing might help you confirm the visible look of UI parts. It’scommonly used to test photographs and textual content edits. Snapshot Testing may also be useful with charts, complicated UI, and look adjustments.

iOS snapshot testing

Let’s briefly discover widespread screenshot testing options. 

The preferred software for snapshot testing on iOS apps is iOSSnapshotTestCase⁴, beforehand referred to as FBSnapshotTestCase. This framework was an open-source challenge by Fb; now Uber maintains it. You need to use iOSSnapshotTestCase inside your unit checks and confirm UIViews and CALayers. It’s potential to configure an acceptance threshold for every screenshot. Though iOSSnapshotTestCase is designed for unit checks, you should utilize this software inside your UI checks.⁵

 We’ll additionally discover some attention-grabbing options if we check out the forks of this software.⁶ A few of the options can add coloration distinction calculations to stabilize flaky checks. Some forks allow macOS and XCUIElement assist. Different helpful options embrace higher screenshot naming and picture scaling.

Snap.swift is one other software for snapshot testing.⁷ Along with iOSSnapshotTestCase, it was additionally designed for unit testing and verifies UIViews. It information reference screenshots tremendous quick and performs effectively with FastLane. This software additionally generates far more detailed logs than iOSSnapshotTestCase and provides the precise screenshot, reference, and distinction presentation as attachments.

Now let’s spotlight the attention-grabbing options of another framework. Unreal Engine has built-in Screenshot Comparability Instruments with computerized delays and retries if picture verification fails.⁸ It additionally makes use of completely different thresholds for native and world variations. And you may evaluate the sunshine depth whereas ignoring colours altogether.

We packed essentially the most attention-grabbing options into SUITCase – Snapshot UI Take a look at Case. This software is designed particularly for UI Testing in Xcode, retaining XCUIElement assist in thoughts. It supplies handy recording and generates informative logs. It saves disk area by scaling screenshots and has a number of comparability strategies that can assist you steadiness accuracy and stability. And we proceed enhancing SUITCase on GitHub.¹

Options

XCUIElement Assist

SUITCase helps XCUIElements and permits many configurable methods to enhance snapshot testing on iOS apps. Since this software is designed for UI testing, you’ll be able to run extra check circumstances in comparison with the beforehand talked about unit checks frameworks. With SUITCase it’s straightforward to confirm the looks of the entire gadget display screen and explicit parts whereas ignoring among the different parts. This function is nice for stability and implements the single-responsibility precept. 

From left to proper: the entire screenshot, the cropped cell, and the entire display screen with out texts.

Recording

SUITCase routinely saves reference, urged, and surprising photographs. If we allow file mode, SUITCase will save the reference photographs. If there is no such thing as a reference screenshot, SUITCase will save the collected screenshot in a separate folder in case you need to use it as a reference later. You’ll be able to set a threshold, which permits among the collected pixels to vary from the reference ones. If the distinction is decrease than the edge worth, verification is handed. If the distinction is increased than the edge you set, SUITCase will save the surprising screenshot and connect it to a check log, in addition to to the reference screenshot and graphical illustration of the distinction.

The circulate chart of potential eventualities.

Scaling

For accuracy, it’s finest to maintain high-resolution photographs and evaluate all pixels. The draw back is we’ve got to make use of vital space for storing. But when we scale down  screenshots, we will save space for storing. Twenty two completely different iPhones assist iOS 11 or increased. However due to Apple’s well-known small fragmentation, solely eight completely different rendered pixels resolutions exist.⁹ The rendered pixels decision numbers look chaotic till we divide them by the display screen scales to get the logical resolutions in factors. Right here we will see that iPhone 11 and iPhone 11 Professional Max (and their predecessors) use the identical logical decision, so we will use shared reference for a full-screen comparability. iOS gadgets additionally use solely 5 widths – 428, 414, 390, 375, and 320. So if we need to evaluate medium-size parts akin to navigation bars, toolbars, and tab bars, we solely have to maintain 5 completely different screenshots. If we evaluate fixed-size parts like buttons, we’ll solely use one reference screenshot.

iPhone fashions, their precise and logical resolutions.

5 comparability strategies

That can assist you steadiness accuracy and stability, SUITCase supplies 5 completely different comparability strategies. Strict technique compares unchanged authentic photographs and is the one technique that saves high-resolution references. Utilizing the With Tolerance technique, we will enable small coloration adjustments. You’ll be able to learn how we tuned this technique in a separate article.¹⁰ One other comparability technique referred to as Greyscale solely compares mild depth, ignoring the colours. DNA is impressed by Microsoft’s PhotoDNA algorithm designed to determine unlawful photographs.¹¹ This technique considerably downscales photographs, releasing up lots of disk area. The final technique Common Colour is the least correct, however it does come in useful in some circumstances.

Let’s see how these strategies work. The subsequent two photographs present Apple’s Shortcuts app localized in English and German. Other than the completely different texts, we’ve got 4 shortcuts with mismatched colours and icons. 

The screenshots of Apple’s Shortcuts app in English and German with completely different shortcuts.

If collected and reference photographs don’t match, SUITCase will connect each anticipated and precise colours for Common Colour technique. For the Strict, With Tolerance, Greyscale, and DNA strategies, SUITCase will connect a distinction visualization. 

The subsequent photographs present the visualizations. The Strict technique highlights all non-matching pixels, and the With Tolerance technique ignores the colour change within the first shortcut. The Greyscale technique highlights texts and icons ignoring the colours of each shortcuts, whereas DNA solely exhibits the most important adjustments. For the Greyscale and DNA strategies, intermediate picture processing steps are proven.

Distinction visualizations.

Outcomes

Snapshot testing with SUITCase improved our check circumstances as a result of now we’re capable of carry out test ups that  had been beforehand unavailable.

Due to options like XCUIElement Assist, Scaling, and the With Tolerance technique, we will confirm the cropped picture from the start of this text, reusing the reference picture whereas testing many alternative gadgets. By the best way, the image was shot at our workplace in Porto.

The cropped screenshot of the Images app.

Conclusion

SUITCase is a good way to enhance your iOS, iPadOS, and tvOS checks. It has options like XCUIElement assist, handy recording, scaling, and 5 comparability strategies to cowl numerous check circumstances. 

Even when you automate checks for different platforms, you’ll be able to implement a number of SUITCase options into your checks. Our software is open-source software program and obtainable on GitHub.¹ We might be blissful to obtain contributors’ points and pull-requests. SUITCase is another software you should utilize to enhance your check automation. Decide your favourite instruments, frameworks, and utilities and make apps higher than ever earlier than!

Which instruments do you utilize to automate screenshot testing? How do you steadiness accuracy and stability?

References

1. Devexperts staff. “devexperts/suitcase” GitHub Web site, GitHub, Inc., 2020, https://github.com/devexperts/suitcase[↩][↩][↩]
2. Roman Zakharov. “Thirty Seconds to Go: Efficiency Comparability of iOS UI Testing Frameworks” Devexperts Weblog, Devexperts LLC, 2020, https://devexperts.com/weblog/ios-ui-testing-frameworks-performance-comparison[↩]
3. Apple Builders Instruments staff. “XCUIElementAttributes | Apple Developer Documentation” Apple Developer Web site, Apple Inc., 2020, developer.apple.com/documentation/xctest/xcuielementattributes[↩]
4. Alan Zeino, Dustin Barker, Adam Ernst, Daniel Doubrovkine, Reid Essential and different contributors. “Snapshot View Unit Checks for iOS” GitHub Web site, GitHub, Inc., 2020, github.com/uber/ios-snapshot-test-case[↩]
5. Steve Moser. “Learn how to Set Up Xcode UI Testing for At present’s Extensions” Useful resource Library, Levvel, 2020, levvel.io/resource-library/xcode-ui-testing-today-extensions[↩]
6. GitHub customers. “Forks · uber/ios-snapshot-test-case” GitHub Web site, GitHub, Inc., 2020, github.com/uber/ios-snapshot-test-case/community/members[↩]
7. Oscar Antonio, Duran Grillo and different contributors. “Snapshot testing in a Snap” GitHub Web site, GitHub, Inc., 2020, github.com/skyweb07/Snap.swift[↩]
8. Unreal Engine staff. “Screenshot Comparability Software | Unreal Engine Documentation” Unreal Engine Documentation, Epic Video games, Inc., 2020, docs.unrealengine.com/en-US/Programming/Automation/ScreenShotComparison/index.html[↩]
9. PaintCode staff. “The Final Information To iPhone Resolutions” PaintCode Web site, PixelCut LLC, 2020, paintcodeapp.com/information/ultimate-guide-to-iphone-resolutions[↩]
10. Roman Zakharov. “Attending to the Backside of the RGB Pixels. Half 1: Colour Distinction” Devexperts Weblog, Devexperts LLC, 2020, https://devexperts.com/weblog/getting-to-the-bottom-of-the-rgb-pixels-part-1-color-difference[↩]
11. Microsoft staff. “PhotoDNA | Microsoft” Microsoft Web site, Microsoft Company, 2020, microsoft.com/en-us/photodna[↩]