AI Agents Passed the Tests. Real Devices Didn’t. with Nick Metcalfe

Nick leads Cambrionix’s global Technical Sales Engineering team, bringing ten years of experience in technical sales engineering and hands-on customer support.

Working closely with SQA and engineering teams, Nick helps customers design, build and optimise real-device testing environments. His work spans lab architecture, hardware integration, automation scripts and APIs, as well as integrating device infrastructure into existing CI/CD pipelines. He also provides hands-on troubleshooting at the physical layer, helping teams identify when hardware, connectivity or device behaviour is affecting the reliability of their test results.

Supporting customers across a wide range of lab sizes and testing requirements gives Nick a broad understanding of how real-device testing environments behave at scale, the common challenges SQA teams encounter, and the practices that help create reliable, repeatable and maintainable test infrastructure.

Nick combines this technical experience with a strong ability to simplify complex information, delivering technical guidance, product demonstrations, training and on-site support to help customers successfully integrate Cambrionix solutions into their existing test environments.

Cambrionix SQA solutions: https://us.cambrionix.com/pages/sqa

Webinar: How to Build a Real Device Testing Lab That Pays for Itself (featuring Snap): https://us.cambrionix.com/pages/media/how-to-build-a-real-device-testing-lab

Blog: Why test failures caused by USB infrastructure are often mistaken for code defects: https://us.cambrionix.com/blogs/blog/why-usb-infrastructure-failures-are-mistaken-for-software-defects

ThunderSync5-C16: https://us.cambrionix.com/products/thundersync5-c16

When a test fails, stop blaming your code first

Nick opens with a habit almost every engineer shares. When a test goes red, we assume our latest change broke something and dig deep into our own scripts. But in device labs, the failure often comes from the setup itself: how devices are connected, the cables, the power, or the host system. Asking “how are these devices connected, and where did this failure really come from?" can save hours of misdirected debugging.

AI is tripling your code. Can your lab keep up?

Teams Nick works with report AI is increasing their output by roughly three times, meaning three times the pull requests and updates that need validation. Nick also warns about AI “checking its own homework." A test that passes only proves the outcome matched what the AI expected, not that it's correct. More code means you need more testing capacity, and scaling emulators alone means scaling your compute costs right along with it.

Emulators vs real devices: use both

Nick recommends a hybrid model. Emulators and simulators are a fast, cheap first pass for pull requests. But once code passes there, run it on real devices, because your users live in the real world. Things like battery swelling in a four or five year old phone, device temperature, and how long since a device was rebooted can all cause failures that users will blame on your app, and emulators won't catch them.

Don't cheap out on cables

Nick sees teams buy thousands of dollars of Mac Studios and flagship phones, then plug them in with whatever cable is lying on the floor. His recommendation: cables rated for up to 100W and USB 3.2, which typically cost between $5 and $10 each. Cambrionix hubs can even flag a faulty cable with a purple LED and report the exact error through the API.

Cloud device farm vs in house lab

Nick is fair about cloud providers like Sauce Labs and BrowserStack. They're a solid option for small scale testing. But at scale the math changes. One customer was looking at about a million dollars a month to outsource their real device testing. Bringing it in house brought that down to around $300K, even after factoring in hardware engineers, space, and setup time. Beyond cost, in house means no queue, full control, and full visibility.

Telling infrastructure failures from code failures

Because Cambrionix sits at the hardware layer, it can capture low level USB communication that you'd never see plugging devices straight into a computer. If the OS reboots the USB tree or shuffles ports, the system records exactly when it happened so you can line it up against your test failure. Nick shares a story of a device maker whose new product stopped behaving correctly. Cambrionix tools revealed the device was negotiating power in a nonstandard way outside the USB spec, something the team never would have suspected.

Automating the painful stuff through an API

Most test teams don't want a pretty UI, they want an API. Cambrionix offers a lightweight local API with Swagger docs that works with Python, JSON, and your CI pipeline. Standout use cases include putting up to 16 Apple devices into DFU mode at once (no more holding button combinations on each device), scripting devices out of restore mode after a crash, and ADB integration for Android with custom power profiles for devices like the Pixel 6 that behave differently.

Scaling without running out of power

Nick shares a California customer that simply couldn't get more power into their building. They were running four or five phones per Mac, filling rooms with host systems. Switching to ThunderSync hubs took them to 16 devices per Mac instantly, freeing up power and space to grow their lab. In his own test, Nick wiped and updated 60 iPhones in about 13.4 minutes using hubs connected to a single Mac mini.

Why Snap tests on legacy devices

Snapchat is heavily used by kids on hand me down phones, so the app has to work on older devices that most cloud labs don't keep. That's why Snap built an in house lab with over 12,000 devices running 700,000 tests a day using Cambrionix hubs.

Nick's number one tip

Set it up once and set it up right. Nick regularly sees teams buy the cheapest option available, then email him two years later with problems, having spent more money than planned because they had to build the setup twice.

---

Key Takeaways

• When a test fails, check your infrastructure before assuming your code is broken.
• AI generated code increases testing demand, and AI validating its own work doesn't prove correctness.
• Use emulators for fast first pass checks and real devices for release confidence.
• Cloud device farms work well at small scale, but in house labs often win on cost, control, and turnaround at enterprise scale.
• Hardware level visibility helps separate flaky tests from real defects.
• Automating device resets, DFU mode, and recovery removes hours of manual lab work.
• Invest in quality cables and infrastructure up front to avoid rebuilding later.

---

Frequently Asked Questions

Is a cloud device farm or an in house device lab better for mobile testing?
Cloud device farms are a good fit for small scale testing and quick access to devices. At larger scale, an in house lab often costs significantly less and gives you full control, no queue times, hardware level visibility, and the ability to test legacy or custom devices cloud vendors don't carry.

Are emulators good enough for mobile testing?
Emulators and simulators are a fast, low cost first pass. But they can't fully reproduce real world conditions like battery degradation, heat, and power negotiation behavior. A hybrid approach that runs on emulators first and real devices before release gives the most confidence.

How do you tell if a failing test is caused by hardware or code?
You need visibility into the hardware layer. Tools that capture USB level events, like port resets or controller reboots, let you match infrastructure events to the exact moment a test failed.

What do you need to set up an in house device lab?
According to Nick, you can start with a dedicated host such as a Mac mini or small Windows or Linux PC, a multi port hub like the ThunderSync5-C16, quality cables rated for up to 100W and USB 3.2, and the Cambrionix API installed on the host.

Can you automate DFU mode on multiple Apple devices?
Yes. With Cambrionix hubs and a Connect Premium licence, you can put up to 16 supported Apple devices into DFU mode at once through the API instead of pressing button combinations on each one.