AI Agents Passed the Tests. Real Devices Didn’t. with Nick Metcalfe

About This Episode:

Cloud device farms make it easy to get started with real device testing. But what happens when you need to scale? Costs climb fast, you wait in a queue for the devices you need, and you lose visibility into what’s actually happening when a test fails.

In this episode, Nick Metcalfe, Head of Technical Sales Engineering at Cambrionix, joins Joe to explain why more testing teams are bringing their device labs in house.

Nick shares how one team was quoted roughly a million dollars a month to outsource their real device testing, then cut that cost to around $300K by building their own lab.

And cost is only part of the story.

You’ll discover why an in house device lab can outperform a cloud lab:

• Lower cost at scale. Cloud pricing works for small test runs, but at enterprise scale an in house lab can save hundreds of thousands of dollars every month.
• No waiting in line. Your devices are yours, available whenever you need them, with no queue and faster turnaround.
• Full control and visibility. See exactly what’s happening at the hardware level, down to the millisecond a USB controller reboots, so you can tell a real bug from an infrastructure failure.
• Test the devices your users actually have. Keep legacy phones, budget devices, and hand me downs in your lab, just like Snap does to support its user base across 12,000+ devices and 700,000 tests a day.
• Test custom hardware. Point of sale systems, VR headsets, smart devices, and prototypes that a cloud vendor simply won’t have.
• Automate the hands on work. Reboot devices, recover from restore mode, and put 16 Apple devices into DFU mode at once, all through an API that plugs into your CI pipeline.
• Protect your device investment. Hold batteries at a set charge level to reduce wear and extend device life for years.

Nick also covers the honest tradeoffs, including the setup and resources an in house lab requires, and why a hybrid approach of emulators first and real devices second gives you the most confidence before release. Plus, his number one tip for building a lab: do it once, do it right.

If your team is scaling mobile test automation, fighting flaky tests, or questioning your cloud device farm bill, this episode is for you.

About This Episode’s Sponsor: Cambrionix

Have you ever chased a failing mobile test for an hour, only to find the real problem was a device that dropped off its USB hub? In large device labs, flaky infrastructure often looks like a code defect. That wastes your team’s time and erodes trust in your automation.

Cambrionix builds intelligent USB hubs and device management software for software QA teams that test on real devices at scale. Whether you’re validating an OS release, regression testing an app update, or flashing firmware across hundreds of devices, Cambrionix helps keep every device charged, connected, and ready for testing.

Here’s what Cambrionix brings to your device lab:

• Scale without the dropouts. Their Thunderbolt powered ThunderSync hubs get around standard USB endpoint limits, so you avoid the crashes and disconnects that happen under heavy load.Run more devices in parallel. Connect up to 16 devices to a single hub, then daisy chain up to six hubs to run as many as 96 devices at once.
• Protect your batteries. Intelligent charging delivers up to 60W per port while preventing overcharging, so devices don’t die mid test or wear out from constant charging.
• One setup for every platform. iOS and Android devices work side by side, and the hubs are compatible with Mac, Windows, and Linux systems.
• Manage your fleet remotely. Cambrionix Connect software lets you monitor, restore, and manage every hub and device from one interface, with role based access for your team.

Real world proof: Snap moved from an outsourced device lab costing about $1M a month to an in house lab supporting 12,000+ physical devices and 700,000 daily test executions, saving over $2M a year. Teams at Apple, Google, Uber, SpaceX, and Snap all rely on Cambrionix.

👉 See how Cambrionix can help your QA team test faster on real devices:  https://testgld.link/realdevice

Episode Summary

When a test fails, stop blaming your code first

Nick opens with a habit almost every engineer shares. When a test goes red, we assume our latest change broke something and dig deep into our own scripts. But in device labs, the failure often comes from the setup itself: how devices are connected, the cables, the power, or the host system. Asking “how are these devices connected, and where did this failure really come from?" can save hours of misdirected debugging.

AI is tripling your code. Can your lab keep up?

Teams Nick works with report AI is increasing their output by roughly three times, meaning three times the pull requests and updates that need validation. Nick also warns about AI “checking its own homework." A test that passes only proves the outcome matched what the AI expected, not that it’s correct. More code means you need more testing capacity, and scaling emulators alone means scaling your compute costs right along with it.

Emulators vs real devices: use both

Nick recommends a hybrid model. Emulators and simulators are a fast, cheap first pass for pull requests. But once code passes there, run it on real devices, because your users live in the real world. Things like battery swelling in a four or five year old phone, device temperature, and how long since a device was rebooted can all cause failures that users will blame on your app, and emulators won’t catch them.

Don’t cheap out on cables

Nick sees teams buy thousands of dollars of Mac Studios and flagship phones, then plug them in with whatever cable is lying on the floor. His recommendation: cables rated for up to 100W and USB 3.2, which typically cost between $5 and $10 each. Cambrionix hubs can even flag a faulty cable with a purple LED and report the exact error through the API.

Cloud device farm vs in house lab

Nick is fair about cloud providers like Sauce Labs and BrowserStack. They’re a solid option for small scale testing. But at scale the math changes. One customer was looking at about a million dollars a month to outsource their real device testing. Bringing it in house brought that down to around $300K, even after factoring in hardware engineers, space, and setup time. Beyond cost, in house means no queue, full control, and full visibility.

Telling infrastructure failures from code failures

Because Cambrionix sits at the hardware layer, it can capture low level USB communication that you’d never see plugging devices straight into a computer. If the OS reboots the USB tree or shuffles ports, the system records exactly when it happened so you can line it up against your test failure. Nick shares a story of a device maker whose new product stopped behaving correctly. Cambrionix tools revealed the device was negotiating power in a nonstandard way outside the USB spec, something the team never would have suspected.

Automating the painful stuff through an API

Most test teams don’t want a pretty UI, they want an API. Cambrionix offers a lightweight local API with Swagger docs that works with Python, JSON, and your CI pipeline. Standout use cases include putting up to 16 Apple devices into DFU mode at once (no more holding button combinations on each device), scripting devices out of restore mode after a crash, and ADB integration for Android with custom power profiles for devices like the Pixel 6 that behave differently.

Scaling without running out of power

Nick shares a California customer that simply couldn’t get more power into their building. They were running four or five phones per Mac, filling rooms with host systems. Switching to ThunderSync hubs took them to 16 devices per Mac instantly, freeing up power and space to grow their lab. In his own test, Nick wiped and updated 60 iPhones in about 13.4 minutes using hubs connected to a single Mac mini.

Why Snap tests on legacy devices

Snapchat is heavily used by kids on hand me down phones, so the app has to work on older devices that most cloud labs don’t keep. That’s why Snap built an in house lab with over 12,000 devices running 700,000 tests a day using Cambrionix hubs.

Nick’s number one tip

Set it up once and set it up right. Nick regularly sees teams buy the cheapest option available, then email him two years later with problems, having spent more money than planned because they had to build the setup twice.

---

Key Takeaways

• When a test fails, check your infrastructure before assuming your code is broken.
• AI generated code increases testing demand, and AI validating its own work doesn’t prove correctness.
• Use emulators for fast first pass checks and real devices for release confidence.
• Cloud device farms work well at small scale, but in house labs often win on cost, control, and turnaround at enterprise scale.
• Hardware level visibility helps separate flaky tests from real defects.
• Automating device resets, DFU mode, and recovery removes hours of manual lab work.
• Invest in quality cables and infrastructure up front to avoid rebuilding later.

---

Frequently Asked Questions

Is a cloud device farm or an in house device lab better for mobile testing?
Cloud device farms are a good fit for small scale testing and quick access to devices. At larger scale, an in house lab often costs significantly less and gives you full control, no queue times, hardware level visibility, and the ability to test legacy or custom devices cloud vendors don’t carry.

Are emulators good enough for mobile testing?
Emulators and simulators are a fast, low cost first pass. But they can’t fully reproduce real world conditions like battery degradation, heat, and power negotiation behavior. A hybrid approach that runs on emulators first and real devices before release gives the most confidence.

How do you tell if a failing test is caused by hardware or code?
You need visibility into the hardware layer. Tools that capture USB level events, like port resets or controller reboots, let you match infrastructure events to the exact moment a test failed.

What do you need to set up an in house device lab?
According to Nick, you can start with a dedicated host such as a Mac mini or small Windows or Linux PC, a multi port hub like the ThunderSync5-C16, quality cables rated for up to 100W and USB 3.2, and the Cambrionix API installed on the host.

Can you automate DFU mode on multiple Apple devices?
Yes. With Cambrionix hubs and a Connect Premium licence, you can put up to 16 supported Apple devices into DFU mode at once through the API instead of pressing button combinations on each one.

About Nicholas Metcalfe

Nicholas “Nick” Metcalfe, Head of Technical Sales Engineering, Cambrionix

Nick leads Cambrionix’s global Technical Sales Engineering team, bringing ten years of experience in technical sales engineering and hands-on customer support.

Working closely with SQA and engineering teams, Nick helps customers design, build and optimise real-device testing environments. His work spans lab architecture, hardware integration, automation scripts and APIs, as well as integrating device infrastructure into existing CI/CD pipelines. He also provides hands-on troubleshooting at the physical layer, helping teams identify when hardware, connectivity or device behaviour is affecting the reliability of their test results.

Supporting customers across a wide range of lab sizes and testing requirements gives Nick a broad understanding of how real-device testing environments behave at scale, the common challenges SQA teams encounter, and the practices that help create reliable, repeatable and maintainable test infrastructure.

Nick combines this technical experience with a strong ability to simplify complex information, delivering technical guidance, product demonstrations, training and on-site support to help customers successfully integrate Cambrionix solutions into their existing test environments.

Connect with Nicholas Metcalfe

• • Company: Cambrionix  www.us.cambrionix.com
  • Blog: www.us.cambrionix.com
  • LinkedIn: www.cambrionix
  • Twitter: www.cambrionix
  • YouTube: www.cambrionix-tech

---

Resources Mentioned

• Cambrionix SQA solutions: https://us.cambrionix.com/pages/sqa
• Webinar: How to Build a Real Device Testing Lab That Pays for Itself (featuring Snap): https://us.cambrionix.com/pages/media/how-to-build-a-real-device-testing-lab
• Blog: Why test failures caused by USB infrastructure are often mistaken for code defects: https://us.cambrionix.com/blogs/blog/why-usb-infrastructure-failures-are-mistaken-for-software-defects
• ThunderSync5-C16: https://us.cambrionix.com/products/thundersync5-c16

Rate and Review TestGuild

Thanks again for listening to the show. If it has helped you in any way, shape, or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.