You've got three conversion rate optimization (CRO) agency proposals open in three tabs, and they all promise the same thing.
Each one says research-led testing, each one shows off a logo wall, and their monthly fees aren't even in the same neighborhood. Nothing in those decks tells you which agency will still be finding wins in month nine.
So before you compare them, pin down what you're actually buying.
What is an ecommerce CRO agency?
An ecommerce CRO agency is an outside specialist team that researches why shoppers don't buy, tests fixes on live traffic, and ships the winners to grow revenue per visitor.
It isn't a design shop that hands over prettier pages, and it isn't a media buyer that sends more traffic. When it's done well, every test teaches the team something about your buyers, which makes the next test sharper. And if you work with a brand-aligned conversion optimization agency, you'll get designers who design beautiful tests and landing pages that elevate your website too.
Now, an ecommerce-specific agency is important for DTC brands: optimizing product pages, bundles, subscriptions, carts, checkouts and landing pages is not the same as optimizing lead-gen forms and SaaS trials.
How do you choose an ecommerce CRO agency?
To choose an ecommerce CRO agency when every deck sounds the same, run every candidate through the same five checks. At SplitBase, we call them the SPLIT Test:
- S: Source of ideas. Test ideas come from research on your customers, not from a best-practice list.
- P: Process. Tests target the weakest parts of your funnel and are planned around your business goals, not launched ad hoc for the sake of testing.
- L: Losses. They plan for losing tests and can show you what one taught them.
- I: Implementation and impact. They ship the winners into your store and report revenue, not just conversion rate.
- T: Team. Specialists in research, data, design and testing run your program, matched to what your brand needs, not one generalist carrying the whole strategy in their head.
Full disclosure before you read on: SplitBase is a CRO agency. These checks work on any agency, ours included, so near the end of this guide we run SplitBase through them and you can judge the answers for yourself.
In this guide:
What should you look for when hiring an ecommerce CRO agency?
When hiring an ecommerce CRO agency, look for proof that it diagnoses before it prescribes. In other words, it studies your data and listens to your customers before it proposes a single test.
The reason is something anyone who has seen a doctor already knows. If the prescription came out before anyone asked where it hurt, you'd walk out, and you'd be right to. Yet plenty of CRO agencies work exactly that way: the test roadmap is already sitting in the proposal, written before anyone has read a single customer survey or watched one session recording.
A test plan written before the research is a guess with a nice cover page. A good agency also needs a process tied to your goals, a plan for losing tests, a way to ship winners and the right team. The SPLIT Test helps you evaluate the CRO agencies you're talking to on each of these, and find out whether they're the right fit for your needs.
| Check | What to ask | A strong answer sounds like | A red flag sounds like |
|---|
| S: Source of ideas | Where will our first ten test ideas come from? | Analytics and funnel data, on-site polls, post-purchase surveys, review mining and recordings, summarized in a findings document before the roadmap | "We'll start with proven winners from other clients." Can't name a single research method |
| P: Process | How do you find the weakest parts of our funnel, and how do tests tie to our business goals? | A view of where the funnel is weak and strong, from landing page to checkout, and a roadmap worked backward from goals like subscription take rate or AOV | Tests picked from whatever came up that week, or a number of tests per month sold as the deliverable |
| L: Losses | What happens when a test loses? | A written hypothesis, primary metric and runtime set before launch, plus a real example of a loss that changed direction | "Our tests almost always win." Tests stopped the day they look positive |
| I: Implementation and impact | Who ships the winners, and how do you report results? | Their developers build tests and ship winners into your theme. Reports show revenue per visitor, AOV and profit, with brand review before launch | Winners handed to your developers as a slide deck. Reports count tests run and clicks, not dollars |
| T: Team | Who works on our account, and what does each person specialize in? | Specialists in research, data, design, copy, development and QA, shaped around your goals, plus a plain answer on when they'd turn a brand down | One generalist carrying the strategy in their head, and "We work with everyone" |
How does the SPLIT Test work?
The SPLIT Test is a five-point checklist SplitBase uses to evaluate ecommerce CRO agencies: Source of ideas, Process, Losses, Implementation and impact, and Team. Each check targets a way CRO engagements commonly go wrong, so an agency that passes all five has removed the usual reasons a program stalls.
Let's take them one at a time, starting with where the ideas come from.
S: Where do the test ideas come from?
Test ideas should come out of research on your customers, and that research should be in your hands before the first test ships. If an agency can't show you where an idea came from, then they're probably just guessing and playing the lottery with your website.
Look for two kinds of research
Research means two kinds of evidence. Behavior data shows where shoppers drop off: analytics, funnel and segment analysis, heatmaps and session recordings.
Voice data shows why: on-site polls, post-purchase surveys, customer interviews and review mining helps you understand what goes on inside the head of your customers.
You want both, because the where without the why leaves you guessing at the fix. SplitBase's guide to conversion research methods for ecommerce walks through how each one turns into a test idea.
Example: the research behind Dr. Squatch's 54% lift
Dr. Squatch shows what that looks like in practice. When the brand came to SplitBase in 2019, on-site A/B testing at the brand was virtually non-existent. Then the research turned up something hiding in plain sight: analytics and customer surveys both showed that many shoppers bought more than one bar of soap, yet the product pages had no quantity field. Shoppers wanted more than one bar, and the page gave them no easy way to say so.
The test that followed made two bars the default, and it lifted revenue per user by up to 54% for soap purchasers, as explained in the Dr. Squatch case study. Copying that test won't get your store the same number, though, because this came directly from quantitative and qualitative patterns unique to Dr. Squatch.
Why copying another brand's winner rarely works
What works for one brand doesn't always work for another.
For example, for years we've worked with the L'Oréal group of companies, including Kiehl's, Armani, NYX, Lancôme, and many more. While all brands operated in the beauty and cosmetics vertical, it wasn't uncommon for one test to lift revenue for one brand, yet to do the opposite for another.
Research shouldn't slow down the first test
With all that said, research shouldn't become an excuse for a slow start. A first test can go live in week two, with deeper research running alongside it. After all, testing done well is research too.
P: Is there a real testing process?
A CRO strategy comes from a process, not from a pile of test ideas. So ask how the agency finds the weakest parts of your funnel, and how it plans tests against your business goals.
Why ad hoc testing adds up to very little
Plenty of agencies run on reaction. Someone spots an idea, a test goes live, and what runs next depends on whatever came up that week. Each test can look busy on a report, yet they add up to very little, because nothing ties them to a goal you actually care about.
A real process starts with your funnel's weak spots
A good process starts by finding out which parts of your funnel are weakest and which are already strong. Ask any team to raise conversion rate and they'll come up with a hundred test ideas, but plenty of those would poke at pages that already work, or at problems that don't really exist. Without a way to see what's working and what isn't, from landing page to product page to cart and checkout, it's hard to build a test plan that actually moves your goals.
Then it works backward from your business goals
From there, a real process puts the weak spots next to the goal, such as raising subscription take rate or AOV before Q4, and works backward into a roadmap. At SplitBase, testing is planned in rolling three-month sprints built around each client's goals and updated weekly as new data comes in, as described in the FAQ. The plan bends when a test surprises you, but it doesn't get rewritten every Monday. If an agency can't tell you which weak spot a test fixes and which goal it serves, it's testing for the sake of testing.
L: What happens when a test loses?
Most tests don't win, so how an agency handles losses tells you more than its best case study. Anyone can show you a winner. The real question is what happens the rest of the time.
Even experts can't predict which ideas will win
And "the rest of the time" is most of the time. SplitBase's own guide to what a CRO program should include puts it plainly: roughly two-thirds of well-designed tests don't win. And you can't dodge the losses by only testing the "good" ideas, because nobody, experts included, can reliably tell in advance which ideas will win. In their 2017 Harvard Business Review article, Ron Kohavi and Stefan Thomke tell how a Bing employee suggested a small change to how ad headlines were displayed. Managers ranked it low priority among hundreds of ideas, and it sat untouched for more than six months. When an engineer finally tested it, revenue rose 12%, more than $100 million a year in the US alone. If Bing's own experts misjudged an idea that badly, an agency's gut feel won't do better. That's why a good agency tests, expects most ideas to lose, and learns from every result.
Rules set before launch protect you from false winners
With losses guaranteed, what protects you from false winners is discipline. Before a test launches, a serious agency writes down the hypothesis, the primary metric, the guardrail metrics and how long the test will run. That discipline is covered at length in Trustworthy Online Controlled Experiments (Kohavi, Tang and Xu, Cambridge University Press, 2020).
How long a test runs should come from a sample size calculation, like the one in Evan Miller's sample size calculator, not from the day the graph happens to look good.
Agencies that use Bayesian analysis rely less on a fixed sample size, but they should still set their stopping rules before launch, such as a minimum number of full weeks and the probability they need before calling a winner. And when a result comes back flat, the agency should be able to explain how it reads an inconclusive A/B test before deciding what to do next.
The one question to ask about losses
If you only ask one question from this section, make it this one: "Show us one losing test and what it changed." An agency that can't produce one either isn't testing much or isn't telling you everything.
Be wary of results promised in the first month
The same math is why you should be wary of an agency that promises results within a single month. It can happen, but the odds are against it.
A Wingify study (the company behind VWO) found that only about 1 in 7 A/B tests, roughly 14%, produce a winner. Experienced teams do better, and SplitBase's guide to A/B testing mistakes puts most mature programs between 20% and 35%, but even that doesn't guarantee a win from one month of tests.
An agency that promises one anyway is either short on experience or throwing everything at the wall, and either way it's playing the lottery with your business.
I: Do they ship winners and measure revenue?
A winning test only makes you money once it's live on your site, so ask who builds tests, who QAs them and who ships the winners. A win that lives in a slide deck doesn't sell a single extra order.
The agency should be able to build and ship winners
The strongest agencies build every variation themselves, QA it across devices and can ship the winner into your theme once the result is in. Not every brand wants an agency coding in its live theme, and that's fine, but the option should be there when you want it. Weaker ones hand your team a deck and a Figma file. And then what? The win waits in a backlog behind three product launches.
Judge tests on revenue, not conversion rate alone
Impact also means keeping the right scoreboard. Conversion rate on its own can lie to you: a discount can lift it while shrinking average order value and margin. So ask which primary metric each test is judged on and which guardrails they watch, such as revenue per visitor, AOV, subscription take rate and repeat purchase rate.
Testing shouldn't water down your brand
Your brand belongs on that scoreboard too. If you've ever worried that testing means watering down the look you spent years building, here's the good news: it doesn't have to.
A celebrity-led beauty brand with strict design guidelines saw a 406% ROI on its experimentation program in its first three months with SplitBase, plus 33% more email signups, without loosening those guidelines (celebrity beauty brand case study).
SplitBase's post on how a CRO agency should strengthen your brand covers the guardrails in more depth.
T: Who's on the team, and is it the right one for you?
The team assigned to your account decides the output you get, so ask who's on it and what each person specializes in. Is your strategy coming out of one person's head, or out of a team of specialists who each know their part?
A program needs five or six specialists, not one generalist
A CRO program needs five or six distinct skills: a strategist, a researcher or data analyst, a conversion designer, a copywriter, a test developer and someone doing QA.
When CRO is sold as one line item next to SEO, email and paid media, those roles tend to collapse into one generalist and a freelancer or two.
It's like a clinic where the same doctor runs the lab work, reads the scans and performs the surgery. So ask what each person actually does all day. Is the analyst a data person, does the designer work on CRO and landing pages every day, or is everyone a generalist wearing several hats?
The team should be shaped around your goals
The right team also depends on what you need. A brand whose biggest gap is paid-traffic landing pages needs a different mix of people from one fixing a subscription funnel, so ask how the agency would shape the team around your goals.
Ask what they need from your side
Then flip the question around and ask what they need from your side. A program run by the agency, not merely supported by it, should need one internal owner who can approve variants and join a weekly or bi-weekly call. If an agency needs your developers for every test, you've hired advice, not a team.
Ask when they would turn a brand down
Fit cuts both ways, and one question reveals a lot: ask when the agency would turn a brand down. A specific answer, such as a minimum number of monthly orders, is a good sign. "We work with everyone" means they'll take your retainer whether or not testing can pay back at your traffic.
Curious where your own store stands before you talk to any agency? The free CRO assessment scores your site and points to what to fix first.
What questions should you ask a CRO agency before you sign?
Here are the questions we'd recommend asking the agencies on your shortlist. You don't need to cover every one, but the answers will tell you a lot about how each agency really works.
The questions are grouped by SPLIT check, so you can see at a glance which part of the test each answer covers.
- (S) What research will you run before the first test, and what will we see from it?
- (S) Walk us through one past test, from the research finding to the hypothesis to the result.
- (S) How do you rank test ideas once you have them?
- (P) How do you find the weakest and strongest parts of our funnel, and how does that turn into a plan tied to our business goals?
- (P) How does a test get from idea to launch, and who decides what runs next?
- (P) What happens to the plan when a test result surprises you?
- (L) What share of your tests win, and what happens to the ones that lose?
- (L) Show us a losing test that changed your direction.
- (L) How do you set sample size and test length before launch, and when do you stop a test early?
- (I) Do you build winning variations into our theme, and how long does that take?
- (I) Which metric judges each test, and which guardrail metrics do you watch?
- (I) How do you keep variants on-brand, and who on our side approves them?
- (T) Who works on our account and what does each person specialize in?
- (T) How would you shape the team around our goals, and who builds the tests: your own developers or contractors?
- (T) When would you tell a brand like ours not to hire you?
A good answer is full of specifics: named methods, named people, real numbers and a real losing test. Generalities like "we're data-driven" and "we follow best practices" are answers to a different question. If you have to squint to find a fact in an answer, there probably isn't one.
What are the red flags when hiring a CRO agency?
The biggest red flag is a test plan that exists before any research on your customers. Most of the other warning signs follow from it: an agency that skips the diagnosis tends to sell volume, promise lifts and report activity, because it doesn't have much else to show you.
Some of these show up in the sales process. Others only appear once you're a few weeks in.
Red flags before you sign
Before you sign, walk away from an agency that:
- Sends a test roadmap with the proposal, before it has seen your data
- Guarantees a specific lift, or prices on a percentage of lift
- Sells a number of tests per month as the main deliverable, with no plan tied to your business goals
- Reports only conversion rate, never revenue per visitor, AOV or profit
- Can't tell you which specialists work on your account, or has one generalist covering every role
- Can't show a single losing test
- Leaves building the winners to your developers
- Promises results within the first month
- Claims it has never lost a test, or reports a win rate above 50%. Every industry stat says otherwise, so they're either testing obvious things, taking too little risk, "testing" bugs that should simply be fixed, or misreading the statistics and calling false positives
Red flags after you sign
After you sign, act fast if you see:
- No research findings in the first two to four weeks
- Tests launched without a written hypothesis, primary metric and planned runtime
- Reports that lead with activity, such as tests run and hours logged
- Winners called early or on thin data, with no check on how they perform once they're live
- Discount, urgency or positioning changes made without your sign-off
A single flag doesn't mean you should fire them. One of these deserves a hard question; two or more, and you're probably paying for a test-running shop dressed up as a CRO partner.
Should you hire in-house, an agency, or a freelancer for CRO?
For most DTC brands doing $10M to $100M a year, a CRO agency plus one internal owner gets further than a single in-house CRO hire. The reason comes down to simple math: a program needs five or six skills, and one hire covers one of them.
That doesn't make an agency right for everyone, though. The table compares five ways to staff CRO, with what each option can and can't do, so you can match the model to your team instead of the other way around. For a closer look at three of these options, see our comparison of CRO consultant vs agency vs in-house.
| Option | What it can do | What it can't do | Best fit when |
|---|
| In-house CRO team | Keeps every learning inside the company and moves at the speed of your roadmap | Can't be hired quickly. Costs several salaries plus tools, and one resignation can stall the program | Testing is central to your strategy and you can recruit and keep four or five specialists |
| One in-house CRO specialist | Owns the roadmap, runs the tools and coordinates simple tests | Can't research, design, build, QA and analyze alone. Tests queue behind your developers | You already have design and development capacity to lend them |
| CRO agency | Brings a full team from week one, patterns from many stores, and builds and ships winners | Can't know your brand on day one. Can't fix product, pricing or traffic problems by itself. Learning leaves with them unless the contract says you own it | You're at $10M+ a year and 3,000+ orders a month, with a lean team that needs the program run for it |
| Freelancer or CRO consultant | Audits, advises and sets up a testing process at a lower cost | Rarely sustains a steady test cadence alone. Usually needs your designers and developers to execute | You need a defined project or a second opinion, and your team can act on the advice |
| Testing tool alone | Runs the tests your team designs and builds | Can't tell you what to test or why. Won't catch a broken test or a flawed read | An internal team already does the research and analysis |
Why a testing tool alone isn't enough
One row deserves a closer look, because it's the most tempting shortcut. Buying a testing tool and expecting conversion to rise is like buying a gym membership and expecting to get fit. The tool is the equipment. Someone still has to write the program and show up.
AI makes that shortcut even more tempting. Anyone can now build a landing page or set up an A/B test in an afternoon, with no design or coding skills, but launching was never the hard part.
Visual editors have come with testing tools from the start, and no-code page builders have been around for years, yet brands using them still launch tests and pages every week without seeing results. The bottleneck isn't the deliverables. Without the right strategy and process, you can run a hundred tests and never move the needle, or launch a dozen landing pages and never grow the business.
The hybrid model: one internal owner plus an agency
Many brands end up with a hybrid: one internal owner who knows the business, plus an agency that runs research, testing and builds. Transparent Labs is a good example. The premium supplement brand had run internal A/B tests before, but testing volume was low and the process wasn't formalized, according to the Transparent Labs case study. The team knew it was leaving wins on the table. What it didn't have was the bandwidth or the system to find them.
So it brought in SplitBase to run a formal testing program. The payoff went beyond the tests themselves: the internal team came out of the engagement with sharper testing instincts of its own. A good agency should leave your team sharper, not more dependent.
How much does an ecommerce CRO agency cost?
Most ecommerce CRO agencies charge a monthly retainer for ongoing programs and a fixed fee for one-off audits or redesigns, and the price moves with your traffic and the scope of work. That's the short answer. The longer one is that the pricing model tells you as much as the number on the invoice.
That's because each model rewards the agency for something different, and you want it rewarded for the same thing you are. Below is what each common model buys you, and what to watch for.
| Pricing model | What you get | What to watch for | Fits when |
|---|
| Monthly retainer | An ongoing program: research, tests, builds, analysis and shipped winners | The minimum term, and whether design, development and copy are included or billed on top | You have steady traffic and want results that compound |
| Fixed-fee audit or research project | Findings and a ranked plan for the next 60 to 90 days | An "audit" that's a heuristic checklist with no customer research, and no plan for who executes it | You want to test the relationship, or you aren't sure a full program will pay back yet |
| Project fee for a redesign or landing pages | A defined build, ideally based on research and test results | A redesign built on opinion, with no measurement plan after launch | Your theme or design system is what's blocking testing |
| Hourly or per-test billing | Specific tasks, common with freelancers | You pay for activity, research is often excluded, and you manage the work | The task is small and well defined |
| Performance-based or % of lift | A fee tied to a reported lift | Lift is a range, not an exact number, and promotions, seasonality and stock levels move it. Incentives favor short-term wins and invoice disputes | Rarely. Treat it as a warning sign |
What drives the price
Inside any model, three things drive the price. Traffic and order volume come first: more orders mean tests reach a result faster, so an agency can run more of them each month. Scope comes next, meaning whether design, development, copy and implementation are included. Research depth and the number of stores, markets or languages make up the rest.
How SplitBase prices its programs
SplitBase, for example, prices its programs as a monthly retainer set by your traffic and scope, and its ecommerce CRO services page lays out what that retainer includes. We don't offer performance-based pricing, and we only take on brands where we see a path to at least a 3x ROI in 3 months. Other agencies price differently, so ask every candidate what their fee assumes about test volume and scope.
Compare the fee to the right alternative
Whatever the model, get the price, the minimum term and what you keep if you part ways in writing before you sign.
Then compare the fee to the right alternative. It's tempting to weigh an agency's fee against doing nothing at all. The real comparison isn't agency versus nothing. It's agency versus the salaries of the five or six people a program needs, plus their tools and the months it takes to hire them.
A free CRO diagnostic call walks through your numbers and shows what a program would cost and return for your store.
When is hiring a CRO agency the wrong move?
Hiring a CRO agency is the wrong move when your traffic can't support testing, or when the real problem sits outside the website. No amount of testing rescues a product people don't want at its price, or a store without enough orders to read a result.
Then comes the uncomfortable part, and a good agency will say it to you before you sign:
- Too few orders. SplitBase sets its own line at $10M a year and 3,000 orders a month. Below that, tests take too long to reach reliable results, and research, landing pages and high-confidence fixes usually pay back more than an A/B testing program.
- No product-market fit yet. If shoppers don't want the product at its price, no page change fixes it, and an agency will have a much harder time getting results. Once you have clear Product-Market Fit and qualified traffic, a good agency can amplify what's already working.
- No one can approve decisions. When stakeholders can't agree on goals or who signs off on a variant, test velocity collapses whoever you hire.
- You want results overnight. Tests need weeks to reach a reliable result, and programs need months to compound.
Even the right agency has limits
Even the right agency has limits. It starts out knowing far less about your brand than you do, so the first few weeks go into learning your customers, your margins and what's been tried before.
It also can't control what happens off the site: if your team discounts its way through Q4, no test will protect your margin. And unless your contract says the research and test archive belong to you, much of what the agency learns leaves with it when the engagement ends.
What should the first 90 days with a CRO agency look like?
By day 90, a good CRO agency should have shipped its first winners and reported the results in dollars. Before that, you should see a first test live by the end of week two, a findings document you can actually read, and new tests launching every month, each with a written report.
Think of those 90 days as checkpoints, not one big reveal at the end:
- Weeks 1 to 2: tracking and analytics audit, research underway, and the first test live by the end of week two
- Weeks 2 to 6: a findings document you can read, and a test roadmap ranked by expected revenue impact
- Months 1 to 3: new tests launching every month, a written report on every result, and winners built into your theme
- Day 90: a clear view of whether the program is paying for itself, with revenue per visitor and AOV next to conversion rate
If you reach day 90 and still can't tell whether the program pays for itself, that deserves a direct conversation with the agency. SplitBase's 30, 60, 90 and 180-day CRO checklist goes deeper on what to hold an agency to at each stage.
How does SplitBase score on the SPLIT Test?
SplitBase is a CRO agency for DTC brands doing $10M+ a year, launched in 2015, so it's one of the agencies you could put through this checklist. Fair's fair: here are its answers to each check.
- Source of ideas: every test idea comes out of the 3Ps method. Patterns studies how visitors behave and what customers say, Perception filters ideas through how the brand is seen next to competitors, and Proof is the A/B test.
- Process: testing is planned in rolling three-month sprints built around each client's business goals, and the plan is updated weekly as new data comes in.
- Losses: winners and losers are defined before launch with a minimum meaningful lift, and every test gets a written report, win or lose.
- Implementation and impact: tests run in the brand's own testing tool, such as Intelligems, VWO or Convert. SplitBase's developers build the winners into the theme, and reporting comes through a live dashboard and weekly or bi-weekly calls.
- Team: each account gets a team of specialists rather than one strategist: strategists, conversion designers, test developers, data analysts, copywriters and QA. Brands under $10M a year or 3,000 orders a month get a plain no.
What a client says after comparing agencies
Of course, any agency can write flattering answers about itself. What counts more is what a client says after comparing the options.
Transparent Labs' CMO, Joe Cornfield, described why the brand picked SplitBase after talking to other agencies: "We talked to several agencies, and decided to work with SplitBase based on their highly process-driven approach to strategy, and analytical rigor when it came to testing and measurement." After a year of testing, and a rebuild based on what those tests proved, subscription take rate grew from 22% to 38%+ and average sitewide conversion rate improved by over 12%.
Those results come from one brand, and from a CRO program and a redesign together, as reported in the Transparent Labs case study, so treat them as an example rather than a forecast.
Who shouldn't hire SplitBase
Just as important is who shouldn't hire us. SplitBase isn't the right hire if you're under $10M a year or 3,000 orders a month, run a B2B or lead-gen site, or want ad management in the same retainer.
For brands that fit, it's an alternative to building a full in-house CRO team and a complement to an internal ecommerce team or a media agency, not a replacement for either. If that's you, request a CRO proposal and ask us the same questions.
We'd expect nothing less.
Frequently asked questions
How do I choose a CRO agency?
Choose a CRO agency by checking five things before price: where test ideas come from, how tests are planned against your goals, how losing tests are handled, whether winners get built and measured in revenue, and which specialists do the work. SplitBase calls this the SPLIT Test. Then ask each agency the same questions and compare the answers, not the decks.
What's the most important thing to look for in an ecommerce CRO agency?
The most important thing to look for is a research phase before any test plan. An agency that proposes tests before studying your data and your customers is running a template. Ask to see an anonymized findings document and how it turned into hypotheses.
How much does an ecommerce CRO agency cost?
Most ecommerce CRO agencies charge a monthly retainer for ongoing programs, plus fixed fees for audits and redesigns. The price depends mostly on your traffic and the scope of work. A retainer usually rises with order volume, because more orders let an agency run more tests each month, and with how much design and development it includes. Steer clear of fees tied to a percentage of lift, because lift can't be measured precisely enough to bill on.
When does a CRO freelancer or consultant make more sense than an agency?
A CRO freelancer or consultant makes more sense for a defined project, an audit or a second opinion, especially when your own team can build and ship the changes. Freelancers rarely sustain a steady test cadence alone, because research, design, development and analysis are separate skills.
How many orders do I need before hiring a CRO agency?
SplitBase uses about 3,000 orders a month, and $10M+ in annual revenue, as the threshold for a full A/B testing program. Below that, tests take too long to reach reliable results, and research, landing pages and high-confidence fixes usually pay back more. A good agency tells you this before you sign.
How long does it take a CRO agency to deliver results?
A well-run CRO agency launches its first test in about two weeks and shows early wins and clear learnings within one to two months. SplitBase aims for its programs to pay for themselves within roughly the first 90 days, depending on traffic, revenue and opportunity size. Compounding results take six to twelve months of steady testing.
What's the difference between a CRO agency and an A/B testing agency?
A CRO agency owns the whole decision process, from customer research to shipping winners, and A/B testing is the validation step inside it. An A/B testing agency focuses on setting up and running tests. If you're comparing testing-focused vendors, SplitBase's guide to choosing an A/B testing agency covers the testing criteria in more depth.
Should a CRO agency guarantee results?
No, a CRO agency shouldn't guarantee a specific lift. Results depend on traffic, seasonality, promotions, stock levels and your customers, so a guarantee is a sales tactic rather than a forecast. What an agency can commit to is a process, plus a clear case for why the program should pay for itself at your traffic.
What percentage of A/B tests should win?
Expect most A/B tests not to win. SplitBase estimates that roughly two-thirds of well-designed tests don't, which is why what an agency learns from losses matters as much as its winners. An agency that reports nearly every test as a winner is probably stopping tests early or judging them on the wrong metric.
Who should own the research and test data when you work with a CRO agency?
Your brand should own the research, the test archive with results, the design files and the code for shipped winners. Write it into the contract, keep platforms like Shopify and Google Analytics in your name with access granted to the agency, and ask for a documented handover in the final month.
Can a CRO agency work alongside our in-house team or media agency?
Yes, a CRO agency can work alongside an internal team or a media agency when roles are set at the start. The CRO agency owns the on-site experience and the tests, and the media agency owns the traffic. Landing pages for paid traffic are where the two meet, so agree early on who builds and tests them.
Do we need a CRO audit before hiring an agency?
Not always. A one-time audit is a sensible first step when you want to test the relationship or aren't sure a full program will pay back, and it should map the next 60 to 90 days of work. Brands with steady traffic and clear goals often go straight into a retainer, with research built into the first weeks.
Related reading
