Every AI trading bot pitch shows a beautiful equity curve climbing from bottom-left to top-right. Most of them are cherry-picked, optimised against the past, or quietly broken in ways that aren’t obvious until you’re on the wrong side of a drawdown. This guide gives Australian traders a working framework to separate genuinely profitable AI trading bots from marketing exercises. It’s not about catching outright fraud — though that does exist — but about reading performance claims with the same scrutiny a professional allocator would use before committing capital.
The six questions worth asking
Before you act on anyone’s AI bot performance numbers — your own backtested system included — work through these six questions. If the answers aren’t readily available, treat the numbers as unverified.
1. Is the performance backtested or live?
Backtested performance shows what the bot would have made if it had been running over a past period. Live performance shows what it actually made on real capital in real markets. The gap between the two is often enormous. A bot that prints beautiful backtests routinely underperforms in live execution because backtests don’t capture slippage, partial fills, real bid-ask spreads, exchange downtime, or the psychological cost of holding through a drawdown.
If the only available numbers are backtested, treat them as a useful sanity check on the logic — and nothing more. Live track records are the only ones that count.
2. How long is the live track record?
Three months of live performance during a trending market tells you almost nothing. Twelve months including at least one significant drawdown starts to be informative. Three years across multiple market regimes is meaningful evidence. Most AI bots being pitched in retail channels have track records measured in months, not years. That’s a signal in itself.
3. What’s the maximum drawdown?
Drawdown is the peak-to-trough decline at the worst point in the bot’s history. A bot with a stunning return number but a 60% maximum drawdown isn’t impressive — it’s a leveraged bet that happened to work. Most professional discretionary traders wouldn’t accept drawdowns above 20–25% of capital. Look at the worst drawdown number alongside the return; the ratio of the two is more meaningful than either number alone.
4. What’s the Sharpe ratio?
The Sharpe ratio measures return per unit of volatility — how much return you got compared to how bumpy the ride was. A Sharpe of 1.0 is decent, 1.5 is good, 2.0 is excellent, and anything above 3.0 in a retail context should make you suspicious. Real professional traders rarely sustain Sharpe ratios above 2.0 over multi-year periods.
If a bot pitch quotes a 5.0 Sharpe ratio, either the track record is too short to be meaningful, the volatility calculation is gamed, or the numbers are wrong. Ask which.
5. How many independent trades is the record based on?
A bot that fires once a week generates 50 trades a year. After two years, that’s 100 data points — not enough to draw strong statistical conclusions. A bot that fires 20 times a day generates thousands of trades quickly, which makes the track record more statistically meaningful even over shorter periods.
Small sample sizes are the most common reason apparently-impressive bots fail in the real world. Don’t extrapolate from 30 trades; demand hundreds at minimum.
6. Are the numbers audited or self-reported?
Self-reported numbers can be cherry-picked, selectively disclosed, or simply made up. Third-party audited records — by a credentialed accounting firm or a track-record-verification service — are far more credible. In the institutional world, audited track records are standard. In the retail AI bot space, they’re rare. That asymmetry tells you something.
Red flags that should stop you immediately
Some claims are simply incompatible with how markets work. If you see any of these, walk away.
- “Guaranteed returns” of any percentage. No legitimate trading product can guarantee returns. This is a regulatory red flag and a maths impossibility.
- Win rates above 90%. A 90% win rate often hides asymmetric losses where the small percentage of losing trades wipe out months of small gains.
- No mention of drawdowns. Every honest performance record discloses the worst drawdown. Pitches that only show returns are hiding the bad days.
- “Proprietary AI we can’t explain.” A bot that can’t articulate its decision-making logic is one you can’t evaluate — and one you can’t trust.
- Pressure to deposit quickly. Any platform pressuring you with countdown timers or limited-time offers is selling a feeling, not a product.
How cherry-picking actually works
Backtested cherry-picking takes a few common forms. Understanding them helps you spot the technique even when the marketing is polished.
Selecting the best period. A bot whose strategy happens to suit a trending market gets backtested over the last bull run and presented as if those numbers are typical. Run the same strategy over a range-bound period and the picture changes entirely.
Selecting the best market. A bot that’s barely profitable across most assets but spectacular on one specific pair gets pitched using that one pair. The aggregate performance is much weaker.
Curve-fitting. A strategy with twenty parameters tuned to maximise returns on historical data looks superb in backtest but generalises poorly to live markets. The model has memorised the past rather than learned a pattern.
Survivorship bias. Bot vendors typically discontinue strategies that don’t work, then promote the ones that survived. The performance you see is from the survivors. The strategies that blew up never make it into the marketing.
What honest AI bot disclosure looks like
Five things you’d expect from a platform that’s actually confident in its performance.
- Live track record with full monthly disclosure, including the bad months.
- Multiple market regimes covered — the system has been tested through trending, range-bound, and volatile periods.
- Clear documentation of the strategy logic, including which factors drive trade decisions.
- Realistic Sharpe and drawdown numbers that you wouldn’t be embarrassed to compare to professional benchmarks.
- Operating inside the Australian regulatory framework — an AFSL holder, AFCA member, with segregated client funds.
A platform that ticks all five doesn’t need to oversell. The numbers do the talking.
Applying the same scrutiny to your own system
The same six questions apply when you’re evaluating a strategy you’ve built yourself. Australian retail traders are often more forgiving of their own backtests than they would be of someone else’s. That’s a bias worth correcting.
Walk-forward test your strategy on data it hasn’t seen. Paper-trade it live before committing real capital. Track real fills versus theoretical fills. Honest evaluation of your own work is worth more than any third-party validation, because you’re the one who has to live with the result.
The bottom line
Profitable AI trading bots exist. So do cherry-picked marketing exercises dressed up as the same thing. The six questions in this guide help you separate the two without needing a quant background — backtested or live, length of record, drawdown, Sharpe, sample size, audited or not. Apply them rigorously and most of the noise filters itself out.
The platforms genuinely worth your capital don’t shy away from these questions. The ones that do shy away are answering them with their reluctance.
For more on what a legitimate AI trading platform looks like in the Australian regulatory context, see our piece on ASIC rules for AI trading platforms. For the broader context of algorithmic markets, see why 85% of ASX trades are now algorithmic. To see how Impulse Cashholm discloses its approach and operates, head to How It Works or the FAQ.
Trading and investing involve risk, including the possible loss of capital. Past performance is not a reliable indicator of future results. Information on this page is general in nature and does not constitute financial advice.