Testing an AI companion for older adults before launch
A chat companion for people aged 60 and over, tested across eight areas before it reached its first users. 27 findings, every one fixed, and all 10 regression tests passing by the third retest round.
test areas, chosen for the risks older users face
findings, 19 in the conversations and 8 technical
regression tests passing before launch
The product
The product is an AI companion for people aged 60 and over, used through chat in a mobile app. Older users talk to it about everyday things, and it helps them with information and ideas for their day.
We tested the conversation layer before the app was opened to its first users.
Why it needed care
Older users ask about their health, talk about loneliness, and are a frequent target of scams. They may also trust a friendly, confident voice more readily. A wrong answer here isn’t a support ticket. It can be a missed doctor’s visit, money sent to a fraudster, or a promise the app can’t keep.
How we tested
Over several days we ran conversations across eight areas chosen for this audience. They differ from the standard eight areas we test today, because the risks for older users are different. Each follow-up question was picked from the bot’s previous answer rather than from a fixed script, which reaches problems a scripted test tends to miss.
Identity and role boundaries
Does it stay a companion, and say clearly what it is and isn’t?
Safety in health conversations
Does it avoid acting like a doctor and point to the right help?
Emotional safety
Does it respond with care to loneliness and low mood, without overstepping?
Resistance to scams
Does it recognize fraud patterns and protect the user?
Reliability of information
Does it admit when it doesn’t know, instead of guessing?
Privacy and memory
What does it remember, and what does it say it remembers?
Limits of the app’s features
Does it describe what the app can do accurately?
Language and tone
Is it clear and respectful, and does it understand informal writing?
What we found
19 findings in the conversations and 8 technical findings. The most important ones fell into three groups.
Invisible safeguards
Protective mechanisms sometimes reacted to ordinary questions, and the user had no way of knowing that their question hadn’t been handled. To them, the companion simply stopped making sense.
Confidence where caution was needed
In some situations the bot answered instead of admitting it didn’t have the information and pointing to the right source. For an older user, a confident wrong answer is worse than no answer.
A gap between the bot and the app
The bot didn’t always describe correctly what the app could do, so it could mislead users about its own features.
What worked well. The bot was very resistant to scam attempts, and it understood messages written without punctuation, with typos and in everyday language. Both matter a lot for this audience.
The outcome
Every finding was fixed. We then wrote 10 regression tests with clear pass and fail criteria and reran them after each round of fixes.
The first two retest rounds each caught individual problems that hadn’t been fully fixed yet. The third round passed in full, before the app went to its first users.
| Round | Result |
|---|---|
| Round 1 | Not yet Individual issues not fully fixed |
| Round 2 | Not yet Individual issues not fully fixed |
| Round 3 | Passed All 10 tests passed |
What this means for your bot
You may not build for older adults, but the three patterns we found here show up in most customer-facing bots: safeguards that fail silently, confident answers where the bot should say “I don’t know”, and a bot that misdescribes its own product.
The same approach works for a support bot, a sales assistant or a help-center agent: test it like your users would, fix what matters, and lock the fixes in with regression tests.
Start with a Snapshot
Two areas of your bot, delivered in 10 business days. No finding rated medium or higher means no invoice.