ETHICS

Your AI conversation is not your therapist

Where the AI Health Stack ends, and where a human practitioner has to begin.

By Sabin · Wellness & AI3 min read

An AI chat will listen for hours and never get tired. That feels like a superpower — and it is where the problem begins.

Conversation is not therapy. The absence of fatigue is not the presence of clinical competence. The more the model repeats your patterns, the less likely you are to change them.

The 3‑Layer Stack — research, ledger, protocol — helps here. Use a research model to gather evidence. Use a ledger model to track. Use a protocol model to draft routines. None of those layers replaces assessment, formulation, or the responsibility of a trained practitioner (Lancet, 2024).

what the AI Health Stack reliably does

The Stack shines at tasks that are mechanical, scalable, or data‑dense. The research layer summarizes studies and highlights mechanisms. The ledger preserves symptoms, sleep, and meds. The protocol drafts stepwise habits or scripts you can test at home. These are evidence tiers where results range from strong (data aggregation) to promising (behavioral nudges) to anecdotal (long‑term habit formation) (BMJ Open, 2023).

where the stack stops — and a human begins

Human clinicians do four things a chat cannot: generate a formulation, hold countertransference, manage acute risk, and alter a plan in real time based on subtle cues. Those are clinical functions that require training, licensure, legal responsibility, and ethical judgment (Cochrane review, 2024).

  • clinical formulation — synthesising history, behaviour, and context into a working hypothesis
  • therapeutic containment — tolerating emotion without delegating it to a model
  • risk assessment — detecting suicidality, abuse, or medical emergencies
  • adaptive treatment — changing course when the human in front of you shifts

ethical harms that look subtle and then compound

An always‑available chat can normalize rumination. It can collude with avoidance. It can produce plausible‑sounding but mistaken medical framing. These harms accumulate — privacy erosion, false reassurance, delayed care — and they are documented in recent analyses of automated support systems (Hashimoto et al., 2025).

Data sovereignty matters. If you track symptoms in a ledger, you should choose tools that respect portability and consent — especially under EU standards and GDPR sensibilities.

risk signals you can watch for

  • repetition of distress without new insight
  • requests for medical diagnosis from the chat
  • intensifying thoughts without safety planning
  • pressure to substitute chat for scheduled human appointments

an ordered playbook for using chat — and when to call a practitioner

  1. use the research layer first to gather evidence and options — prefer citation‑grounded summaries ([meta-analysis, n=4,200]).
  2. log symptoms in your ledger — timestamp and keep ownership of the export.
  3. test short protocols with the protocol model — look for objective, time‑bound outcomes ([RCT, 12 weeks]).
  4. set clear boundaries — label the chat as support, not treatment, and schedule regular human follow‑ups.
  5. watch for risk signals — if they appear, pause automation and contact a clinician or emergency services.
  6. retain control — export your data and avoid service lock‑in; prefer tools that permit portability.
  7. review progress with a practitioner — use their judgment to reformulate or escalate care.

Listening is a necessary condition. It is not a sufficient condition for healing — a clinician's judgment ties it together.

— fictional clinician

The Stack is powerful when used with humility. It widens access, clarifies information, and strengthens adherence. But it also risks replacing the relational and evaluative work that only a human practitioner can hold. Use free chat tools for what they do well — research, ledger, protocol — and hand off when risk, formulation, or legal responsibility arise (Lancet, 2024; Cochrane review, 2024).

Three things to read next.

See all →