AI applications / Evaluation tooling
Guest Support Agent
Connecting guest conversations, property knowledge and human expertise in one support workflow.
GnG Vacation · Sole developer
~10,000
guest messages handled per month
The problem
Guests need property-specific answers throughout a stay: check-in instructions, parking details and help when something goes wrong. The same questions recur, while exceptions still need someone who knows the property. I built the support agent and the tooling around it as the sole developer.
How it works
- 01
Guest message
Reservation and property context
- 02
Agent + tools
Property knowledge and web search
- 03
Answer or escalate
Guest reply or staff handoff in Lark
- 04
Feedback
Reviewed answers become reusable knowledge
What I built
- Tool-based property lookup and local recommendations, with automated communications around arrival, the stay and departure.
- Lark escalation for unresolved questions, connecting staff answers back to the guest-support workflow.
- Response-scoring tools covering dialogue quality, policy compliance, tool use and hallucination control.
- Episode instrumentation for tool traces, verifier decisions, send outcomes and human draft reviews.
- A repeat-question evaluation harness reporting precision and recall on synthetic labeled cases.
Engineering decisions
Keep people in the knowledge loop
Staff answers can become FAQs and structured property updates. Sensitive fields require approval, and overwrites use confidence checks.
Inspect behavior across the conversation
Immediate and delayed feedback record corrections, repeated questions and staff resolution. These signals support investigation; silence alone is not a satisfaction label.
Separate implementation from measured quality
The evaluation harness and scoring tools are implemented. A configured acceptance threshold is a test target; quality scores and resolution-rate improvements need measured results before being reported.