AI call logs and success rates

AI call logs record the action type, question, data source, token consumption, latency, and execution result of every AI request, for troubleshooting, cost observation, and quality evaluation. The AI assistant also provides a success rate report that aggregates turn data by scenario and failure reason.

Entry points

Feature	Entry
AI call logs	Account Center → AI Service → AI call logs
Success rate report	AI Assistant → left configuration menu → success rate report

Both entries show only data for the current project. The backend isolates data by project context and does not provide cross-project aggregation.

AI call logs

Recorded actions

Logs are written asynchronously by each AI call chain and cover the following actions:

Action code	Source	Description
GENERATE	SQL generation	Generate a draft SQL from a question
OPTIMIZE	SQL optimization	Read-only optimization based on the current SQL
QUERY	Natural language data query	Generate SQL and run a preview
INSIGHT	Result insight	Generate conclusions from query results
AGENT_CHAT	AI assistant	Tool calls and answers for one conversation turn
AGENT_CONFIRM	AI assistant	Continuation turn after the user confirms a sensitive operation
AGENT_TURN	AI assistant observability	Turn summary row; the data source of the success rate report
SEMANTIC_RESOLVE	Observability instrumentation	Semantic layer resolution
LLM_STEP	Observability instrumentation	Single-step model call
SQL_GEN	Observability instrumentation	SQL generation stage within a scenario
ANSWER_GEN	Observability instrumentation	Answer generation stage within a scenario
CLOSURE_REPAIR	AI assistant	Message sequence closure repair alert, success=0 for easier monitoring

Action display in the list

The "Action type" filter in the list offers only "All / Generate / Optimize / Query", which map to the three SQL-related actions. Actions such as AGENT_* and INSIGHT still appear in the list, but are shown with their raw action codes and cannot be filtered individually through that dropdown.

List fields

Column	Meaning
User ID	ID of the user who initiated the call
Action type	Action code; generate/optimize/query are shown as tags
Question	Question text entered by the user
Data source type	Data source type of this call
Token consumption	total_tokens, the sum of prompt and completion tokens
Duration (ms)	Duration of this call; 500–999 ms is shown in orange, ≥1000 ms in red
Result	Green for success, red for failure
Call time	Log creation time

Filtering and pagination

1. Keyword: fuzzy match on question content;
2. Action type: all, generate, optimize, query;
3. Time range: quick options for the last hour, last 24 hours, and last seven days, or a custom start and end time;
4. Click "Query" to reload with the current conditions; the page number returns to 1.

Pagination defaults to 10 rows per page and can be switched to 10 / 20 / 50 / 100; the backend caps a single page at 100.

Design rationale

• Asynchronous writes: the request thread assembles the project, user, and question information, then hands it to the aiAuditExecutor thread pool for insertion; write failures are only logged as alerts and do not affect the main business flow;
• Project pinning: asynchronous threads do not inherit the project context, so projectId must be written into the log entity within the request thread;
• Text truncation: long texts such as the question, current SQL, and generated SQL are limited to 4000 characters, and error messages to 500 characters, so a single log entry does not grow too large;
• Fields not shown in the UI: the table also stores the session ID, turn ID, matched scenario, actual model, mode, generated SQL, schema, referenced tables, tool call chain, prompt/completion tokens, time to first event, and failure reason. These fields are used for troubleshooting and reporting; they are not shown on the current page and can be inspected in the database as needed;
• No automatic cleanup: there is currently no log archiving or periodic cleanup task, so logs keep accumulating. In production, define a retention strategy based on database capacity, or partition/archive on the database side.

Troubleshooting tips

Symptom	What to check first
An AI request failed	Rows whose "Result" is failure, combined with error_msg and fail_reason in the database
Slow response	Records highlighted in the duration column; use first_event_ms to tell whether the first byte is slow or the whole request is slow
Rising token cost	Filter by question keyword and time range, and observe the total_tokens distribution
AI assistant operations have no effect	Check the AGENT_CHAT / AGENT_CONFIRM records and the tool call chain

Success rate report

The success rate report is located in the left configuration menu of the AI Assistant. It aggregates AGENT_TURN turns by scenario + failure reason to observe the answer quality of the AI assistant.

How to use

1. Open the left configuration menu on the AI Assistant page;
2. Select "Success rate report";
3. Optionally select a date range (by day, inclusive of the start and end dates);
4. Click "Query" to refresh the data.

If no date is selected, all historical data is aggregated.

Report fields

Column	Description
Scenario	Matched scenario code; the same scenario may have multiple rows
Failure reason	Success rows show a "Success" tag; failure rows show reasons such as timeout, rate limiting, and tool failure
Count	Total number of turns for this scenario + failure reason combination
Scenario success rate	Sum of successes across all groups of the scenario ÷ total turns
Average tokens	Average total tokens for the group
Average duration (ms)	Average duration for the group

Scenario success rate is color-coded: ≥90% green, ≥70% orange, below 70% red.

Failure reason reference

Reason code	Meaning
MODEL_TIMEOUT	Model call timed out
MODEL_429	Model service rate limited
MODEL_ERROR	Model returned an error
TOOL_ERROR	Tool execution failed
USER_REJECTED	User rejected the confirmation
EXPIRED	Confirmation expired
CANCELLED	User cancelled
STEP_LIMIT	Exceeded the maximum number of reasoning steps

Design rationale

• The report aggregates only AGENT_TURN summary rows; ordinary SQL generation/optimization logs are not included in the statistics;
• The scenario success rate is not taken from a single row: the success counts and totals of all failure reason groups under the same scenario are summed separately before calculating, to avoid distortion at the group level;
• Average tokens and average duration are aggregated by the database; the frontend only displays them and applies color grading;
• Both the report and the logs are project-isolated, and the report API requires the AI assistant feature to be enabled.

Limitations and notes

• Log writes are asynchronous, so a call that has just finished may take a few seconds to appear in the list;
• Logs record call metadata and do not include full model request or response bodies;
• Question text may contain business data, so access to the log database should be managed as sensitive information;
• The log scope follows the project switch; historical project data does not mix into the current project;
• CLOSURE_REPAIR is an anomaly alert record; when it appears, investigate the message persistence chain instead of treating it as a normal user operation.

Related pages

• AI data query and SQL analysis
• AI assistant
• Security and traffic governance