Token usage

Turn per-call token counts into cost per caller, logged in a form that sums.

Every response includes what the call consumed - the usage dictionary described under invoke. The service below turns those numbers into cost per department, logged per call in a form that sums with standard tools.

Prerequisites. An LLM connection named Shared LLM - the one connection several departments share is exactly the case where per-caller cost matters.

The service

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

# The provider's price per one million tokens, in USD - from its price list
_price_per_million_input = 0.15
_price_per_million_output = 0.60

class AskWithAccounting(Service):
    """ Answers a question and logs what it cost, attributed to the caller.
    """
    name = 'example.ai.ask-with-accounting'

    input = 'department', 'question'
    output = 'answer', 'cost_usd'

    def handle(self):

        conn = self.llm['Shared LLM']

        response = conn.invoke(self.request.input.question)

        # The two counts are always present, regardless of the model that answered
        input_tokens = response['usage']['input_tokens']
        output_tokens = response['usage']['output_tokens']

        input_cost = input_tokens * _price_per_million_input / 1_000_000
        output_cost = output_tokens * _price_per_million_output / 1_000_000
        cost_usd = round(input_cost + output_cost, 6)

        # One parseable line per call - key=value, so any log tooling can sum it
        self.logger.info('llm-cost department=%s input_tokens=%d output_tokens=%d cost_usd=%f',
            self.request.input.department, input_tokens, output_tokens, cost_usd)

        self.response.payload.answer = response['text']
        self.response.payload.cost_usd = cost_usd

Run the service

curl localhost:17010/example/ask -d '{"department":"billing","question":"What is an integration platform?"}'

Expected output - the cost of this call is in the response:

{"answer": "An integration platform connects...", "cost_usd": 0.000141}

And the server log now has the line that accounting tools aggregate:

llm-cost department=billing input_tokens=14 output_tokens=213 cost_usd=0.000141

Summing a day's spend per department is one command over the log:

grep -h 'llm-cost' ~/env/qs-1/server1/logs/server.log | \
    awk -F'department=| input' '{print $2}' | sort | uniq -c

The REST channel:

channel_rest:
  - name: example.ai.ask-with-accounting
    service: example.ai.ask-with-accounting
    url_path: /example/ask

Chat costs

In a chat, input_tokens covers everything the provider received - the history included. The number grows turn by turn until trimming caps it, which is the cost ceiling max_history_turns sets.

Failure behavior

A failed call raises before any tokens are consumed on the reply, so nothing is logged for it here - but the failure itself is in the audit log with its duration and error. The one case that spends tokens without a log line is a call that succeeds while the service crashes later - which is why the cost line above is written immediately after the call, before anything else can fail.

See also

FeatureWhat it does
Invoking LLMsThe usage dictionary on every response
Cost and limitsThe limits that cap what a call may spend
Provider failuresWhat failed calls cost and where they are recorded