Tokenizer

Calculate DeepSeek model token usage offline — no API calls needed, powered by Transformers.js.

@deepseek-kit/tokenizer is a standalone package for offline token counting. It bundles the DeepSeek tokenizer vocabulary and uses Transformers.js to encode text locally — no API key, no network requests, no API costs. This is useful for estimating prompt costs, validating context length limits, and monitoring token usage before sending requests.

Installation

npm
pnpm
bun
yarn
npm i @deepseek-kit/tokenizer

Basic Usage

Counting Tokens

Use countTokens() to get the token count of a text:

import { countTokens } from '@deepseek-kit/tokenizer'

const count = await countTokens('Hello, world!')
console.log(count)
// 4

This is the quickest way to estimate how many tokens a piece of text will consume when sent to the DeepSeek API.

Use Cases

Estimating Prompt Cost

Before sending a request, estimate the token count to calculate the approximate cost:

import { countTokens } from '@deepseek-kit/tokenizer'

const prompt = 'Explain how async/await works in JavaScript.'

const tokenCount = await countTokens(prompt)
console.log(`Prompt tokens: ${tokenCount}`)

Validating Context Length

DeepSeek models have a maximum context length. Check before sending long texts:

import { countTokens } from '@deepseek-kit/tokenizer'

const MAX_CONTEXT_TOKENS = 64000

async function validateContextLength(text: string) {
  const count = await countTokens(text)
  if (count > MAX_CONTEXT_TOKENS) {
    throw new Error(`Text exceeds maximum context length (${count}/${MAX_CONTEXT_TOKENS} tokens)`)
  }
  return count
}

Monitoring Multi-turn Conversations

Track token accumulation across conversation turns:

import { countTokens } from '@deepseek-kit/tokenizer'

const conversation: string[] = [
  'User: What is TypeScript?',
  'Assistant: TypeScript is a typed superset of JavaScript...',
  'User: How does it differ from JavaScript?',
]

const totalTokens = await Promise.all(conversation.map(countTokens))
const sum = totalTokens.reduce((acc, n) => acc + n, 0)
console.log(`Conversation tokens: ${sum}`)

API Reference

countTokens()

Counts the number of tokens in a text.

function countTokens(text: string): Promise<number>
textrequiredstring
The input text to count tokens for.

Returns a Promise<number> — the total token count.

LangChain Integration
Use deepseek-kit alongside LangChain.js — DeepSeek handles text reasoning and tool calling while LangChain handles multimodal and complex workflows.