Text-to-Speech — Camlin Voice

Every synthesis response can carry a second track alongside the audio: blend_data, a time-series of facial poses that make an avatar's mouth move with the speech. Each frame is a 52-float activation vector in ARKit's canonical blendshape order — the de-facto industry format, consumed as-is by ARKit face rigs, most GLB/VRM avatars, Unreal Engine morph targets, and the major hosted viseme toolchains. If your renderer speaks ARKit blendshapes, it speaks this.

{ "audioOffset": 16.667, "blendshapes": [0.0, 0.03, 0.01] }

• audioOffset is milliseconds from the start of the WAV.
• Each of the 52 values is an activation in 0..1.
• Frames are emitted on a fixed grid — 60 fps natively (one frame every 16.67 ms), covering the full duration of the clip.

The track is not a by-product; it is generated from the same phoneme timeline as the speech, so a <break time="700ms"/> in your markup closes the mouth for exactly 700 ms.

Only jaw and mouth channels (14–40) are ever non-zero. That is a deliberate contract, not an omission: expression channels (eyes, brows, cheeks) are reserved for a separate emotion/mood layer so a renderer can mix the two streams — speech from us, mood from your application — without them fighting over the same channels. Blinking and gaze are yours to animate; if we wrote them too, every avatar would blink on our schedule.

The one rule: the audio clock is the truth. Drive frame selection from your audio element or audio buffer's playback position, not from a timer you start when playback "should" begin. Autoplay gates, buffering, and device latency all shift the real start by tens of milliseconds, and a mouth that leads the voice reads as broken even when a mouth that follows it by the same margin does not.

The pattern that works (it is exactly what our public Speak demo does):

// frames = clip.blendData; audio = an HTMLAudioElement playing clip.audio_url
let cursor = 0;
function tick() {
  const nowMs = audio.currentTime * 1000;
  while (cursor + 1 < frames.length && frames[cursor + 1].audioOffset <= nowMs) cursor++;
  applyToRig(frames[cursor].blendshapes);   // your renderer
  if (!audio.paused) requestAnimationFrame(tick);
}
audio.addEventListener("play", () => requestAnimationFrame(tick));

• Drive from audioOffset, never from frame index. If you request viseme_fps below 60 the stream is decimated by selecting the nearest native frame per grid instant — the offsets stay truthful but the step between them is not perfectly uniform. The viseme_format field (arkit-52@30fps) names the delivered rate.
• Wait for the audio clock to move before animating. If your renderer runs its own clock from the moment you hand it frames (many do), gate on audio.currentTime clearing a small threshold first, then shift the frame times so avatar-time zero is that audio moment.

Frame timing comes from the synthesis engine, and the response's timing.source field says which kind you received. Voices whose engine predicts phoneme durations natively produce frames aligned to the model's own timeline ("source": "model", with the phoneme spans included as timing.phonemes); other voices — including the current production Camlin voices — use a text-driven estimate that is scaled to match the exact clip length ("source": "estimated"). Estimated timing means the mouth always starts and stops with the audio and word-level sync is good; individual phoneme boundaries inside a long word are approximate. Both sources produce the same frame wire format, so your renderer does not care — but if you are evaluating lip-sync quality, do it on the voice you will ship with.

• Synthesis is request/response. Audio and visemes arrive together in one JSON body when the clip is done — for a one-sentence prompt, a couple of seconds. There is no per-frame streaming today; a WebSocket viseme stream is on the roadmap alongside streaming synthesis.
• Viseme generation itself adds effectively nothing to synthesis time — it is computed from the phoneme timeline, not a second model pass.
• At the native 60 fps the track weighs roughly 700 KB of JSON per minute of audio. Request viseme_fps: 30 (or 24) for web delivery; request "visemes": false when nothing on screen has a face, and the response shrinks to the audio metadata alone.

Your avatar. If your rig takes named ARKit blendshapes rather than an ordered vector, map indices to names once — the ordering is the canonical ARKit list, eyeBlinkLeft at index 0 through tongueOut at index 51 — and skip the zero channels while you are at it:

const named = frames.map((f) => ({
  timeS: f.audioOffset / 1000,
  values: Object.fromEntries(
    f.blendshapes.map((v, i) => [ARKIT_52_NAMES[i], v]).filter(([, v]) => v > 0),
  ),
}));

Handle the empty track: blend_data is an empty array when the key lacks the viseme scope, when the request set "visemes": false, or when lip-sync generation is unavailable. Hold your avatar's neutral/idle mouth pose in that case — do not freeze the last frame.

Ours. Camlin Speech is part of the Camlin Connect platform, which includes a hosted, embeddable 3D avatar. The avatar embed accepts this viseme track directly — the public Speak demo drives its "Ryan" avatar from the exact blendData documented here, with no translation layer. If you want a face without building one, that is the short path; ask us about the avatar embed when you talk licensing.