Are you sure you want to delete this access key?
Refer to the SeamlessM4T README for an overview of the M4T models.
Inference is run with the CLI, from the root directory of the repository.
The model can be specified with --model_name
seamlessM4T_v2_large
, seamlessM4T_large
or seamlessM4T_medium
:
S2ST:
m4t_predict <path_to_input_audio> --task s2st --tgt_lang <tgt_lang> --output_path <path_to_save_audio> --model_name seamlessM4T_v2_large
S2TT:
m4t_predict <path_to_input_audio> --task s2tt --tgt_lang <tgt_lang> --model_name seamlessM4T_v2_large
T2TT:
m4t_predict <input_text> --task t2tt --tgt_lang <tgt_lang> --src_lang <src_lang> --model_name seamlessM4T_v2_large
T2ST:
m4t_predict <input_text> --task t2st --tgt_lang <tgt_lang> --src_lang <src_lang> --output_path <path_to_save_audio> --model_name seamlessM4T_v2_large
ASR:
m4t_predict <path_to_input_audio> --task asr --tgt_lang <tgt_lang> --model_name seamlessM4T_v2_large
Inference calls for the Translator
object instantiated with a multitask UnitY or UnitY2 model with the options:
and a vocoder:
vocoder_v2
for seamlessM4T_v2_large
.vocoder_36langs
for seamlessM4T_large
or seamlessM4T_medium
.import torch
from seamless_communication.inference import Translator
# Initialize a Translator object with a multitask model, vocoder on the GPU.
translator = Translator("seamlessM4T_large", "vocoder_36langs", torch.device("cuda:0"), torch.float16)
Now predict()
can be used to run inference as many times on any of the supported tasks.
Given an input audio with <path_to_input_audio>
or an input text <input_text>
in <src_lang>
,
we first set the text_generation_opts
, unit_generation_opts
and then translate into <tgt_lang>
as follows:
S2ST and T2ST (speech output):
# S2ST
text_output, speech_output = translator.predict(
input=<path_to_input_audio>,
task_str="S2ST",
tgt_lang=<tgt_lang>,
text_generation_opts=text_generation_opts,
unit_generation_opts=unit_generation_opts
)
# T2ST
text_output, speech_output = translator.predict(
input=<input_text>,
task_str="T2ST",
tgt_lang=<tgt_lang>,
src_lang=<src_lang>,
text_generation_opts=text_generation_opts,
unit_generation_opts=unit_generation_opts
)
Note that <src_lang>
must be specified for T2ST.
The generated units are synthesized and the output audio file is saved with:
# Save the translated audio output:
import torchaudio
torchaudio.save(
<path_to_save_audio>,
speech_output.audio_wavs[0][0].cpu(),
sample_rate=speech_output.sample_rate,
)
S2TT, T2TT and ASR (text output):
# S2TT
text_output, _ = translator.predict(
input=<path_to_input_audio>,
task_str="S2TT",
tgt_lang=<tgt_lang>,
text_generation_opts=text_generation_opts,
unit_generation_opts=None
)
# ASR
# This is equivalent to S2TT with `<tgt_lang>=<src_lang>`.
text_output, _ = translator.predict(
input=<path_to_input_audio>,
task_str="ASR",
tgt_lang=<src_lang>,
text_generation_opts=text_generation_opts,
unit_generation_opts=None
)
# T2TT
text_output, _ = translator.predict(
input=<input_text>,
task_str="T2TT",
tgt_lang=<tgt_lang>,
src_lang=<src_lang>,
text_generation_opts=text_generation_opts,
unit_generation_opts=None
)
Note that <src_lang>
must be specified for T2TT
Press p or to see the previous file or, n or to see the next file
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?