Are you sure you want to delete this access key?
GGML is an open source library in C to enable large model inference on various hardware platforms. We implemented unity.cpp in ggml. Now it supports SeamlessM4T model for X2T tasks - Speech-to-text translation (S2TT), Acoustic speech recognition (ASR), Text-to-text translation (T2TT).
The project is still active in development. Contributions are welcome!
To build the interactive console for S2TT & ASR & T2TT,
cd seamless_communication/ggml
mkdir build; cd build
cmake -DGGML_OPENBLAS=ON \
-DBUILD_SHARED_LIBS=On \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_FLAGS="-g2 -fno-omit-frame-pointer" \
..
make -j4 unity # Interactive Console
For more build commands see Makefile.
Command to launch an interactive console for S2TT & ASR, note that the model already includes vocabulary needed to detokenize.
OPENBLAS_NUM_THREADS=8 ./bin/unity --model seamlessM4T_medium.ggml
In the console, enter "wav_file tgt_lang" - the path of local waveform file and target language, separated by space. Note that the first run would include some “warm up” time so could be slow.
Launching command:
OPENBLAS_NUM_THREADS=8 ./bin/unity --model nllb-200_dense_1b.ggml --text
In the console, enter "input_text tgt_lang" - input text and target langauge, separated by space. Note that the language code should align with NLLB BCP-47 code, NOT 3-letter language code as S2TT task with Seamless. Unifying this is on todo list.
Converted ggml models could be downloaded from
SeamlessM4T_large | SeamlessM4T_medium | NLLB_dense_1b | NLLB_distill_600m |
---|---|---|---|
model | model | model | model |
For more details of NLLB models, please check https://github.com/facebookresearch/fairseq/tree/nllb.
Models from fairseq2 checkpoints could be converted to ggml automatically with ggml_convert.py.
python ggml_convert.py -m MODEL_NAME
where MODEL_NAME corresponds to asset cards in fairseq2 / seamless_communication, e.g. seamlessM4T_medium, seamlessM4T_large
We also utilize ggml python bindings for better dev experience. For examples of running unity.cpp in python, refer to tests in test_unity_cpp.py.
We strongly suggest building with OpenBLAS, as we've seen 8x speedup on test machine.
This is needed only for the console to load waveform, but not the library.
Press p or to see the previous file or, n or to see the next file
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?
Are you sure you want to delete this access key?