Duration-Aware Pause Insertion Using Pre-Trained Language Model for Multi-Speaker Text-to-Speech

Authors: Dong Yang, Tomoki Koriyama, Yuki Saito, Takaaki Saeki, Detai Xin, Hiroshi Saruwatari
arxiv preprint
poster



Abbr. Meaning
RPs Respiratory pauses, at word transitions without punctuation.
PIPs Punctuation-indicated pauses, at punctuation marks.
RPI Proposed respiratory pause insertion model, predicting position of RPs.
CPI Proposed categorized pause insertion model, predicting position and category of RPs and PIPs.
CPI (Position) CPI model, but only predicting position of RPs and PIPs.
RPI* RPI model with unmatched speaker embedding.
CPI* CPI model with unmatched speaker embedding.

1. Short utterance

These utterances are not used in our subjective evaluations.

FastSpeech2 CPI
1. (speaker id: 783, female speaker) Opposite the printed words "pay to" were written
the words "kleins sporting goods."
2. (speaker id: 1182, male speaker) Foreign witnesses, too, were in favor of publicity.

2. Long-form utterance

These utterances are used in our subjective evaluations.

FastSpeech2 Baseline RPI CPI CPI (Position) RPI* CPI*
1. (speaker id: 637, male speaker) If the congress wishes to be the voice of the revolutionary democracy it must not sit with folded hands before the developing civil war, the result of which may be a dangerous outburst of counter revolution. The possibility of a peaceful outcome lies in the formation of a united democratic authority.
2. (speaker id: 770, male speaker) Love with the shield of the broken heart ever his bow doth bend, with a single shaft for a single prize, and the ultimate bolt that parts and flies comes with a thunder of split skies, and a sound of souls that rend.
3. (speaker id: 1779, female speaker) Early on the following day the whole town was on its feet, and everyone wondered how and where the stranger would build the wonderful palace. The princess alone was silent and sorrowful, and had cried all night till her pillow was wet, so much did she take the fate of the beautiful youth to heart.
4. (speaker id: 2004, female speaker) It is heart racking to struggle day by day, amid incessant interruption and melee, to snatch out of the hurly burly some shreds of humour or pathos or (dare one say?) beauty, and phrase them intelligibly.

3. Samples of CPI

These utterances are not used in our subjective evaluations.

Text: But now there has grown up a large enough leisure class possessed of such wealth that any aspersion on the score of enforced manual employment would be idle and harmless calumny; and the corset has therefore in large measure fallen into disuse within this class.

(speaker id: 70, male speaker): but now there has grown up a large enough leisure class possessed of such wealth sp2 that any aspersion on the score of enforced manual employment sp2 would be idle and harmless calumny sp2 and the corset has therefore in large measure fallen into disuse within this class sp3

(speaker id: 1649, female speaker): but now there has grown up a large enough leisure class possessed of such wealth sp1 that any aspersion on the score of enforced manual employment sp1 would be idle and harmless calumny sp2 and the corset has therefore in large measure fallen into disuse within this class sp3