What A Hindi Speaking AI Sounds Like

A few posts ago we transcribed an Indian English video using a Speech-To-Text Indian English transcription Artificial Intelligence (AI). Today we are going to take the same idea forward, and now going to translate the video into Hindi.

To recap, we used a sample clip from Rajya Sabha TV which as (a) transcribed using a speech-to-text AI, and (b) cleaned up by a human. We will now clean it up a little bit more, and then translate the content into Hindi. Finally, we will use a Text-To-Speech AI to speak out the Hindi content.

Its going to be awesome. We hope this post gives you some ideas about how to use our technology. Let’s begin!

What Are We Going To Be Working With? Our Source Video Is Already Transcribed

We will use the video we produced at the end of this post. It is posted below.

First though, a bit of theory! Just like school!

The Look-Ahead And Forward Algorithms - The Theory

One of the challenges in working with AI is the Look-Ahead and Forward algorithm usage. The majority of translation AI’s use the look forward algorithm to translate text content.

From our trusty source of all knowledge (Wikipedia), Look-Ahead (backtracking) is 'In backtracking algorithms, look ahead is the generic term for a sub-procedure that attempts to foresee the effects of choosing a branching variable to evaluate one of its values. The two main aims of look-ahead are to choose a variable to evaluate next and the order of values to assign to it.'

Also, and slightly more technically, a Forward Algorithm is 'The forward algorithm, in the context of a hidden Markov model (HMM), is used to calculate a 'belief state': the probability of a state at a certain time, given the history of evidence. The process is also known as filtering. The forward algorithm is closely related to, but distinct from, the Viterbi algorithm.'

The Look-Ahead And Forward Algorithms - What Do I Need To Know?

Okay - what does the above mean? Basically, the translation AI looks for a full stop. Once it finds the full stop, it translates the sentence.

No really - what? You need to add grammar, primarily full stops and comma’s so that the text-to-text translation AI works properly

To do this, make sure you check your transcript for grammatical correctness. This is shown below. Practically this means:

1. Add lots of full stops everywhere! Humans are very good at understanding each other, so we don’t need to be told where one sentence ends and where another sentence starts. AI’s - not so much.

2. If you watch the video embedded below, the sentences have all been converted into shorter sentences which are in third person. An AI can translate the shorter sentences way better, but take care to not loose meaning.

3. It is expected that a human subject matter expert translator will do post editing after the AI has done its first pass. However, simplifying means that you can translate into many languages quickly, and the post-editing can be reduced. Use your judgement!

4. The grammar was changed - shown below. If you watch through the video below, you will notice the open captions do not match what is being said. Note that this step is being taken primarily in preparation for the translation step.

   

5. After the process was complete we add an Auto-Overlay so you can see the full outcome. Note that the words are a little bit different, tense had been changed to past tense for everything and sentences are shorter.

6. Very cool. Now we can begin the translation process. Nicely done!

Translating The Content To Hindi

1. So the first thing we need to do is translate the content. Click Action -> Translate and translate this video into Hindi.

   

2. It’ll take a few seconds and then you will see the translated file in your Root pane. It is hightlighted in yellow in the below image.

   

3. Open it up, and in the Captions tab you will be able to see the translation. In the below image, note that the colour of the text has been set to black so it is easier to see for your post-editing.

   

4. When you do your own video, please ensure you do post editing! Remember, use the AI for the heavy lifting, but use human subject matter experts for the best results.

5. Below, we use the Auto-Overlay so you can see the output. We are using a black text with a yellow highlight at transparency of 80%. The little spinner indicates that the app is applying the Auto-Overlay to the video asset. This is what creates the open captions in the video.

   

6. This is a direct AI translation so some errors are expected. Also, note that the timings are a little bit off. What is happening in the background is our code is trying to do a best fit of the sentences, but manual edits are often required. Please do this as part of the post-editing process.

7. Well done! Your video is now translated. But you thought it was going to speak Hindi?

AI Dubbing In Hindi - Now The Magic Starts!

This is the simple version - we can get much more sophisticated in our outcome, including voices that sound like young or older people, men or women, and changing the volume and speed of the speech.

It is also possible to break the conversation up into different blocks of speech, properly matching the English content spoken.

Drop us an email at hello@videotranslator.ai if you are curious and have questions. Or try the free-trial!

1. Copy the captions from the Video component into the Audio component.

   

2. Now we will use a Text-To-Speech AI to 'speak-out' the content in Hindi. Use the Action -> Transcribe to access the Text-To-Speech AI as shown below. This will give us the AI dubbing.

   

3. Abracadabra!

Next Steps - How To Implement Your Own AI Dubbing

There are many options you have once you AI dubbing is complete. Several clients simply prefer to download the asset and work on it in their preferred audio/video editor. However, much of the same functionality is also present in the app.

1. Mute the original audio soundtrack using the ‘minus’ button as shown below.

   

2. Generally it is recommended you slow-down the AI dubbing. This is because it is an AI, there is limited scope for tone. People expect changes is tone when talking to each other. An AI is unable to do this so when there is less tone, it helps to slow down the speech to about 90%. This helps people to understand the words.

3. When you insert the AI speech into the video as an Audio-Overlay use the 'plus' button to add the AI dubbing *.mp3 file.

   

4. Above, the highlights show (a) the plus button at the bottom to add additional audio overlays, (b) the original soundtrack muted and new soundtrack added, (c) the new soundtrack has its speed reduced to 80%, and (d) start and end time can be managed on a per conversation block basis.

5. The final outcome is below. This has several errors, and needs more work. To see the error forward to 1:57 and you will hear the sound stops. This is because we have just used one mp3 Voice Overlay for simplicity to explain the workflow.

6. Congratulations! We hope you found this post useful. This is quite a simplified workflow - let us know if you have any questions.

How To Think About This

• Generally, you would break up the transcript into multiple conversation blocks. This helps space out the AI speaking nicely.

• Select the appropriate gender of the AI if there is a conversation happening between a man and a woman.

• If two men or two women are speaking in a conversation, use multiple AI voices. This helps your viewers understand what is happening.

• Remember, the key win is SEO. When deploying the asset to some social media channel like LinkedIn or YouTube, make sure to translate the relevant metadata. This helps search engines index your content making it reachable in other languages.

Conclusion

Please connect with us on LinkedIn, YouTube or Facebook for any comments, questions, or just to keep up to date with the work we do!

We are very grateful for your support!

Curious about how we could help your business? Check out our managed service, or try our app for free! Alternately drop us an email at hello@videotranslator.ai.