Open Captions: Enhancements November 2019

Open Captions: Enhancements November 2019

> Open Captions: Enhancements November 2019 Tat Banerjee [November 20, 2019]

Working with a prospect, we added a number of enhancements to our open captions workflow. If you are interested in what open captions are and what its use cases are please have a look at this blog post.

Very simply open captions are generally used as an accessibility tool focused on people who have hearing impairments. More recently due to the rise of streaming services, open captions have been making a come back.

  • As an example, a simple use case would be a new mum at home with her child. If the child is sleeping, the mum can watch video content (on TV or online) with open captions on mute.

  • Another example is people watching video on a social media platform, where the video content is often auto-played but in a muted environment.

We have added one simple enhancement to the Auto-Overlay functionality. The new functionality allows a user to add the open captions anywhere in the video.

We will be using the same RSTV sample which we worked with in an earlier blog post. To see how we used an AI to caption the Indian English video, please read this blog post.

Context + The Enhancement

A client we have been working with for a while had the following requirement. He wanted to add the open captions in a specific place in the video. Historically this was possible but was very hard to use effectively.

The enhancement is that we now allow X and Y coordinates to be placed, allowing the user to have fine grained control over the placement of the overlay.

What Used To Be Possible?

Assuming you followed the steps in this visual guide, you would be aware we can add open captions using the Auto-Overlay feature.

Basically we used the Auto-Overlay feature as shown below. This is a two step process - we already have the captions from the previous steps.

  1. Use Action -> Auto Overlay and choose a text colour, size and font. We have chosen black colour text, with font size 36 and font Ubuntu.

    Open Captions: Step 1 in the Auto Overlay
    Open Captions: Step 1 in the Auto Overlay
  2. In the second step, it used to only be possible to choose from three options Top, Centre and Bottom. This is shown below.

    Open Captions: Step 2 in the Auto Overlay
    Open Captions: Step 2 in the Auto Overlay
  3. It is important to understand what is happening here. Its basically a mathematical function called a text writer that is writing-out (or baking-in) the open captions into the video. The yellow highlight was probably not the best choice but its useful in demo.

    Open Captions: After the Auto Overlay process has finished

  4. Ok - now we are going to make some changes. First we use Action -> Remove Auto-Overlay to take away the open captions as we want to move them around. This looks like the below image.

    Open Captions: Remove the Auto Overlay
    Open Captions: Remove the Auto Overlay

How Has Auto-Overlay Changed?

The change is slightly different for (a) the stripe, and (b) the highlight in the Auto-Overlay function. First, lets talk about the stripe and then we will look at the highlight.

Use the Action -> Auto-Overlay and we use the pre-filled values for Step 1. The below images are from the point-of-view of Step 2.

No Stripe Or Highlight

In this use case we don’t want a stripe or a highlight. This is shown in the image below.

Open Captions: No Stripe or Highlight
Open Captions: No Stripe or Highlight

In this case, the video output looks like below. As you can see, depending on the colour scheme of the underlying asset the lack of a stripe or highlight may not offer the best user experience.

Open Captions: No Stripe or Highlight

Using A Stripe

In this case we want a stripe. The idea of the stripe is a bar across the screen. Here there are two options, show the stripe always, or show it only where there is text on the screen.

First of course, go ahead and remove the no-stripe/no-highlight overlay from your asset. Then, trigger the Auto-Overlay process again.

Text Only Vs No Text Only

Why would you use this? Theuse case is to do with the speed of the speech in your video.

  • In the case where the speech is very quick, with minimal breaks in the speaking, we would recommend No Text Only
  • Alternately, if there are gaps in the speech, use Text Only.

The idea is when the underlying speech is relativity fast the flickering of the stripe looks terrible, so you want the stripe to be in place irrespective of whether the text is showing or not showing.

The below example is of a Stripe with Text Only and we are using the blue colour.

Open Captions: Stripe with Text Only
Open Captions: Stripe with Text Only

This does not look amazing but you can see the effect of the Stripe with Text Only.

Open Captions: Stripe with Text Only

Using A Highlight

In this case we want to use a highlight. The idea here is a highlight around the text in question. Using the highlight option looks like below. Text only is not a thing with the highlight for obvious reasons.

Open Captions: Highlight
Open Captions: Highlight

In the video below, we used the Centre option to make it very obvious what is happening. Transparency is also turned down for the highlight itself.

Open Captions: Highlight but centred with no transparency

The Big Change: Advanced Options - Stripe

The stripe example is the slightly more complicated of the two options. The first thing to note is the actual dimensions of your video. In our case, the video is 640 x 360 pixels. This is shown in the image below.

Open Captions: Work out the dimensions of your video
Open Captions: Work out the dimensions of your video

Add a stripe with the below advanced settings. Note that these numbers are actually wrong, but very useful to understand what is happening.

Open Captions: Add X and Y coordinates
Open Captions: Add X and Y coordinates

What we are doing here is setting X and Y coordinates for the text at 20, 20. We are also setting the X and Y for the stripe at 40, 40. Normally you would set the stripe to X and Y coordinates similar to the text.

Open Captions: Stripe with incorrect advanced options

Ok - so that was nice, but not really useful. Lets try again with the below settings.

Open Captions: Add different X and Y coordinates
Open Captions: Add different X and Y coordinates

This looks much better. The idea here is to put the stripe at the bottom of the video. But we have made a mistake in the alignment of the text.

Open Captions: Stripe with less incorrect advanced options

With the examples above, hopefully it makes sense how to use this functionality.

Important: If you are using the stripe, please note that the height of the stripe is 10% of the height of the video itself. You may need to manually fix up the captions so that all the captions are single lines only.

The Big Change: Advanced Options - Highlight

Highlights are easier to use than the Stripe - mostly because we only require one set of X and Y coordinates. The setup used is shown below.

Open Captions: Add X and Y coordinates - just one set for the text
Open Captions: Add X and Y coordinates - just one set for the text

This results in the following video using the highlight. Please note this setup is not optimised and make the changes required on your side.

Open Captions: Highlight is much easier to use!

Important: If you are using the highlight, it is well worth making sure that each sentence is roughly the same length and centring the open captions into the video nicely.

Conclusion

We hope that this visual guide is helpful to you! Open Captions are very useful, and a critical part of the value proposition our video translator offers to clients.

If you are interested in trying out our technology, please try our platform or drop us an email at support@q6a.com.au.