BLOGS
The Best Way to Measure Translation Quality with Metrics
November 30, 2021
How can you maintain high-quality translated content year on year? A solid approach to translation quality metrics can help you out.
When you're managing localization for a large global company, it can be hard to know if your translations are really "good." You invest a lot of ongoing budget translating content for your international markets, but you don't know how to tell if the quality matches the investment.
You create the content, you get it translated, you send it out… and then you just have to hope that the translations are suitable.
Sometimes your in-country reviewers send back lots of revisions. Sometimes they don't. But, you can't tell what that means for the overall quality of the translations.
The best way to measure translation quality is by approaching quality metrics in the right way.
What is translation quality?
Translation quality is a measure of how well translated content meets its intended requirements. It can incorporate both objective aspects like grammar, punctuation, and spelling, and subjective aspects like style, purpose, and tone. There is no definitive method for measuring translation quality.
Across the translation industry, people describe quality in various ways. One group of researchers, when attempting to derive an objective measure of translation quality, admitted that they couldn't even agree amongst themselves how the term should be defined!
For most companies, the question of translation quality comes up when there are problems.
You might assume that quality issues occur when a translator has made a mistake. However, while such mistakes can happen, this type of quality error is relatively uncommon when you are using competent translators.
It's much more common for quality issues to crop up in other ways. For example, if a particular translation would cause a compliance issue for your company it's a quality issue even if the translation is linguistically perfect.
Translation quality metrics and how to track them
Before you can find the right quality metrics for you, you first need to identify your company's unique quality requirements.
When we start a new project with a client, we ask them a lot of questions to help us identify the requirements of their particular piece of content.
We ask questions like:
- Who is the intended audience of the content?
- What's the communicative purpose of the content?
- Will it only be used internally or will customers consume it?
- Is there specific terminology that must be translated in a specific way?
- What locales will the translated content be used in?
We then identify metrics that will be relevant for the specific content based on the answers.
This is what the Multidimensional Quality Metrics (MQM) framework calls a "functionalist" approach to quality. This framework provides a vocabulary for categorizing translation quality issues. The project is headed by the European Commission and aims to improve mass machine translation of important documents.
MQM provides a set of categories for translation quality issues, including:
- Accuracy
- Style
- Fluency
- Locale Conventions
- Terminology
- and others
If you're struggling to select metrics for your content, your translation provider should be able to help you identify suitable ones.
Check the source! Why quality management is everyone's job
Whose responsibility is it to avoid translation quality issues? Is it just the translator's job?
While your translation provider certainly holds a lot of responsibility for avoiding quality issues, they cannot do it alone.
At Rubric, we find that a lot of quality issues originate with the source content. You need to create content so that it works well with translation. Authoring your content with translation in mind helps reduce quality issues down the line.
Susannah Eccles, Rubric's Operations Director, describes how we used metrics with one client to reduce quality issues:
"We often use metrics to gather data to illustrate particular problems to clients."For example, one client had 7 rounds of reviews for their translated content. Usually, most of the changes should be made during the first review round because that's the lowest cost point. However, in many of the company's markets, they weren't submitting changes until review round 3 or later."The management wasn't really aware of this problem. By gathering data about how many markets were doing each review round and how many changes were being made in each round, we were able to illustrate the problem and then help the localization manager fix that issue."This improved the process, reduced the cost of translation, and improved the Translation Memory quality."
The Translation Memory (TM) is a database of all past translations made by your company. Submitting the changes in earlier review rounds means that the TM only stores translations that had been properly reviewed. This helps to ensure that low-quality translations are not saved.
When your content creators understand the translation process, they can also create the content in ways that work well with translation.
How to evaluate translation quality in your business
There are many ways you could start tracking and evaluating translation quality in your business.
However, a good question to get yourself started is "Why do we want to evaluate translation quality?"
As Susannah's client example above shows, quality tracking often starts when an existing problem is raised. If everything about your translated content is working perfectly, there is probably no need to start tracking new metrics just for the sake of it.
For example, perhaps you feel like you always receive lots of change requests from a particular reviewer…
In this case, you might work with your translation provider and start by tracking:
- The number of change requests from all reviewers to get a comparison.
- During which review rounds the changes arise.
- The types of changes.
You might then evaluate the data. From this, you might decide to track other metrics, such as those related to accuracy, style, or other factors.
Finally, you would use the data to identify where the problem actually lies. Perhaps the translator needs to talk to the original content creator. Perhaps the reviewer requires some guidance in how to review effectively. Perhaps there is another problem that is causing the issues.
Only when you have seen the data will you know what to do next.
The key to ongoing translation quality control
Translation quality metrics are not like other localization metrics, which you might track for many years just to keep an eye on your translation process.
With quality metrics, you should usually be using them to identify and solve a particular process problem.
This way, you and your translation provider can continually improve the quality of your localization process. In turn, this will improve the quality of all your translated content.
If you would like some help identifying the best localization metrics for your business, fill in this form and one of our strategists will be happy to give you some advice.
And for more tips, see our eBook on streamlining your localization process.
DOWNLOAD OUR FREE EBOOK
10 Profitable Tips to Streamline Your Translation Workflow
Like any manufacturer, you know that a streamlined workflow is the key to a successful business. However, the idea of having a “streamlined translation workflow” might seem unusual.