eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Since their initial development in the seminal AI research paper Attention Is All You Need, transformer-based architectures have completely redefined the field of Natural Language Processing (NLP) and set the state of the art for numerous AI benchmarks and tasks.
What are transformer models? They’re an advanced artificial intelligence model that has benefited from an “education” the likes of which some dozen humans might gain in a lifetime.
Transformer architectures are typically trained in a semi-supervised manner on a massive amount of text—think English Wikipedia, thousands of books, or even the entire Internet. By digesting these massive corpora of text, transformer-based architectures become powerful language models (LM) capable of accurately understanding and performing predictive analytics based on textual analysis.
In essence, this level of exhaustive training allows transformer models to approximate human text cognition – reading – at a remarkable level. That is, not merely simple comprehension but (at best) making upper level connections about the text.
Recently, it has been shown that these impressive learning models can also quickly be fine-tuned for upper level tasks such as sentiment analysis, duplicate question detection, and other text-based cognitive tasks. Additional model training on some separate dataset/task relative to what the model was originally trained on allows the parameters of the network to be slightly modified for the new task.
More often than not, this results in better performance and faster training than if the same model had been trained from scratch on the same dataset and task.
Also see: Top 10 Text Analysis Solutions
Benefits of Transformer Models
1) Great with Sequential Data
Transformer models are excellent at dealing with the challenges involved with sequential data. Because of this, they act as an encoder-decoder framework, where data is mapped to a representational space by the encoder. Then they are mapped to the output by way of the decoder. This makes them scale well to parallel processing hardware like GPUs – a processor that is super-charged to drive AI software forward.
2) Pre-Trained Transformers
Pre-trained transformers can be developed to quickly perform related tasks. This is because transformers already have a deep understanding of language, which allows training to focus on learning whatever goal you have in mind. For example, named-entity recognition, language generation, or conceptual focus. Their pre-training makes them particularly versatile and capable.
3) Gain Out-of-the-Box Functionality
By fine-tuning your pre-trained transformers, you can gain high performance out of the box, without enormous investment. In comparison, training from scratch would take longer, and use orders of magnitude more compute and energy just to reach the same performance metrics.
4) Sentiment Analysis Optimization
Transformer models enable you to take a large-scale LM (language model) trained on a massive amount of text (the complete works of Shakespeare), then update the model for a specific conceptual task, far beyond mere “reading,” such as sentiment analysis and even predictive analysis.
This tends to result in a significantly better performance because the pre-trained model already understands language really well, so it just has to learn the specific task, versus trying to learn both language and the task at the same time.
Looking Ahead: Redefining the Field of NLP
Since their early emergence, transformers have become the de facto standard for tasks like question answering, language generation, and named-entity generation. Though it’s hard to predict the future when it comes to AI, it’s reasonable to assume that transformer models bears close focus as a next-gen emerging technology.
Most significant, arguably, is their ability to allow machine learning models to not only approximate the nuance and comprehension of human reading, but to far surpass human cognition at many levels – far beyond mere quantity and speed improvements.
About the Author:
Dylan Fox is the CEO of AssemblyAI
