Understanding the basics of machine learning based NLP

machine learning based NLP 

In our last article we discussed the heuristic-based approach to NLP. In this article we will understand how machine learning is playing a key role on solving NLP challenges. Like all machine learning problems are applied for other data problems, NLP has a similar approach. On a broad level, we can breakdown machine learning-based NLP task into following categories.

Supervised and unsupervised learning in NLP:

First is supervised machine learning. Supervised learning enables a model to learn based on historical data. Based on this learning, the model produces an output based on data it has not seen. Machine learning techniques such as classification and regression are also used for NLP. An example of supervised learning is when an ML model can learn the classification of news topic based on analyzing the text.

Next, let's talk about unsupervised machine learning. Unsupervised learning enables a model to analyze and cluster unlabeled data. Unsupervised algorithms are given data and they can discover hidden patterns or data grouping without the additional human intervention.

An example of unsupervised learning is when machine learning model goes through unlabeled set of data and is able to categorize that into groups. NLP unsupervised problem-solving is tougher than the supervised learning. We will now look at the overall steps that are involved when leveraging an ML-based approach to NLP.

How machine learning models applies to NLP?

Let's go over the high-level process of any machine learning approach. Any ML approach for NLP, supervised or unsupervised, consists of three common steps. First is extracting features from text. Based on the desired output, we extract the required feature from text. Based on this text, the machine learning model will learn. Next is using feature representation to learn a model.

Once you have done the extraction then for model to learn we will use feature representation. The model will learn based on its ability to learn highly abstract representation of data. And finally, we'll do the evaluation and improving of the model.

Followed by the feature representation and extraction, we will evaluate the efficiency of the model and work towards improving the results. This is an iterative process and will apply in almost all scenarios where machine learning is used for NLP.

Now that we have understood the overall approach to how machine learning can be applied to natural language processing, in the next article, we will look at some of the commonly applied machine learning models. We will briefly discuss some of the commonly used supervised machine learning model used in NLP.

We will discuss this in greater detail in the following article as well. The following models are Naïve Bayes, support vector machines, Hidden Markov Models, and conditional random fields. Let's discuss each of these topics one by one.

Naïve Bayes model:

The Naïve Bayes algorithm relies on Bayes theorem to calculate the probability of a class label belonging to a set of feature in an input data. In the example of news classification, there is an assumption that the word count of domain specific labels are not correlated with one another. That is, the word count are independent.

Support vector machine:

Naïve Bayes is a great starting algorithm as it's simple to understand and fast to train and run. Next is Support Vector Machine. Support Vector Machine is one of the most popular classification algorithm which works in principle of learning a decision boundary that act as a separator between various categories of text.

In the example of news classification it could differentiate between sports and finance, and this can be done using a linear or a non-linear decision boundary. The boundary of the support vector machine should be such that the distance between point across classes is at maximum.

The strength of SVM is that the models are robust even with variations and noise in the data. It is built to handle high dimensional data or high feature data, which is common in NLP. The weakness of the SVM is that it takes lot of time to train and hence scaling when there is a large amount of training data which becomes difficult.

Hidden Markov model:

The other problem is that the results are not explainable. The next model is Hidden Markov model. Hidden Markov model is a statistical model that assumes that there is a hidden state that generates data and that only once the data is generated we can try to model the hidden states.

For instance in POS tagging which assigns part of speech to sentences, Hidden Markov models are used to tag text data in the sentence to part of speed such as subjects, verbs, objects and adverbs. The assumption made here is that each hidden state, which would be part of speech in our case here, is dependent on the previous state.

Conditional Random Fields:

The next model is Conditional Random Fields or CRF as they are properly known to have the ability to model sequential data. Conditional Random Fields outperform Hidden Markov model which rely mostly on sequential nature of language.

For the example of POS tagging, CRFs performed better than Hidden Markov model as CRFs can tag any word by classifying them individually from the pool of all POS tags. We now have a brief understanding of machine learning and how that can be used for NLP application. In the next article we'll try to understand how deep learning can be used for NLP.

Comments

Post a Comment

Popular posts from this blog

What are some ethical implications of Large Language models?

Image
Ethical consideration in LLMs In this article, we're going to address the ethical challenges surrounding the usage of large language models, including potential biases and privacy issues.  Bias in AI system: AI systems, particularly, will deliver biased results. There's no other way to get around it. Search engine technology, for example. It's not neutral, processes a large amount of data, and prioritizes results based on clicks and user preferences. So, a search engine can become essentially an echo chamber. And you can see that it will uphold certain biases of the real world, potentially because looking at the clicks and the user preferences of a majority of users using that search engine. And this is how we can come up with prejudices and stereotypes that are furthered online because of these systems that are in place. So, when we talk about bias in AI, it could be something like as simple as a gender bias, right? So, gender biases should be avoided, but it can't sho...
Read more

Introduction to the fine tuning in Large Language Models

Image
Fine tuning in LLM In this article, we're going to delve into the techniques used to adapt a generic pre-trained model for a specific task or application. Now, pre-trained language models can do really impressive stuff just immediately off the shelf, including things like text generation, summarization, and even providing levels of code. But LLMs are not one-size-fits-all as you can imagine and maybe I want to specialize it even further after the model has been trained. I'd like to make it really good at answering questions based on certain topics such as medicine or travel or code or something. This is what the idea of fine-tuning comes in.  Methods used in Fine tuning: Fine-tuning a large language model refers to the process of taking a pre-trained language model or a model that's already been trained on a large dataset and further training it on a smaller task-specific dataset. Now this process will allow the model to adapt and specialize its capability. And if we take t...
Read more

Understanding the basic architecture of LLMs

Image
Architecture of LLMs In this article, we're going to discover the key components in general architecture that make up typical large language models.  Introduction to LLM Architecture: Now, large language models will have things like the transformer neural networks at their very underlying pieces of it. Remember, these transformers are just a large number of parameters are attached to this transformer model and these LLMs are able to then understand and generate accurate responses because it's using this type of neural network at its core. In fact, what you're going to see is that these LLMs are referred to as foundation models and it's all it means is a foundation model is so large and impactful that it serves as its own class of artificial intelligence so that we can start experimenting with these types. Transformer Model Structure: It's just a way for us to classify these models that are being used today. Now, what exactly is a transformer model? We'll get to ...
Read more