NLP guide towards neural networks

Beginning the journey. This is just intro to refresh. Intended for beginners.

** Statistical Language models

Over the years language related task was used using different techniques. Statistical model was one of the famous model which was used by Google before transforming into different deep learning models. The LM (Language modelling) is based upon probability for next word based upon the given corpus.

differences source:scalr.ai

* * Neural Language models (NLM)

Recently with the better outcome the use of neural network in the development of ml model build has become a preferred choice. NLM is when neural network is being used in natural language processing.

Heuristics for NLM

Size matters. The best models were the largest models, specifically number of memory units.
Regularization matters. Use of regularization like dropout on input connections improves results.
CNN vs Embedding. Character-level Convolutional Neural Network (CNN) models can be used on the front-end instead of word embedding, achieving similar and sometimes better results.
Ensembles matter. Combining the prediction from multiple models can offer large improvements in model performance.