Read on Twitter
Overfitting sucks.
Here are 7 ways you can deal with overfitting in Deep Learning neural networks.
https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread">https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">
Here are 7 ways you can deal with overfitting in Deep Learning neural networks.
https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" title="Overfitting sucks.Here are 7 ways you can deal with overfitting in Deep Learning neural networks.https://abs.twimg.com/emoji/v2/... draggable="false" alt="🧵" title="Thread" aria-label="Emoji: Thread">https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" class="img-responsive" style="max-width:100%;"/>
A quick reminder:
When your model makes good predictions on the same data that was used to train it but shows poor results with data that hasn& #39;t seen before, we say that the model is overfitting.
The model in the picture is overfitting.
https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">
When your model makes good predictions on the same data that was used to train it but shows poor results with data that hasn& #39;t seen before, we say that the model is overfitting.
The model in the picture is overfitting.
" title="A quick reminder: When your model makes good predictions on the same data that was used to train it but shows poor results with data that hasn& #39;t seen before, we say that the model is overfitting.The model in the picture is overfitting. https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" class="img-responsive" style="max-width:100%;"/>
The more data you feed the model, the more likely it will start generalizing (instead of memorizing the training set.)
Look at the relationship between dataset size and error.
(Unfortunately, sometimes there& #39;s no more data.)
Train your model on more dataThe more data you feed the model, the more likely it will start generalizing (instead of memorizing the training set.)Look at the relationship between dataset size and error.(Unfortunately, sometimes there& #39;s no more data.)https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="1⃣" title="Tastenkappe Ziffer 1" aria-label="Emoji: Tastenkappe Ziffer 1"> Train your model on more dataThe more data you feed the model, the more likely it will start generalizing (instead of memorizing the training set.)Look at the relationship between dataset size and error.(Unfortunately, sometimes there& #39;s no more data.)https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" class="img-responsive" style="max-width:100%;"/>
You can automatically augment your dataset by transforming existing images in different ways to make the data more diverse.
Some examples:
You can:
The more complex your model is, the more capacity it has to memorize the dataset (hence, the easier it will overfit.)
Simplifying the model will force it to generalize.
Make your model simplerYou can:https://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of layershttps://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of weightsThe more complex your model is, the more capacity it has to memorize the dataset (hence, the easier it will overfit.) Simplifying the model will force it to generalize.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="3⃣" title="Tastenkappe Ziffer 3" aria-label="Emoji: Tastenkappe Ziffer 3"> Make your model simplerYou can:https://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of layershttps://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of weightsThe more complex your model is, the more capacity it has to memorize the dataset (hence, the easier it will overfit.) Simplifying the model will force it to generalize.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">">
Make your model simplerYou can:https://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of layershttps://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of weightsThe more complex your model is, the more capacity it has to memorize the dataset (hence, the easier it will overfit.) Simplifying the model will force it to generalize.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="3⃣" title="Tastenkappe Ziffer 3" aria-label="Emoji: Tastenkappe Ziffer 3"> Make your model simplerYou can:https://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of layershttps://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of weightsThe more complex your model is, the more capacity it has to memorize the dataset (hence, the easier it will overfit.) Simplifying the model will force it to generalize.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">">
Make your model simplerYou can:https://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of layershttps://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of weightsThe more complex your model is, the more capacity it has to memorize the dataset (hence, the easier it will overfit.) Simplifying the model will force it to generalize.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="3⃣" title="Tastenkappe Ziffer 3" aria-label="Emoji: Tastenkappe Ziffer 3"> Make your model simplerYou can:https://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of layershttps://abs.twimg.com/emoji/v2/... draggable="false" alt="▫️" title="Weißes kleines Quadrat" aria-label="Emoji: Weißes kleines Quadrat">Reduce the number of weightsThe more complex your model is, the more capacity it has to memorize the dataset (hence, the easier it will overfit.) Simplifying the model will force it to generalize.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">">
This is known as "Early Stopping."
Identify when overfitting starts happening and stop the learning process before it does.
Plotting the training and validation errors will give you what you need for this.
Stop the learning process before overfittingThis is known as "Early Stopping."Identify when overfitting starts happening and stop the learning process before it does. Plotting the training and validation errors will give you what you need for this.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="4⃣" title="Tastenkappe Ziffer 4" aria-label="Emoji: Tastenkappe Ziffer 4"> Stop the learning process before overfittingThis is known as "Early Stopping."Identify when overfitting starts happening and stop the learning process before it does. Plotting the training and validation errors will give you what you need for this.https://abs.twimg.com/emoji/v2/... draggable="false" alt="👇" title="Rückhand Zeigefinger nach unten" aria-label="Emoji: Rückhand Zeigefinger nach unten">" class="img-responsive" style="max-width:100%;"/>
Smaller weights can result in a model less prone to overfit.
Rescaling input data is a way to constraint these weights and keep them from increasing disproportionally.
Dropout is a regularization method that randomly ignores some of the outputs of a layer.
This simulates the process of training different neural networks with different architectures in parallel, which is a way to avoid overfitting.
https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/">https://machinelearningmastery.com/dropout-f...
Use DropoutsDropout is a regularization method that randomly ignores some of the outputs of a layer.This simulates the process of training different neural networks with different architectures in parallel, which is a way to avoid overfitting. https://machinelearningmastery.com/dropout-f..." title="https://abs.twimg.com/emoji/v2/... draggable="false" alt="6⃣" title="Tastenkappe Ziffer 6" aria-label="Emoji: Tastenkappe Ziffer 6"> Use DropoutsDropout is a regularization method that randomly ignores some of the outputs of a layer.This simulates the process of training different neural networks with different architectures in parallel, which is a way to avoid overfitting. https://machinelearningmastery.com/dropout-f..." class="img-responsive" style="max-width:100%;"/>
These refer to a technique that penalizes the loss function to keep the weights of the network constrained.
This means that the network is forced to generalize better because it can& #39;t grow the weights without limit.
Is there anything else you use to prevent your models from overfitting?
