\hypertarget{dynamic__training__optimization_8h}{}\doxysection{models/dynamic\+\_\+training\+\_\+optimization.h File Reference}
\label{dynamic__training__optimization_8h}\index{models/dynamic\_training\_optimization.h@{models/dynamic\_training\_optimization.h}}


This is Dynamic Training Optimization for Neural Network.  


This graph shows which files directly or indirectly include this file\+:
\nopagebreak
\begin{figure}[H]
\begin{center}
\leavevmode
\includegraphics[width=220pt]{dynamic__training__optimization_8h__dep__incl}
\end{center}
\end{figure}


\doxysubsection{Detailed Description}
This is Dynamic Training Optimization for Neural Network. 

Copyright (C) 2020 Parichay Kapoor \href{mailto:pk.kapoor@samsung.com}{\texttt{ pk.\+kapoor@samsung.\+com}}

\begin{DoxyDate}{Date}
4 January 2021 
\end{DoxyDate}
\begin{DoxySeeAlso}{See also}
\href{https://github.com/nnstreamer/nntrainer}{\texttt{ https\+://github.\+com/nnstreamer/nntrainer}} 
\end{DoxySeeAlso}
\begin{DoxyAuthor}{Author}
Parichay Kapoor \href{mailto:pk.kapoor@samsung.com}{\texttt{ pk.\+kapoor@samsung.\+com}} 
\end{DoxyAuthor}
\begin{DoxyRefDesc}{Bug}
\item[\mbox{\hyperlink{bug__bug000165}{Bug}}]No known bugs except for N\+YI items \end{DoxyRefDesc}
Dynamic training aims to optimize the cost of applying the gradient. The cost of applying the gradient includes the cost of the optimizer (adam, etc) where the optimizer variables are updated, and the cost of actually updating the weights (which can be non-\/trivial with bigger weights and distributed training).

There are two supported modes\+:
\begin{DoxyEnumerate}
\item Gradient Mode\+: The already calculated gradient is used to estimate if this gradient must be used to update the weight, or if this update must be skipped.
\item Derivative Mode\+: This mode tries to estimate an approximate gradient with low cost in order to save the cost of calculating gradient. This cost of calculating gradient is wasted if the gradient is not going to be applied.
\end{DoxyEnumerate}

There are two supported reduction operations which reduce the gradient and the weight to a single value in order to compare it with a threshold. If the reduced value is less than threshold, the update is performed with some probabilty proportional to the value. If the reduced value is higher than threshold, then the update is always performed. 