If we were not to worry about loss of precision, we could in principle make $h$ as small as possible. However, due to the computed expression in the above program example $$f_0''=\frac{ f_h -2f_0 +f_{-h}}{h^2}=\frac{ (f_h -f_0) +(f_{-h}-f_0)}{h^2},$$ we reach fairly quickly a limit for where loss of precision due to the subtraction of two nearly equal numbers becomes crucial.

If $(f_{\pm h} -f_0)$ are very close, we have $(f_{\pm h} -f_0)\approx \epsilon_M$, where $|\epsilon_M|\le 10^{-7}$ for single and $|\epsilon_M|\le 10^{-15}$ for double precision, respectively.

We have then $$\left|f_0''\right|= \left|\frac{ (f_h -f_0) +(f_{-h}-f_0)}{h^2}\right|\le \frac{ 2 \epsilon_M}{h^2}.$$