The box dimension is not countably stable, so even one exceptional point could drive it up. For example, given $\epsilon>0$, the function $f(x)=x^{1/2} \sin(1/x)$ on $[0,1]$, with $f(0)=0$, is locally Lipschitz on $(0,1]$, yet the graph of $f$ has box dimension at least $1.25$. Indeed, for $j<k$, to cover the graph above $[1/(2\pi(k+j)), 1/(2\pi(k+j+1))]$, at least $ck^{3/2}$ squares of side length $1/(4k^2)$ are needed. Summing over $j<k$ we get $c_1 k^{5/2}$ squares of side length $1/(4k^2)$.
You can get a result of the type you want by working with the Hausdorff or packing dimension of the graph, as these notions are stable under countable union.
In order to prove the packing dimension of the graph of $f$ is at most $2-\alpha$, it suffices to assume that the exceptional set (where the local$\alpha$-H"older condition fails to hold) has packing dimension at most $1-\alpha$. [This condition is sharp, unless you make further assumptions on the behavior of $f$ on the exceptional set.]
To see this, partition the graph of $f$ to the part over the exceptional set (that clearly has packing dimension at most $2-\alpha$, and for each integer $m$, the part $A_m$ where the $\alpha$-H"older condition holds with constant $m$. The usual argument you referred to shows that for each $m$, the box dimension of $A_m$ is at most $2-\alpha$. Then use the connection between packing dimension and Box dimension (see, e.g. [1]. [2] or [3]) to conclude.
[1] Falconer, Kenneth. Fractal geometry: mathematical foundations and applications. John Wiley & Sons, 2004.
[2] Mattila, Pertti. Geometry of sets and measures in Euclidean spaces: fractals and rectifiability. No. 44. Cambridge university press, 1999.
[3] Bishop, Christopher J., and Yuval Peres. Fractals in probability and analysis. Vol. 162. Cambridge University Press, 2017.