If you think of elements in a local Artin $k$-algebra $R$ as, say functions on the origin of $k^n$ which remember some (finite amount of) higher order information in the various $n$ directions, then a small extension $R'$ of $R$ is just another such ring with functions that remember "at most one order higher".
For example, let $R = k[x,y]/((x,y)^2)$ and consider the small extension $R' = k[x,y]/((x,y)^3)$. The elements of $R$ are functions which remember up to 1st order in the directions $x,y$. The elements of $R'$ are what we get if we take functions from $R$ and stick on some 2nd order terms in the $x,y$.
You can also just extend only in one direction. For example, $R'' = k[x,y]/(x^3,xy,y^2)$ is also a small extension of $R$, but the only 2nd order term we added was $x^2$, not $xy$ nor $y^2$.
However, if you take $A = k[x]/(x^2)$ and $A'' = k[x]/(x^4)$, then this is not a small extension because we went two orders up, from 1st order to 3rd order.
In short, I interpret a small extension as one that only thickens our fat point by an order of at most 1 in each direction.