Here is a partial positive answer, going in the opposite direction of Terry´s comment.

Let $X$ be an infinite topological space. Suppose $X$ is first countable at $p \in X$ and for every open neighborhood $U$ of $p$ there is a smaller neighborhood $V \subseteq U$ such that $|U \setminus V|=|U|$. Then there is a group structure on $X$ with identity $p$ such that the operation $(x,y) \mapsto xy^{-1}$ is continuous at $(p,p)$.

Just note that (starting with a $U_0$ of minimal cardinality) we can find $\{U_n : n \in \omega\}$ an open local base at $p$ such that for every $n \in \omega$ we have $U_{n+1} \subset U_n$ and $|U_n \setminus U_{n+1}|=|U_0|$. Now find (e.g. using compactness of first-order logic) a group $G$ containing subgroups $H_0 \supset H_1 \supset H_2 \supset \cdots$, such that $|G|=|X|$ and $|H_n \setminus H_{n+1}|=|H_0|=|U_0|$ for every $n \in \omega$, and then transfer the group structure to $X$ in the obvious way.

Note that we can also change the hypothesis to: *there is an open neighborhood of $p$ with no smaller neighborhoods* (e.g. $p$ is isolated) and the conclusion still holds; just call $U_0$ such neighborhood, find $G$ and $H_0$ as before and ignore the rest.