This is known as Tietze theorem: if $A$ is an open connected set such that for every boundary point there is a locally supporting hyperplane, then $A$ is convex. I don't know what is the standard reference, internet search gave me the following one:
F.A.Valentine. Convex sets. McGraw-Hill, New York, 1964, pp. 51-53.
There are easier proofs than fedja's (at least to geometer's eye). My favorite one is the following. Take $a,b\in A$, there is a polygonal path from $a$ to $b$ in $A$. Suppose this path has $N$ edges. Then there is a shortest polygonal path of at most $N$ edges in the closure of $A$. If it is not a straight segment, the first edge must touch the boundary of $A$ (otherwise one can shorten the path). The first point where it meets the boundary has obvious problems with local supporting hyperplane.
Update. As Zsbán Ambrus pointed out in a comment, vertices on the boundary cause problems, so one should restrict to polygonal paths contained in the interior. But then it is not clear why a minimum exists. Rather that considering a minimum, one can do shorthening by hand: begin with any polygonal path in the interior, choose consecutive serments $[a,b]$ and $[b,c]$ and move $b=b(t)$ to $a$ along the segment. If a segment $[a,b(t)]$ touches the boundary at some moment $t$, observe a contradiction. If not, then the final segment $[a,c]$ is also contained in the interior, so we get a polygonal line with fewer edges. Repeat this procedure until it becomes a single segment.
