If you have a homogeneous space $X$ with structure group $G$ (in your case, the fiber passing through $p$) then left multiplication give you a nice action $L: X \times G \to X$.  Then for a fixed $p \in X$, $L(p) : G \to X$.  The differential of this guy is a map $L(p)_* : TG \to TX$ which takes an element of $T_gG$ to an element of $T_{g \cdot p} X$.  At $g = 1$, this means you get a map $L(p)_* : \mathfrak{g} \to TX$.  This is actually a representation of $\mathfrak{g}$ on the space of vector fields on $X$, and will inherit nice properties depending on how nice the action of $G$ on $X$ is. 


This map $L(p)_*$ is precisely the sharp map you describe: it takes an element $\xi$ of $\mathfrak{g}$ and gives a vector field on $X$ which coincides with infinitesimal multiplication by $\xi$.  In your case, $G$ acts freely so $X$ looks like a copy of $G$.

If $G$ actually has a nice matrix representation then unrolling the definitions, you'll find that you really can write your proposed equation $A^\sharp(p) = A \cdot p$.  For example, look at how the infinitesimal generators of $\mathfrak{so}(3)$ acts on points of the unit sphere by multiplication --- you'll easily see the vector fields whose flows are $\exp(\xi t)$.