For infinite-dimensional algebras this is not true.
As it is mentioned in the comments, the claim is equivalent to the injectivity of the coordinate-wise product map $u:A^I \otimes_A A^J \to A^{I\times J}$ (where now the exponent stands for products, not coproducts).
For a counterexample, consider the commutative, unital, associative algebra $A=\Bbbk [x_i \; (i \in \mathbb{N})]/(x_i x_j \; (i,j \in \mathbb{N}))$, let $I=J=\mathbb{N}$ and $\alpha = (x_i)_{i \in \mathbb{N}} \in A^I$. Then clearly $u(\alpha \otimes \alpha)=0$, but one can prove that $\alpha \otimes \alpha$ is nonzero in $A^I \otimes_A A^I$.
Indeed, $A^I \otimes_A A^I = (A^I \otimes_{\Bbbk} A^I)/U$, where $U$ is spanned by the elements of the form
$$\lambda x_i \otimes \mu - \lambda \otimes x_i \mu,\quad
\lambda x_i \otimes \beta,\quad
\beta \otimes x_i \lambda$$
for all $i \in \mathbb{N}$, $\lambda, \mu\in \Bbbk^I$, $\beta \in N^I$ denoting $N=\mathrm{Span}(x_k \mid k \in \mathbb{N})$. These elements do not span $\alpha \otimes \alpha$ in $A^I \otimes_{\Bbbk} A^I$.
(Edit, response to the comment: If $A$ is finite-dimensional then $u$ is injective. Indeed, given a $\Bbbk$-basis $b_1, \dots, b_d$ in $A$, the elements of the form $\lambda \otimes \mu\, b_k$ span $A^I \otimes_A A^J$, where $k=1,\dots,d$, $\lambda \in \Bbbk^I$, $\mu \in \Bbbk^J$. Hence it is enough to prove the statement for $A=\Bbbk$.
Assume that $u$ vanishes on the nonzero element $\sum_{k=1}^m \lambda^{(k)} \otimes \mu^{(k)}$ where $\lambda^{(k)},\mu^{(k)}$ are as above. We may assume that $\lambda^{(1)},\dots,\lambda^{(m)}$ are independent and similarly for $\mu$. Consequently, $\{(\lambda_i^{(k)})_k\mid i\in I\}$ spans $\Bbbk^m$ by a rank-argument.
On the other hand, as $u$ vanishes on the sum, we have
$\sum_{k=1}^m \lambda_i^{(k)}\cdot \mu_j^{(k)}=0$ for all $i,j$, i.e. $(\mu_j^{(k)})_k$ is orthogonal to a spanning set with respect to the obvious bilinear form on $\Bbbk^m$. This is a contradiction.)
