Let $ß_F(G,H)$ denote your (abelian) group, where the operation is the convolution product.
Let's just run with the recasting I gave in the comments. We will compute some facts about the order $ß_F(G,H)$ (which turns out to always be finite) by relating it to several other groups of morphisms. This won't ultimately resolve exactly what this group and the desired quotient look like, but hopefully it is illuminating and possibly can be developed further to provide the desired answers.
We can equivalently rewrite $B$ as a linear map $FG\to F^H$. We can note that any $B$ which is multiplicative in the first or second argument (while the other is held fixed) trivially satisfies your identity. These correspond to the unital coalgebra and augmented algebra morphisms $FG\to F^H$. So your group must be providing some sort of generalization of both of these concepts (for $G,H$ groups) simultaneously.
Written in the linear form, your identity is then equivalent to
$$\Delta(B(gh))(B(g)B(h)\otimes B(g)B(h)) = (B(gh)\otimes B(gh))\Delta(B(g))\Delta(B(h)),$$
where $FG$ and $F^H$ are given their usual structure as Hopf algebras. The condition $B(g,h)\neq 0$ is equivalent to requiring the linearized version to send group-like elements to invertible elements. The condition $B(x,1)=1=B(1,y)$ is equivalent to requiring that the linearized version respect the unit and counit maps.
The invertibility of $B(g),\ \forall g\in G,$ is the amusing thing here, as we can then rewrite your identity as
$$\Delta(B(g))(B(g)^{-1}\otimes B(g)^{-1}) \cdot \Delta(B(h))(B(h)^{-1}\otimes B(h)^{-1}) = \Delta(B(gh))(B(gh)^{-1}\otimes B(gh)^{-1}).$$
This means that the map $T(B)\colon FG\to F^H\otimes F^H$ given by
$$g\mapsto \Delta(B(g))(B(g)^{-1}\otimes B(g)^{-1})$$
is in fact an algebra homomorphism.
Informally speaking the assignment $B\to T(B)$ satisfies:
- $T(B)$ is an algebra homomorphism if and only if $B$ satisfies your identity.
- $T(B)$ is a morphism of augmented algebras if and only if $B$ satisfies your identity and $B(g,1)=1$ for all $g\in G$.
- $T(B)$ is trivial (identically the identity of $F^H\otimes F^H$) if and only if $B$ is a morphism of coalgebras (equiv, $B$ sends group-likes to group-likes).
- $B$ is a morphism of unital coalgebras if and only if $T(B)$ is trivial and $B(1)$ is the identity of $F^H$.
This is informal because I'm more or less acting as if $$T\colon \text{Hom}_{F}(FG,F^H)\to \text{Hom}_{F}(FG,F^H\otimes F^H),$$ but the formula is only well-defined on those $B$ sending group-likes to invertible elements. But by adjusting the domain we can resolve that issue.
We only need those $B$ that map elements of $G$ to normalized units of $F^H$. Define $U(A)$ to be the multiplicative group of normalized units of the Hopf algebra $A$. We may then replace the domain with $\text{Hom}_{\text{Set}}(G,U(F^H))$, which is equivalently $\text{Hom}_{\text{coalg}}(FG,FU(F^H))$. Note that using normalized units automatically encodes the relation $B(x,1)=1$ for all $x\in G$. Since $F^H$ is commutative, we can in fact endow $\text{Hom}_{\text{Set}}(G,U(F^H))$ with the structure of an abelian group via the convolution product: $(f_1*f_2)(g) = f_1(g) f_2(g)$ for all $g\in G$. Moreover, the convolution product also endows $\text{Hom}_F(FG,F^H\otimes F^H)$ with the structure of an abelian group. We can then view $T$ as a well-defined group homomorphism
$$\text{Hom}_{\text{Set}}(G,U(F^H))\to \text{Hom}_{F}(FG,F^H\otimes F^H).$$
We can adjust the domain further. We also wish to impose the condition that $B(1)$ is the identity, or equivalently that $B(1,y)=1$ for all $y\in H$. As such we may instead think of $T$ as a well-defined group homomorphism
$$\text{Hom}_{\text{Set}}(G\setminus\{1_G\},U(F^H))\to \text{Hom}_{F}(FG,F^H\otimes F^H).$$
The augmented algebra morphisms form a subgroup of the codomain, so it follows that $$ß_F(G,H)=T^{-1}(\text{augmented algebra morphisms}).$$ By dualizing, an augmented algebra morphism $FG\to F^H\otimes F^H$ becomes a unital coalgebra morphism $F(H\times H)\to F^G$, which is equivalent to a set map $H\times H\to \widehat{G}$ sending the identity to the identity, where $\widehat{G}$ is the character group of $G$ over $F$.
We can then collect a number of facts about $ß_F(G,H)$ and $T$.
$\ker(T) = \text{Hom}_{\text{Set}}(G\setminus\{1_G\},\widehat{H})$, which is an abelian group of order $$|\widehat{H}|^{|G|-1}.$$
$\ker(T)\subseteq ß_F(G,H)$ is a (normal) subgroup and $ß_F(G,H)/\ker(T)$ is isomorphic to a subgroup of $\text{Hom}_{\text{Set}}((H\times H)\setminus\{(1_H,1_H)\},\widehat{G})$. This parent group of the quotient has order $$|\widehat{G}|^{|H|^2-1}.$$
$ß_F(G,H)$ is finite and has order dividing $$|\widehat{G}|^{|H|^2-1}|\widehat{H}|^{|G|-1}.$$
As your identity is trivially satisfied when any of $x,y,z,w$ are an identity element, it is easy to verify that
$$ß_F(\mathbb{Z}_2,\mathbb{Z}_2)\cong \widehat{\mathbb{Z}_4},$$
so that the bound can be strict. Indeed, $T(ß_\mathbb{C}(\mathbb{Z}_2,\mathbb{Z}_2))$ has order $2$ while the augmented algebra morphisms in question are a group of order $8$, and $\ker(T)$ is precisely the bicharacters. So in general the image of $T$ need not contain all possible augmented algebra homomorphisms. This also shows that your group may contain maps which are not convolution products of unital coalgebra morphisms and augmented algebra morphisms. I currently leave it unresolved as to what, exactly, the image of $T$ is in general. Determining that would in principle resolve your questions, though doesn't seem much easier at first glance.
To get to a characterization of $B$ being a group homomorphism you can observe that we could have equivalently rewritten it as a linear map $FH\to F^G$, and this map will be precisely the linear dual of the first one. We then have the obvious analogue to $T$, call it $\check{T}$, and by the preceding case we'd have that $B$ is a group homomorphism if and only $B\in \ker(T)$ and $B^*\in\ker(\check{T})$. Here we're identifying $B\colon FG\to F^H$, so that $B^*\colon FH\to F^G$ is the (equivalent) dualized map. So the group homomorphisms are precisely $\ker(\check{T})^*\cap \ker(T)$.
We can also exploit this to apply our results about $T$ to $\check{T}$ as well, which combine to give a better set of divisors that bound the order of $ß_F(G,H)$.
$\ker(\check{T}) = \text{Hom}_{\text{Set}}(H\setminus\{1_H\},\widehat{G})$, which is an abelian group of order $$|\widehat{G}|^{|H|-1}.$$
$\ker(\check{T})\subseteq ß_F(G,H)$ is a (normal) subgroup and $ß_F(G,H)/\ker(\check{T})$ is isomorphic to a subgroup of $\text{Hom}_{\text{Set}}((G\times G)\setminus\{(1_G,1_G)\},\widehat{H})$. This latter group has order $$|\widehat{H}|^{|G|^2-1}.$$
$ß_F(G,H)$ is finite and has order dividing $$|\widehat{H}|^{|G|^2-1}|\widehat{G}|^{|H|-1}.$$
Combining with the previous results using $T$, and the fact that $\ker(T)\
\cap\ker(\check{T})^*$ is the subgroup of bicharacters, we have
$|ß_F(G,H)|$ is divisible by $\text{lcm}(|\widehat{G}|^{|H|-1}|,\widehat{H}|^{|G|-1})$ and $$\frac{|\widehat{G}|^{|H|-1}|\widehat{H}|^{|G|-1}}{|\text{Hom}_{\text{group}}(G,\widehat{H})|}$$.
$|ß_F(G,H)|$ divides $$\gcd(|\widehat{G}|^{|H|^2-1}|\widehat{H}|^{|G|-1}, |\widehat{H}|^{|G|^2-1}|\widehat{G}|^{|H|-1})$$.
When $G=H$ this is of limited help, but if $\widehat{G},\widehat{H}$ have coprime orders it tells us the full order of the group:
- If $\widehat{G},\widehat{H}$ have coprime orders then $$|ß_F(G,H)| = |\widehat{G}|^{|H|-1}|\widehat{H}|^{|G|-1}.$$
So $|ß_\mathbb{C}(\mathbb{Z}_2,\mathbb{Z}_3)| = 12$, for example. Indeed, it is generated by the unital coalgebra morphisms and the augmented algebra morphisms, which intersect trivially.
