I've just realized that I don't understand something important and basic about the Weil-Deligne group and its representations. (I'm not very surprised by this).
Following Deligne's article, Section 8 of "Les Constantes des Equations Fonctionnelles Des Functions L" from the 1972 Antwerp volume, one is led to consider representations of the Weil-Deligne group in the following algebraic sense: for any local nonarchimedean field, there is a group scheme $W'$, defined over $Q$, which is a semidirect product of the Weil group scheme $W$ and the additive group scheme $G_a$. This is a non-affine group scheme, and the Weil subgroup scheme $W$ is obtained as a countable disjoint union of affine subschemes (the cosets of inertia).
So, as is now standard, we consider algebraic representations of the group scheme $W'$, over various fields $E$ of characteristic zero, as such representations provide a unified framework (thanks to results of Grothendieck, Deligne, Serre) for $\lambda$-adic representations that arise from arithmetic.
A crucial piece of this is to restrict attention (or semisimplify) to the semisimple representations of the Weil-Deligne group. And presumably, such semisimple representations form a Tannakian category (over any base field of characteristic zero).
And so to my question... what is the algebraic group associated to this Tannakian category? Or am I just confused? And how does the (non-affine) Weil-Deligne group scheme relate to this (affine) algebraic group obtained by restricting attention to these semisimple representations? Does this involve one of these awfully large group schemes like $Spec(E[E^\times])$, where $E$ is a characteristic zero field (something like the semisimple algebraic hull of the discrete group $Z$)? Any references?