I just want to clarify, the $Ext$ you are considering are Sheaf-Ext, not global section Ext, right?

Regardless, I think the answer is probably you can't determine that just from vanishing / support. Consider for example $X = \text{Spec } k[x, y]$ and $Z = V(x)$ and $Z'= V(x^2)$. They have the same vanishing behavior for Ext, but one is reduced and the other is not. There are some ways in which a scheme fails to be reduced which can be detected by $Ext$ though, for example $k[x,y]/(x^2, xy)$ has a more complicated series of $Ext$s.

What you can detect is the (non-)Cohen-Macaulay'ness (or more generally, facts about the depth) of $Z$. See for example, Corollary 3.5.11 in Bruns-Herzog Cohen-Macaulay rings. Basically, if all the exts that vanish do vanish, then you are Cohen-Macaulay.

There are different ways a variety can be non-reduced. It can have associated points that are not generic points, ie the ring can have associated primes which are not minimal.

Non-reducedness in the form of having these non-generic associated points is an obstruction to being Cohen-Macaulay. In particular, Cohen-Macaulay rings are unmixed. Therefore, those obstructions can be found in Ext-groups, but there can be other reasons that a ring is not Cohen-Macaulay. Therefore, other things can be seen in the Ext-groups as well.

Of course, at some level all the Ext groups put in a complex, $R Hom_X(O_Z, \omega_X)$ isn't really any different data than $O_Z$ itself (apply $R Hom_X( - , \omega_X)$ again and you get $O_Z$ back). Thus reducedness can be seen from that complex. But I don't think you can see it just from vanishing / support.

I just want to clarify, the $Ext$ you are considering are Sheaf-Ext, not global section Ext, right?

Regardless, I think the answer is probably you can't determine that just from vanishing / support. Consider for example $X = \text{Spec } k[x, y]$ and $Z = V(x)$ and $Z'= V(x^2)$. They have the same vanishing behavior for Ext, but one is reduced and the other is not. There are some ways in which a scheme fails to be reduced which can be detected by $Ext$ though, for example $k[x,y]/(x^2, xy)$ has a more complicated series of $Ext$s.

What you can detect is the (non-)Cohen-Macaulay'ness (or more generally, facts about the depth) of $Z$. See for example, Corollary 3.5.11 in Bruns-Herzog Cohen-Macaulay rings. Basically, if all the exts that can vanish do vanish, then you are Cohen-Macaulay.

There are different ways a variety can be non-reduced. It can have associated points that are not generic points, ie the ring can have associated primes which are not minimal.

Non-reducedness in the form of having these non-generic associated points is an obstruction to being Cohen-Macaulay. In particular, Cohen-Macaulay rings are unmixed. Therefore, those obstructions can be found in Ext-groups, but there can be other reasons that a ring is not Cohen-Macaulay. Therefore, other things can be seen in the Ext-groups as well.

Of course, at some level all the Ext groups put in a complex, $R Hom_X(O_Z, \omega_X)$ isn't really any different data than $O_Z$ itself (apply $R Hom_X( - , \omega_X)$ again and you get $O_Z$ back). Thus reducedness can be seen from that complex. But I don't think you can see it just from vanishing / support.

I just want to clarify, the $Ext$ you are considering are Sheaf-Ext, not global section Ext, right?

Regardless, I think the answer is probably not. Consider for example $X = \text{Spec } k[x, y]$ and $Z = V(x)$ and $Z'= V(x^2)$. They have the same vanishing behavior for Ext, but one is reduced and the other is not.

What you can detect is the Cohen-Macaulay'ness (or more generally, facts about the depth) of $Z$. See for example, Corollary 3.5.11 in Bruns-Herzog Cohen-Macaulay rings. Now, certainly non-reducedness in the form of having higher associated primes, can be an obstruction to being Cohen-Macaulay. In particular, Cohen-Macaulay rings are unmixed. Therefore, those obstructions can be found in Ext-groups, but it is not the only obstruction. Other things can be seen in the Ext-groups as well.

I just want to clarify, the $Ext$ you are considering are Sheaf-Ext, not global section Ext, right?

Regardless, I think the answer is probably you can't determine that just from vanishing / support. Consider for example $X = \text{Spec } k[x, y]$ and $Z = V(x)$ and $Z'= V(x^2)$. They have the same vanishing behavior for Ext, but one is reduced and the other is not. There are some ways in which a scheme fails to be reduced which can be detected by $Ext$ though, for example $k[x,y]/(x^2, xy)$ has a more complicated series of $Ext$s.

What you can detect is the (non-)Cohen-Macaulay'ness (or more generally, facts about the depth) of $Z$. See for example, Corollary 3.5.11 in Bruns-Herzog Cohen-Macaulay rings. Basically, if all the exts that vanish do vanish, then you are Cohen-Macaulay.

There are different ways a variety can be non-reduced. It can have associated points that are not generic points, ie the ring can have associated primes which are not minimal.

Non-reducedness in the form of having these non-generic associated points is an obstruction to being Cohen-Macaulay. In particular, Cohen-Macaulay rings are unmixed. Therefore, those obstructions can be found in Ext-groups, but there can be other reasons that a ring is not Cohen-Macaulay. Therefore, other things can be seen in the Ext-groups as well.

Of course, at some level all the Ext groups put in a complex, $R Hom_X(O_Z, \omega_X)$ isn't really any different data than $O_Z$ itself (apply $R Hom_X( - , \omega_X)$ again and you get $O_Z$ back). Thus reducedness can be seen from that complex. But I don't think you can see it just from vanishing / support.