First, I just want to consider the requirement that $X_t$ be a *local* diffeomorphism, which by the inverse function theorem is equivalent to requiring the Jacobian of $X_t$ to never vanish. We have
$$ X_t = id + \int_0^t V_s\,ds = id + \int_0^t \left(v_0 + \int_0^s f_r(X_r)\,dr\right)\, ds \\= id + v_0 t + \int_0^t \int_0^s f_r(X_r)\,dr\,ds$$
and so, taking Jacobians,
$$D X_t = I + (D v_0)t + \int_0^t \int_0^s D f_r(X_r) \bullet D X_r\,dr\,ds$$
where $\bullet$ denotes matrix multiplication. Mirroring the proof of Picard-Lindelöf, we can define a sequence converging to $DX_T$ by "repeatedly plugging the whole RHS into the term inside the integral", which should converge by the Banach fixed point theorem, assuming $f$ is Lipschitz. So we get an infinite sum that looks like
$$DX_t = I + (Dv_0)t + \int_0^t \int_0^s D f_r(X_r) + r\,D f_r(X_r)\bullet D v_0\,dr\,ds \\
+ \int_0^t\int_0^s\int_0^r\int_0^u Df_v(X_v)+v\,Df_v(X_v)\bullet D v_0\,dv\,du\,dr\,ds + \cdots $$

Now, it seems to me that a very natural condition to impose is that $Dv_0$ be nilpotent. After all, consider the case where $f = 0$, so $DX_t = I + D v_0\,t$. This matrix will be non-invertible as soon as $t$ hits $-1/\lambda$ for any nonzero eigenvalue $\lambda$ of $Dv_0$. So if $Dv_0$ has any nonzero eigenvalues, $DX_t$ will be singular in some finite time.*

Once we start thinking this way, it's natural to continue and require $D f_t$ to be nilpotent at each point, and to commute with $Dv_0$, so that their product is also nilpotent. We will then have $DX_t = I + nilpotent$, so that it's unipotent, and in particular a local diffeo.

Going from that to a *global* diffeo is a much much harder problem. Based on preliminary googling (search for example 'global univalence'), this is an active area of research that a lot of work has been done in, but which still has a lot of open conjectures -- it's quite close to the famous Jacobian conjecture for polynomial maps. One conjecture I encountered here on page 5 is the following:

Conjecture 3 ($C^1$ Unipotent Jacobian Univalence Conjecture). If $f: \mathbb R^n \to \mathbb R^n$ is $C^1$ and $Df$ is unipotent, then $f$ is injective.

which would apply in this case. This seems to have been proven for $n = 2$, and there are other results about so-called "P-matrices" and "N-matrices" where it is also known. Surjectivity also seems to be an easier problem than injectivity. In any case, I think the moral is that you'll probably have to do a dig through the literature to see if you can find something that applies in your case.

*I'm going off your question that we want $X_t$ to be defined for all time, positive and negative. If you only care about $t>0$, you could make the weaker assumption that the eigenvalues of $Dv_0$ and $Df_t(X_t)$ are always positive (and still that they commute), which would related to Conjecture 2 from the same paper, since the eigenvalues of $DX_t$ would be bounded away from 0.