The answer is Yes.

In your theory, let us assume that "finite" means equinumerous with a finite von Neumann ordinal, where that means a transitive set that is well-ordered by $\in$ and having a largest element and no limit ordinals. 

We can prove that the finite ordinals are well-ordered, and therefore the principle of induction works on them. Using this, we can prove that every finite set has a power set, since the power set of a set of size $n+1$ is obtained from the power set of a size $n$ subset, by adding the extra element or not to every subset. 

If every set is finite, it follows that every set has a power set.

Otherwise, there is an infinite set $X$. By $\in$-induction, this set has an ordinal rank, which cannot be finite, since by induction every finite-rank set is finite. And so $\omega$ exists. 

Now argue by induction that for every $n$, the collection of size-$n$ subsets of $X$ exists. This is easily true for $n=0$. If it is true for $n$, then we can prove that it is also true for $n+1$, since the size $n+1$ subsets of $X$ arise from a subset of the product of the size $n$ subsets with $X$ itself, adding an additional element in all possible ways to the size $n$ subsets. (Note: products exist by suitable instances of collection to get all the sections existing and then put them together.)

Finally, by an instance of collection using $\omega$, we can collect together all the size $n$ subset families over all finite $n$, and from this construct the set of finite subsets of $X$, as desired.