The classes of ring you look at are precisly the rings of finitistic dimension zero for the question with projective instead of free. I will give a large class of example for finite dimensional algebras.

Let $A$ be a local finite dimensional (over a field $K$) non-selfinjective algebra. Then every non-projective module has infinite projective dimension and thus the only modules of finite projective dimension are the projective modules, which are all free.
Concrete example:
Take a quiver $Q$ with one point and $n \geq 2$ loops and let $KQ/I$ the quiver algebra with the relations $I$ such that $I=J^2$ consists of all paths of length 2. Alternatively, this is $K[x_1,...,x_n]/I$ with $I$ the ideal generated by all monomials of length 2.

So the "smallest" example is probably the 3-dimensional algebra $K[x,y]/(x^2,y^2,xy)$.

For finite dimensional algebras, the local finite dimensional non-selfinjective algebras are exactly those with the property that every module of finite projective dimension is free.
When you look instead at the property "every module of finite projective dimension is projective", then you arrive at the finite dimensional algebras with finitistic dimension zero (the finitistic dimension is defined as the supremum of all projective dimensions of modules having finite projective dimension), which is a very large class that contains for example all selfinjective and all local algebras.

You need two results about finite dimensional algebras:

1.For a local algebra, every projective module is free.

- Local algebras have finitistic dimension zero.

The first result is well known and 2. is easy to prove, but I will search for a reference now. (https://link.springer.com/chapter/10.1007/978-3-0348-8658-1_8 proposition 2.1. states that the finitistic dimension of local algebras is 0, this is not a good reference but I leave it here until a better one is found).