With $5$ you can using the following moves:

```
...
...
oo.
ooo
```
```
...
oo.
oo.
..o
```

```
...
oo.
.oo
..o
```

```
...
ooo
.oo
...
```

```
.oo
ooo
...
...
```

```
oo.
ooo
...
...
```

So the only number of pieces which can't go to infinity is $3$.