EDIT: Installing it was surprisingly easy! Really just pip install nougat-ocr as said in the doc. But using it (on a WSL Ubuntu) is another story:

darij@skogen:/mnt/c/worksnap/temprepth/2$ nougat scan.pdf -o towber
WARNING:root:No GPU found. Conversion on CPU is very slow.
downloading nougat checkpoint version 0.1.0-small to path /home/darij/.cache/torch/hub/nougat-0.1.0-small
config.json: 100%|█████████████████████████████████████████████████████████████████████| 557/557 [00:00<00:00, 3.00Mb/s]
pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████| 956M/956M [03:41<00:00, 4.53Mb/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████| 96.0/96.0 [00:00<00:00, 770kb/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████| 2.04M/2.04M [00:00<00:00, 15.1Mb/s]
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 639kb/s]
/home/darij/.local/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
  0%|                                                                                            | 0/49 [00:00<?, ?it/s]INFO:root:Processing file scan.pdf with 49 pages
  2%|█▋                                                                               | 1/49 [03:07<2:30:22, 187.96s/it]WARNING:root:Found repetitions in sample 0
[nltk_data] Downloading package words to /home/darij/nltk_data...
[nltk_data]   Unzipping corpora/words.zip.
WARNING:root:Skipping page 2 due to repetitions.
  6%|████▉                                                                            | 3/49 [08:15<2:03:54, 161.63s/it]WARNING:root:Found repetitions in sample 0
WARNING:root:Skipping page 4 due to repetitions.
  8%|██████▌                                                                          | 4/49 [11:00<2:02:05, 162.78s/it]

I can live with the slowness (apparently Witcher 3 has no problems finding my GPU), but if it skips half the pages I'm not sure how much I can get out of it.