What AI-powered or multi-modal LLM (Large Language Model) apps/interfaces are available to edit/revise pdf or Word documents by automatically producing LaTex for the math expressions within the document files or images of the docs?
$\begingroup$
$\endgroup$
4
-
4$\begingroup$ The comments here were not at all shining light on the problem, rather a lot of heat. And the question didn't need reference to now-deleted comments and past versions that were not meeting community expectations. I think everyone should just take a breather, the question has gathered some interesting answers and down the track people will be able to engage here dispassionately. Maybe something will be invented in the near future that will be really, really useful. $\endgroup$– David Roberts ♦Commented Jan 18 at 5:03
-
$\begingroup$ I'm looking forward to efficient, effective voice-prompted LLMs to deal with the problem that have a teachable interface such as alleged for the Rabbit R1. $\endgroup$– Tom CopelandCommented Jan 18 at 20:26
-
1$\begingroup$ The question reads much better now, and is clear. +1! $\endgroup$– Carl-Fredrik Nyberg BroddaCommented Jan 19 at 7:43
-
$\begingroup$ Here's an example of related work from 2015: Search on 'Algebraic Geometry (Latex)' at karpathy.github.io/2015/05/21/rnn-effectiveness. Karpathy also did a recent introductory vid on LLMs for those familiar with the general idea of a neural net: youtube.com/watch?v=zjkBMFhNj_g $\endgroup$– Tom CopelandCommented Jan 22 at 18:31
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
1
Since you ask specifically about LLM's: A customized version of ChatGPT is available for this functionality, see Document to LaTeX Custom GPT. Here is a promotional.
I have not tried it, you need a paid ChatGPT account.
An older open source LaTeX-OCR based on neural network technology (ViT) was developed by Lukas Blecher. It's a learning-based system, so it will require training of the neural network.
$\begingroup$
$\endgroup$
3
For AI-based PDF-to-LaTeX conversion, there is a Python package called Nougat. I always plan to play around with it in the coming week, but somehow this coming week never comes. If it is as good as it claims, we should brace ourselves for a wave of re-editions of classic texts.
EDIT: Installing it was surprisingly easy! Really just pip install nougat-ocr as said in the doc. But using it (on a WSL Ubuntu) is another story:
darij@skogen:/mnt/c/worksnap/temprepth/2$ nougat scan.pdf -o towber
WARNING:root:No GPU found. Conversion on CPU is very slow.
downloading nougat checkpoint version 0.1.0-small to path /home/darij/.cache/torch/hub/nougat-0.1.0-small
config.json: 100%|█████████████████████████████████████████████████████████████████████| 557/557 [00:00<00:00, 3.00Mb/s]
pytorch_model.bin: 100%|█████████████████████████████████████████████████████████████| 956M/956M [03:41<00:00, 4.53Mb/s]
special_tokens_map.json: 100%|████████████████████████████████████████████████████████| 96.0/96.0 [00:00<00:00, 770kb/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████| 2.04M/2.04M [00:00<00:00, 15.1Mb/s]
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████| 106/106 [00:00<00:00, 639kb/s]
/home/darij/.local/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3526.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
0%| | 0/49 [00:00<?, ?it/s]INFO:root:Processing file scan.pdf with 49 pages
2%|█▋ | 1/49 [03:07<2:30:22, 187.96s/it]WARNING:root:Found repetitions in sample 0
[nltk_data] Downloading package words to /home/darij/nltk_data...
[nltk_data] Unzipping corpora/words.zip.
WARNING:root:Skipping page 2 due to repetitions.
6%|████▉ | 3/49 [08:15<2:03:54, 161.63s/it]WARNING:root:Found repetitions in sample 0
WARNING:root:Skipping page 4 due to repetitions.
8%|██████▌ | 4/49 [11:00<2:02:05, 162.78s/it]
I can live with the slowness (apparently Witcher 3 has no problems finding my GPU), but if it skips half the pages I'm not sure how much I can get out of it.
-
5$\begingroup$ I've tried Nougat on a couple of documents. My experience is that it does fairly well with modern LaTeX generated output, but also with older documents with high quality professional typesetting that were cleanly scanned in high resolution, and are mostly in English. Unfortunately, it did obviously worse for noisy scans and/or uncommon notation, non-English text, and poor quality typesetting (typewriter/handwriting combinations). Output quality can be improved by passing it through ChatGPT, in the role of proof-reader/copy-editor. In summary: Worth a try! $\endgroup$ Commented Jan 14 at 23:09
-
2$\begingroup$ @IgorKhavkine: I also like the output that I'm getting when I'm getting it! If it wouldn't skip over half the pages due to mysterious "repetitions" it would be wonderful. (I've opened a bug ticket.) $\endgroup$ Commented Jan 14 at 23:15
-
$\begingroup$ Thanks Darij, for leading constructive input to this Q and expanding from a comment to this answer. $\endgroup$ Commented Jan 18 at 15:19
$\begingroup$
$\endgroup$
1
FineReader PDF empowers professionals to maximize efficiency in the digital workplace. Featuring ABBYY’s latest AI-based OCR technology, FineReader PDF makes it easier to digitize, retrieve, edit, protect, share, and collaborate on all kinds of documents in the same workflow.
AI-powered document automation
Quickly and accurately convert PDFs and images to searchable, exportable, and machine readable text.
-
$\begingroup$ Nice. SNIP bypasses the need for a stylus if one prefers a camera or scanner for conversion from actually handwritten to LaTex expressions. $\endgroup$ Commented Jan 14 at 16:53
$\begingroup$
$\endgroup$
5
Not exactly what you are looking for, but related, and in my opinion, worth bringing up to the wider community.
Recently, I tried using ChatGPT4 to translate an old French paper and provide English LaTeX code for it. It was sort of successful (quite good at the translation, but the LaTeX had very many mistakes, though still it was good scaffolding that I could tweak fairly easily into being correct):
The original French PDF is also in the Overleaf project (compile the document so the LaTeX PDF shows up on the right window, and then open the French PDF on the left window to compare them).
Thanks to my friend who used his ChatGPT plus access to help out. The method was screenshotting pieces of the PDF (asking it to just OCR the PDF, it refused, saying it can't perform OCR, though of course it can, on the screenshots) and passing into ChatGPT with prompt "Convert the following image into English LaTeX, I don't need to see the intermediate translation, just show me the English LaTeX code"
Additional notes from my friend:
"In total there was one fatal render error that I had to fix, there was another render error that just resulted in weirdly shifted equations, and I caught three hallucinations where it got to the end of a page that implied there was a theorem or proposition to follow and then inserted a comment saying to insert the theorem or proposition. Which is strictly speaking is a hallucination, but it doesn't render bc it's a comment, and it's a useful comment
I think I caught at least one mathematical translation error that I didn't fix. At the begining there was an exponent (1) which was instead translated as alpha
I also used the GPT builder, which was told to make sure LaTeX code was accurate. I noticed that if I didn't include this on previous documents I've tried transcribing it would generate more errors"
-
4$\begingroup$ I think this is a good and important use case for this technology. And performance should be expected to improve with newer language models. Just to be inclusive, I've had reasonable success in related experiments with alternatives to ChatGPT, of which one example is Claude. Its advantage is that the free version allows a much larger context size than the free version of ChatGPT. $\endgroup$ Commented Jan 15 at 8:25
-
$\begingroup$ Ask your friend if he can / would use a 'jailbreak' technique (see the LLN video) to liberate OCR for the pdf. // Having a trained AI assistant to perform these types of tasks efficiently and effectively (via voice prompts) surely will be a huge benefit in the near future. $\endgroup$ Commented Jan 15 at 14:09
-
$\begingroup$ @IgorKhavkine, nice, I took the liberty to ask Claude about this question and the responses and pasted Claude's response here. He could only respond from inferences from the title in the URL, but it is a very relevant 'hallucination'. $\endgroup$ Commented Jan 15 at 14:45
-
$\begingroup$ @TomCopeland we tried various prompts to do the OCR, but ultimately at the last minute it would revert course and go back to refusing to do OCR $\endgroup$– D.R.Commented Jan 15 at 17:53
-
$\begingroup$ Maybe a potential issue with copyright infringement underlies the obstruction.Try asking someone proficient with LLMs for a workaround/hack--someone familiar with jailbreak maneuvers. I suspect we'll have to wait a year or two before really efficient and effective interfaces are available for assisted editing such as reformatting, error detection and correction, and semi-automatic spot checking for the mathematics. $\endgroup$ Commented Jan 15 at 19:52