Using recollindex update for backup files

Question

I was starting with inotify to monitor changes in my directories and launch a backup script using rclone. So I thought that, taking advantage of the fact that I'm using Recoll in my debian, I could use the changes that Recoll detected to launch rclone, without having inotify working.

I haven't been able to find in Recoll a clear log file that tells me the last files that have been modified or created, every time I update the index. I've been trying the recoll_status script, or trying to read xapiandb using xapian-delve, but without success.

Any idea how to get a list of the last added or updated documents in the Recoll index?

Nathaniel M. Beaver · Accepted Answer · 2020-08-04T13:24:35.360

Using recollindex to check for changes to files is an interesting idea, but it shouldn't be necessary if I am understanding your use-case. Rclone already looks at file size and modification time to decide what to update:

Normally rclone will look at modification time and size of files to see if they are equal. If you set this flag then rclone will check the file hash and size to determine if files are equal.

https://rclone.org/docs/

This is essentially the same thing that recollindex does.

Use mtime instead of ctime to test if a file has been modified. The time is used in addition to the size, which is always used.

https://www.lesbonscomptes.com/recoll/usermanual/webhelp/docs/RCL.INSTALL.CONFIG.RECOLLCONF.TERMS.html

Take a look at the source code here for details:

// File signature and up to date check. The sig is based on
// m/ctime and size and the possibly new value is checked against
// the stored one.

https://framagit.org/medoc92/recoll/-/blob/d8edbcbc55cf7658af025212b7f01acd293d7884/src/index/fsindexer.cpp#L651

So rclone is doing essentially the same comparison as recollindex, except instead of updating the index, rclone updates the remote backup. You can just run rclone the same way you are currently running recollindex.

Furthermore, using recoll's index to decide what needs backing up would have a number of drawbacks, such as:

Depending on the skippedNames entry in ~/.recoll/recoll.conf, recollindex will skip indexing many files such as PNG and JPEG files. Presumably you will want these files backed up.
The index does not reflect the current state of the filesystem; in fact, the filesystem most likely changed while recollindex was running. So the index cannot be used as a reliable way to decide what needs updating.

By the way, when started in real time monitoring mode recoll isn't doing anything special to detect changes; it's using Inotify or FAM/GAMIN behind the scenes. Here's a link to the relevant part of the source code if you're curious:

https://framagit.org/medoc92/recoll/-/blob/ef1b8343fad5ef1b04714a3031b207bee458ca40/src/index/rclmonprc.cpp

https://framagit.org/medoc92/recoll/-/blob/ef1b8343fad5ef1b04714a3031b207bee458ca40/src/index/rclmonrcv.cpp

The recoll_status.py script just polls ~/.recoll/idxstatus.txt. This won't have the full list of changed files, either; it just shows what file is currently being indexed, and only updates occasionally, not for every single file.

// Update the status file. Avoid doing it too often. Always do
// it at the end (status DONE)

https://framagit.org/medoc92/recoll/-/blob/d8edbcbc55cf7658af025212b7f01acd293d7884/src/index/recollindex.cpp#L122

Using recollindex update for backup files

1 Answers1