0

I am working to get a data transfer service up to which I can copy only newly created files in /h/data/share/(company share) to /g/data/global/ (public dropbox), however the files in /g/data/global are being moved, which is causing me some issues because it keeps copying and syncing the same files over and over, which the other end is getting the many duplicates.

I have tried some of the following:

    cd /h/data/share/
       find . -newerct -1 > /h/data/sync.txt
       rsync -avzh --no-R --from-file=/h/data/sync.txt /g/data/global/

I keep getting too many arguments after running, this is supposed to be a service which can be auto run with a cronjob. Also, this will be moving thousands of .mp3, .jpg, .png, .doc, etc per day.

David Lashua
  • 39
  • 6
  • I am guessing the find command will find directories which have been changed (I have not seen -newerct before), and the rsync -a will cause all of these directories to be recursively copied, which may be rather more than you want. I suggest you add `-type f` to your find command. – icarus Nov 06 '19 at 08:21
  • What are the file systems behind it? `rsync` relies on change times and for e.g. `FAT` these are two-second intervals. Thus you would have to increase a time window there. [see here](https://unix.stackexchange.com/questions/267204/rsync-to-usb-flash-drive-always-transferring-all-data) – FelixJN Nov 06 '19 at 09:19
  • You mention Dropbox in the title, but there's no mentioning of it in the text. BTW, are you allowed to copy files from the _company_ share to a _public_ cloud storage (which the title implies you're trying to do)? – Kusalananda Nov 06 '19 at 10:33
  • Yes I am allowed, this is a file share we are sharing data with other entities. Only data that goes into this share will be shared out, however they want to pass it through the DMZ which is where my Linux box resides so it can handle the data passing. /h/data/share is mounted to a files ahre inside our firewalls, and /g/data/global/ is on the exportfs out to certain ips. – David Lashua Nov 06 '19 at 12:43
  • -newerct deals with created date rather than mt which is modified date. I think I might need to do a cp of some kind. – David Lashua Nov 06 '19 at 12:44
  • So far I have a cifs share inside the network which is being mounted to /h/data/share/ and the /g/data/global/ is a nfs share to a specific ip. – David Lashua Nov 06 '19 at 12:45

0 Answers0