List files not stored in Git repos

Question

I want to find out what files in a certain directory are not managed by Git. This is so I because I use Git for backups, and I want to eventually get all my files personal in there.

What unix tool could accomplish this? Is there a way to use find to do this in a reasonably efficient way?

Example:

I have a folder where I store RFC-related stuff, it looks like this:

.
├── TheFile
└── tests
    ├── 4180
    │   └── data
    │       ├── bad
    │       └── good
    │           └── linebreaks.csv
    ├── get-rfc.sh
    ├── .git <contents omited>
    ├── LICENSE
    ├── README
    └── rfc4180.txt

I'm looking for a command that will output me:

TheFile

Answer 1

find the_starting_dir \( -type d -exec test -d '{}'/.git \; -prune \) -o -print

Not the most portable of find invocations, but works with GNU find.

Find walks the directory tree. The term -prune returns true but stops find from further processing the subtree. So the left hand side of the -o says "if this is a directory, and if test says there is a sub-directory called .git and if prune returns true then we are done processing this thing in the tree". The right hand side says "otherwise print it"

If you don't want directories printed then change -print to \( ! -type d -print \) but then you will not get any indication about empty directories.

You can change the -print to -ls to get listings, -printf see manual etc etc.

Answer 2

I don't understand why you don't want to use git...

git status --untracked-files

[edit]

As I understand the request, there is a directory tree. Inside this tree there are multiple git repos, but there is not one covering the whole tree. The example shows a git repo in the tests sub directory. Therefore your suggested command would get a git error fatal: Not in a git repository rather than TheFile as requested.

find . -name .git  -exec echo  $(realpath '{}') \; |\
sed 's/\(.*\)\(.git\)/git --git-dir=\1\2 --work-tree=\1 status --untracked-files/g' | bash  

Answer 3

There are a lot of specific cases in your request.

1. Files actually outside a git-managed directory.
   • Your TheFile fits this case.
2. Files inside a directory managed by Git, with some .git marker. .git is not always a directory. It can be a file as well, with a path to the real GIT_DIR. We can further break these files down as follows:
   1. Known files, those present in the Git index.
   2. Ignored files, those files matching a pattern per gitignore(5):
      • .gitignore
      • $HOME/.config/git/ignore
      • $GIT_DIR/info/exclude
   3. Files under an actual $GIT_DIR directory, but NOT part of the repo.
      • .git/hooks are the most likely
      • Could also be malware

So the most reliable case, is going to be generating TWO lists, relative to your given base directory $D, and comparing them (be sure to sort them and remove duplicates beforehand).

I can't think of a reliable way to generate the sub-list for 2.3 above, so I leave that as an open problem (I'd love to know about it, because I've lost hooks before).

Shell script to list known files per 2.1 above:

for g in $(find $D -name .git) ; do
  echo $g 
  p=${g%/.git} g2=`readlink -f $g` ;
  ( cd $p && GIT_DIR=$g2 \
  git ls-files --exclude-standard --full-name ) \
  | sed "s,^,${p}/,g" ; 
done > list-2.1

Shell script to list ignored files per 2.2 above:

for g in $(find $D -name .git) ; do
  p=${g%/.git} g2=`readlink -f $g` ;
  ( cd $p && GIT_DIR=$g2 \
  git ls-files \
  --others -i --exclude-standard ) \
  | sed "s,^,${p}/,g" ; 
done > list-2.2

Shell script to list files per 2.3 above:

TODO > list-2.3

Shell script to process the lists and find what's not on side B:

comm -23 <(find $D ! -type d |sort) <(sort 2.1 2.2 2.3 | uniq)

Answer 4

find <root_dir> -type d -name ".git"

is this what you are looking for ?

you could also do something like

find <root_dir> -type d -name ".git" -print0 | xargs -0 -r dirname

to output just the directories names without the /.git part ; you can even avoid dirname and just use the output of find plus /../ at the end of each folder name .

find <root_dir> -type d -name ".git" -print0 | xargs -0 -r printf "%s/../"

EDIT

You can negate the -name parameter and play with the -maxdepth value; for example I can easily list all the directories in my kernel tree that do not have a name that starts with a c ( uppercase or lowercase ) .

find linux-4.8.2/ -type d ! -iname "c*" -maxdepth 1

I still don't really get what you mean with :

I'm looking for the exact opposite of this, namely a command which find all files not in a Git repo;

simply because the files that are in the same directory on the same filesystem of the .git root dir might not be even be part of a repository, it just means that that literally there are files in the same dir ( or subdirs ) where there is a dir named .git too . Remember that git has the concept of staging and in general what you see on a disk may not be your git repository .

Take a look at https://libgit2.github.com/ which is the official C lib for git and there are a lot of bindings for it plus a lot of docs, examples and how to .