different results on terminal vs qsub submission

Question

I'm trying to run a command on the terminal and also submit it to the cluster but I am getting different results.

When I type on the terminal this :

$ for i in *_1.fastq.gz; do echo $i >> t.txt; zcat $i | \
    grep "GCTGGCAAAAAGAAGGTAACATGTTTT" >> t.txt ; echo >> t.txt ; done

I get the output like this

adrenal_4a_ERR315335_1.fastq.gz
GCANAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA

adrenal_4a_ERR315452_1.fastq.gz
GCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA
CAAGAACAGAATGAAGAAAGTCAGGGGGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTT

adrenal_4c_ERR315392_1.fastq.gz

adrenal_4c_ERR315450_1.fastq.gz

and so on..

This is what the output is expected.

When I submit the same command to the HPC cluster via qsub I'm getting a completely different result:

$ qsub -l h_vmem=4G -cwd -j y -b y -N n_tr -R y \
    "for i in *_1.fastq.gz; do echo $i >> t.txt; zcat $i | \
       grep "GCTGGCAAAAAGAAGGTAACATGTTTT" >> t.txt ; echo >> t.txt ; done"

adrenal_4a_ERR315452_1.fastq.gz
GCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA
CAAGAACAGAATGAAGAAAGTCAGGGGGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTT

adrenal_4a_ERR315452_1.fastq.gz

appendix_4a_ERR315437_1.fastq.gz
GCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA
CAAGAACAGAATGAAGAAAGTCAGGGGGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTT

adrenal_4a_ERR315452_1.fastq.gz
GCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA
CAAGAACAGAATGAAGAAAGTCAGGGGGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTT

adrenal_4a_ERR315452_1.fastq.gz
GGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGA

appendix_4a_ERR315465_1.fastq.gz
GCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA
CAAGAACAGAATGAAGAAAGTCAGGGGGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTT

adrenal_4a_ERR315452_1.fastq.gz

appendix_4b_ERR315345_1.fastq.gz
GCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA
CAAGAACAGAATGAAGAAAGTCAGGGGGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTT

adrenal_4a_ERR315452_1.fastq.gz
GCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGGAACTATGTAGCATAGTGTCTTAACACCTCAGTAAAGAGATCGGAAGAGCACA
CAAGAACAGAATGAAGAAAGTCAGGGGGACTGCAAAGGCCAATGTTGGTGCTGGCAAAAAGAAGGTAACATGTTTTAAGAAACTATGTAGCATAGTGTCTT

What is it that I'm doing wrong here?

The weak quoting (`"..."`) is allowing the invoking shell to expand `$i`? — steeldriver, Jul 09 '18 at 18:02
Are you talking about the double quotes for the qsub command? — user3138373, Jul 09 '18 at 18:03

score 6 · Answer 1 · answered Jul 09 '18 at 18:12

You're using the wrong sort of quotes for your command.

The double quotes ("...") allow the shell to evaluate your variables such as $i, and because you've already used that variable and it has been left with the value adrenal_4a_ERR315452_1.fastq.gz, means your qsub sees this for execution:

for i in *_1.fastq.gz; do echo adrenal_4a_ERR315452_1.fastq.gz >> t.txt; zcat adrenal_4a_ERR315452_1.fastq.gz | grep GCTGGCAAAAAGAAGGTAACATGTTTT >> t.txt ; echo >> t.txt ; done

Change the outermost quotes to single quotes ('...') and it'll work as expected:

qsub -l h_vmem=4G -cwd -j y -b y -N n_tr -R y 'for i in *_1.fastq.gz; do echo $i >> t.txt; zcat $i | grep "GCTGGCAAAAAGAAGGTAACATGTTTT" >> t.txt ; echo >> t.txt ; done'

slm · Answer 2 · 2018-07-09T18:16:54.637

Try doing this instead:

$ qsub -l h_vmem=4G -cwd -j y -b y -N n_tr -R y \
    'for i in *_1.fastq.gz; do echo $i >> t.txt; zcat $i | \
        grep "GCTGGCAAAAAGAAGGTAACATGTTTT" >> t.txt ; echo >> t.txt ; done'

Notice the difference?

I changed your double quotes (") to single quotes (').

 'for i in ....t.txt ; done'
 ^^^                     ^^^

This will guard them from your local shell expanding anything inside. When you use double quotes the *_1.fastq.gz is susceptible to being expanded locally vs. on the computer cluster's node.

different results on terminal vs qsub submission

2 Answers2