Shub is not recognized as an internal or external command operable program or batch file

root@f71f06b4874d:/home# ./PEMA_v1.bds
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
Fatal error: ./PEMA_v1.bds, line 68. Directory '/mnt/analysis/mydata' does not exists

do you recognise the problem?

Haris Zafeiropoulos

@hariszaf

Hi @Matthias!

Do you have your raw data in a folder called mydata inside your analysis directory?

yes, the folder is on the Desktop

Haris Zafeiropoulos

@hariszaf

So you have something like this
/Users/matthias/Desktop/my_analysis/

and there you have your parameters file and a folder called
mydata
is that right?

Desktop/analysis_directory/mydata/

the parameters file is under analysis_directory, and the fastq file is under mydata

Haris Zafeiropoulos

@hariszaf

and you run
docker run -it -v /Desktop/analysis_directory:/mnt/analysis

I am asking because you need to mount the whole analysis_directory not just your data.

i hadnavigates to the desktop therefore i only had "/anlaysis_directory" as path. Now i copied your command and get this: "docker run" requires at least 1 argument.
See 'docker run --help'.

Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container

Haris Zafeiropoulos

@hariszaf

docker run -it -v /Desktop/analysis_directory:/mnt/analysis hariszaf/pema

if you run this what do you get?

Haris Zafeiropoulos

@hariszaf

ok. could you run
ls -la /mnt
and then
ls -la /mnt/analysis

root@3069a1f860d3:/home# ls -la /mnt
total 12
drwxr-xr-x 1 root root 4096 May 24 16:19 .
drwxr-xr-x 1 root root 4096 Jun 17 12:41 ..
drwxr-xr-x 2 root root 40 Jun 17 12:41 analysis
drwxr-xr-x 2 root root 4096 May 24 16:19 databases

Haris Zafeiropoulos

@hariszaf

:thumbsup:

total 4
drwxr-xr-x 2 root root 40 Jun 17 12:41 .
drwxr-xr-x 1 root root 4096 May 24 16:19 ..

Haris Zafeiropoulos

@hariszaf

this is from the second command???

Haris Zafeiropoulos

@hariszaf

ok please run exit.
then go to your desktop, run pwd and please tell me the exact path you get

C:\Users\xobsma\Desktop>pwd
'pwd' is not recognized as an internal or external command,
operable program or batch file.

18 replies

Hi Haris, I am back. I found the problem with the Mac installation. You have to add a "~" before the path to the analysis directory in the github tutorial
'''docker run -it -v ~/Desktop/analysis_directory/:/mnt/analysis hariszaf/pema''',

...and i got another error when executing PEMA_v1.bds:

Fatal error: ./PEMA_v1.bds, line 161, pos 1. Task/s failed.

29 replies

Haris Zafeiropoulos

@hariszaf

Hi @/all ! FYI I am about to integrate the convertIllumunaRawDataToEnaFormat.sh script in the main PEMA code. It seems like an unessecary burden! If you have any other suggestions in terms of improvement of PEMA there are more than welcome!

@hariszaf . I got this error and i feel that this has something to do with the parameter.tsv file

all the first steps are done! clustering is about to start!
Fatal error: /home/PEMA_v1.bds, line 557, pos 84. Map 'paramsOfTable' does not have key 'clusteringAlgoForCOI_ITS'.
PEMA_v1.bds, line 557 : if ( paramsOfTable{'clusteringAlgoFor16S_18SrRNA'} == "algo_swarm" || paramsOfTable{'clusteringAlgoForCOI_ITS'} == " algo_SWARM") {

ProgramCounter.pop(100): Node ID does not match!
PC : PC: size 4 / 0, nodes: 1422 -> 4690 -> 4691 -> 4698
Node Id : 4699
bdsNode Id : 4698
ProgramCounter.pop(100): Node ID does not match!
PC : PC: size 3 / 0, nodes: 1422 -> 4690 -> 4691
Node Id : 4698
bdsNode Id : 4691
ProgramCounter.pop(100): Node ID does not match!
PC : PC: size 2 / 0, nodes: 1422 -> 4690
Node Id : 4691
bdsNode Id : 4690
ProgramCounter.pop(100): Node ID does not match!
PC : PC: size 1 / 0, nodes: 1422
Node Id : 4690
bdsNode Id : 1422

i have checked the clusteringAlgoForCOI_ITS and set it into algo_CROP, but the error still persist. Also I am not really sure if I have to comment (#) the 'clusteringAkgofor_16S_18SrRNa part as I am using only COI for the moment . Thanks @hariszaf . So sorry for the trouble...

Haris Zafeiropoulos

@hariszaf

Hi @ajiwahyu ! Thank you for the feedback! You should not comment out any of the parameters! PEMA needs to know that it has access to all of them even if you do not use them. So, you could check that all the parameters are on and make sure that your parameters file is tab separated. I would suggest to go clusteringAlgoFor16S_18SrRNA algo_vsearch and clusteringAlgoForCOI_ITS algo_SWARM for a first run; especially if you have many samples as the CROP algorithm will take quite a long!

It is essential to have a tab seperated file.. This is a quite common error but for the time being we cannot avoid this format.. :/

Thanks @hariszaf. I just tried your suggestion and the pipeline seemed working with SWARM but spit the following error when I used CROP. Also a question with Database, for COI what database are you using?. Is there any setting that I need to change if I want to use different data base?.Also for setting a targetlength at the trimmomatic part, how should one decide it?. IAm using COI, with COI primer dgHCO2198, expecting a final merge reads at 300-330 bp.

This is the error i got when using CROP COI; cropPath is: /mnt/analysis/16S_final_test/7.gene_dependent/gene_COI/CROP
/home/scripts/croParameters.sh: line 49: cd: tools/: No such file or directory
toolsPath is: /mnt/analysis
/home/scripts/croParameters.sh: line 57: /mnt/analysis/CROP/CROPLinux: No such file or directory
cp: cannot stat 'CROP_output.cluster.list': No such file or directory
sed: can't read CROP_output.cluster.list: No such file or directory
split: cannot open 'CROP_output.cluster.list' for reading: No such file or directory
rm: cannot remove 'CROP_output.cluster.list': No such file or directory

sed: can't read /mnt/analysis/16S_final_test/7.gene_dependent/gene_COI/CROP/abundances/x: No such file or directory
rm: cannot remove '/mnt/analysis/16S_final_test/7.gene_dependent/gene_COI/CROP/abundances/x.abundances': No such file or directory
cat: 'x': No such file or directory
rm: cannot remove 'x': No such file or directory
tail: cannot open 'CROP_output.cluster' for reading: No such file or directory
cp: cannot stat 'CROP_output.cluster.list': No such file or directory
sed: can't read SWARM.swarms: No such file or directory
Fatal error: /home/PEMA_v1.bds, line 971, pos 3. Exec failed.
Exit value : 2
Command : bash /home/scripts/cropOutputTranformations.sh
PEMA_v1.bds, line 804 : if ( paramsOfTable{'gene'} == 'gene_COI' || paramsOfTable{'gene'} == "gene_ITS") {
PEMA_v1.bds, line 927 : if ( paramsOfTable{'clusteringAlgoForCOI_ITS'} == 'algo_CROP' ) {
PEMA_v1.bds, line 971 : sys bash $path/scripts/cropOutputTranformations.sh

Haris Zafeiropoulos

@hariszaf

Hi @ajiwahyu! Thanks a lot for this! There was a bug on the script that calculates the CROP parameters. I am about to rebuild an image fixing that.

Regarding the COI marker gene, PEMA uses the MIDORI database, which you may find here.

For the time being you cannot use an in-house reference database, however that is going to be included in the next version of PEMA.

Haris Zafeiropoulos

@hariszaf

When it comes to the Trimmomatic parameters, you should always check on its manual
As you can see there, targetLength is a MAXINFO related parameter. I think that by reading this and by combining this with the special feautures of your dataset, you may end up with a value for this parameter.
Some users have mentioned that they use a value a bit smaller than the merged sequence they're expecting, but I do not know if that is always the case.

Thanks @hariszaf !, ill be waiting the update from the bug!

10 replies

Haris Zafeiropoulos

@hariszaf

:plus1:
thanks again for the feedback! i ll fix it asap!

After running PEMA with vsearch, the pipeline stopped because my metadata file was named as metadata1.csv. I renamed it as metadata.csv and restarted the pipeline at phyloseq.chp. It resulted in an empty OTU table and so phyloseq did not run. As we discussed, I believe I should have restarted at tableStep.chp?

Haris Zafeiropoulos

@hariszaf

Hi @migdesmarais! it is most likely that this is what went bad. If you run the tableStep.chp and it is ok it would be great if you'd share

Haris Zafeiropoulos

@hariszaf

Hi @ajiwahyu , hi @all.
PEMA v.1.3 is now up and running.
The CROP bug has been resolved and now you can run CROP in multiple threads as well.

@ajiwahyu many thanks for the feedback once again.

thanks @hariszaf !. Am i correct to assume that i need to do another "singularity pull" again so my PEMA container is up to date?

M Danie Al Malik

@daniealmalik354

Hello @hariszaf, I have trouble to run my data on Windows OS from Docker. When I run the data the result like this: Fatal error: ./pema_latest.bds, line 80, pos 27. Map 'paramsFirstAssignment' does not have key 'outputFolderName'. Please help me if you have a suggestion. Thank you

Haris Zafeiropoulos

@hariszaf

thanks @hariszaf !. Am i correct to assume that i need to do another "singularity pull" again so my PEMA container is up to date?

Yes you are absolutely right. You now need to run
singularity pull shub://hariszaf/pema:v.1.3

Haris Zafeiropoulos

@hariszaf

Hello @hariszaf, I have trouble to run my data on Windows OS from Docker. When I run the data the result like this: Fatal error: ./pema_latest.bds, line 80, pos 27. Map 'paramsFirstAssignment' does not have key 'outputFolderName'. Please help me if you have a suggestion. Thank you

Hi @daniealmalik354 ! Download the parameters.tsv file from here. It would be better to do that using the wget command.
There is a chance that you have completed the parameters file using spaces instead of tabs. Please make sure that you are using tabs.

3 replies

Haris Zafeiropoulos

@hariszaf

Hi @/all !
The convertion of Illumina's raw data to the ENA formt is now part of the PEMA code.
pema:v.1.3.1 is now available as both Docker and Singularity containers and from now on you do not have to convert your raw data on your own! You just need to set the parameter EnaData to No and PEMA will do the convertion automatically.