text:
Installing Sibelia for Ragout on Mac OSX
I am trying to use Ragout: https://github.com/fenderglass/Ragout
to fill the gaps in my de novo genome assembly. You can access the article freely here: https://www.ncbi.nlm.nih.gov/pubmed/24931998
For this, I first need to install Sibelia.
I tried doing so by cloning the GitHub repository into output/software/ragout/ and then running the following command:
git clone https://github.com/fenderglass/Ragout output/software/ragout

python2 output/software/ragout/scripts/install-sibelia.py

Installing Sibelia
Downloading source...
-- The CXX compiler identification is AppleClang 9.0.0.9000037
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is AppleClang 9.0.0.9000037
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test HAVE_UNKNOWN_WALL
-- Performing Test HAVE_UNKNOWN_WALL - Success
-- Performing Test HAVE_UNKNOWN_FOMIT_FRAME_POINTER
-- Performing Test HAVE_UNKNOWN_FOMIT_FRAME_POINTER - Success
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for strings.h
-- Looking for strings.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Performing Test HAVE_INLINE
-- Performing Test HAVE_INLINE - Success
-- Performing Test HAVE___INLINE
-- Performing Test HAVE___INLINE - Success
-- Performing Test HAVE___INLINE__
-- Performing Test HAVE___INLINE__ - Success
-- Performing Test HAVE___DECLSPEC_DLLEXPORT_
-- Performing Test HAVE___DECLSPEC_DLLEXPORT_ - Failed
-- Performing Test HAVE___DECLSPEC_DLLIMPORT_
-- Performing Test HAVE___DECLSPEC_DLLIMPORT_ - Failed
-- Check size of uint8_t
-- Check size of uint8_t - done
-- Check size of int32_t
-- Check size of int32_t - done
-- Looking for PRId32
-- Looking for PRId32 - found
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/build
Scanning dependencies of target divsufsort
[  4%] Building C object libdivsufsort-2.0.1/lib/CMakeFiles/divsufsort.dir/divsufsort.o
[  8%] Building C object libdivsufsort-2.0.1/lib/CMakeFiles/divsufsort.dir/sssort.o
[ 12%] Building C object libdivsufsort-2.0.1/lib/CMakeFiles/divsufsort.dir/trsort.o
[ 16%] Building C object libdivsufsort-2.0.1/lib/CMakeFiles/divsufsort.dir/utils.o
[ 20%] Linking C static library libdivsufsort.a
[ 20%] Built target divsufsort
Scanning dependencies of target Sibelia
[ 24%] Building CXX object CMakeFiles/Sibelia.dir/sibelia.cpp.o
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/sibelia.cpp:8:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.h:10:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/align.h:43:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/sequence.h:103:
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/sequence/string_set_dependent_generous.h:38:9: warning:
      'SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_' is used as a header
      guard here, followed by #define of a different macro [-Wheader-guard]
#ifndef SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/sequence/string_set_dependent_generous.h:39:9: note:
      'SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUSH_' is defined here; did you
      mean 'SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_'?
#define SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUSH_
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/sibelia.cpp:8:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.h:10:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/align.h:44:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score.h:48:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score/score_matrix.h:40:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/file.h:70:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system.h:82:
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system/system_sema.h:120:27: warning:
      'sem_init' is deprecated [-Wdeprecated-declarations]
            SEQAN_DO_SYS(!sem_init(hSemaphore, 0, init));
                          ^
/usr/include/sys/semaphore.h:55:42: note: 'sem_init' has been explicitly marked
      deprecated here
int sem_init(sem_t *, int, unsigned int) __deprecated;
                                         ^
/usr/include/sys/cdefs.h:176:37: note: expanded from macro '__deprecated'
#define __deprecated    __attribute__((deprecated))
                                       ^
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/sibelia.cpp:8:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.h:10:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/align.h:44:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score.h:48:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score/score_matrix.h:40:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/file.h:70:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system.h:82:
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system/system_sema.h:124:27: warning:
      'sem_destroy' is deprecated [-Wdeprecated-declarations]
            SEQAN_DO_SYS(!sem_destroy(hSemaphore));
                          ^
/usr/include/sys/semaphore.h:53:26: note: 'sem_destroy' has been explicitly
      marked deprecated here
int sem_destroy(sem_t *) __deprecated;
                         ^
/usr/include/sys/cdefs.h:176:37: note: expanded from macro '__deprecated'
#define __deprecated    __attribute__((deprecated))
                                       ^
3 warnings generated.
[ 28%] Building CXX object CMakeFiles/Sibelia.dir/postprocessor.cpp.o
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.cpp:9:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.h:10:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/align.h:43:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/sequence.h:103:
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/sequence/string_set_dependent_generous.h:38:9: warning:
      'SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_' is used as a header
      guard here, followed by #define of a different macro [-Wheader-guard]
#ifndef SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/sequence/string_set_dependent_generous.h:39:9: note:
      'SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUSH_' is defined here; did you
      mean 'SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_'?
#define SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUSH_
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        SEQAN_SEQUENCE_STRING_SET_DEPENDENT_GENEROUS_H_
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.cpp:9:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.h:10:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/align.h:44:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score.h:48:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score/score_matrix.h:40:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/file.h:70:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system.h:82:
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system/system_sema.h:120:27: warning:
      'sem_init' is deprecated [-Wdeprecated-declarations]
            SEQAN_DO_SYS(!sem_init(hSemaphore, 0, init));
                          ^
/usr/include/sys/semaphore.h:55:42: note: 'sem_init' has been explicitly marked
      deprecated here
int sem_init(sem_t *, int, unsigned int) __deprecated;
                                         ^
/usr/include/sys/cdefs.h:176:37: note: expanded from macro '__deprecated'
#define __deprecated    __attribute__((deprecated))
                                       ^
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.cpp:9:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/postprocessor.h:10:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/align.h:44:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score.h:48:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/score/score_matrix.h:40:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/file.h:70:
In file included from /Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system.h:82:
/Users/cr517/Documents/phd/project/sibelia-build/Sibelia-master/src/include/seqan/system/system_sema.h:124:27: warning:
      'sem_destroy' is deprecated [-Wdeprecated-declarations]
            SEQAN_DO_SYS(!sem_destroy(hSemaphore));
                          ^
/usr/include/sys/semaphore.h:53:26: note: 'sem_destroy' has been explicitly
      marked deprecated here
int sem_destroy(sem_t *) __deprecated;
                         ^
/usr/include/sys/cdefs.h:176:37: note: expanded from macro '__deprecated'
#define __deprecated    __attribute__((deprecated))
                                       ^
3 warnings generated.
[ 32%] Building CXX object CMakeFiles/Sibelia.dir/indexedsequence.cpp.o
[ 36%] Building CXX object CMakeFiles/Sibelia.dir/util.cpp.o
[ 40%] Building CXX object CMakeFiles/Sibelia.dir/outputgenerator.cpp.o
[ 44%] Building CXX object CMakeFiles/Sibelia.dir/blockfinder.cpp.o
[ 48%] Building CXX object CMakeFiles/Sibelia.dir/blockinstance.cpp.o
[ 52%] Building CXX object CMakeFiles/Sibelia.dir/bifurcationstorage.cpp.o
[ 56%] Building CXX object CMakeFiles/Sibelia.dir/bulgeremoval.cpp.o
[ 60%] Building CXX object CMakeFiles/Sibelia.dir/dnasequence.cpp.o
[ 64%] Building CXX object CMakeFiles/Sibelia.dir/edge.cpp.o
[ 68%] Building CXX object CMakeFiles/Sibelia.dir/fasta.cpp.o
[ 72%] Building CXX object CMakeFiles/Sibelia.dir/serialization.cpp.o
[ 76%] Building CXX object CMakeFiles/Sibelia.dir/synteny.cpp.o
[ 80%] Building CXX object CMakeFiles/Sibelia.dir/test/unrolledlisttest.cpp.o
[ 84%] Building CXX object CMakeFiles/Sibelia.dir/platform.cpp.o
[ 88%] Building CXX object CMakeFiles/Sibelia.dir/stranditerator.cpp.o
[ 92%] Building CXX object CMakeFiles/Sibelia.dir/vertexenumeration.cpp.o
[ 96%] Building CXX object CMakeFiles/Sibelia.dir/resource.cpp.o
[100%] Linking CXX executable Sibelia
[100%] Built target Sibelia
[ 20%] Built target divsufsort
[100%] Built target Sibelia
Install the project...
-- Install configuration: "Release"
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/bin/Sibelia
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/NEWS.md
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/ANNOTATION.md
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/README.md
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/USAGE.md
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/INSTALL.md
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/SIBELIA.md
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/C-SIBELIA.md
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/LICENSE.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/C-Sibelia
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/C-Sibelia/Staphylococcus_aureus
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/C-Sibelia/Staphylococcus_aureus/NCTC8325.fasta
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/C-Sibelia/Staphylococcus_aureus/README.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/C-Sibelia/Staphylococcus_aureus/RN4220.fasta
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/C-Sibelia/Staphylococcus_aureus/variant.vcf
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/blocks_coords.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.conf
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.highlight.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.highlight1.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.highlight2.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.highlight3.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.highlight4.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.image.conf
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.png
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.segdup.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.sequences.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/circos/circos.svg
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/coverage_report.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/d3_blocks_diagram.html
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/genomes_permutations.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/Helicobacter_pylori.fasta
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Helicobacter_pylori/README.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/blocks_coords.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos/circos.conf
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos/circos.highlight.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos/circos.image.conf
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos/circos.png
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos/circos.segdup.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos/circos.sequences.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/circos/circos.svg
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/coverage_report.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/d3_blocks_diagram.html
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/genomes_permutations.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/README.txt
-- Installing: /Users/cr517/Documents/phd/project/sibelia-build/share/Sibelia/doc/examples/Sibelia/Staphylococcus_aureus/Staphylococcus.fasta
Traceback (most recent call last):
  File "output/software/ragout/scripts/install-sibelia.py", line 111, in <module>
    sys.exit(main())
  File "output/software/ragout/scripts/install-sibelia.py", line 107, in main
    return int(not install_deps(args.prefix))
  File "output/software/ragout/scripts/install-sibelia.py", line 31, in install_deps
    return install_sibelia(prefix)
  File "output/software/ragout/scripts/install-sibelia.py", line 64, in install_sibelia
    shutil.copy(sibelia_bin_src, sibelia_bin_dst)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 119, in copy
    copyfile(src, dst)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: '/Users/cr517/Documents/phd/project/lib/Sibelia'

As you can see, I get a lot of errors. Could someone help me, please?
I also tried to install Sibelia following their instructions here: https://github.com/bioinf/Sibelia
cd Sibelia/build
➜  build git:(master) cmake ../src
-- The CXX compiler identification is AppleClang 9.0.0.9000037
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The C compiler identification is AppleClang 9.0.0.9000037
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test HAVE_UNKNOWN_WALL
-- Performing Test HAVE_UNKNOWN_WALL - Success
-- Performing Test HAVE_UNKNOWN_FOMIT_FRAME_POINTER
-- Performing Test HAVE_UNKNOWN_FOMIT_FRAME_POINTER - Success
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for strings.h
-- Looking for strings.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Performing Test HAVE_INLINE
-- Performing Test HAVE_INLINE - Success
-- Performing Test HAVE___INLINE
-- Performing Test HAVE___INLINE - Success
-- Performing Test HAVE___INLINE__
-- Performing Test HAVE___INLINE__ - Success
-- Performing Test HAVE___DECLSPEC_DLLEXPORT_
-- Performing Test HAVE___DECLSPEC_DLLEXPORT_ - Failed
-- Performing Test HAVE___DECLSPEC_DLLIMPORT_
-- Performing Test HAVE___DECLSPEC_DLLIMPORT_ - Failed
-- Check size of uint8_t
-- Check size of uint8_t - done
-- Check size of int32_t
-- Check size of int32_t - done
-- Looking for PRId32
-- Looking for PRId32 - found
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/cr517/Documents/phd/project/Sibelia/build
➜  build git:(master) make
Scanning dependencies of target lagan
[  3%] Creating directories for 'lagan'
[  6%] Performing download step (DIR copy) for 'lagan'
[  9%] No patch step for 'lagan'
[ 12%] No update step for 'lagan'
[ 15%] No configure step for 'lagan'
[ 18%] Performing build step for 'lagan'
rightinfluence.cpp:21:2: error: reference to 'end' is ambiguous
        end.score = -2;
        ^
rightinfluence.cpp:3:18: note: candidate found by name lookup is 'end'
Fragment origin, end;
                 ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iterator:1768:1: note:
      candidate found by name lookup is 'std::__1::end'
end(const _Cp& __c)
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iterator:1611:1: note:
      candidate found by name lookup is 'std::__1::end'
end(_Tp (&__array)[_Np])
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iterator:1760:1: note:
      candidate found by name lookup is 'std::__1::end'
end(_Cp& __c)
^
rightinfluence.cpp:22:22: error: reference to 'end' is ambiguous
        origin.totalScore = end.totalScore = 0;
                            ^
rightinfluence.cpp:3:18: note: candidate found by name lookup is 'end'
Fragment origin, end;
                 ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/iterator:1768:1: note:

...

There might be two issues, one is a hard coded or an incorrect path, the other one might be due to not using sudo or root rights

Hi, I tried doing the commands with sudo but I still get similar errors. The first failure is: -- Performing Test HAVE___DECLSPEC_DLLEXPORT_ -- Performing Test HAVE___DECLSPEC_DLLEXPORT_ - Failed -- Performing Test HAVE___DECLSPEC_DLLIMPORT_ -- Performing Test HAVE___DECLSPEC_DLLIMPORT_ - Failed

@Llopis, how can I improve my attempt at installing this software with non-hard-coded path and correct path?

This is a fairly obscure package, I have a feeling that you'll end up needing to post this as a github issue and hope that the developers reply there.

I posted the issue on Github here: https://github.com/fenderglass/Ragout/issues/23

@charlesdarwin modify the source code before installing or via a patch in github. However, make the issues self-contained, include the version number, the machine description and everything that might be relevant

@Llopis, sorry, what do you mean by modifying the source code? Modify the code that's present in here: https://github.com/fenderglass/Ragout ? And how to I get the version number? I will add my machine specs: Mac OSX Sierra version 10.12.6

Yes, that is the source code of the application you want to install

OK, I'll have a look at it and study it.

There is a link to compiled binaries for OSX on the project homepage

Indeed, there are also Ragout binaries available on github. The easiest solution to all of this is to not compile and instead just use the binaries.

Hi, regarding Sibelia, I have done the following: In R: download.file('https://sourceforge.net/projects/sibelia-bio/files/3.0.6/Sibelia-3.0.6-Source.tar.gz', 'output/software/sibelia/Sibelia-3.0.6-Darwin.tar.gz')
In bash: tar -xvzf output/software/sibelia/Sibelia-3.0.6-Darwin.tar.gz -C output/software/sibelia/ chmod +rwxrwxrwx output/software/sibelia/Sibelia-3.0.6-Source/src/csibelia/C-Sibelia.py
This gives me access to the C-Sibelia.py software but I cannot find any Sibelia.py in my computer. Do you know if I am downloading the right thing?

I tried installing Ragout from the binaries:
In R: download.file('https://github.com/fenderglass/Ragout/releases/download/2.0/ragout-2.0-macos-x86_64.tar.gz', 'ragout-2.0-macos-x86_64.tar.gz')
In bash:
tar -xvzf ragout-2.0-macos-x86_64.tar.gz  chmod +rwxrwxrwx ragout-2.0-macos-x86_64/ragout.py ragout-2.0-macos-x86_64/ragout.py -h The help page is then opened so I consider it installed.


text:
What software/tool can I use to visualise a multiple sequence alignment in the xmfa format as produced by command-line tool Mauve?
What software/tool can I use to visualise a multiple sequence alignment in the xmfa format as produced by command-line tool Mauve?
I am particularly interested in seeing the single nucleotide polymorphisms on the screen.
I am doing a multiple sequence alignment of about a dozen whole genomes, all in a single contig.
Link to Mauve

What sort of alignment? How may sequences? Are you thinking reads against genome, or is this multiple full length genomic sequences aligned against each other? Since you mention mauve, should we assume whole genome alignments?

Whole genome alignments of about a dozen full bacterial chromosomes.

Typically I use the Mauve viewer to visualize the alignments.  Are there things you find lacking in that viewer?
There are multiple threads about converting xmfa to other formats

I would like to see the sequences more zoomed in, at the base level. Do you know if it's possible with the Mauve viewer?

I don't think so.  Mauve is a designed to find large segments of homology and is not really for per base alignments

I supervised a student a while ago, and she made a tool to focus on (i.e., visualize) SNPs in multiple alignments. The tool is called ADOMA, and is quite simple. It uses clustal for alignment, so I am not sure it if it may work for your Mauve data. If your data are xmfa, you'll probably have to make fasta of it first.
So I am not sure if this is a solution, but maybe the tool can be of use for your research.

Can ADOMA be used for whole genome alignments, e.g. bacterial genomes? Or is just for gene-size alignments?

Sorry I am not sure. The tool was made by a very good (independent working) student. It uses clustal, and if I recall correctly, clustal can also take aligned data already. So I guess your aligned data can maybe be formatted to the right input for clustal? And then adoma could in theory show only SNPs in the alignment. I think whole genome data would make very large output files, so I am not sure if it would be feasible.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/

sort: votes


text:
How can I not show insertions in the Integrative Genome Viewer (IGV)?
I am using IGV 2.5.2 and would not like to see the insertions  in my aligned reads. How can I do that? I have managed to remove mismatched bases because there is an option to do so when I right-click on the window ("Show mismatched bases"). Now I would like to do the same with these purple insertions.

3rd January 2020 update

Thanks to a member's response, I found some parameter in the preferences that might be helpful to hide the indels. However, I have tried different numbers from 0 to 1000 and the indels are still shown. What numbers should I put to the parameters "Show indel threshold (bases)" and should I click the box or not?

Check the box and try 1 as the threshold.

Preferences -> Alignments -> Hide indels < 'x' bases.

Credit to Pierre Lindenbaum

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Problem installing last build of Sniffles
I have followed the steps described here to install Sniffles on my Mac Book Pro running macOS High Sierra.
However, I am struggling with this step:
cmake -D CMAKE_C_COMPILER=/opt/local/bin/gcc-mp-4.7 -D CMAKE_CXX_COMPILER=/opt/local/bin/g++-mp-4.7 ..

I do not have an /opt folder, so I get the following output:
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:2 (project):
  The CMAKE_C_COMPILER:

    /opt/local/bin/gcc-mp-4.7

  is not a full path to an existing compiler tool.

  Tell CMake where to find the compiler by setting either the environment
  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
  the compiler, or to the compiler name if it is in the PATH.

CMake Error at CMakeLists.txt:2 (project):
  The CMAKE_CXX_COMPILER:

    /opt/local/bin/g++-mp-4.7

  is not a full path to an existing compiler tool.

  Tell CMake where to find the compiler by setting either the environment
  variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.

-- Configuring incomplete, errors occurred!
See also "/Users/xxx/Documents/xxx/xxx/Sniffles-master/build/CMakeFiles/CMakeOutput.log".
See also "/Users/xxx/Documents/xxx/xxx/Sniffles-master/build/CMakeFiles/CMakeError.log".

Unsurprisingly, the error file says that /opt/local/bin/gcc-mp-4.7 does not exist.
So I tried:
cmake -D CMAKE_C_COMPILER=/usr/bin/gcc -D CMAKE_CXX_COMPILER=/usr/bin/gcc ..

I get another long message but the most important bit is:
CMake Error at /usr/local/Cellar/cmake/3.8.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find OpenMP (missing: OpenMP_C_FLAGS OpenMP_CXX_FLAGS)

I do not know what to do to solve this error.
I do have gcc here: /usr/bin/gcc and cmake version 3.8.1.
I have created an issue on the Sniffles GitHub repository.
Update 13th June 2018
I installed the latest version of gcc:
brew install gcc
Warning: gcc 8.1.0 is already installed and up-to-date
To reinstall 8.1.0, run `brew reinstall gcc`

Then, I tried to make cmake use gcc and g++ in the /usr/local/bin folder.
cmake -D CMAKE_C_COMPILER=/usr/local/bin/gcc-8 -D CMAKE_CXX_COMPILER=/usr/local/bin/g++-8 ..

This recreated the same problem:
CMake Error at /usr/local/Cellar/cmake/3.8.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find OpenMP (missing: OpenMP_C_FLAGS OpenMP_CXX_FLAGS)

Do I need to set the OpenMP_C*_FLAGS to something?

Do you have OpenMP installed? It's required.

@DevonRyan OpenMP is installed as part of GCC. The problem is that OP isn’t using GCC.

You need to install GCC, or a newer version of Clang.
Your compiler — though the binary is called gcc — is actually Clang, and the Clang version that’s shipping with macOS/Xcode still does not support OpenMP.
The easiest course of action is probably to install GCC via Homebrew (brew install gcc) and then configure Sniffles to use Homebrew’s GCC (i.e. setting the CMAKE_C*_COMPILER paths to /usr/local/bin/g{cc,++}-$version).

I have tried what you said, unsuccessfully. I have updated my original question.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
igv_plotter error: Xvfb did not start
I am trying to get screenshots from alignments of reads to a genome in IGV (Integrated Genome Viewer) using igv_plotter.
igv_plotter --genome ref.fa readsSorted.bam --locus-file locus_file.txt --max-memory 10G

Traceback (most recent call last):
  File "/Users/user/anaconda/envs/python3env/bin/igv_plotter", line 140, in <module>
    igv_robot.execute()
  File "/Users/user/anaconda/envs/python3env/lib/python3.6/site-packages/igv_api.py", line 256, in execute
    self._execute_impl(self.__command_queue)
  File "/Users/user/anaconda/envs/python3env/lib/python3.6/site-packages/igv_api.py", line 387, in _execute_impl
    fake_display.start()
  File "/Users/user/anaconda/envs/python3env/lib/python3.6/site-packages/xvfbwrapper.py", line 79, in start
    raise RuntimeError('Xvfb did not start')
RuntimeError: Xvfb did not start

locus_file.txt only contains this: I:10,000-20,000. From reading online, it seems that this Xvfb is what I need to be able to use IGV without a GUI. I am working on Mac High Sierra and I don't know if I can install Xvfb on Mac OS X or how I can do it. I was quickly out of my depths on the online posts about Xvfb on Mac. I checked that I installed igv_plotter correctly by successfully pulling out the help with igv_plotter -h. I could not find Xvfb using brew.
Alternatively, is there another programmatic way of taking images of aligned reads to a reference, similar to the one below?
Opening IGV and making a screenshot is not a viable option for me, as I would like to export hundreds or thousands or regions.

Hi, What is your question getting igv_plotter to work or find an alternative? If the first, why don't you use the gui to make the screenshots?

@Llopis Maybe because the result is unusable (importing it in Illustrator yields rubbish that’s very time-consuming to unravel).

Did you install XQuartz on your macOS? It should include Xvfb.

@Llopis, I want to automate this for thousands of regions, so opening the GUI is not an option.

@KonradRudolph XQuartz is installed on my macOS. Not sure what it is for but I can open it and it works like a Terminal.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
How do you download stuff from NCBI fast?
I have 2 .csv files that contains a list of accession codes. For example for this experiment a .csv file will contain:
SAMN10870244
SAMN10870243
SAMN10870242
SAMN10870241
SAMN10870240
SAMN10870239
SAMN10870238
SAMN10870237
SAMN10870236
SAMN10870235
SAMN10870234
SAMN10870233

Then I run the following
for i in $(cat *.csv); do echo ${i}; /usr/bin/sratoolkit/bin/fastq-dump --gzip ${i} & done

Some of my downloads takes a lot of time days. Is there a faster way to get stuff from NCBI?
Are there mirrors of NCBI to accelerate the process?
Why is it so slow?

EDIT 2024
You might want to look into:

ncbi-acc-download: A CLI tool for efficient downloads from NCBI Entrez.
NSDPY: A Python library for batch downloading with additional features like taxonomic sorting.

Answers to this question can be found on biostars. To give you a short summary:

Faster way might be to use the parallel-fastq-dump, as suggested in this answer. I never tested that tool though, my own experience is that prefetch is more stable than fastq-dump command.
In that same post it seems that the Japanese mirror seems faster, don't know if that is still valid.
The reason this is never optimized might be that downloading from SRA is not a task you'll have to do repeatedly, as described here in this post.

Thanks, comprehensive answer. Is it this site https://www.ddbj.nig.ac.jp/  ?

https://www.ddbj.nig.ac.jp/dra/index-e.html, yes it is the same.

You can try wget to download SRA files from NCBI server. This seems to be 2x faster in terms of CPU time than prefetch and has the advantage of not clogging your /home/ncbi/public/sra/ directory in case you have space limitation. The tutorial for wget downloading method is available at https://github.com/johanzi/ncbi_tutorial.
Here my timing of the prefetch method:
time prefetch SRR3300113
real    3m5.609s
user    0m6.332s
sys     0m2.076s

Versus the wget method:
time wget ftp://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR330/SRR3300113/SRR3300113.sra
real    3m4.696s
user    0m0.456s
sys     0m3.332s

That makes a user+sys time of 8.402s for prefetch and 3.788s for wget. Not so much difference for real time though (about a second).

Hello Johan, welcome to BIoinformatics.SE. Is there a reason to think that wget would be faster? Intuitively I would guess that it's the server that is clogged, then using a different program to query the server does not sound like something that would help. (by the way, my inability to see how it could help does not mean that it does not :-)

Hi Kamil, the wget process is about 2x faster in terms of CPU time than prefetch method based on my test:

user + sys sort of tells you how long CPUs were active during the real time of execution (i.e. for user real is what matters, you can check this). So, what I would get out of your nice test is, that the wget is a more optimized tool, but both of them are fetching the data from exactly the same place and the limiting factor is the speed of their ftp.

Yes. I would say the main advantage of wget is that you can choose where to put your downloaded SRA files while it goes in default into your home directory /home/user/ncbi/public/sra with prefetch. It can be a problem when you work on a server and you have limited disk usage for your home directory.

You can choose where to put the cache, at least as of a few versions ago, via vdb-config. I agree it's a total pain that it defaults to the home directory instead of the current one, but it's a fixable problem.


text:
How to cluster the human genes by pathways/system-biology/metabolic properties?
I would like to make a chord plot from my data. I have a list of genes that according to my experiments are divided into 64 clusters of enrichment pattern.
I would combine my 64 clusters with a second clustering. A clustering of all the human genes into subgroups of "interacting" genes (e.g. both are pathway, co-expressed) will allow me to combine the two clustering data on a chord plot.
In the analogy of the following graph the continents will be my 64 clusters of the genes and the edges/chords will be assigned based on the biological function  clustering. I guess that such a functional clustering can be done ore has been done using platforms such as: KEGG, genescf, or reactome.org.

Is there a biological pathway clustering of genes for lymphoblastoid or breast cell lines?
Are there any suggestions how to build the clustering between genes whose functions are strongly correlated in the cell?

Edit: the question is about getting the clustering from a system-biological perspective not how to plot chord diagrams.
Is there such a metric to quantify how likely two genes are correlated in their enrichment, function etc. For example, using STRING we can see that PIK3CA and PTEN are more co-functioning than PIK3CA and SF3B1. How can I do something like String on genome wide scale.

Its difficult to understand the experiment, so not easy to recommend something other than generic correlation/covariance

@MichaelG. Would like something that measure the distance between two genes on the multigraph of pathways.

To measure if two genes are functional similar I developed the BioCor package. It calculates a similarity score between genes by the measuring the amount of pathways  shared. However, it doesn't take into account the strength of the connection between two proteins. You could multiply the similarity by a value between 0 and 1 according to the number of evidences of the relation between the two proteins.
It is optimized for large amounts of data, you could even perform them in parallel, but I doubt you find information about pathways for most of the genes (In my informal test, I think that I could calculate similarities for about 1600 genes not the ~20000 of the human genome).
When I am interested in co-expression data I calculate the co-expression by correlation and add the similarity of the genes calculated with BioCor. The resulting distance matrix can be used for clustering. Maybe the section about using weighted co-expression network analysis and BioCor can be helpful.
To have a "biological pathway clustering of genes for lymphoblastoid or breast cell lines", you would need to have a specific signature/pathway of the cell lines. Currently I am not aware of accepted standard signature database. The single cell atlas could be useful to find expression of your cells, but I don't know if they provide a signature for cell line

Is there a python version for it?

No, but you can use it in python by using rpy2

Production of a chord diagram is described in detail here. You use the R package with either,
library(chorddiag)
library (circulize)

The first level clusters are predefined and the second layer of clustering is what you need to work out. The obvious stat is correlation, secondly (better approach IMO) you might look at covariance, although I don't exactly know what this "interaction" constitutes, which makes it difficult.
My personal view is the diagramatic representation doesn't really matter, its the robustness of the stats behind the diagram that counts.
I was a bit surprised to learn ggplot2 doesn't do circular diagrams :-(

Lymphoblastoid Cell lines

I think you need to clarify the experiment on these cells lines, I assume its a time course, i.e. time series.

At a guess this is RNA expression, if you are using a chip then its between predetermined genes (i.e. spotted on the chip). If it is RNA seq then it can be anything. I suspect you are treating the cells and monitoring changes in RNA expression as a function of their biological (immunological?) activity. If the basic measure is time, i.e. a time series - thus two genes being expressed in the same unit time the best opening statistic is covariance. If all you want to do is make a chord diagram out of it, I think thats fine, this will form a 2x2 variance matrix for which any standard clustering will work (Jaccard distances [there will be better ones]) or direct plotting from the matrix. If you looking at unique expression behaviour to attempt to unlock some sort of pathway then you are into more serious regression time series (autoregression that sort of thing - this is complicated). If its oscillations then its a separate branch of stats based on sinosoids (sine waves) called periodicity analysis. HOWEVER, if the interaction is genetic similarity (gene families) e.g. mRNA then you want a phylogenetics measure, which if its clustering neighbor-joining is fine.

My personal view if it is the experiment I think, I would,

Construct a matrix of the covariance of the (I assume) time series
Make sure the covariance is standarised between -1 to 1 (Pearson
approach), R-cran.
Assign a tight cut-off (tricky because there's no "hard cut-off") but say 0.7 (you need to experiment a bit)
Plot the raw covariance on the chord diagram.

Easy, simple and highly visual.

For my experiment, I know how to cluster it. My question is independent! I would to find a metric that assign a distance to two genes according to their functions. If they happen to function together in the same context/pathways they will be close. For example, the proteins of NRAS and PIK3CA will be closer than NRAS and SF3B1 (http://version10.5.string-db.org/cgi/network.pl?taskId=KbO2wnULZxKQ)
I am looking for a resource to be able to build such thing on a genome wide scale.


text:
How to analyze and visualize CRISPR-CAS9 screening results?
I have a whole-gene CRISPR screening of an assay at different time points. I created a table of the gene number and the count of its reads:
gene_id, count_t0, count_t1, count_t3
ABCA1       3        6          7

What would be a standard way to process such type of data to inference information on up-regulated/down-regulated genes.
I saw this tool: http://crispr-analyzer.dkfz.de/, but it requires my NGS raw data and these file are very large in my case. I'd expect that the information for analysis purposes can be derived solely from the counting table above.

Have you tried DESeq2? It's one of the options from the tool you linked to even.


text:
How do you query and explore ENCODE data?
I am looking for a modular way to query data from ENCODE.
For example, I would like to get CHiP-seq or similar tracks for a specific cell line. What's the proper way to do it?
Finally, is there an API to do it?

I don't know whether there is an API, but ENCODE's website does provide an interactive data matrix where you can filter data based on assay and sample type, place data sets in a "shopping cart", and then proceed to "checkout" to download the files of interest.

I'll add the fact that I've never used this website before today, so the fact that I was able to navigate it so easily is a credit to the group(s) that created and maintain the website!

You can just add /?format=json to any page to get the JSON output.
ENCODE REST API documentation: https://www.encodeproject.org/help/rest-api/
Example scripts: https://github.com/ENCODE-DCC/submission_sample_scripts

So let's say I want to get the bed file of this experiment using curl https://www.encodeproject.org/experiments/ENCSR000CKC/

If you look at the JSON for that experiment, there are multiple BED files listed there: https://www.encodeproject.org/experiments/ENCSR000CKC/?format=json

So let's say I want to get the bed file of this experiment using curl encodeproject.org/experiments/ENCSR000CKC
You can also search for files that belong to a given experiment, e.g. https://www.encodeproject.org/search/?type=File&dataset=/experiments/ENCSR000CKC/

Then further select file properties (status, format, etc.) using the facets on the left.
This basically builds up a search URL for you, which is also possible to do manually. For example all BED files in ENCSR000CKC with GRCh38 assembly, returned as JSON: https://www.encodeproject.org/search/?type=File&dataset=/experiments/ENCSR000CKC/&file_format=bed&assembly=GRCh38&format=json (See properties tab here https://www.encodeproject.org/profiles/file.json for all of the file fields you can use.)
You can also specify a field parameter to return only fields you are interested in. For example https://www.encodeproject.org/search/?type=File&dataset=/experiments/ENCSR000CKC/&file_format=bed&assembly=GRCh38&field=s3_uri&field=cloud_metadata.url&field=dataset&field=file_format&format=json only returns the s3_uri, cloud_metadata.url, dataset, and file_format fields. These objects are returned in the @graph list:

Putting this all together to parse in Python:
>>> import requests
>>> import pandas as pd
>>> url = (
...     'https://www.encodeproject.org'
...     '/search/?type=File'
...     '&dataset=/experiments/ENCSR000CKC/'
...     '&file_format=bed'
...     '&assembly=GRCh38'
...     '&field=file_format'
...     '&field=assembly'
...     '&field=s3_uri'
...     '&field=cloud_metadata.url'
...     '&format=json'
... )
>>> r = requests.get(url)
>>> files = r.json()['@graph']
>>> df = pd.DataFrame(files)
>>> df['cloud_metadata'] = df['cloud_metadata'].apply(lambda x: x.get('url'))
>>> df[['@id', 'assembly', 'file_format', 's3_uri', 'cloud_metadata']]
                   @id assembly file_format                                                                                s3_uri                                                                                cloud_metadata
0  /files/ENCFF368QQU/   GRCh38         bed  s3://encode-public/2016/08/04/10493464-917f-4290-bb13-c5fec8319887/ENCFF368QQU.bed.gz  https://encode-public.s3.amazonaws.com/2016/08/04/10493464-917f-4290-bb13-c5fec8319887/ENCFF368QQU.bed.gz
1  /files/ENCFF554NVI/   GRCh38         bed  s3://encode-public/2016/08/04/865f220e-7a47-427a-a7c1-9bdc049d4658/ENCFF554NVI.bed.gz  https://encode-public.s3.amazonaws.com/2016/08/04/865f220e-7a47-427a-a7c1-9bdc049d4658/ENCFF554NVI.bed.gz
2  /files/ENCFF217CRP/   GRCh38         bed  s3://encode-public/2016/08/04/43aa693e-1581-4886-8ace-7e1609f45790/ENCFF217CRP.bed.gz  https://encode-public.s3.amazonaws.com/2016/08/04/43aa693e-1581-4886-8ace-7e1609f45790/ENCFF217CRP.bed.gz

The s3_uri and cloud_metadata.url both point to the same thing. You can use the s3_uri to download using awscli or boto3. You can pass the https link to cURL.

There are two places:

encodeproject.org
screen.encodeproject.org

Details can be found in the ENCODE3 papers published in July 2020


text:
How to get the GO information for all the human genes?
I would like to retrieve a list of all the human genes and their proteins’ functional classification.
The Gene type attribute on ensembl.org is not specific enough, because it assign the label coding protein to all the proteins for which I am trying to classify. Also the KEGG Pathway and Enzyme ID option doesn't seem to help.
I have translated all the gene names to their GO representation, but now I am not sure how to query for their cellular function, biological process, and cellular component.
Is there such a platform that can retrieve this info from PANTHER?
Sadly, the reference proteome of humans doesn't contain a functional classification either:
 zcat UP000005640_9606.fasta.gz | grep -i mYOSIN-2
>sp|Q9UKX2|MYH2_HUMAN Myosin-2 OS=Homo sapiens OX=9606 GN=MYH2 PE=1 SV=1

# >db|UniqueIdentifier|EntryName ProteinName OS=OrganismName OX=OrganismIdentifier [GN=GeneName ]PE=ProteinExistence SV=SequenceVersion

Is there any way to use bgee, QuickGO, GONet, or Uniport to cluster genes using the customize columns?
EDIT:
I think that getting the most specific (GO term name) and least specific (GO domain) gene's GO values can be retrieved from ensmebl:

If you really need ALL genes then you won't find such a database, because we don't know all of their functions.

Have you looked at GO annotations? They are an hierarchical annotation for gene function. For enzymes the EC numbers are also an option. PFam is for structural domains, which generally align with function.

GO terms are used routinely to cluster genes for enrichment analysis. There are three categories, location, function and process, and a gene/protein will have one or more descendants (it's a hierarchical ontology). There are online tools that classify genes, such as GOrilla or Reactome. Programmatically you can use say Goatools in Python. There is a R package (don't recall name). And I think a Julia.

@MatteoFerla thanks. Let me know if you have a working example. I found GOATOOLS: https://www.nature.com/articles/s41598-018-28948-z


text:
Are there genomics databases for neurological and psychiatric disorders?
Are there databases that relate genetic disorders with neurological disorders?
I am thinking of something like COSMICS for cancer? AFAIK cancer usually develops due to a combination of 5-10 mutations.
Also if you can comment how genomics/bioinformatics in neurological contexts is compared to the how bioinformaticians study cancer?
Related: https://www.nimhgenetics.org/resources/psychencode

Try with PsyGeNet:

PsyGeNET (Psychiatric disorders Gene association NETwork) is a resource for the exploratory analysis of psychiatric diseases and their associated genes

It is based on literature mining and curated by experts.
Here is the publication if you want more details:
PsyGeNET: a knowledge platform on psychiatric disorders and their genes
and they even have a Bioconductor package for programatic queries:
psygenet2r: a R/Bioconductor package for the analysis of psychiatric disease genes


text:
CRISPR screening and systems biology
I am not sure how to tackle this one, so I’ll explain the general idea I have in mind.
Given results of a CRISPR knockout screening experiment (like in this paper) with two types of assays: control vs. treated with 5 time points per each of the assays.
Is there a way to use the screening data to perform systems biology network analysis? Something which is related to signal transduction, transcriptional regulatory, or metabolic networks?
Maybe something which uses Bayesian inference? Or something similar to differentially expressed representation of the CRISPR screening? Alternatively, a GWAS analysis similar to this paper?

do you have a list of genes that you would like to investigate through network analysis?

@gabrielet, yes I do have such a list. Even a few related such lists. But I also want to incorporate the information from the up regulation and down regulation values I got in the analysis. Not just the list of genes

To investigate genes through a network based approach you can use several tools. I don't know which level of programming you have or which tool you prefer but here is a brief overview of the possibilities you have.
You can look into established databases containing information about interactions like, for instance, STRING. Or, you can build networks using the wgcna method with R.
Then, once you have your network you can analyse its properties and here you also have several tools available such as standalone, java-base software like cytoscape, R libraries such as igraph, python libraries such as networkX online tools like centiserver...
Also you can perform gene set enrichment analsysis, which means adding a further layer of biological information to you set of genes like metabolic pathways (from KEGG, for example) and molecular functions (from GeneOntology, for instance) (gsea) as here, here, and here.
Here there is also a nice tutorial about network analysis with R.

What do you call screening results?
Do you have a kind of array with protein or RNA data?
If so, you can use several open-source tools available on the internet, like Enrichr or network analyst to better understand the pathways affected by your treatment.

https://www.nature.com/articles/s41467-018-06676-2

what's network analyst?

https://www.networkanalyst.ca


text:
How to convert a .fastq chip-seq file into a wig/bigwig track and how to extract chip-seq peaks?
I have a reference .fasta file and a raw .fastq file with chip-seq data. I am trying to create a bigwig track from the and .fastq and .fasta ref file of the raw signal. Then I would like to do some peak analysis for the chip-seq track.
Can someone help me with googling the write things, I feel really lost while trying to google it and not sure where to start from. I was thinking to do .fastq -> .bam -> wig -> bigwig and then somehow to do peak detection analysis.

Maybe follow some tutorials first? On Biostars many links to helpful tutorials.

Before you proceed, it is usual to do quality control on the .fastq data.  The FASTQC program is a good place to start.
Once you have checked the quality, trimming is the typical next step to remove low-quality data.  There are options such as Trimomatic, trimGalore for this.
Once you are sure you have clean data, your first step is alignment - align the sequences in the .fastq file to the reference sequence in the .fasta file.  bowtie2 is one option for this process.  This gets you to the .bam stage.
These steps will take a little time, some research, and may be somewhat frustrating, but they are the foundation for what you want to do. One option for somewhat automating these steps is to use the Galaxy platform (usegalaxy.org), otherwise these are mostly command-line tools.
The bam to wig and wig to bigwig steps are relatively simple conversions, and Biostars has answers for these.

What about peak calling once I have a bigwig file?

I have heard of the R package BayesPeak, but ChIP-Seq is not really my thing.

macs2 should do the work.


text:
How to import a large amount of .bed, .gff, .vcf, .paf, .sam files into an SQL database?
Are there best practices to load different bioinformatics file formats such as VCF, BED, GFF, and SAM to SQL databases? I am wondering how people out there do that efficiently.
All of these three formats are tab-separated files, so basically the following should work. I feel weird about it since most people I know don't use MySQL to work with these files.
LOAD DATA LOCAL INFILE 'bed.bed'
INTO TABLE bed-file
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS (list of the columns)
SET creation_date  = STR_TO_DATE(@creation_date, '%m/%d/%y');

Did you ever find a satisfactory answer to this question? I'd love to hear more about what sort of solution, if any, you ended up with.

@JessimeKirk not really. I've heard recently about Glow. Maybe it will be helpful to you.

The clodius tool puts tab-delimited data from a BED file into an rtree in a sqlite3 database:
https://github.com/higlass/clodius/blob/c98bb16ade93402fcea3b749d705c52ea165b609/clodius/cli/aggregate.py#L457-L777
Data are stored in this database at multiple "zoom levels", which allows for fast queries at wide genomic extents:
https://github.com/higlass/clodius/blob/develop/clodius/tiles/beddb.py
Aggregation (input) and querying (output) seem efficient in practice, but efficiency may depend on aspects of your specific dataset, i.e., whether you mean efficiency in execution time of INSERT or SELECT statements, or in space efficiency in the overall database size.
Other data can be converted to BED via vcf2bed, sam2bed, gff2bed, etc. and then aggregated. This is not the only way to do things, of course, but hopefully this can suggest directions to explore.

You might also take a look at this genomics-focused sqlite extension: https://github.com/mlin/GenomicSQLite
As described in the repository README:

This SQLite3 loadable extension adds features to the ubiquitous
embedded RDBMS supporting applications in genome bioinformatics:

genomic range indexing for overlap queries & joins
in-SQL utility functions, e.g. reverse-complement DNA, parse "chr1:2,345-6,789"
automatic streaming storage compression (also available standalone)
reading directly from HTTP(S) URLs (also available standalone)
pre-tuned settings for "big data"


text:
Are there open-sourced graph-based variant callers?
I would like to compare GATK with graph-based variant callers. I have seen the Fast and accurate genomic analyses using genome graphs paper from SevenBridges and the paragraph tool by Illumina, though it does not look like folks use their platform or it is not open-sourced (in the case of SevenBridges's GRAF).
The idea is to see if graph-based approaches detect better both SNPs and structural variants compared with existing methods. Additionally, I believe that if there exists some novel graph-based aligner/assembler one could use the metadata downstream to identify variants.

Not sure about variant callers, but since you asked about aligners, you should check out vg.

@user438383 thanks, it's worth mentioning WhatsHap, GraphAligner, and Minigraph as well.

Yes, there are. There are some suggestions in the comments (VG, WhatsHap, GraphAligner, Minigraph).
However, to be clear, the current default variant calling algorithm used in the GATK (the haplotype caller) is a graph-based algorithm. It constructs a De Bruijn graph from mapped reads for local assembly around variants. Other tools that also follow this strategy include Platypus and freebayes.
You might also consider multi-color genome assemblers. For example, Cortex and Corticall use colored De Bruijn graphs to aggregate reads from multiple genomes and call variants.

I think there is a difference between using de Bruin graph and represent a genome as a graph. The colored graph assemblers are interesting. Thank you.

I completely agree.


text:
How can I build a protein network pathway from a gene expression quantification file?
Assume I have found the top 0.01% most frequent genes from a gene expression file.
Let's say, these are 10 genes and I want to study the protein protein interactions, the protein network and pathway.
I thought to use string-db or interactome, but I am not sure what would be a plausible way to approach this problem and to build the protein network etc. Are there other more suitable databases?
How can I build a mathematical graph or network for these data?

There are multiple ways to do this, and multiple protein interaction databases besides the ones you mentioned, such as BioGRID or IntAct. Interaction databases are different in how interactions are defined, sometimes it can be experimental evidence of interaction, sometimes coexpression, orthology-based predictions, etc.
There is no single solution to your problem. For String-DB you can use their R package STRINGdb.
# string-db
library(STRINGdb)

# retrieve full graph for H. sapiens
string_db <- STRINGdb$new(version="10", species=9606,
                          score_threshold=400, input_directory="" )

# define genes of interest
genes.of.interest <- ...

# get their neighbors
neighbors <- string_db$get_neighbors(genes.of.interest)

# get the subgraph of interest (your genes + their neighbors)
my.subgraph <- string_db$get_subnetwork(c(genes.of.interest, neighbors))

# look how many genes and interactions you have
my.subgraph

If you prefer GUI, you might consider using Cytoscape. They have multiple addons for interaction databases, e.g. for STRINGdb. There you can just provide your gene list and get a network in a few clicks.

Note that STRING is very dirty. if you want to use it toi build a network it is essential to filter the results by detection method (it has many, many inferred interactions).

Many interaction databases now work with PSI format files. Most of the main databases can do this and the EBI has set up PSICQUIC View, a very useful page where you can query multiple databases at once.
Note that it is very important to limit the results according to the detection method. There is a lot of noise in protein interaction databases. Depending on what you want to do you could limit to only experimentally verified interactions or to only direct, binary interactions
(so exclude the results of, for example, ChIP analyses which can also find complexes) etc.
That said, here's a simple example script that will query the APID database using its PSICQUIC service:
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;

my @proteins;
## Read a list of target proteins, one per line (this expects UniProt names)
open(my $fh, "$ARGV[0]") or die "Need a list of proteins as the 1st argument: $!\n";
while (<$fh>) {
    chomp;
    push @proteins, $_;
}
close($fh);
## Get human interactions only
my $species="9606";

## Get the interactions for each target protein
foreach my $protein (@proteins) {
    my $queryUrl= "http://cicblade.dep.usal.es/psicquic-ws/webservices/current/search/query/$protein";
    $queryUrl  .= "?taxidA:$species%20ANDtaxidB:$species";

    my $tries=1;
    my $content = get $queryUrl;
    while ($tries<=10) {
        if (defined($content)) {
            $tries=11;
        } else {
            print STDERR "Could not retrieve $queryUrl, retrying($tries)...\n";
            $content = get $queryUrl;
        }
        $tries++;
    }

    # Now list all interactions
    my @lines = split(/\n/, $content);
    my $LINES= @lines;
    my $count = 0;
    for my $line (@lines) {
        $count++;
        my @flds = split(/\t/, $line); # split tab delimited lines
        # split fields of a PSIMITAB 2.5 line
        my ($idA, $idB, $altIdA, $altIdB, $aliasA, $aliasB, $detMethod, $author, $pub, $orgA, $orgB, $intType, $sourceDb, $intID, $conf) = @flds;
        ## Here you can add logic to limit the interactions by specific detection method codes ($detMethod)
        ## or database of origin etc.

        ## Print
        print "$line\n"
    }
}

I tested it by using a file with the UniProt ID for human TP53 (P04637) and it returns a list of 3988 interactions (showing 10 randomly selected results below):
$ foo.pl names.txt | shuf -n 10
uniprotkb:Q9BWC9    uniprotkb:P04637    -   -   uniprotkb:CCDC106(gene_name)    uniprotkb:TP53(gene_name)   psi-mi:"MI:0007"(anti tag coimmunoprecipitation)    Zhou, J. et al.(2010)   pubmed:20159018 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0469"(intact)    intact:EBI-7812926  -
uniprotkb:P04637    uniprotkb:O15126    -   -   uniprotkb:TP53(gene_name)   uniprotkb:SCAMP1(gene_name) psi-mi:"MI:0018"(two hybrid)    Lim, J. et al.(2006)    pubmed:16713569 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0463"(biogrid)   biogrid:720622  -
uniprotkb:P63165    uniprotkb:P04637    -   -   uniprotkb:SUMO1(gene_name)  uniprotkb:TP53(gene_name)   psi-mi:"MI:0018"(two hybrid)    Minty, A. et al.(2000)  pubmed:10961991 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0463"(biogrid)   biogrid:262339  -
uniprotkb:P04637    uniprotkb:P31350    -   -   uniprotkb:TP53(gene_name)   uniprotkb:RRM2(gene_name)   psi-mi:"MI:0416"(fluorescence microscopy)   Xue, L. et al.(2003)    pubmed:12615712 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0465"(dip)   dip:DIP-40167E  -
uniprotkb:P04637    uniprotkb:Q00987    -   -   uniprotkb:TP53(gene_name)   uniprotkb:MDM2(gene_name)   psi-mi:"MI:0004"(affinity chromatography technology)    Dai, MS. et al.(2004)   pubmed:15308643 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0463"(biogrid)   biogrid:478073  -
uniprotkb:Q99576    uniprotkb:P04637    -   -   uniprotkb:TSC22D3(gene_name)    uniprotkb:TP53(gene_name)   psi-mi:"MI:0428"(imaging technique) Ayroldi, E. et al.(2015)    pubmed:25168242 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0463"(biogrid)   biogrid:1255896 -
uniprotkb:P04637    uniprotkb:Q00987    -   -   uniprotkb:TP53(gene_name)   uniprotkb:MDM2(gene_name)   psi-mi:"MI:0415"(enzymatic study)   Lui, K. et al.(2013)    pubmed:23572512 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0463"(biogrid)   biogrid:859223  -
uniprotkb:P25685    uniprotkb:P04637    -   -   uniprotkb:DNAJB1(gene_name) uniprotkb:TP53(gene_name)   psi-mi:"MI:0004"(affinity chromatography technology)    Qi, M. et al.(2014) pubmed:24361594 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0463"(biogrid)   biogrid:938952  -
uniprotkb:P04637    uniprotkb:Q8IW41    -   -   uniprotkb:TP53(gene_name)   uniprotkb:MAPKAPK5(gene_name)   psi-mi:"MI:0424"(protein kinase assay)  Sun, P. et al.(2007)    pubmed:17254968 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0469"(intact)    intact:EBI-1202077  -
uniprotkb:Q13526    uniprotkb:P04637    -   -   uniprotkb:PIN1(gene_name)   uniprotkb:TP53(gene_name)   psi-mi:"MI:0096"(pull down) Mantovani, F. et al.(2007)  pubmed:17906639 taxid:9606(Homo sapiens)    taxid:9606(Homo sapiens)    -   psi-mi:"MI:0469"(intact)    intact:EBI-6112688  -

Thank you, I am not sure what's EBI nor how to convert/map gene names to list of proteins.

@0x90 EBI is just the European Bioinformatics Institute. Mapping is always complicated, I am afraid, and one of the main practical problems for this sort of analysis is that both genes and proteins have many, many different identifiers in the different databases. However, there are many tools that can help you map. For example: 1, 2.

If you run this Rscript using the gene name as an argument, you'll get a file with the pathway written to the directory.
#!/usr/bin/Rscript
args = commandArgs(trailingOnly=TRUE)
library(paxtoolsr)
id <- args[1]
write.table(graphPc(source=id,kind='neighborhood',
            format='BINARY_SIF',verbose=TRUE),
            file=paste(id,'pathway',sep='_'), sep='\t',quote=FALSE)

Why not just use string-db's online tool? There you can adjust all parameters of interest such as interaction confidence as well as pull in extra protein interactions.
If you make a user you can also upload a custom background gene-set set which means the analysis of go-terms and interconnectedness that sting provides is usable.


text:
How does Cellminer's "Cross-correlations of transcripts, drugs, and microRNAs" work
I have seen one of Cellminer tools. I am not sure how do they calculate the cross correlation of the genes, what does it actually mean? Based on what databases?
For example if I take their example input (HUGO format):
abcb1
BRCA2
CNBP

I get the following cross correlation matrix (based on what signals).
Identifier      abcb1   BRCA2   CNBP
abcb1           1      -0.142   0.069
BRCA2          -0.142   1       0.176
CNBP           0.069    0.176   1

You're probably best asking the authors

In the article describing Cellminer says about the cross-correlations:

... a cross-correlation table of the resultant z-scores can be
  generated ...

On page 3503
And the z-scores are defined previously as:

The tool output includes relative transcript intensity presented
  as z-scores...


text:
EMT/EMT-like processes in bioinformatics to study cancer progression
First I am not sure why, but it seems EMT can be used to infer the status of cancer progression.
What kind of databases are there for the EMT process in the context of cancer?
I saw this book which mentions ArrayExpress, but I need some help to understand what kind of data it stores.
Are there similar/better resources? How can I use EMT in the context of cancer progression study?

Are there other process like EMT that are being used to study cancer progression?
Are there EMT like events in leukemia progression?

Please note that the fields of Bioinformatics and Cancer Genomics move very quickly with new technologies. A source from 2010 would be regarded by many as out of date.

@TomKelly so you don't recommend on ArrayExpress?

Not unless you need to access the data from a particular study. Large scale genomics projects have far higher sample sizes and used newer technologies. What exactly are you trying to do? It does not seem like you are familiar with this field.

EMT describe cells' change in their state from being epithelial to the mesenchymal class. So, if a cancer cell line is gaining properties that allow it to move, it might reach the metastasis phase, were a cell can move to any point of the body and start a new tumor there.  As in the wikipedia page you linked:

EMT has also been shown to occur in wound healing, in organ fibrosis and in the initiation of metastasis in cancer progression.

I am not aware of any kind of database for the EMT process, the most relevant resource might be the gene ontology, which has a term for it, which lists 185 (human) genes as related to this biological process.
ArrayExpress is a database of transcriptomics, it :

stores data from high-throughput functional genomics experiments

You can evaluate the role of EMT in the data stored in the database.

What do you mean by evaluate the role of EMT in the data stored in the DB?

@0x90 Use one of the gene set enrichment techniques to the data, from doing a PCA with those genes to perform GSEA (like in the broad institute), or look for different expressed pathways like in SIPA or ...

thanks, if you have a good reference for similar work it would be much appreciated.

Similar work to what? To use analyze a pathway in a dataset?

yes to see expressed pathways in dataset which will be adjustable for my case.

This is a hard topic, the pathways are loosely described and there isn't any canonical pathway database, but GSVA is a good software tool that let you convert your matrix of genes x samples to pathways x sample, so you can use it to evaluate if a pathway is more expressed than another

Note that EMT is not the motility of cells. It is a change in their state from being epithelial to the mesenchymal class. In development, this is important in the formation of different tissues. In cancer, it is a disruption of regulatory processes, including cellular polarity and checkpoints on growth. Essential the tumour cells are becoming more like stem cells and less like the epithelial tissue. This is not "metastasis" that is another process by which cancer cells spread but EMT can contribute towards progression of disease towards that.

The epithelial-mesenchymal is a process by which cells lose some of the structures specific to their cell type. This is normal during embryonic development or wound healing for the formation of new cells. However, it is very uncommon in healthy adults and is thereby associated with the cancers in which it occurs. EMT allows cancer cells to become more like stem cells and could facilitate metastasis or immortal cell growth, key processes for cancer progression.
If you can detect the molecular components involved in EMT, you may be able to make some inferences about the progression of the patient’s disease. This process is specific to cancers in epithelial tissues (breast, stomach, and skin cancers for example). It is just one of vastly many pathways by which cancers can grow and develop. Many of these are specific to different cancers types by tissue or molecular subtype. Cancer research is therefore very challenging and requires a deep understanding of the particular system you are focusing on.
Gene expression data sets are a great resource for this. You can compare many pathways to clinical properties from previous studies. Microarray datasets such as ArrayExpress and Gene Expression Omnibus (GEO) were very popular for this but are now largely out of date. Many databases now support higher quality RNASeq datasets with larger sample sizes and fewer batch effects. The Cancer Genome Atlas (TCGA) and International Cancer Genome Consortium (ICGC) have among the largest Cancer databases. These are a tool for researchers to investigate any genes or pathways. They will not specifically address EMT, unless you focus on the genes involved in a relevant cancer type.

What do you mean by They will not specifically address EMT, unless you focus on the genes involved in a relevant cancer type. when referring to TCGA and ICGC. How can I find EMT related stuff there?

Exactly the same way as you would with ArrayExpress data. None of them are designed for this purpose: they all simply store expression data. It is only with 3rd party databases or knowledge of the biological context that you could extract this information from expression datasets of all genes.

Pathway analysis is a broad topic, far beyond the scope of an SE question. There are entire courses dedicated to it. Please be more precise in your questions if you wish to get meaningful answers. An example of how the data would be used would be valuable.


text:
How to simulate biological pathways?
I would like to simulate a biological pathway in Matlab, Mathematica or Python. To study the response of the pathway using Control Theory. Is there a standard way to simulate pathways by using differential equations or any other formalism.
For example I would like to try to simulate this KEGG pathway map. Is there something more recent than this work from 2005?
For example to simulate Petri networks?

What do you want to simulate, the concentration of the metabolites through time? Do you have initial concentrations or do  you want to study the effect of a condition (drug, variant, food,...) in a pathway? What kind of data do you have?

At the end of the day it's just a system of differential equations, so I'm not sure how helpful some external "pathway simulation" package would be (just use the base matlab or python functions).

Some equations cannot be solved analytically so they must use numeric approximations that are not trivial. This usually requires specific software on how to deal with the equations. (I am not aware if matlab or python are able to do this, but it is in my opinion a good question). I remember using polymath to solve some differential equations, but it was not used for pathway simulation but for bioreactors

In matlab this can be done using the SimBiology toolbox.
In mathematica this can be done using the SystemModeler.
In python there are multiple packages, e.g. (in no particular order) PySB, Tellurium, PySD

Is there a tool to translate a pathway from KEGG into a set of ODEs or the input formats of SimBiology or PySB?

I think what one can do is to download the KGML of the pathway from KEGG as described here and then to load it using KEGGtranslator and export it to SMBL. PySB and Matlab should be able to load SMBL files.


text:
Is there a tool to convert a variableStep .wig to fixedStep one?
I have a .wig file in this format:
variableStep chrom=chr1 span=20
59965   7
59985   10
60005   10
60025   10

and I would like to format it to something like this format:
#bedGraph section chr16:0-103584
chr16   0       59901   0
chr16   59901   59927   1
chr16   59927   59931   2

Please assume, I am working with a reference hg19 genome, in case the info isn't deducible from the variableStep .wig file. I couldn't find anything in WiggleTools

To me it is not obvious if these two files are consistent. What is the first position 59965 converted to ? (Perhaps it is because I'm unfamiliar with these type of files..)

@llrs they aren’t of the same data. It’s just to demonstrate the format

Could you please change one or the other so that they represent the same data?

When I had to do something similar previously, I had used the following method using sed and awk. It is not as robust in terms of error detection and correction as the python solution but it is a simple one-liner.
$ cat temp.wig
variableStep chrom=chr1 span=20
59965   7
59985   10
60005   10
60025   10

$ sed -r 's/variableStep chrom=(.*) span=([0-9]*)/\1\t\2/g' temp.wig \
        | awk 'BEGIN{FS="\t";OFS="\t"}{if ($1~/^chr/) {c=$1 ; s=$2} else {print c,$1,$1+s,$2}}' \
    > temp.bedgraph

$ cat temp.bedgraph
chr1    59965   59985   7
chr1    59985   60005   10
chr1    60005   60025   10
chr1    60025   60045   10

This is a temporary solution I came up with that worked for my specific case:
def convert_line(line, chrom, span, start=None, fixedStep=None, count=None):
        new_line=''
        if fixedStep is None or count is None or start is None:
                line_arr  = line.split()
                start = line_arr[0]
                end   = int(line_arr[0]) + int(span)
                val   = line_arr[1]
                new_line = '{0}\t{1}\t{2}\t{3}\n'.format(chrom, start, end, val)
        else:
                line_arr  = line.split()
                val   = np.float(line_arr[0])
                start = int(start) + int(fixedStep)*int(count)
                end   = start + int(fixedStep)
                new_line = '{0}\t{1}\t{2}\t{3}\n'.format(chrom, start, end, val)

        return new_line

def convert_file(input_file, output_file):
        chrom=None
        span=None
        fixedStep=None
        count=None
        start=None
        with open(input_file, 'r') as fin, open(output_file, 'w') as fout:
                for line in fin:
                        if len(line)<1 or line[0] == '#':
                                continue
                        elif 'variableStep' in line or 'fixedStep' in line:
                                for key in line.split():
                                        if 'chrom=' in key:
                                                chrom = key.split('chrom=')[-1]
                                        if 'span=' in key:
                                                span = key.split('span=')[-1]
                                        if 'step=' in key:
                                                fixedStep = key.split('step=')[-1]
                                        if 'start=' in key:
                                                start = key.split('start=')[-1]
                                                count = 0
                        else:
                                line_out = convert_line(line, chrom, span, start, fixedStep, count)
                                fout.write(line_out)
                                if count is not None:
                                        count += 1

input:
variableStep chrom=chr16 span=20
59965   7
59985   10
60005   10

output:
chr16   59965   59985   7
chr16   59985   60005   10
chr16   60005   60025   10
chr16   60025   60045   10


text:
How to convert ChiP-seq track in .bam file format into an array at fixed bin resolution?
Is there a out of the box to convert a .bam file to a .wig file at specific resolution/bin size?
I was thinking to use igvtools, but I am not sure how it works:
igvtools count -z 5 -w 25 -e 250 alignments.bam  alignments.cov.tdf  hg19
What is the diffrence between -w,-z,-e?
-z num  Specifies the maximum zoom level to precompute. The default
               value is 7 and is sufficient for most files. To reduce file
               size at the expense of IGV performance this value can be
               reduced.

 -e, --extFactor num

The read or feature is extended by the specified distance in bp prior to counting. This option is useful for chip-seq and rna-seq applications. The value is generally set to the average fragment length of the library minus the average read length.

-w, --windowSize num

The window size over which coverage is averaged. Defaults to 25 bp.

Using deepTools to get a bin size of 10 and 20 threads:
bamCoverage -of bedgraph -bs 10 -b input.bam -p 20

You'll probably prefer the default bigWig format if you need random access or to use something like IGV.
Regarding the options to igvtools, -w is equivalent to -bs/--binSize in deepTools and denotes the resolution of the output data. -e (it's the same option in bamCoverage) denotes whether and how much to extend a read in its 3' direction. This done such that the output represents the sequences fragments. -z is specific to things like bigWig files (and apparently tdf files), where there are precomputed values that IGV will load as you zoom out (the various resolutions are "zoom levels", which is parlance taken from the bigWig format).


text:
How to create a .bed file from .fasta?
I have some problems in creating a .bed file for hg19, so I will be able to visualize the .bed file in IGV.
The .fasta file contains rows of this form:
>chr1:0-1000
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
>chr1:1000-2000
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
>chr1:2000-3000
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
>chr1:3000-4000
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN

I tried both with biopython:
from Bio import SeqIO

with open(f1) as in_f, open(f1+'.seq','w') as out_f:
    for record in SeqIO.parse(in_f, 'fasta'):
        out_f.write('{}\t0\t{}\n'.format(record.id, len(record))

and awk:
samtools faidx file.fasta
awk 'BEGIN {FS="\t"}; {print $1 FS "0" FS $2}' file.fasta.fai  > file.bed

Both ways gave me the same output which IGV couldn't work with:
chr1:0-1000 0   1000
chr1:1000-2000  0   1000
chr1:2000-3000  0   1000
chr1:3000-4000  0   1000
chr1:4000-5000  0   1000
chr1:5000-6000  0   1000
chr1:6000-7000  0   1000
chr1:7000-8000  0   1000

Now I tried to see what would be a valid .bed file format from UCSC and the format looks different. For example:
#bin    chrom   chromStart  chromEnd
13  chr1    39845715    39846245
19  chr1    87030756    87031903
22  chr1    111148649   111149175
22  chr1    113246065   113246628

Why my methods don't output the .bed in the same format?
How can I make the awk/biopython methods to output a .bed file that IGV can actually load.

As @devon-ryan points out below, both of your BED format examples are incorrect. I think it would be helpful for others if you edited your question to give a correct example. See for example http://genome.ucsc.edu/FAQ/FAQformat#format1

Here is a fun little python script I cooked up for the occasion. It will take a standard fasta file as a command-line argument and turn it into the proper bed format. Using your example fasta:
import pandas as pd
import sys

inFasta = sys.argv[1] # take fasta as command argument

def fastaParser(fasta):
    headers = []
    with open(fasta) as f:
        header = None
        for line in f:
            if line.startswith('>'): # identifies fasta header line
                headers.append(line[1:-1]) # append all of the line that isnt >
                header = line[1:] # in reset header
    newHeader = (header.replace(':',',') for header in headers) # format to be accepted later
    newnewHeader = (header.replace('-',',') for header in newHeader) # format to accept later
    bedHead = (header.split(',') for header in newnewHeader) # separate by comma from format above
    return bedHead

fastaParsed = fastaParser(inFasta) # run the function

headers = list(fastaParsed) # go from generator to list
bedfile = pd.DataFrame(headers) # create dataframe which will output bedfile

bedfile.to_csv('fastabedfile.bed', sep='\t', index=False, header=None) # output bedfile format

to run:
[dkennetz@nodecn202  fasta_parser]$python fastaParser.py fakeFasta.fa

[dkennetz@nodecn202  fasta_parser]$more fastabedfile.bed
chr1    0       1000
chr1    1000    2000
chr1    2000    3000
chr1    3000    4000

I'm assuming the FASTA header contains all the information you want. You can easily generate a proper bed file with Biopython:
from Bio import SeqIO

for record in SeqIO.parse('test.fa', 'fasta'):
    lst = record.id.split(':')
    chr_, (start, end) = lst[0], lst[1].split('-')
    print('\t'.join((chr_, start, end)))

Note that native Python code is only marginally longer:
with open('test.fa') as f:
    for line in f:
        if line.startswith('>'):
            lst = line.rstrip().split(':')
            chr_ = lst[0].lstrip('>')
            start, end = lst[1].split('-')
            print('\t'.join((chr_, start, end)))

Both code snippets output bed format:
chr1    0   1000
chr1    1000    2000
chr1    2000    3000
chr1    3000    4000

The UCSC format you linked to isn't a BED file, your method should never produce it. What you posted as your desired output is also not a BED file. Below is a BED file:
chr1    0       1000
chr1    1000    2000
chr1    2000    3000
chr1    3000    4000

awk and biopython were producing the appropriate thing already. You just need to zoom in on a region in IGV. Since it's not a bigBed file, IGV won't show everything at all zoom levels.
As a demonstration, below is the BED file from above in IGV:

If your headers are always of the form >chrZ:a-b, where chrZ is a chromosome name, a is a zero-indexed, half-open start position, and b is a zero-indexed, half-open stop position:
$ awk '($0 ~ /^>/){ print substr($0,2) }' in.fa | sed 's/[:|-]/\t/g' > out.bed

If your headers contain coordinates using a different index, then you can make adjustments to $1 and $2 in the awk statement to create zero-indexed, half-open coordinates to export to BED, depending on the initial index.

How do I do it with biopython?

I would not use Biopython for this. I recommend using standard Unix tools that are fast and flexible. I would not use a hammer to open a jar.

Assuming that you have a reference genome sequence file (e.g., reference.fasta) available. Getting this kind of file is straightforward.
1) Index the reference genome and map your reads or FASTA sequences to it (for example with bowtie2)
# index reference genome (should be precomputed)
bowtie2-build reference.fasta reference
# map reads
bowtie2 -x  reference -U file.fasta -S file.sam
# compress SAM to a BAM (binary) file
samtools view -bS file.sam > file.bam

2) Create a BED files from the BAM file using bedtools
# extract BED file
bedtools bamtobed -i file.bam > file.bed

Alternatively, for sliding windows you can generate these from a reference sequence provided that you know the length of each chromosome (perhaps there is a way to extract these directly from reference.fasta):
# length per chromosome
samtools view -H file.bam | grep "SQ" | cut -d":" -f2-3 | sed 's/LN://' > file.chr.txt

Create sliding windows (using only the lengths per chromosome)
# create sliding windows
bedtools makewindows -g file.chr.txt -w 1000 > windows.bed


text:
Does this FASTQ data contain single or paired end calls?
I have this fastq data from GEO:
zcat SRR1658526.fastq.gz | head -n 20
@SRR1658526.1 HWI-ST398:296:C1MP4ACXX:1:1101:1093:2094 length=102
GATCTCTATTACTTTTTGAAGGATTNNNNNNNNNNAANTTTTGAATCANNNNNNNNNNNNNNNNNNNNNNNNNNNNNTNNNNNNNNNNNNNNNNNNNNNNNN
+SRR1658526.1 HWI-ST398:296:C1MP4ACXX:1:1101:1093:2094 length=102
@<@FFDFDHHFHHIIIIGHBGGGIG##########10#0:BGDDHHII######################################################
@SRR1658526.2 HWI-ST398:296:C1MP4ACXX:1:1101:1167:2107 length=102
ATAATATTGTAGATATAAATGTTATCTAATCTTATCTGATCAGCTTGCTNNATANNNNNNNNNNNNNNNNNNACNTATGNNNNNNNNNNNNNNNNNNNNNCC
+SRR1658526.2 HWI-ST398:296:C1MP4ACXX:1:1101:1167:2107 length=102
CCCFFFFFHHFHHJJIIIIJIJJHJHJJJJIJJIJJJJIJHIJJJJGII#####################################################

It is supposed to be paired-end sequences. Is the prefix @ and + are the R1 and R2? What's the convention here?

Entries in a fastq file occupy 4 lines each and R1 and R2 are typically in different files. Since that SRA project is 2x51 you seem to have run fastq-dump without the --split-3 option, so both R1 and R2 are merged together. Make your life easier and never use SRA, but instead just download the individual files from ENA.

what's ENA? and how what would be the right fastq-dump command?

ENA is the European Nucleotide Archive. Don't use fastq-dump, don't use SRA at all.

What would be the equivalent command?

wget or curl or just right clicking on the web interface and selecting save.

I see, so ENA is a European mirror of GEO?

Use aspera, just as with SRA.

the link will be here: https://www.ebi.ac.uk/ena/data/view/SRR1658526

The data shown in the fastq files are paired end data.
In fastQ files @ and + do not correspond to R1 and R2.
The wikipedia page for fastq files describe the following conventions:

A FASTQ file normally uses four lines per sequence.
Line 1 begins with a '@' character and is followed by a sequence identifier and an optional description (like a FASTA title line).
Line 2 is the raw sequence letters.
Line 3 begins with a '+' character and is optionally followed by the same sequence identifier (and any description) again.
Line 4 encodes the quality values for the sequence in Line 2, and must contain the same number of symbols as letters in the sequence.

When using fastq-dump
".1" and ".2" stand for read suffices assuming u used the -I flag.
In this case you can see that R1 (".1") and R2 (".2") are within the same file.


text:
Are there databases or resources that contain expression of proteins or genes before and after cancer treatment?
Are there databases that contain information about gene or protein expression before and after targeted therapy or chemotherapy?
I am curious about different cancer cell lines or cancer types.
Any suggestion on a set of resources I should look into?

"Are there databases..." As a general rule, no, there are almost never specialized databases for anything.

@DevonRyan, any idea how to extract expression ratio before and after cancer treatment from https://portal.gdc.cancer.gov/, https://bioinformatics.mdanderson.org/public-datasets/, https://www.immport.org/shared/home, https://cancer.sanger.ac.uk/cosmic, or http://www.cbioportal.org/

Find appropriate samples, match them up, find expression metrics, write a program to do that. Never expect tools for random things like this.

One such database/tool is “[PharmacoGx][1]: an R package for analysis of large pharmacogenomic datasets.” Bioinformatics (Oxford, England).
They brought together a large compendium of experimental data (cell lines and the like if I remember correctly from a talk), standardized these and made available within a special object class. I think there is genomic and transcriptomic data, not sure if there is proteomic data though.

Do you mind to include a simple example?

What you are trying is not a simple task overall, therefore I cannot provide a "simple" example. I can only point you to their vignette, they have worked-out cases there: http://bioconductor.org/packages/release/bioc/vignettes/PharmacoGx/inst/doc/PharmacoGx.pdf

I see. What about the databases COSMICS, TCGA, and immport. Aren’t they supposed to have such things?

Can't talk about others but TCGA is (mostly) treatment naive, you will need to check clinical information tables (of your cancer of interest) if patients got treatment or not though. If you decide go down that road, TCGABiolinks (and especially its vignettes) would be very helpful.