Insert query not working

Version: 2.4.2
Module: quill-jdbc
Database: postgres
Expected behavior
I'm trying to insert a new row into a table. My code looks like this
Future {
      run {
        quote {
          decks.insert(lift(deck)).returning(_.id)
        }
      }
    }

Actual behavior
I'm noticing two issues :
First of all, this code gets compiled as a "Dynamic query", but I don't see why it couldn't be compile as a static quotation. I've used similar code in other places in the project and some of them get compiled as static quotations
Second of all, and most importantly, when actually running the code, the insert query is wrong. Here's what I get :
19:21:25.474 [blocking-IO-1] DEBUG io.getquill.context.jdbc.JdbcContext - INSERT INTO courses.decks (name,owner_type,owner_id,language,description,category_id,is_premium,visibility,created_at,updated_at) VALUES (name, owner_type, owner_id, language, description, category_id, is_premium, visibility, created_at, updated_at)
[error] knoko - ERROR: column "name" does not exist
  Indice : There is a column named "name" in table "decks", but it cannot be referenced from this part of the query.
  Position : 139

Here we can see that instead of placing the acual values in the VALUES part of the query, quill repeats the column names, resulting in a postgres error
I have no idea why it does that for this particular case class and not all of them
@getquill/maintainers

Please provide reproducible example using scastie. The information that you provided is insufficient.

Sure! I made a scastie like you asked. We can see in the scastie that the query gets parsed as a dynamic query

@gbogard Query is dynamic because you have specify the type of quotation, please remove that and it will be compile static, see http://getquill.io/#quotation-compile-time-quotations for more information.
However the wrong query is produced because the type of insertMeta is also explicitly set.
@gbogard To resolve your issue please remove explicit type in quote and insertMeta as in https://scastie.scala-lang.org/9CUJtp1WSi6kNWL5cUTgtA

Opened issue to add compiler error when explicit type of insertMeta is set #1105


About the cause of playback errors in Windows 10

Have you read the Tutorials?
Yes
Have you read the FAQ and checked for duplicate open issues?
Yes
What version of Shaka Player are you using?
3.0.6
What browser and OS are you using?
Electron Windows10
Please ask your question
Please forgive the machine translation.
When some users try to stream and play videos protected by Widevine DRM, the following error occurs and the video cannot be played.
The error occurs before a license is issued.
player.load error: shaka.util.Error {
    "severity": 2,
    "category": 6,
    "code": 6002,
    "data": [
        "Failed to execute 'createMediaKeys' on 'MediaKeySystemAccess': CreateCdmFunc not available."
    ],
    "handled": false
}

I looked at the list of errors, but I couldn't find the specific cause.
What exactly causes this error?

This looks like something specific to both Chrome and Windows.  I'm not familiar with the implementation, but I believe there's a Windows-specific CDM in Chrome.
@xhwang-chromium, can you help out here?  What would cause this error?  How can it be resolved?

We are not seeing this error in Chrome. Which version of Chromium is this based on? Any difference in how it's using the CDM?

We are not seeing this error in Chrome. Which version of Chromium is this based on?

We are using downstream_electron v0.4.0.
If you check the package.json, the electron version is v6.0.0, so the Chromium version is probably M76.
https://www.electronjs.org/ja/docs/latest/tutorial/electron-timelines

Any difference in how it's using the CDM?

I don't think there is any difference.
This page was used as a reference when implementing it.

https://shaka-player-demo.appspot.com/docs/api/tutorial-ui.html
https://shaka-player-demo.appspot.com/docs/api/tutorial-drm-config.html

Excerpt of code
// HTML
<video id="video" ref="video" data-shaka-player :poster="content.image_url.package" style="width: 100%; height: 100%;">
</video>

// Vue JS
created() {
    require('shaka-player/dist/controls.css')
    this.shakaUi = require('shaka-player/dist/shaka-player.ui')
    document.addEventListener('shaka-ui-loaded', this.initApp)
},
methods: {
    initApp: async function() {
        this.video = this.$refs.video
        const ui = this.video['ui']
        const controls = ui.getControls()
        this.player = controls.getPlayer()
    },
    playVideo(manifestUrl, customData, currentTime) {
        const licenseServerUrl = 'https://◯◯◯.com/?custom_data=' + customData
        this.player.configure({
            drm: {
                servers: {
                    'com.widevine.alpha': licenseServerUrl
                }
            }
        })
        this.manifestUrl = manifestUrl // mpd file url
        this.player.load(manifestUrl).then(() => {
            this.video.play()
        }).catch((error) => {
            onPlayerError('player.load error: ' + error)
        })
    },
}

@ogawa-ma DOWNstream Electron version you're using is outdated, can you try the most recent one?
@mackode do you have any other suggestions?

Please allow some time for a response.


Whitelist ip-ranges

Ability to whitelist ip-ranges for which the proxy doesn't need to be unset.

Implemented in release 1.2


Complete list of Font Awesome, including Pro and Brand

Nice one! Out of curiosity how did you manage to get the pro icon names?
I made this comment in the issue but the feedback is relevant to this PR

I'm certainly open to supporting this, here is what I'd expect/suggest:
* create/find/procure a list of the font-awesome  pro-icon-names to unicode from the font-awesome website or similar; https://github.com/FortAwesome/Font-Awesome/tree/master/metadata might be helpful. Ideally store the  pro-cons in a separate file to the free ones.

* add a flag to the script to toggle using the pro icons

* update the README with instructions  for installing the package with the pro icons and a way to check if  the pro-version is installed

* a warning message if a pro icon-name is used in the config but the pro-flag is not enabled

Also,  can this icon list please be sorted? A side effect of parsing a file of json with the module is that the output  is sorted, eg.
python -m  json.tool file_of_massive_json

Ah, very simple: I have a Pro license :sweat_smile:
And the list is sorted in my latest commit.


Connectivity - Provide a mapping between connection types and bandwidths

This is a

[ ] Bug
[x] Feature Request

Which plugin does this impact:

[ ] Battery
[x] Connectivity
[ ] Contacts
[ ] DeviceInfo
[ ] ExternalMaps
[ ] Geolocator
[ ] Media
[ ] Permissions
[ ] Settings
[ ] Text To Speech
[ ] Vibrate
Other:

Version Number of Plugin: 2.1.2
Device Tested On: Windows 10 desktop and mobile
Simulator Tested On:
Feature Request
There is a property for ConnectionTypes and a property for Bandwidths, but no mapping between the two... that feature would be nice so that if I need a user to send logs I can detect what connection type they were using and what the bandwidth of that connection was.  This request is related to #255

Moved to: https://github.com/jamesmontemagno/ConnectivityPlugin/issues/7


Chore(stress-chaos): Support stress-chaos experiment with custom experiment name

Signed-off-by: udit<EMAIL_ADDRESS>
What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Checklist:

[ ] Fixes #
[ ] PR messages has document related information
[ ] Labelled this PR & related issue with breaking-changes tag
[ ] PR messages has breaking changes related information
[ ] Labelled this PR & related issue with requires-upgrade tag
[ ] PR messages has upgrade related information
[ ] Commit has unit tests
[ ] Commit has integration tests
[ ] E2E run Required for the changes

stress-chaos-logs.txt


Is it possible to use continuous pressure?

I am using wayland, and I would like to use the keyboard to control the course, I already managed to do that now my question is how can I do so that when I press an assigned key it invokes the command continuously, something like what happens when you hold down a key for example the "a" key.
This is the configuration I currently have
h = command(runuser -l think-crag -c 'wlrctl pointer move -40 0')
j = command(runuser -l think-crag -c 'wlrctl pointer move 0 40')
k = command(runuser -l think-crag -c 'wlrctl pointer move 0 -40')
l = command(runuser -l think-crag -c 'wlrctl pointer move 40 0')
i = command(runuser -l think-crag -c 'wlrctl pointer move 40 -40')
u = command(runuser -l think-crag -c 'wlrctl pointer move -40 -40')
m = command(runuser -l think-crag -c 'wlrctl pointer move 40 40')
n = command(runuser -l think-crag -c 'wlrctl pointer move -40 40')

what i really want to do is the mousekeys functionality

command() is not repeatable. Native mouse support is on the agenda, but in the meantime you might want to check out warpd.


Calling unwear after picking up sword and riding vehicle causes vehicle to snap to unwear position

https://user-images.githubusercontent.com/18633264/171322529-2c818d6d-a2bd-4235-8dde-0471bbfe80d6.mp4

Fixed in z-targeting


Implementation of ADMM-IVA

Summary

This PR implements the ADMM-based BSS method proposed in the Acoustic Society of Japan.
In addition, I've created a base class of PDSBSS and ADMMBSS as ProxBSS.

Codecov Report
Attention: 32 lines in your changes are missing coverage. Please review.
:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Files
Coverage Δ

ssspy/bss/iva.py
96.23% <ø> (ø)

ssspy/linalg/prox.py
85.71% <50.00%> (-3.76%)
:arrow_down:

ssspy/bss/pdsbss.py
91.66% <70.00%> (-3.21%)
:arrow_down:

ssspy/bss/proxbss.py
87.37% <87.37%> (ø)

ssspy/bss/admmbss.py
84.69% <84.69%> (ø)

... and 1 file with indirect coverage changes
:loudspeaker: Thoughts on this report? Let us know!.


[Suggestions] - [Modules] - Natural moon, gas giant and more

I suggested this in a module because if people want to disable, will be easy.
I want to know if you planned to add for natural spawn of gas planet (there already image) and natural moon generation with several types like the planets but moon may be have atmosphere so you can build construction like on planets (and on moons) and moons without atmosphere just like now moon construction.
Also for gas giant, construction can may be like planets but haven't mine of metal and crystal.
After this if possible, may be create a another module for create world-ship.
Good luck and continuation.

Very good idea. I am dreading to have to make an account, use it as a bot, and spawn it with planets and moons.

yes possible to build :)

Hi! You idea is interesting.
Its not planned to add this features. In my option a bot framework should not included to this game.
I have start write a bot in 2011. Its not full feature. Maybe there are some inspirations for you.
https://www.dropbox.com/s/le8uwdl98i0q2mp/BOT.zip?dl=1


[rom_e2e,dv] increase test timeouts for boot_policy_valid test_unlocked0 tests

Tests in the test_unlocked0 LC state take longer to run.
Signed-off-by: Timothy Trippel<EMAIL_ADDRESS>

All boot_policy_valid_* tests have now passed in DV.


[WFLY-5105] Migrate discovery-group attribute in cluster-connection

JIRA: https://issues.jboss.org/browse/WFLY-5105

Linux Build 7521 is now running using a merge of 3cbfc132d8b63ddbb28a7c5c0d9cc4a3fcf22d17

Linux Build 7521 outcome was SUCCESS using a merge of 3cbfc132d8b63ddbb28a7c5c0d9cc4a3fcf22d17
Summary: Tests passed: 3103, ignored: 369 Build time: 00:49:46


TPX_Perpetual_Dex

Title of your project
Taiko
Short (1-2 sentence) description of your project
Perpetual Dex on Taiko
Link to your project
https://tpx.fi/
Live on Testnet
Yes (default)
If your project is live on one of Taiko's testnets, please provide evidence of that here (eg. links, screenshots). If your project is NOT live, please provide your plan for integration here.
https://github.com/tpxfi
Upload a quality logo to your project
https://drive.google.com/file/d/1fcve7cxB4Thwxg0r71ypfDAtH19229A7/view?usp=sharing

Will there be an actual product in the future?


[fix] gws/schedule: import csv

#8

Coverage remained the same at 55.876% when pulling f68f1d8f56dac0170970ad28e1572cc2c04c6d03 on tany:schedule-csv into c22f147ec6cb2cb15951d76a3c25a52337cb1511 on shirasagi:master.


Add support for Groq LLM

I'll give it a try. This might already work using Langchain::LLM::OpenAI since the Groq technical FAQ says:

How to transition from OpenAI to Groq?
It's incredibly simple to transition! If you have an application already using OpenAI's API, all it takes is updating 3 variables: the API key, the endpoint, and the model!

@mattlindsey There's no way they're hosting OpenAI models, because other than GPT-2 none of them are open source. Only open-source ones: https://console.groq.com/docs/models.

Actually the above example seems to work great with this:
class OpenAI < Base
DEFAULTS = {
n: 1,
temperature: 0.0,
chat_completion_model_name: "llama3-8b-8192",
embeddings_model_name: "text-embedding-3-small"
}.freeze
So it should work by fixing something.

@andreibondarev I take that back.  It seems to be working fully now as-is.  Maybe I just needed to wait for the Groq key to 'activate.'

@mattlindsey I don't think it's a good idea. It would be kind of odd to instantiate an LLM::OpenAI class and then interface with mixtral model via Groq.

@andreibondarev That seem to be what they tell you to do in their FAQ, and it works (with a fix, it seems).  But maybe you'd prefer a Groq LLM class.

I discovered using parameters as follows works currently, in case anyone needs it now:
llm = Langchain::LLM::OpenAI.new(api_key: 'your_groq_key_here', llm_options: { uri_base: 'https:/
/api.groq.com/openai/'})

llm.chat(messages: [{role: "user", content: "Tell me a story?"}], model: 'mixtral-8x7b-32768').completion

For now (for chat completion) it seems Groq Cloud is matching OpenAI's API. So the Groq adapter could use the underlying ruby-openai for the subset of API features currently available on https://api.groq.com/v1/openai
The test/fixtures/vcr_cassettes folder of groq-ruby should give some examples of API calls that are available.
Currently it is:

https://api.groq.com/openai/v1/models
https://api.groq.com/openai/v1/chat/completions

@andreibondarev The following should work pretty easily, and I can do a PR if you want:
module Langchain::LLM
  # LLM interface for Groq OpenAI compatible services
  #
  # Usage:
  #    groq = Langchain::LLM::GroqOpenAI.new(
  #      api_key: ENV["GROQ_API_KEY"],
  #      llm_options: {},
  #      default_options: {}
  #    )
  class GroqOpenAI < OpenAI

For anyone else who just wants a quick Groq solution who might not know where the diff defaults/overrides go (e.g. me) this seems to work:
  @groq = llm = Langchain::LLM::OpenAI.new(
    api_key: ENV["GROQ_API_KEY"],
    llm_options: {
      uri_base: "https://api.groq.com/openai/"
    },
    default_options: {
      chat_completion_model_name: "llama3-70b-8192"
    }
  )

  messages = [{:role=>"system", :content=>"I like to solve maths problems."}, {"role"=>"user", "content"=>"What is 2+2?"}]
  @groq.chat(messages:).completion
  # => "That's an easy one! The answer is... 4!"


Different numbers of final read counts in _1 and _2 FASTQ files

Hi, I'm using FreeHiC for simulating Cow Hi-C data.
For fastqFile data using during training steps, SRR6691720 data (https://www.ncbi.nlm.nih.gov/sra/?term=SRR6691720) was used and other parameters are below.
train=1
simulate=1
postProcess=1
coreN=20
mismatchN=3
gapN=1
mismatchP=""
gapP=""
chimericP=""
simuN=157702285
readLen=80
resolution=40000
lowerBound=80000
refragU=800
ligateSite="AAGCTAGCTT"
After simulating Hi-C data, I checked the output of the simulation and found that read IDs in each read file did not match. Also, the numbers of reads in read 1 and read 2 files are different.
There was a warning message in log but I already checked that my input FASTQ files do not have any problems.
(All reads are sorted equally in both input read files and all IDs are matched)
Read id in read 1 and read 2 file does not match. Please check the input read files and sort them correctly.
Could you tell me how to fix this problem?

Hello!
May I know how did you download the data from GEO? Sometimes when you download a paired-ended sequencing data and split into two end fastq files, a suffix such as  "-1" or "_1" may be appended to the read name to indicate this is end 1. In our case, we don't want this to happen and the read name should be exactly the same and in the same order.
The corresponding python codes to check the read name is as below, if they are not exactly the same, you will see the error message.
if r1.qname == r2.qname:
                if r1.is_unmapped == True and r2.is_unmapped == True:
                    unmapped_count += 1
                    continue
                elif r1.is_unmapped == False and r2.is_unmapped == False:
                    mapped_count += 1
                    if is_uniread(r1) == True and is_uniread(r2) == True:
                        uni_count += 1
                        (r1, r2) = sam_flag(r1, r2, readEnd1, readEnd2)
                        outfile.write(r1)
                        outfile.write(r2)

                else:
                    singleton_count += 1
            else:
                print("Read id in read 1 and read 2 file does not match. Please check the input read files and sort them correctly.")
                sys.exit()

Otherwise......may I see a fraction of your input fastq files?
Thanks,
Ye

I downloaded the data from SRA using fastq-dump and split 2 paired-end files using --split-files option
fastq-dump --split-files SRR6691720
These are some parts of my input FASTQ files
<SRR6691720_1.fastq>
[sy0819@europa00 Cow]$ head SRR6691720_1.fastq
@SRR6691720.1 1 length=80
GTTCTNTTTAGTTGAGAATTCTATTTCAGTCTCTTTTATTCATCTGGTCTCCTTCTNCANTANNNNNNNCANNNNNNNNN
+SRR6691720.1 1 length=80
AAAAA#EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAAEAEEEEEEEEEEEE#AE#EE#######<A#########
@SRR6691720.2 2 length=80
ATCAANTATGTGATCAGTGCATGATCAGTCACGTGATCAGTCGTGATCAGTGCATGNTCNGANANNNNNTCNNNNNNNNN
+SRR6691720.2 2 length=80
6AAAA#EEEEEEEEEEEEEEEEEEEEEEEE/EEEEEEEEEEEEAE//EEEAAE/AE#E<#</#E#####EE#########
@SRR6691720.3 3 length=80
GATCCNTTACCACTGAGCCACCAGGGAAGCCCACACAGTAGATGCCTGGGCTTCAAAGTNAAAGANNNNGGANNNNNNNN

<SRR6691720_2.fastq>
@SRR6691720.1 1 length=80
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
+SRR6691720.1 1 length=80
################################################################################
@SRR6691720.2 2 length=80
NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
+SRR6691720.2 2 length=80
################################################################################
@SRR6691720.3 3 length=80
NANNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN

I also checked SAM files, which is the output of FreeHi-C rawDataTraining steps.
These files are in the rawDataTraining/s1_alignment/ directory and part of SAM files are below.
SRR6691720_1.sam
SRR6691720.1    4       *       0       0       *       *       0       0       GTTCTNTTTAGTTGAGAATTCTATTTCAGTCTCTTTTATTCATCTGGTCTCCTTCTNCANTANNNNNNNCANNNNNNNNN        AAAAA#EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEAAEAEEEEEEEEEEEE#AE#EE#######<A#########
SRR6691720.2    4       *       0       0       *       *       0       0       ATCAANTATGTGATCAGTGCATGATCAGTCACGTGATCAGTCGTGATCAGTGCATGNTCNGANANNNNNTCNNNNNNNNN        6AAAA#EEEEEEEEEEEEEEEEEEEEEEEE/EEEEEEEEEEEEAE//EEEAAE/AE#E<#</#E#####EE#########
SRR6691720.3    4       *       0       0       *       *       0       0       GATCCNTTACCACTGAGCCACCAGGGAAGCCCACACAGTAGATGCCTGGGCTTCAAAGTNAAAGANNNNGGANNNNNNNN        AAAAA#EAEEEEEEEAEEEEEAEEEEEEEEEEEEEEEEEEE/EEEEEEEEEEEAEEEEA#EEEEE####EEE########
SRR6691720.4    4       *       0       0       *       *       0       0       AAAAGNCCCTGATGCTGGGACAGATTGAAGGCAGGAGGAGAAGGGGATGTCAGATGAGANGTTTANNNNGCANNNNNNNN        AAAAA#EEEAEEEAE6AEEEEEEE6EE/E/EAAAAEEAAE/AEAEEEEEAEEA//E/EE#E/EEE####/<E########
SRR6691720.5    4       *       0       0       *       *       0       0       ATGTTNTAAGGTAATTAGCCTCTAATTAAAATAAATTTAATTAGAAAAATAAAAATAACNTTGACNNNNCTGNNNNNNNN        AAAAA#EAEEEEEEEEEEAAEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE#EEEEE####EEE########

SRR6691720_2.sam
SRR6691720.1    4       *       0       0       *       *       0       0       NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN        ################################################################################
SRR6691720.2    4       *       0       0       *       *       0       0       NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN        ################################################################################
SRR6691720.3    4       *       0       0       *       *       0       0       NANNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN        #A##############################################################################
SRR6691720.4    4       *       0       0       *       *       0       0       NGNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN        #A##############################################################################
SRR6691720.5    4       *       0       0       *       *       0       0       NCNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN        #A##############################################################################

I don't think that there are some problems with read names in both input FASTQ files and output mapping files.
Could you tell me what is the problem??
Thanks,
Sue

yea.....looks fine to me as well. Can you also share the original complete log or screen output information with the error messages? Looks to me that this error may not appear in the training section but in the simulation sequence data processing section. In that case, there must be something wrong with the simulation.

Here is the original complete log information!
I had to simulate multiple Hi-C data sets, so I completed the training step first and simulate multiple sets with training=0 option
[ Training step log ]
projDir=/mss5/RACA2/programs/FreeHiC
fastqFile=/mss5/RACA2/read_simulation/hic/realData/Cow/SRR6691720
ref=/mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa
refrag=/mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed
outDir=/mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x
simuName=bosTau9_10x
summaryFile=/mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/summary/bosTau9_FreeHiC.summary
bwa=bwa
samtools=samtools
bedtools=bedtools
train=1
simulate=1
postProcess=1
coreN=20
mismatchN=3
gapN=1
mismatchP=
gapP=
chimericP=
simuN=394259238
readLen=80
resolution=40000
lowerBound=80000
refragU=800
ligateSite=AAGCTAGCTT
Pre-checking the input parameters......
  1. FreeHi-C package directory exist.
  2. /mss5/RACA2/read_simulation/hic/realData/Cow/SRR6691720 will be downloaded.
  3. Reference genome file exists and is not empty.
  4. Restriction enzyme fragment file exists and is not empty.
  5. Simulation run name is provided.
  6. Output directory is provided.
  7. Summary file path is provided.
  8. Path to BWA executable file is provided. Checking bwa help manual......

Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.17-r1198-dirty
Contact: Heng Li<EMAIL_ADDRESS>
Usage:   bwa <command> [options]

Command: index         index sequences in the FASTA format
         mem           BWA-MEM algorithm
         fastmap       identify super-maximal exact matches
         pemerge       merge overlapping paired ends (EXPERIMENTAL)
         aln           gapped/ungapped alignment
         samse         generate alignment (single ended)
         sampe         generate alignment (paired ended)
         bwasw         BWA-SW for long queries

         shm           manage indices in shared memory
         fa2pac        convert FASTA to PAC format
         pac2bwt       generate BWT from PAC
         pac2bwtgen    alternative algorithm for generating BWT
         bwtupdate     update .bwt to the new format
         bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.
      There are three alignment algorithms in BWA: `mem', `bwasw', and
      `aln/samse/sampe'. If you are not sure which to use, try `bwa mem'
      first. Please `man ./bwa.1' for the manual.

  9. Path to samtools executable file is provided. Checking samtools help manual......

Program: samtools (Tools for alignments in the SAM format)
Version: 1.9 (using htslib 1.9)

Usage:   samtools <command> [options]

Commands:
  -- Indexing
     dict           create a sequence dictionary file
     faidx          index/extract FASTA
     fqidx          index/extract FASTQ
     index          index alignment

  -- Editing
     calmd          recalculate MD/NM tags and '=' bases
     fixmate        fix mate information
     reheader       replace BAM header
     targetcut      cut fosmid regions (for fosmid pool only)
     addreplacerg   adds or replaces RG tags
     markdup        mark duplicates

  -- File operations
     collate        shuffle and group alignments by name
     cat            concatenate BAMs
     merge          merge sorted alignments
     mpileup        multi-way pileup
     sort           sort alignment file
     split          splits a file by read group
     quickcheck     quickly check if SAM/BAM/CRAM file appears intact
     fastq          converts a BAM to a FASTQ
     fasta          converts a BAM to a FASTA

  -- Statistics
     bedcov         read depth per BED region
     depth          compute the depth
     flagstat       simple stats
     idxstats       BAM index stats
     phase          phase heterozygotes
     stats          generate stats (former bamcheck)

  -- Viewing
     flags          explain BAM flags
     tview          text alignment viewer
     view           SAM<->BAM<->CRAM conversion
     depad          convert padded BAM to unpadded BAM

  10. Path to bedtools executable file is provided. Checking bedtools manual......
bedtools is a powerful toolset for genome arithmetic.

Version:   v2.28.0
About:     developed in the quinlanlab.org and by many contributors worldwide.
Docs:      http://bedtools.readthedocs.io/
Code:      https://github.com/arq5x/bedtools2
Mail:      https://groups.google.com/forum/#!forum/bedtools-discuss

Usage:     bedtools <subcommand> [options]

The bedtools sub-commands include:

[ Genome arithmetic ]
    intersect     Find overlapping intervals in various ways.
    window        Find overlapping intervals within a window around an interval.
    closest       Find the closest, potentially non-overlapping interval.
    coverage      Compute the coverage over defined intervals.
    map           Apply a function to a column for each overlapping interval.
    genomecov     Compute the coverage over an entire genome.
    merge         Combine overlapping/nearby intervals into a single interval.
    cluster       Cluster (but don't merge) overlapping/nearby intervals.
    complement    Extract intervals _not_ represented by an interval file.
    shift         Adjust the position of intervals.
    subtract      Remove intervals based on overlaps b/w two files.
    slop          Adjust the size of intervals.
    flank         Create new intervals from the flanks of existing intervals.
    sort          Order the intervals in a file.
    random        Generate random intervals in a genome.
    shuffle       Randomly redistribute intervals in a genome.
    sample        Sample random records from file using reservoir sampling.
    spacing       Report the gap lengths between intervals in a file.
    annotate      Annotate coverage of features from multiple files.

[ Multi-way file comparisons ]
    multiinter    Identifies common intervals among multiple interval files.
    unionbedg     Combines coverage intervals from multiple BEDGRAPH files.

[ Paired-end manipulation ]
    pairtobed     Find pairs that overlap intervals in various ways.
    pairtopair    Find pairs that overlap other pairs in various ways.

[ Format conversion ]
    bamtobed      Convert BAM alignments to BED (& other) formats.
    bedtobam      Convert intervals to BAM records.
    bamtofastq    Convert BAM records to FASTQ records.
    bedpetobam    Convert BEDPE intervals to BAM records.
    bed12tobed6   Breaks BED12 intervals into discrete BED6 intervals.

[ Fasta manipulation ]
    getfasta      Use intervals to extract sequences from a FASTA file.
    maskfasta     Use intervals to mask sequences from a FASTA file.
    nuc           Profile the nucleotide content of intervals in a FASTA file.

[ BAM focused tools ]
    multicov      Counts coverage from multiple BAMs at specific intervals.
    tag           Tag BAM alignments based on overlaps with interval files.

[ Statistical relationships ]
    jaccard       Calculate the Jaccard statistic b/w two sets of intervals.
    reldist       Calculate the distribution of relative distances b/w two files.
    fisher        Calculate Fisher statistic b/w two feature files.

[ Miscellaneous tools ]
    overlap       Computes the amount of overlap from two intervals.
    igv           Create an IGV snapshot batch script.
    links         Create a HTML page of links to UCSC locations.
    makewindows   Make interval "windows" across a genome.
    groupby       Group by common cols. & summarize oth. cols. (~ SQL "groupBy")
    expand        Replicate lines based on lists of values in columns.
    split         Split a file into multiple files with equal records or base pairs.

[ General help ]
    --help        Print this help menu.
    --version     What version of bedtools are you using?.
    --contact     Feature requests, bugs, mailing lists, etc.

  11. Action is defined for training raw input Hi-C sequencing data.
  12. Action is defined for simulate Hi-C sequencing data.
  13. Action is defined for whether to process simulated Hi-C sequencing data.
  14. Number of processing core is specified.
  15. Number of maximum alignment mismatches allowed is specified.
  16. Number of maximum alignment gaps allowed is specified.
  17. No number for the percentage of simulated reads that have mismatches is specified. FreeHi-C will use the default percentage that is equal to that of the input raw Hi-C sequencing data.
  18. No number for the percentage of simulated reads that have gap is specified. FreeHi-C will use the default percentage that is equal to that of the input raw Hi-C sequencing data.
  19. No number for the percentage of simulated reads that are chimeric reads is specified. FreeHi-C will use the default percentage that is equal to that of the input raw Hi-C sequencing data.
  20. Sequencing depth of simulation is specified.
  21. Read length of simulation is specified. Please make sure it does not exceed the read length of input raw Hi-C reads.
  22. Resolution for processing Hi-C reads is specified.
  23. Lower bound for valid long-range interactions is specified.
  24. Upper bound for the distance to the assigned restriction enyzme cutting site is specified.
  25. Sequences at the ligation site is provided.
Input parameter checking is successfull!!
Start FreeHi-C......
I. Training: 1 - Alignment!
(We assume the reads length are the same within the same fastq file.)......

Step1.3 - Trim unmapped reads until the restriction enzyme cutting site.
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 73810881 reads
##Fastq file: /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_1.fastq
##Restriction sites:
AAGCTAGCTT
##Output File: /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_1.fastq
##Trimmed reads: 3676

Step1.4 - Rescue chimeric reads by re-aligning trimmed unmapped reads.
[bwa_aln_core] calculate SA coordinate... 6.69 sec
[bwa_aln_core] write to the disk... 0.00 sec
[bwa_aln_core] 2551 sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa aln -n 3 -o 1 -t 20 /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_filter_1.fastq
[main] Real time: 6.780 sec; CPU: 10.564 sec
[bwa_aln_core] convert to sequence coordinate... 4.56 sec
[bwa_aln_core] refine gapped alignments... 0.84 sec
[bwa_aln_core] print alignments... 0.00 sec
[bwa_aln_core] 2551 sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa samse /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_filter_1.sai /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_filter_1.fastq
[main] Real time: 7.280 sec; CPU: 5.413 sec
step1.5 - Merge chimeric reads with mapped full-length reads.
Start alignment of read end 2
Step1.1 - BWA alignment
[bwa_aln_core] calculate SA coordinate... 328.69 sec
[bwa_aln_core] write to the disk... 0.10 sec
[bwa_aln_core] 262144 sequences have been processed.
.
.
.
[bwa_aln_core] convert to sequence coordinate... 4.88 sec
[bwa_aln_core] refine gapped alignments... 0.95 sec
[bwa_aln_core] print alignments... 0.46 sec
[bwa_aln_core] 203342006 sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa samse /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_2.sai /mss5/RACA2/read_simulation/hic/realData/Cow/SRR6691720_2.fastq
[main] Real time: 8567.442 sec; CPU: 6501.996 sec
Step1.2 - Filter and get unmapped alignment sam file.
Step1.3 - Trim unmapped reads until the restriction enzyme cutting site.
[M::bam2fq_mainloop] discarded 0 singletons
[M::bam2fq_mainloop] processed 89078768 reads
##Fastq file: /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_2.fastq
##Restriction sites:
AAGCTAGCTT
##Output File: /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_2.fastq
##Trimmed reads: 4339

Step1.4 - Rescue chimeric reads by re-aligning trimmed unmapped reads.
[bwa_aln_core] calculate SA coordinate... 6.39 sec
[bwa_aln_core] write to the disk... 0.00 sec
[bwa_aln_core] 3067 sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa aln -n 3 -o 1 -t 20 /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_filter_2.fastq
[main] Real time: 11.036 sec; CPU: 9.074 sec
[bwa_aln_core] convert to sequence coordinate... 3.82 sec
[bwa_aln_core] refine gapped alignments... 0.67 sec
[bwa_aln_core] print alignments... 0.01 sec
[bwa_aln_core] 3067 sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa samse /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_filter_2.sai /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_unmapped_trim_filter_2.fastq
[main] Real time: 12.245 sec; CPU: 4.506 sec
step1.5 - Merge chimeric reads with mapped full-length reads.
I. Training: 2 - Joining read-ends!
Read 1 file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_1.sam
Read 2 file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720_2.sam
Output file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720.sam
Summary of alignment results =  True
Summary file path =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/summary/bosTau9_FreeHiC.summary
Training =  1
Verbose =  True
Trainning!
------------Begin joining ends from read 1 file and read 2 file------------
# of reads processed:  1000000
.
.
.
# of reads processed:  203000000
I. Training: 3 - Categorize read-pairs!
RE fragment =  /mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed
Paired-ended read alignment file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s1_alignment/SRR6691720.sam
Output directory =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/rawDataTraining/s2_validPairs
RE cutting site distance lower bound =  50
RE cutting site distance upper bound =  800
Minimum intrachromosomal contact distance =  80000
Binning method =  window
Binning resolution =  40000
Summary =  True
SummaryFile =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_10x/summary/bosTau9_FreeHiC.summary
Verbose =  True
## Loading Restriction File Intervals '/mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed'...
dict_keys(['chr1', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr2', 'chr20', 'chr21', 'chr22', 'chr23', 'chr24', 'chr25', 'chr26', 'chr27', 'chr28', 'chr29', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr999'])
------------Begin reading paired-ended file------------
------------Categorize aligned read pairs------------
I. Training: 4 - Remove duplicates!
I. Training: 5 - Binning!

[Simulation & Post-process step log]
projDir=/mss5/RACA2/programs/FreeHiC
fastqFile=/mss5/RACA2/read_simulation/hic/realData/Cow/SRR6691720
ref=/mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9.soft-masked.rename.fa
refrag=/mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed
outDir=/mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x
simuName=bosTau9_05x
summaryFile=/mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/summary/bosTau9_FreeHiC.summary
bwa=bwa
samtools=samtools
bedtools=bedtools
train=0
simulate=1
postProcess=1
coreN=20
mismatchN=3
gapN=1
mismatchP=
gapP=
chimericP=
simuN=157702285
readLen=80
resolution=40000
lowerBound=80000
refragU=800
ligateSite=AAGCTAGCTT
Pre-checking the input parameters......
  1. FreeHi-C package directory exist.
  2. /mss5/RACA2/read_simulation/hic/realData/Cow/SRR6691720 will be downloaded.
  3. Reference genome file exists and is not empty.
  4. Restriction enzyme fragment file exists and is not empty.
  5. Simulation run name is provided.
  6. Output directory is provided.
  7. Summary file path is provided.
  8. Path to BWA executable file is provided. Checking bwa help manual......

Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.17-r1198-dirty
Contact: Heng Li<EMAIL_ADDRESS>
Usage:   bwa <command> [options]

Command: index         index sequences in the FASTA format
         mem           BWA-MEM algorithm
         fastmap       identify super-maximal exact matches
         pemerge       merge overlapping paired ends (EXPERIMENTAL)
         aln           gapped/ungapped alignment
         samse         generate alignment (single ended)
         sampe         generate alignment (paired ended)
         bwasw         BWA-SW for long queries

         shm           manage indices in shared memory
         fa2pac        convert FASTA to PAC format
         pac2bwt       generate BWT from PAC
         pac2bwtgen    alternative algorithm for generating BWT
         bwtupdate     update .bwt to the new format
         bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.
      There are three alignment algorithms in BWA: `mem', `bwasw', and
      `aln/samse/sampe'. If you are not sure which to use, try `bwa mem'
      first. Please `man ./bwa.1' for the manual.

  9. Path to samtools executable file is provided. Checking samtools help manual......

Program: samtools (Tools for alignments in the SAM format)
Version: 1.9 (using htslib 1.9)

Usage:   samtools <command> [options]

Commands:
  -- Indexing
     dict           create a sequence dictionary file
     faidx          index/extract FASTA
     fqidx          index/extract FASTQ
     index          index alignment

  -- Editing
     calmd          recalculate MD/NM tags and '=' bases
     fixmate        fix mate information
     reheader       replace BAM header
     targetcut      cut fosmid regions (for fosmid pool only)
     addreplacerg   adds or replaces RG tags
     markdup        mark duplicates

  -- File operations
     collate        shuffle and group alignments by name
     cat            concatenate BAMs
     merge          merge sorted alignments
     mpileup        multi-way pileup
     sort           sort alignment file
     split          splits a file by read group
     quickcheck     quickly check if SAM/BAM/CRAM file appears intact
     fastq          converts a BAM to a FASTQ
     fasta          converts a BAM to a FASTA

  -- Statistics
     bedcov         read depth per BED region
     depth          compute the depth
     flagstat       simple stats
     idxstats       BAM index stats
     phase          phase heterozygotes
     stats          generate stats (former bamcheck)

  -- Viewing
     flags          explain BAM flags
     tview          text alignment viewer
     view           SAM<->BAM<->CRAM conversion
     depad          convert padded BAM to unpadded BAM

  10. Path to bedtools executable file is provided. Checking bedtools manual......
bedtools is a powerful toolset for genome arithmetic.

Version:   v2.28.0
About:     developed in the quinlanlab.org and by many contributors worldwide.
Docs:      http://bedtools.readthedocs.io/
Code:      https://github.com/arq5x/bedtools2
Mail:      https://groups.google.com/forum/#!forum/bedtools-discuss

Usage:     bedtools <subcommand> [options]

The bedtools sub-commands include:

[ Genome arithmetic ]
    intersect     Find overlapping intervals in various ways.
    window        Find overlapping intervals within a window around an interval.
    closest       Find the closest, potentially non-overlapping interval.
    coverage      Compute the coverage over defined intervals.
    map           Apply a function to a column for each overlapping interval.
    genomecov     Compute the coverage over an entire genome.
    merge         Combine overlapping/nearby intervals into a single interval.
    cluster       Cluster (but don't merge) overlapping/nearby intervals.
    complement    Extract intervals _not_ represented by an interval file.
    shift         Adjust the position of intervals.
    subtract      Remove intervals based on overlaps b/w two files.
    slop          Adjust the size of intervals.
    flank         Create new intervals from the flanks of existing intervals.
    sort          Order the intervals in a file.
    random        Generate random intervals in a genome.
    shuffle       Randomly redistribute intervals in a genome.
    sample        Sample random records from file using reservoir sampling.
    spacing       Report the gap lengths between intervals in a file.
    annotate      Annotate coverage of features from multiple files.

[ Multi-way file comparisons ]
    multiinter    Identifies common intervals among multiple interval files.
    unionbedg     Combines coverage intervals from multiple BEDGRAPH files.

[ Paired-end manipulation ]
    pairtobed     Find pairs that overlap intervals in various ways.
    pairtopair    Find pairs that overlap other pairs in various ways.

[ Format conversion ]
    bamtobed      Convert BAM alignments to BED (& other) formats.
    bedtobam      Convert intervals to BAM records.
    bamtofastq    Convert BAM records to FASTQ records.
    bedpetobam    Convert BEDPE intervals to BAM records.
    bed12tobed6   Breaks BED12 intervals into discrete BED6 intervals.

[ Fasta manipulation ]
    getfasta      Use intervals to extract sequences from a FASTA file.
    maskfasta     Use intervals to mask sequences from a FASTA file.
    nuc           Profile the nucleotide content of intervals in a FASTA file.

[ BAM focused tools ]
    multicov      Counts coverage from multiple BAMs at specific intervals.
    tag           Tag BAM alignments based on overlaps with interval files.

[ Statistical relationships ]
    jaccard       Calculate the Jaccard statistic b/w two sets of intervals.
    reldist       Calculate the distribution of relative distances b/w two files.
    fisher        Calculate Fisher statistic b/w two feature files.

[ Miscellaneous tools ]
    overlap       Computes the amount of overlap from two intervals.
    igv           Create an IGV snapshot batch script.
    links         Create a HTML page of links to UCSC locations.
    makewindows   Make interval "windows" across a genome.
    groupby       Group by common cols. & summarize oth. cols. (~ SQL "groupBy")
    expand        Replicate lines based on lists of values in columns.
    split         Split a file into multiple files with equal records or base pairs.

[ General help ]
    --help        Print this help menu.
    --version     What version of bedtools are you using?.
    --contact     Feature requests, bugs, mailing lists, etc.

  11. Action is defined for training raw input Hi-C sequencing data.
  12. Action is defined for simulate Hi-C sequencing data.
  13. Action is defined for whether to process simulated Hi-C sequencing data.
  14. Number of processing core is specified.
  15. Number of maximum alignment mismatches allowed is specified.
  16. Number of maximum alignment gaps allowed is specified.
  17. No number for the percentage of simulated reads that have mismatches is specified. FreeHi-C will use the default percentage that is equal to that of the input raw Hi-C sequencing data.
  18. No number for the percentage of simulated reads that have gap is specified. FreeHi-C will use the default percentage that is equal to that of the input raw Hi-C sequencing data.
  19. No number for the percentage of simulated reads that are chimeric reads is specified. FreeHi-C will use the default percentage that is equal to that of the input raw Hi-C sequencing data.
  20. Sequencing depth of simulation is specified.
  21. Read length of simulation is specified. Please make sure it does not exceed the read length of input raw Hi-C reads.
  22. Resolution for processing Hi-C reads is specified.
  23. Lower bound for valid long-range interactions is specified.
  24. Upper bound for the distance to the assigned restriction enyzme cutting site is specified.
  25. Sequences at the ligation site is provided.
Input parameter checking is successfull!!
Start FreeHi-C......
## *****************************************
## II. Simulating interaction read sequences
## *****************************************
II. Simulating!
Feature (chr23:52498684-52498764) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498577-52498657) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498553-52498633) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498591-52498671) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498556-52498636) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498609-52498689) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498652-52498732) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498611-52498691) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498651-52498731) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498551-52498631) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498666-52498746) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498547-52498627) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498640-52498720) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498584-52498664) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498544-52498624) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498663-52498743) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498648-52498728) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498580-52498660) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498570-52498650) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498594-52498674) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498647-52498727) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498599-52498679) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498578-52498658) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498553-52498633) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498598-52498678) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498629-52498709) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498651-52498731) beyond the length of chr23 size (52498615 bp).  Skipping.
Feature (chr23:52498538-52498618) beyond the length of chr23 size (52498615 bp).  Skipping.
check1
check2
check3
check4
RE fragment =  /mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed
Fragment interaction file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/rawDataTraining/s2_validPairs/SRR6691720.validPairs.fragFreq
Output directory =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuSequence
Output file name =  SRR6691720
Total number of interactions to simulate =  157702285
Read length to simulate =  80
Maximum distance away from restriction site =  800
Path to bedtools =  bedtools
Path to reference genome fasta file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9.soft-masked.rename.fa
Path to summary file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/summary/bosTau9_FreeHiC.summary
Verbose =  True
Read in restriction fragment dictionary!
Read in interaction frequency!
Convert dictionary and list into C structure!
Sample fragment interaction!
Read in trained parameters!
243797050
101747200
2285627
2097474
Simulate end 1!
0
.
.
.
157500000
Get fasta from the reference genome!
Add mutation!
500000
.
.
.
65500000
Add insertion!
Add deletion
Simulate end 2!
0
.
.
.
157500000
Get fasta from reference genome!
Add mutation!
500000
.
.
.
65500000
Add insertion!
Add deletion!
II. Simulation Post-processing: 1 - Alignment!
(We assume the reads length are the same within the same fastq file.)......

Start alignment of read end 2
Step1.1 - BWA alignment
[bwa_aln_core] calculate SA coordinate... 53.93 sec
[bwa_aln_core] write to the disk... 0.07 sec
[bwa_aln_core] 262144 sequences have been processed.
.
.
.
[bwa_aln_core] calculate SA coordinate... 48.07 sec
[bwa_aln_core] write to the disk... 0.06 sec
[bwa_aln_core] 11505705 sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa aln -n 3 -o 1 -t 20 /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuSequence/SRR6691720_2.fastq
[main] Real time: 161.632 sec; CPU: 2509.953 sec
[bwa_aln_core] convert to sequence coordinate... 8.88 sec
[bwa_aln_core] refine gapped alignments... 0.69 sec
[bwa_aln_core] print alignments... 0.45 sec
[bwa_aln_core] 262144 sequences have been processed.
.
.
.
[bwa_aln_core] convert to sequence coordinate... 4.38 sec
[bwa_aln_core] refine gapped alignments... 0.59 sec
[bwa_aln_core] print alignments... 0.40 sec
[bwa_aln_core] 11505705 sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa samse /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuProcess/s1_alignment/SRR6691720_2.sai /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuSequence/SRR6691720_2.fastq
[main] Real time: 263.851 sec; CPU: 262.748 sec
Choose not to rescue chimeric reads or the chimeric reads have been trimmed before alignment!
[bam_sort_core] merging from 0 files and 20 in-memory blocks...
[bam_sort_core] merging from 0 files and 20 in-memory blocks...
II. Simulation Post-processing: 2 - Joining read-ends!
Read 1 file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuProcess/s1_alignment/SRR6691720_1.sam
Read 2 file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuProcess/s1_alignment/SRR6691720_2.sam
Output file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuProcess/s1_alignment/SRR6691720.sam
Summary of alignment results =  True
Summary file path =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/summary/bosTau9_FreeHiC.summary
Training =  0
Verbose =  True
------------Begin joining ends from read 1 file and read 2 file------------
Read id in read 1 and read 2 file does not match. Please check the input read files and sort them correctly.
II. Simulation Post-processing: 3 - Categorize read-pairs!
RE fragment =  /mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed
Paired-ended read alignment file =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuProcess/s1_alignment/SRR6691720.sam
Output directory =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuProcess/s2_validPairs
RE cutting site distance lower bound =  50
RE cutting site distance upper bound =  800
Minimum intrachromosomal contact distance =  80000
Binning method =  window
Binning resolution =  40000
Summary =  True
SummaryFile =  /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/summary/bosTau9_FreeHiC.summary
Verbose =  True
## Loading Restriction File Intervals '/mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed'...
dict_keys(['chr1', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr19', 'chr2', 'chr20', 'chr21', 'chr22', 'chr23', 'chr24', 'chr25', 'chr26', 'chr27', 'chr28', 'chr29', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chr8', 'chr9', 'chr999'])
------------Begin reading paired-ended file------------
------------Categorize aligned read pairs------------
II. Simulation Post-processing: 4 - Binning!

If you need any other information, please ask me!
Thanks,
Sue

Can you check the number of reads in /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuSequence/SRR6691720_1.fastq and /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuSequence/SRR6691720_2.fastq ? It seems that more than 157500000 reads are simulated for each end by the following log.
Simulate end 1!
0
.
.
.
**157500000**

However, only 1763658 reads of end 1 are aligned by bwa aln. Similarly for end 2. Much fewer reads are processed by bwa.
[bwa_aln_core] **1763658** sequences have been processed.
[main] Version: 0.7.17-r1198-dirty
[main] CMD: bwa aln -n 3 -o 1 -t 20 /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9.soft-masked.rename.fa /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9_05x/bosTau9_05x/simuSequence/SRR6691720_1.fastq
[main] Real time: 41.735 sec; CPU: 383.532 sec

If the fastq files do not contain 157702285 reads, then there should be something wrong with the simulation section. BTW, have you tried simulating using the demo data and demo runs?
Ye

There are 157,702,265 reads in simuSequence/SRR6691720_1.fastq file
and 157,702,277 reads in  simuSequence/SRR6691720_2.fastq file.
Yes, I have already run the demo data and got simulation data. It seems that the demo run did not have any problems in outputs.

Hi,
I ran the simulation section with 10 times fewer numbers (15,000,000) as you recommended
There are 14,999,999 reads in both _1.fastq and _2.fastq files.
However, the read ids were not matched in this simulation data, and also the same error messages emerged.
Read id in read 1 and read 2 file does not match. Please check the input read files and sort them correctly.
These are the examples of final simulation data
[SRR6691720_1.fastq]
[sy0819@europa00 simuSequence]$ head -n 12 SRR6691720_1.fastq
@0(-)
ggggtagaggggtgggatgaagagattgggattgacatatatatacatgattgatactattcataaaatagataagtaat
+
EAEEAEEEEEEEEEEEEEEEEEEEEEEAEEEEEEEEEEEEEEE/AEEEEEEEE//AEEEEAEEEEEEEE/EEEEEE/EE/
@2(+)
CTGTTGAACCATATGATGATTGTTGGCCTCCTAACCttcccccttaacagttcagcacttgtctccttagaacaagggct
+
AAAEEEEEE/EE6EEE/EEEEEEEEEEEEEE<EEEEAEEEEEEEEEEEEEEEEE/EEEE/EEEEEEEEEEEEEEEAAE/E
@3(-)
acatattaatcaaattaacaaagatcaaacacaaagaacaaatattaaaagcagcaagggaaaaacaacaaataacacac
+
AAEEAEEEEEE/EEEEEEEAEEEE//AEAEEEEEEAEEEEEEEAE6/EEEEEEEEE6EEEE/EEEEEEEEEEE<EAAAAE

[SRR6691720_2.fastq]
[sy0819@europa00 simuSequence]$ head -n 12 SRR6691720_2.fastq
@0(+)
agaaaatcaatgaaactaaaagttgcttctttgaaaagattaataaaattgataaatgtctagccactactaaaaaaaat
+
EA/AAAEEE6EEEEAEEAEEAEE6EEEEEEEEEE<EE/EEEEEEEEEEEE<AEEEEEEEEEEAEEEEEEAEEE<EAAAEA
@1(-)
ttgctggcatattgagtgcagcactttcacagcatcatctttcaggatttggaatagctcaactggaattccatcacctc
+
AEEEAEAEEEEEEEEEEEEEEAEEEEEEEEE/E/EEEEE/EEEEEEEEAEEEE//EEEEEE/EE/EE6EEEEE/EAEAA<
@3(+)
gcttccctcctggccctctcccctcactcctccacctgctcccattcctcaaacacaccatgctcccttccgcttcgaga
+
AE/EAEEEEEEEEAAEEEE6EEEEAAEEEEAEEEEEEEEEEEEEEEEEEEEEAEEEEEEEAEEEEEE/AAEEEEE/AAEA

Please let me know if you need other information
Thanks,
Sue

I see!! I just realized that bedtools has updated its getfasta function in recent released versions!! A quick solution is that if you can download and install 2.25.0 version of bedtools: https://github.com/arq5x/bedtools2/releases/tag/v2.25.0 and still use current freehic software, it should solve the problem. Otherwise, I will update the freehic repository later this week to accommodate the new output from the latest version of bedtools.
Let me know how it goes! Thanks for bringing it up! Appreciate your time and tries!
Ye

I was using bedtools 2.28.0 version before and I downloaded bedtools 2.25.0 version as you advised
I corrected the bedtools path in the parameter files and re-ran the simulation
bedtools="/mss5/RACA2/programs/bedtools2/bin/bedtools"
[sy0819@europa00 bosTau9]$ /mss5/RACA2/programs/bedtools2/bin/bedtools -version
bedtools v2.25.0

But I still get the same error messages before and the numbers of reads in both files are still different
Read id in read 1 and read 2 file does not match. Please check the input read files and sort them correctly.
Total line counts in both reads and examples of FASTQ files are below
[sy0819@europa00 simuSequence]$ wc -l *
  59999996 SRR6691720_1.fastq
  60000000 SRR6691720_2.fastq
[sy0819@europa00 simuSequence]$ head SRR6691720_1.fastq
@0
TGAGCCTGTGTTATCCCCCTCTTCTCATGTGTCCCCCTGCTAGAGGTGCAGCTCTAACTAGATCACTTCTCCTTTACTGG
+
EAAEAA/EEE6EAA/EEEEEEEA6EEEEEEEAEAAEEEE//EE/EEEEEEEEEEEEEEEEEEEEEEAEE<E/E/EAA6AA
@2
acttctgcaaagaagacatatagatggccaataaacacatgaaaagattctcgacatcactaactattCAGTTCAGTTCa
+
AE6EAAEEEEEEE/EEEEEEEEEEEEEEEEEEE//EE/EEAEEEEE/EEEEEEEEEEEE/EEEEEAE/EEEE<EEAEAAA
@4
AATAATAATAACAGTGAAAGAATGATTCTTCACCATTGGTGTGGTGTTTTTTCCCTTTCATGccccaaacacatgtgtat
[sy0819@europa00 simuSequence]$ head SRR6691720_2.fastq
@1
gagaatcccatggaccgaggaacctggtaggctgcagtccatgaggtcgctaagagtcggacacgactgagagacttcac
+
AEA/AEEEEAEEAAEEEEAEEEEEEEEEEEEE/E<EEEEAEE/EEEEEAEE/EEE/E<EE/EEEEEAEEEEEEAEA/AAA
@2
AATAGAAAATGGAAAGTAATTGAGGAGACATGTCCATATATTGTCAAACTAAAAATAACTAACCCATGGATGCTTTGAAA
+
AA//AEEEEAEEEEEEEEEEEAE/E<EEEEEEE/E/E/E/EEEEEEEEAE6EEE/EAEEEEE/EEEE<EEEAEEEEEEAE
@3
AAAAATATACCTTTTAAATAAACATCTATAAAAGTCTAAAATAATTATAAGTATTGAAAAAAATCAGGAATTTTGCTTTC

I think bedtools could affect the final results but read unmatching problem is not that case.
Could you check it again please?
Sue

Hello Sue,
I tried many ways but I still cannot reproduce the problem you have using demo data and other data on my side. Do you mind sharing the following two datasets or tell me where you downloaded online so that I can re-run your data and see what is happening?
ref=/mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa
refrag=/mss5/RACA2/read_simulation/hic/FreeHiC/cow_hindiii.bed

BTW, I noticed that you claim to use 20 cores for parallel alignment. Can you confirm that the computing resources are available on your side?
Apologize for the trouble that delays your studies!
Ye

Hello Ye,
I used the bosTau9 chromosome level assembly which is soft-masked and changed the name of chrX to chr999.
[sy0819@europa00 FreeHiC]$ grep ">" bosTau9.soft-masked.rename.fa
>chr10
>chr11
>chr12
>chr13
>chr14
>chr15
>chr16
>chr17
>chr18
>chr19
>chr1
>chr20
>chr21
>chr22
>chr23
>chr24
>chr25
>chr26
>chr27
>chr28
>chr29
>chr2
>chr3
>chr4
>chr5
>chr6
>chr7
>chr8
>chr9
>chr999

You could just download the data from the link below and filtering regular chromosome sequences only (chr1~29 + chrX).
After that, just rename chrX into chr999.
http://hgdownload.soe.ucsc.edu/goldenPath/bosTau9/bigZips/bosTau9.fa.gz
BTW, I renamed the chromosome X as I got some error messages while using FreeHiC because of chrX.
It would be great if FreeHiC supports non-integer chromosome names including chromosome X or Y also :)
cow_hindiii.bed was made by HiC-Pro's digest_genome.py script.
$PATH/HiC-Pro_2.11.1/bin/utils/digest_genome.py -r hindiii -o cow_hindiii.bed /mss5/RACA2/read_simulation/hic/FreeHiC/bosTau9/bosTau9.soft-masked.rename.fa

Or I could just share these data by email.
So if you want me to send these data directly to you, please ask me!
Thanks,
Sue

While I am running, if you are interested, you can also check out the lite version of FreeHiC that we recently developed. It is an approximation of the original FreeHi-C simulation pipeline but much faster. ;-)

Sounds great!
I'll try the lite version also!
If you have any progress, please let me know! :)
Thanks,
Sue

Hello, ye!
Did you reproduce the same results from mine with the data I told you before?
It'll be great if I could know any progress about it!
Thanks,
Sue

Apologize for the delay! I should be able to finish the runs later this week.
Ye

Hello Sue,
I have tried on your data and simulated at 15000 and 15000000. Both works successfully on my side: I got the same number of reads in fastq files and matching read query name after alignment and sorting (instead of checking the simulated fastq file, it is more reliable to check the read id in sam file as reads in fastq file are not sorted by read id yet.)
>wc -l *fastq
  60000000 cowData_demo_1.fastq
  60000000 cowData_demo_2.fastq
 120000000 total

>less cowData_demo_1.sam
@HD     VN:1.6  SO:queryname
@SQ     SN:chr1 LN:158534110
@SQ     SN:chr2 LN:136231102
@SQ     SN:chr3 LN:121005158
@SQ     SN:chr4 LN:120000601
@SQ     SN:chr5 LN:120089316
@SQ     SN:chr6 LN:117806340
@SQ     SN:chr7 LN:110682743
@SQ     SN:chr8 LN:113319770
@SQ     SN:chr9 LN:105454467
@SQ     SN:chr10        LN:103308737
@SQ     SN:chr11        LN:106982474
@SQ     SN:chr12        LN:87216183
@SQ     SN:chr13        LN:83472345
@SQ     SN:chr14        LN:82403003
@SQ     SN:chr15        LN:85007780
@SQ     SN:chr16        LN:81013979
@SQ     SN:chr17        LN:73167244
@SQ     SN:chr18        LN:65820629
@SQ     SN:chr19        LN:63449741
@SQ     SN:chr20        LN:71974595
@SQ     SN:chr21        LN:69862954
@SQ     SN:chr22        LN:60773035
@SQ     SN:chr23        LN:52498615
@SQ     SN:chr24        LN:62317253
@SQ     SN:chr25        LN:42350435
@SQ     SN:chr26        LN:51992305
@SQ     SN:chr27        LN:45612108
@SQ     SN:chr28        LN:45940150
@SQ     SN:chr29        LN:51098607
@SQ     SN:chr999       LN:139009144
@PG     ID:bwa  PN:bwa  VN:0.7.15-r1140 CL:bwa-0.7.15/bwa samse cowData/data/bosTau9.soft-masked.rename.fa cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_1.sai cowData/results/bosTau9_test/simuSequence/cowData_demo_1.fastq
@PG     ID:samtools     PN:samtools     PP:bwa  VN:1.10 CL:samtools_v1.10/bin/samtools sort -n -@ 4 -o cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_1.sorted.sam cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_1.sam
@PG     ID:samtools.1   PN:samtools     PP:samtools     VN:1.10 CL:samtools_v1.10/bin/samtools sort -n -@ 4 -o cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_1.sorted.sam cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_1.sam
0       0       chr2    101240873       37      80M     *       0       0       GGAAACCCAAAGAGGTGAAGAAGGAAATAGCAACCATAAGGCAACAAATTAAAAAAAGAGGAGTCAGCATAAAGCCTGGA        EAA<EEE/E/EEEEE/EEE/EAEEEEEEEEEEE/EEE6EEEEEEEEEEE6EEEEEEAEEEEEEAEEEEEEEEEEEAA/AA        XT:A:U  NM:i:1  X0:i:1  X1:i:0  XM:i:1  XO:i:0  XG:i:0  MD:Z:55T24
1       16      chr9    31279648        37      81M     *       0       0       GCTCTTCCCAGAATCAGCTTTAGCCTTGGCTACATCTTCCTGAGCGTTTCTCTTCTGGCTTTGGACCGTCTGTCAAATGAA       AEAEEA/EEEE6EEEAEEE6EEEA/EEEEEE/EEEEEEEEEEE6/EEEEEEEEEEEEAE<EEEEAAEAEE/EEEEEEEAAA       XT:A:U  NM:i:2  X0:i:1  X1:i:0  XM:i:2  XO:i:0  XG:i:0  MD:Z:28A51C0
2       0       chr3    29509100        37      80M     *       0       0       TCTACATCTGTTTCACTTTGGTGGTACATTTCACAGGGTCTTTAACACTAGGCAGATCTCCCCATTCTTACCACTTGTTA        AAEA/EEEEEEEEE<E6EEEEEE/EEEEE/EAEEEEEEEE/EEEEAEAEEEEEEEEEEEEEEEEEE/EEE/EE/EEAEA/        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80
3       16      chr21   25837445        37      80M     *       0       0       TTCAACCATCTCATCCTCTGTCACTGTCTTCTGCCCTCAATCTCTCACAGCATCAGGGTCTTTTCCAATGAGTTGGCTCT        AEAAAEEA<6/EEEEEEEEEEE/EEAEEEEEEAE/EEEEAEEEEEE/EEEEEEAEEEEEEEEEEEE<EEEEEEEEAAAEA        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80
4       16      chr9    35227839        37      80M     *       0       0       TTGTAAATCCCTTGACCTGGTCAAGCTTATTTCCTACATTCTAAATTTTATGCTCAAGAATTACTAACTTCCACAAATCT        AAAAAE/EEEEEEEEE/E/EEEEEEEEAEAE/EAEEEEEE/EEEEEEEE/EAEAAEEEEEEEEEE/AE/E<EE/EA/AEA        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80
5       0       chr11   21856404        37      80M     *       0       0       TAGGAGCTCAAAGCATGTATCCCATTCCCCTTTCCCTGGGTCTGTTACACATGATAAGTTCTAAGAACAATCTAAGACTA        AAEAE6E/EEEEAEEAEEEE//EE/EEEEEEEEEEEEEEEEE/AE6A/E/6AEEEEAE/EEEEA6EEEEE/EEEEAEA<A        XT:A:U  NM:i:1  X0:i:1  X1:i:0  XM:i:1  XO:i:0  XG:i:0  MD:Z:28T51
6       16      chr11   21855034        37      80M     *       0       0       AGCTGTGTGGATCTATTTTACATCATGGATCTCCTTCAGAGAGTAGAGTTCTACCAGAGAATGTTGCAAAGAGTAACCAG        AEAAAEEEAAEE/EEEAEE/E/EEEEEEEE/A<EEEEEEEEEEEEEEEEEAEEEEEEEAEEEEAEAE6/EEEEE/EEEAA        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80
7       0       chr29   5819250 37      80M     *       0       0       TAAACAGTGCTATGATGAACATTGGGGTACAGATGTCTCTTTCAATTCTGGTTTCTTCGGTGTGTATGCCCAGCAGTGGG        /EEEEEEEEE/EEEEEEEE/EEEEEEEAEEEEEEEEAE/EEEAE<EE/EEEEE/EEEEEEEAEEEEEEEEEEEEEAEAE/        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80

>less cowData_demo_2.sam
@HD     VN:1.6  SO:queryname
@SQ     SN:chr1 LN:158534110
@SQ     SN:chr2 LN:136231102
@SQ     SN:chr3 LN:121005158
@SQ     SN:chr4 LN:120000601
@SQ     SN:chr5 LN:120089316
@SQ     SN:chr6 LN:117806340
@SQ     SN:chr7 LN:110682743
@SQ     SN:chr8 LN:113319770
@SQ     SN:chr9 LN:105454467
@SQ     SN:chr10        LN:103308737
@SQ     SN:chr11        LN:106982474
@SQ     SN:chr12        LN:87216183
@SQ     SN:chr13        LN:83472345
@SQ     SN:chr14        LN:82403003
@SQ     SN:chr15        LN:85007780
@SQ     SN:chr16        LN:81013979
@SQ     SN:chr17        LN:73167244
@SQ     SN:chr18        LN:65820629
@SQ     SN:chr19        LN:63449741
@SQ     SN:chr20        LN:71974595
@SQ     SN:chr21        LN:69862954
@SQ     SN:chr22        LN:60773035
@SQ     SN:chr23        LN:52498615
@SQ     SN:chr24        LN:62317253
@SQ     SN:chr25        LN:42350435
@SQ     SN:chr26        LN:51992305
@SQ     SN:chr27        LN:45612108
@SQ     SN:chr28        LN:45940150
@SQ     SN:chr29        LN:51098607
@SQ     SN:chr999       LN:139009144
@PG     ID:bwa  PN:bwa  VN:0.7.15-r1140 CL:bwa-0.7.15/bwa samse cowData/data/bosTau9.soft-masked.rename.fa cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_2.sai cowData/results/bosTau9_test/simuSequence/cowData_demo_2.fastq
@PG     ID:samtools     PN:samtools     PP:bwa  VN:1.10 CL:samtools_v1.10/bin/samtools sort -n -@ 4 -o cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_2.sorted.sam cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_2.sam
@PG     ID:samtools.1   PN:samtools     PP:samtools     VN:1.10 CL:samtools_v1.10/bin/samtools sort -n -@ 4 -o cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_2.sorted.sam cowData/results/bosTau9_test/simuProcess/s1_alignment/cowData_demo_2.sam
0       16      chr2    103068449       37      80M     *       0       0       ACTTGACATTCTAAAAGCAGGATCTTTACATTTTTTGTTCATTTAGAAAAAACATTTTAGAGAAAGATTCCTACCAAATA        EAAA/EEEAEAE/EEEEEEE/EEEEEEEEEEEEE/E<EAEEEEEEEAEEEEEEEEAE/AEE/EEA/EEE/6E/EEA/EEA        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80
1       0       chr9    31688430        37      80M     *       0       0       GGAGATGGAAGATCTTTGCTTGGTGAGGCATTCCCTGAGAAAGCAATCCTGGCTAGTTCCTTTGCTCTGCAGAATGCGGA        A6E/<EEEEEAEEEEEEEE<EEEEEEEEEEEEEEEEEEEEEEEEEEEEE<EEEEAAEEEEAEEEEEEEEEEEEEEAAEA<        XT:A:U  NM:i:1  X0:i:1  X1:i:0  XM:i:1  XO:i:0  XG:i:0  MD:Z:77A2
2       0       chr3    30227271        37      80M     *       0       0       GAAAAAATATGAGTCTTTTAAAAATTCATTCTTCCATTCTCAGAAGGTACATCTGCACTGCACCTTGTCCTCATGGAAGA        EAAAAEEEEEEE/EEEEEEEEEEEEEEEEEEEEEEEEEEEEEAEAEEEEEEEEEEAEEAE/EEEEEEEEEEEEEEAEAEA        XT:A:U  NM:i:1  X0:i:1  X1:i:0  XM:i:1  XO:i:0  XG:i:0  MD:Z:51A28
3       16      chr29   22485763        37      80M     *       0       0       CAAAAATATGTTCAACTGCATATATAGCTCACCATTTAGGGTATTCATGAATCCAAAACCCAACCATAGCTTTTAAACCA        AEEAAAAEEEEEEEEEAEE//EEEE/EEEEEAEEEEAE/EEEAAEEEEEAE/EEEEEEEEEEEAEAEEEEEEEE<EA/EA        XT:A:U  NM:i:1  X0:i:1  X1:i:0  XM:i:1  XO:i:0  XG:i:0  MD:Z:67G12
4       16      chr9    37358606        37      80M     *       0       0       CCTTCACTATCTCCCGGAAGTTGTTCAAATTCACATTCACTGAGTTGGTGATGCCATCCAACTATTTCATCCTCTATTGT        EAE6/EEEEEE/EEEEE/EEEEE/EEEEAAEEEEAEEEE<EEEAEEEEEEAEEEEEAEEEEAEEEEEEE/EAEEEEAAAE        XT:A:U  NM:i:1  X0:i:1  X1:i:0  XM:i:1  XO:i:0  XG:i:0  MD:Z:23A56
5       16      chr999  11396651        37      80M     *       0       0       TACAGGGTCAGTTCATTCTATCACTCTGCAGCTTACCAGGTATCTAGGATTTGGGACAGAGTCATAGAAGACAACATGAA        AAAAAEEEEEEEEEEAEEEEEAEEEEEEEEEEEEEEEEEAAEE/EEEEEEEEEEEEEEAEEE6EEA<EEE/EE/EEA6A/        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80
6       0       chr999  11396545        37      80M     *       0       0       ATCTCTGTTTTACACATGAGGAAACTGAGGCTTATGGAGGTTAATAAGTTGTCCAAAGTCTTTCATCCAGAAAGCAACAG        /AAAE/EEEEEEEEAE/EAEE6EEEEEAAE/EEEEEEEEEE/EAEE/EEEEEEEEEEEEEEEAEE/EE/EEEAE/AEAAA        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80
7       0       chr29   7194098 37      80M     *       0       0       ATGCACAGTGCCCTAAGTGAGCTCAGCTGTTACATAACGATTATTTACAAAGTGTGGTGAGAGGGTGGGGTGAGAGAGCC        A66E/EEEAEEEEAEAEEEEAEEAEEE/EEEEEAEEEEA6EE/EEEEEEEEEAEEEEEE/EEEEEEEEEEEEEEEAAAAA        XT:A:U  NM:i:0  X0:i:1  X1:i:0  XM:i:0  XO:i:0  XG:i:0  MD:Z:80

Therefore, the issue you came across is still not quite clear to me.....may be related to your computing resources? Maybe you can try using 1 core, larger memory resource, clear the /tmp folder where sorting temporary files are stored, and simulate 10000 reads or smaller to start.
Otherwise, if this read id mismatches issue continues, maybe the following filtering may help. The rationale is to match read query name from both ends in the sam file. I added right after the alignment step in the simulation session. You can replace the original freeHiC.sh by the newly added freeHiC_v2.sh and see if it will help. Please let me know how it goes and I can merge it with the original pipeline! Thanks!
## match sam queryname from both ends %%%%
        $samtools view ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_1.sam | awk '{print $1}' >${outDir}/${simuName}/simuProcess/s1_alignment/queryName_1
        $samtools view ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_2.sam | awk '{print $1}' >${outDir}/${simuName}/simuProcess/s1_alignment/queryName_2
        cat ${outDir}/${simuName}/simuProcess/s1_alignment/queryName_1 ${outDir}/${simuName}/simuProcess/s1_alignment/queryName_2 | sort | uniq -c | awk '$1 == 2 {print $2}' >${outDir}/${simuName}/simuProcess/s1_alignment/queryNam\
e.common
        queryLen1=$(cat ${outDir}/${simuName}/simuProcess/s1_alignment/queryName_1 | wc -l)
        queryLen2=$(cat ${outDir}/${simuName}/simuProcess/s1_alignment/queryName_2 | wc -l)
        queryLenC=$(cat ${outDir}/${simuName}/simuProcess/s1_alignment/queryName.common | wc -l)

        if [[ "$queryLen1" != "$queryLenC" ]] || [[ "$queryLen2" != "$queryLenC" ]] || [[ "$queryLen1" != "$queryLen2" ]]; then
            echo "Matching simulation query names between end 1 and end2......"
            ${samtools} view -H ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_1.sam >${outDir}/${simuName}/simuProcess/s1_alignment/${name}_1.sam.header
            ${samtools} view -H ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_2.sam >${outDir}/${simuName}/simuProcess/s1_alignment/${name}_2.sam.header

            awk 'NR==FNR{a[$0]=1;next}a[$1]' ${outDir}/${simuName}/simuProcess/s1_alignment/queryName.common ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_1.sam  >>${outDir}/${simuName}/simuProcess/s1_alignment/${name}_1.\
sam.header
            awk 'NR==FNR{a[$0]=1;next}a[$1]' ${outDir}/${simuName}/simuProcess/s1_alignment/queryName.common ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_2.sam  >>${outDir}/${simuName}/simuProcess/s1_alignment/${name}_2.\
sam.header
            mv  ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_1.sam.header ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_1.sam
            mv  ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_2.sam.header ${outDir}/${simuName}/simuProcess/s1_alignment/${name}_2.sam

        fi
       rm -rf ${outDir}/${simuName}/simuProcess/s1_alignment/queryName*

Best,
Ye


Better handling for manually created source files and compiler APIs

Search Terms
type checker source file cannot ready property 'members' of undefined
Suggestion
When a user of the compiler API manually creates a source file with ts.createSourceFile instead of retrieving it from their program, and then asks the program for type inference on a contained node, this can crash. See https://github.com/Microsoft/TypeScript/issues/8136 / https://github.com/vuejs/vue-cli/issues/2712 / https://github.com/angular/tsickle/issues/151 / https://github.com/general-language-syntax/TS-GLS/issues/39.
It seems like one of three interpretations might be best:

This is explicitly unsupported behavior, but for the sake of performance & simplicity, no checks should happen
This is explicitly unsupported behavior, and a more explicit error should be thrown
This should become supported behavior, and the program should dynamically create source files as requested

Use Cases
Auto-generated TypeScript files, such as .vue snippets, still want access to a type checker, such as for TSLint rules.
Examples
https://github.com/palantir/tslint/issues/4273
Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript / JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. new expression-level syntax)

This is by design currently, but we should probably attempt to do something better than crash (ideally we'd capture the difference between the trees in the API somehow) - maybe we can set a bit on synthetic source files that only gets unset when it becomes correctly bound and checked, and we use that bit to throw early on in the checker if it's set? I'm unsure.

maybe we can set a bit on synthetic source files that only gets unset when it becomes correctly bound and checked, and we use that bit to throw early on in the checker if it's set?

That probably won't work for SourceFiles from different Programs or Nodes that were previously part of a SourceFile but no longer are because of incremental parsing through updateSourceFile.

We could set a random ID on all nodes in a program and assert if the node the checker receives has an ID not equal to the hosting program's, perhaps?

@weswigham I haven't come up with a better idea than that 😄. Would you accept a PR for the program-specific IDs?
Clarifying questions:

Would each round of parsing (first the initial creation, then incremental parsing) use a new ID created by, say Math.random() + "-" + ts.timestamp()?
Is the performance cost of updating all nodes something you're concerned about? Meaning: would you want a PR to include timing tests showing no real change, and if so, how do you TypeScript folks normally create those?

Would each round of parsing (first the initial creation, then incremental parsing) use a new ID

I'd say so, yes.

Is the performance cost of updating all nodes something you're concerned about?

Setting it at the same time we refresh and set .parent pointers in the parser/binder should be relatively cheap, since those need to be rewritten in the same way.

Edit: oh, and should this be an opt-in feature, so that the CLI doesn't enable it, but external tools programmatically calling TS APIs would?

I'd be temped to tie it to the diagnostics flag - but it's probably best to only tie it to a flag if it has measurable perf impact.

Meaning: would you want a PR to include timing tests showing no real change, and if so, how do you TypeScript folks normally create those?

We have some internal tools for informally measuring perf changes of some samples we have (and analyzing the resulting perf traces) - unfortunately we've never made them public. The closest thing we use in public is the aggregate timing data collected by the --extendedDiagnostics flag, but that's only for tsc. For measuring, eg, the impact on LS operations and incremental parse, we don't have anything anywhere (at least that I know of). We store execution time of tests when they're run in parallel so we can more optimally schedule them on future runs, so that tracks, eg, fourslash test duration. You could maybe shoehorn that into measuring a fourslash test that edits and rechecks a file a bunch of times... but yeah - we have no formal infra for measuring LS perf generally right now. When a perf issues is reported, we usually just get a repro and then open the build or LS in the chrome devtools and run a performance trace - that's not great for trying to benchmark a LS change, though.

While i am not too familar with typescript internals, the fork-ts-checker-webpack-plugin, somehow managed to allow typechecking for .vue file: https://github.com/Realytics/fork-ts-checker-webpack-plugin/blob/master/src/VueProgram.ts


Ensure dev mode works with GNAP

[ ] Figure out a way to run tests without requiring generating test keys in https://github.com/trustbloc/wallet/pull/1784
[ ] Check if auth returns bootstrap data in dev mode and fix any issues
[ ] Analyze if we need to hard code token expiry used in wallet unlocking for dev mode, or whether we can use the value from bootstrap data returned by auth

Firstly, the auth server url for dev mode seems to be broken. I am unable to reach https://localhost:8044, and replacing agentOpts['hub-auth-url'] = 'https://localhost:8044'; -> agentOpts['hub-auth-url'] = 'https://auth.trustbloc.local:8044'; in options.js gets me past the first error and to the login page.
Secondly, I tried removing this check for production before updating profileOpts with data from getBootstrapData, but after that call, profileOps.bootstrap.data.user is empty, although user is set with hardcoded values in initOpts. Maybe it is related to fact that Object.assign cannot deal with nested values and overrides the data.user and data.tokenExpiry that was set from before, I'm not 100% sure.


Infinity Cursor Value

Hi again😁
I planned to scrap all tweets for specified user and did try to user<PHONE_NUMBER>284815873 which just having 166 tweets. After all tweets has been fetched, the cursor still returning value and list with pinned tweet.
Sample code:
const delay = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const fetchAllTweets = async (id: string) => {
  let hasNextPage = true;
  let cursor: string | undefined = undefined;
  const filePath = `${id}tweets.json`;

  const user = await rettiwt.user.details(id);

  // Initialize the file by writing an empty array
  fs.writeFileSync(filePath, JSON.stringify([]));

  while (hasNextPage) {
    try {
      const response = await rettiwt.user.replies(id, 20, cursor);

      if (response && response.list) {
        const tweets = response;

        // Read current file content
        const currentData = JSON.parse(fs.readFileSync(filePath, 'utf8'));

        // Append new tweets to the current data
        currentData.push(tweets);

        // Write updated data to the file
        fs.writeFileSync(filePath, JSON.stringify(currentData, null, 2));

        cursor = response.next.value;
        hasNextPage = !!cursor;
        console.log(response.next, hasNextPage);

        await delay(3000);
      } else {
        console.error(`Invalid response structure for ${id}:`, response);
        hasNextPage = false;
      }
    } catch (err) {
      console.error(`Failed to fetch tweets for ${id}:`, err);
      hasNextPage = false;
    }
  }

};

Sample debug logs response after all tweets fetched and infinity returns pinned tweet:

RESPONSE JSON FILE

I'm trying to work out a fix ASAP.

if (response && response.list)

Here, can you try with the condition as response && response.list.length?
With only response.list, the net logical evaluates to true, even with response.list == []. However, with response.list.length, the net logical evaluates to false, when an empty list is returned, thereby stopping the fetch process.

I think this will still pass cause as you can see the last data is still returining list which contains pinned tweet. and I just realized that the pinned tweet returns on every request
    {
      "list": [
        {
          "id": "1794272810065592439",
          "createdAt": "Sat May 25 07:42:28 +0000 2024",
          "tweetBy": {
            "id": "1794271235284815873",
            "userName": "PEBIWP",
            "fullName": "PEBIWELIANPUTRA",
            "createdAt": "Sat May 25 07:36:25 +0000 2024",
            "isVerified": false,
            "likeCount": 188,
            "followersCount": 278,
            "followingsCount": 137,
            "statusesCount": 166,
            "pinnedTweet": "1794272810065592439",
            "profileBanner": "https://pbs.twimg.com/profile_banners/1794271235284815873/1716624438",
            "profileImage": "https://pbs.twimg.com/profile_images/1794271474011996160/1Bksw3nL_normal.jpg"
          },
          "entities": {
            "hashtags": [],
            "mentionedUsers": [
              "elonmusk"
            ],
            "urls": []
          },
          "media": [
            {
              "url": "https://pbs.twimg.com/media/GOaJmxDbYAAC7uP.jpg",
              "type": "photo"
            }
          ],
          "fullText": "akun debu gue salah apaaa😭? sekedar reply dan rt doang mas @elonmusk https://t.co/oIFLGNd3gZ",
          "lang": "in",
          "quoteCount": 0,
          "replyCount": 4,
          "retweetCount": 0,
          "likeCount": 1,
          "viewCount": 429,
          "bookmarkCount": 0
        }
      ],
      "next": {
        "value": "DAABCgABGOvjMmb__zUKAAIY5o8GzhqgkQgAAwAAAAIAAA"
      }
    }

Okay then the issue seems to be the pinned tweet. A workaround would be to check if the returned result has only the pinned tweet, which can be the exit condition.
A permanent fix will be to just remove the pinned tweet from the result on the library's side. Working on it now.

For now, the only solution seems to be checking the following as exit conditions:

The returned list is empty (in case the user does not have a pinned tweet)
The returned list only has the pinned tweet (in case the user has a pinned tweet).

As for whether the user has a pinned tweet or not, it can be ascertained from the user details.


[DropDownIcon] width is limited

Hello,
I need to use the DropDownIcon, and I am trying to take advantage of the ability to add text next to the icon itself. The reason I am doing this, is because I need something like the DropDownMenu, just with a custom label.
The problem is that the parent div is being limited to 48px width only:

Maybe the style could be moved to the inner span.mui-font-icon, and allow the outer div to be whatever width is needed.
Thank you.

DropDownIcon is deprecated and the new Icon Menus support this feature.


Add webhook resource

Close #7

Oooooh it's officially supported? Will you be back porting the slack integration

Oooooh it's officially supported? Will you be back porting the slack integration

@chb0github Xray team added the public API for their JIRA security integration. We have plans to backport this for Slack and MSTeams integrations.


Allow adding badge name depending on badge type

For artists/exhibitors/press, the ability to add a badge name with manager override (for at-con registration).
Note that for regular attendees, you should not be able to add a badge name when registering At-Con, but that would be desirable for specialty badges (industry, exhibitors, etc).
Need to figure out an elegant way to handle that.

It's very likely that we'll allow setting badge names for at-con registrations next year, which makes the security side of this moot - just need to add badge name to the at-con screen.

Closed in e4f47be55fc0f9e95a513fee50c901b2d332e50e


Nothing happens when i press login?

http://i.imgur.com/UGcXPiN.png
Basically this happens, is the wwwdir supposed to look like this? or am i retarded?
http://i.imgur.com/X0GTh6U.png

It's the webhoster's fault.


Check names for numeric values

Eg self::SOMETHING_SIX = 6; should be disallowed
Will need to work out languages but default to English

Hey,
Nice idea:) I wouldn't put it as strict check but more like warning/suggestion type of check:)


Finding the center location of an object

Reasons why we need

Will fix facing objects above a size of 1

There are probably more reason but these are the two reasons i have ran into.

cc @garyttierney thoughts?

Pretty sure this is needed when clicking on an object/item/npc, the server (in RS) rewrites the path if the entities size is > 1 or if some other factors are met (this is so you move directly in front of or in the center of whatever you clicked), go try it.

@ryleykimmel unsure what you mean. If the client rewrites the path to the origin of the entity (i.e., its actual position in the game world) then why would it be needed?
In the case that we do get messages for objects on all of the tiles that they occupy (and they are not just rewritten to the origin of the object) can we just add those objects to the positions that they occupy and return the original position for the center?
Also, why do we need the center?

@garyttierney this is an example of why we need
http://recordit.co/3mPTKhNfwc
player currently doesn't even face the tree vs how it is on runescape

The server rewrites the path, not the client. This is another issue than what Jesse is addressing.


Question about the input image size

The size of the input image is 3024*4032, the recognition effect is not good, and the recognized text is very dense, is the recognition effect of the model on large-size pictures bad?

@caozhongbjtu Hi, it shouldn't be that bad.
you can try to play with text_low, text_threshold, link_threshold to get best result depend on size of your input image.
also you can get binary image of your image after recog
# make score and link map
score_text = y[0, :, :, 0].cpu().data.numpy()
score_link = y[0, :, :, 1].cpu().data.numpy()

@caozhongbjtu Hi, it shouldn't be that bad.
you can try to play with text_low, text_threshold, link_threshold to get best result depend on size of your input image.
also you can get binary image of your image after recog
# make score and link map
score_text = y[0, :, :, 0].cpu().data.numpy()
score_link = y[0, :, :, 1].cpu().data.numpy()

I'll try it,thanks


[nfs-provisioner] Segfault after extended usage, Ganesha restarts but NFS shares remain stuck

ganesha.nfsd periodically segfaults on my machine, using nfs-provisioner v1.0.8/v1.0.9. Ganesha then restarts (thanks to #91), but it doesn't appear to restart serving old shares - all other containers using the disk get stuck.
Relevant logs from nfs-provisioner container:
E0511 23:11:52.470463       1 reflector.go:315] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:384: Failed to watch *v1.StorageClass: Get https://<IP_ADDRESS>:443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=1405043&timeoutSeconds=362&watch=true: dial tcp <IP_ADDRESS>:443: getsockopt: connection refused
E0511 23:11:52.433869       1 reflector.go:315] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:411: Failed to watch *v1.PersistentVolumeClaim: Get https://<IP_ADDRESS>:443/api/v1/persistentvolumeclaims?resourceVersion=1405043&timeoutSeconds=588&watch=true: dial tcp <IP_ADDRESS>:443: getsockopt: connection refused
E0511 23:11:52.433639       1 reflector.go:315] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:412: Failed to watch *v1.PersistentVolume: Get https://<IP_ADDRESS>:443/api/v1/persistentvolumes?resourceVersion=1405043&timeoutSeconds=501&watch=true: dial tcp <IP_ADDRESS>:443: getsockopt: connection refused
E0511 23:11:56.120782       1 reflector.go:205] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:412: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:default:nfs-provisioner" cannot list persistentvolumes at the cluster scope
E0511 23:11:56.120807       1 reflector.go:205] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:384: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:default:nfs-provisioner" cannot list storageclasses.storage.k8s.io at the cluster scope
E0511 23:11:56.120782       1 reflector.go:205] github.com/kubernetes-incubator/external-storage/lib/controller/controller.go:411: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:default:nfs-provisioner" cannot list persistentvolumeclaims at the cluster scope
E0512 07:03:09.370484       1 main.go:92] NFS server Exited Unexpectedly with err: ganesha.nfsd failed with error: signal: segmentation fault, output:
I0512 07:03:10.383161       1 server.go:129] Running NFS server!

Using vers=4.1 in mount options because it was the only way I could get writing to shares working without "invalid argument" (similar to #447), but that's another issue.
Kubernetes manifests:
kind: StatefulSet
apiVersion: apps/v1beta1
metadata:
  name: nfs-provisioner
spec:
  serviceName: "nfs-provisioner"
  replicas: 1
  template:
    metadata:
      labels:
        app: nfs-provisioner
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      serviceAccount: nfs-provisioner
      terminationGracePeriodSeconds: 0
      nodeSelector:
        kubernetes.io/hostname: rem
      containers:
        - name: nfs-provisioner
          image: quay.io/kubernetes_incubator/nfs-provisioner:v1.0.0
          ports:
            - name: nfs
              containerPort: 2049
            - name: mountd
              containerPort: 20048
            - name: rpcbind
              containerPort: 111
            - name: rpcbind-udp
              containerPort: 111
              protocol: UDP
          securityContext:
            capabilities:
              add:
                - DAC_READ_SEARCH
                - SYS_RESOURCE
          args:
            - "-provisioner=potatofrom.space/nfs"
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SERVICE_NAME
              value: nfs-provisioner
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: export-volume
              mountPath: /export
      volumes:
        - name: export-volume
          hostPath:
            path: /srv/kubes
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: 'true'
  name: nfs
parameters:
  mountOptions: vers=4.1
provisioner: potatofrom.space/nfs
reclaimPolicy: Retain
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-provisioner
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get"]
  - apiGroups: ["extensions"]
    resources: ["podsecuritypolicies"]
    resourceNames: ["nfs-provisioner"]
    verbs: ["use"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-provisioner

Might be helped if some bugs have been fixed in later Ganesha versions (a la #168, though it seems Ganesha is at 2.6.2 now).

Yeah there is not much we can do but maybe update ganesha and hope it gets fixed. Unstable as it is running in this environment, containerized kernel nfs is not without its own problems :]. Theoretically ganesha have no problem restarting old shares since they're all still in its config file and they all have unique fsid but I have no idea.
My recommendation for anyone who's used nfs provisioner & generally happy with its functionality after demoing it but wants stability, is create an nfs server and use the nfs-client provisioner. Or upgrade to a more production-ready, managed, etc. solution

Any updates on this?
If now, where can I find pointers to build a more up-to-date image of https://github.com/nfs-ganesha/nfs-ganesha using 2.7.1 for instance?
Thx

As far as I can see this is still happening.


[Bug] - Won't Create Repo in Organization

When I try and create a new static web app, I select my organization from github as where I want to create it, but if I go to create a new repository, that repo is created under my personal account.

but the repo it creates ends up being under my personal m-sterspace org:

If I select and existing repo in my organization it works, but it seems to always create the new one in my personal org.
Also, if there isn't already a branch in an existing repo, it rather unceremoniously exits, it would be nice if it defaulted to creating a master branch for you.

Hi @m-sterspace
Thanks for using the new extension!  I had a quick question regarding this part of your issue

Also, if there isn't already a branch in an existing repo, it rather unceremoniously exits, it would be nice if it defaulted to creating a master branch for you.

So I was looking into this, and I don't know if the GitHub API will let us create a branch for you in an empty repository since technically nothing has been initialized yet.
We are discussing about having the option of pushing your local workspace project when you create a new repo.  Maybe we can do something similar when we see that a repo is empty?
Thanks for your feedback! 😄

Even just a warning message or something would be helpful ... if I remember correctly, I believe that when I ran through it the first time it just didn't create anything but didn't explain why it hadn't worked or present me with an error? I think I just went on a hunch that it was probably because the repo wasn't initialized yet but that may not be as obvious to someone else! 😃

I think the current experience for me is it gets to the branch step, but has nothing listed (but at allows you to go back)

Which I agree is a non-obvious error since there's no way to know whether or not it just didn't load or some other problem occured.
Thanks very much for your response!  The first part of your issue is resolved and we should be releasing a patch update soon that will include that fix, as well as removing the requirement of using Insiders, so please look forward to that. 😁

Yep, it does the same thing for me, I should have clarified and said "the experience if you just repeatedly hit enter to blindly accept all the defaults", because then you hit enter on that line and nothing happens 😛

Closing as fixed-- other issue has been filed here: https://github.com/microsoft/vscode-azurestaticwebapps/issues/117