Fastlane cert fails with 'Unauthorized Access'

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description
Running fastlane dies in 'cert' with Unauthorized access. I know the username and password are good for Apple Dev Portal, and the user has admin rights.
Complete output when running fastlane, including the stack trace and command used
buildNodeDSW:App jenkins$ fastlane --verbose beta
[✔] 🚀
DEBUG [2018-01-15 12:08:20.10]: Checking if there are any plugins that should be loaded...
Successfully loaded Appfile at path '/path/to/Appfile'

app_identifier: 'com.mycompany.AppID'
apple_id<EMAIL_ADDRESS>

INFO [2018-01-15 12:08:20.41]: ----------------------------------------
INFO [2018-01-15 12:08:20.41]: --- Step: Verifying fastlane version ---
INFO [2018-01-15 12:08:20.41]: ----------------------------------------
INFO [2018-01-15 12:08:20.41]: Your fastlane version 2.75.1 matches the minimum requirement of 2.70.2  ✅
INFO [2018-01-15 12:08:20.47]: ------------------------------
INFO [2018-01-15 12:08:20.47]: --- Step: default_platform ---
INFO [2018-01-15 12:08:20.47]: ------------------------------
INFO [2018-01-15 12:08:20.48]: Driving the lane 'ios beta' 🚀
INFO [2018-01-15 12:08:20.53]: -----------------------------
INFO [2018-01-15 12:08:20.53]: --- Step: unlock_keychain ---
INFO [2018-01-15 12:08:20.53]: -----------------------------
INFO [2018-01-15 12:08:20.60]: Unlocking keychain at path:
INFO [2018-01-15 12:08:20.88]:
INFO [2018-01-15 12:08:20.94]: ------------------
INFO [2018-01-15 12:08:20.94]: --- Step: cert ---
INFO [2018-01-15 12:08:20.94]: ------------------
+-------------------+--------------------------------------+
|                 Summary for cert 2.75.1                  |
+-------------------+--------------------------------------+
| keychain_path     |   |
| keychain_password | ********                             |
| development       | false                                |
| force             | false                                |
| username          |<EMAIL_ADDRESS>       |
| platform          | ios                                  |
+-------------------+--------------------------------------+
INFO [2018-01-15 12:08:21.16]: Starting login with user<EMAIL_ADDRESS>WARN [2018-01-15 12:08:22.61]: Lane Context:
INFO [2018-01-15 12:08:22.61]: {:DEFAULT_PLATFORM=>:ios, :PLATFORM_NAME=>:ios, :LANE_NAME=>"ios beta"}
ERROR [2018-01-15 12:08:22.61]: Unauthorized Access
INFO [2018-01-15 12:08:22.62]: Successfully generated documentation at path '/path/to/fastlane/README.md'
+------+---------------------+-------------+
|             fastlane summary             |
+------+---------------------+-------------+
| Step | Action              | Time (in s) |
+------+---------------------+-------------+
| 1    | Verifying fastlane  | 0           |
|      | version             |             |
| 2    | default_platform    | 0           |
| 3    | unlock_keychain     | 0           |
| 💥   | cert                | 1           |
+------+---------------------+-------------+
ERROR [2018-01-15 12:08:22.64]: fastlane finished with errors
Looking for related GitHub issues on fastlane/fastlane...
Search query: The request could not be completed because:
Unauthorized Access
URL: https://api.github.com/search/issues?q=The request could not be completed because:
	Unauthorized Access+repo:fastlane/fastlane
➡  Fastlane init: Unauthorized Access
https://github.com/fastlane/fastlane/issues/9321 [closed] 3 💬
29 Aug 2017
➡  fastlane init error
https://github.com/fastlane/fastlane/issues/9241 [closed] 25 💬
29 Aug 2017
➡  2 Factor on Travis CI
https://github.com/fastlane/fastlane/issues/9244 [closed] 8 💬
01 Jun 2017
and 5 more at: https://github.com/fastlane/fastlane/search?q=The request could not be completed because:
	Unauthorized Access&type=Issues&utf8=✓
🔗  You can ⌘ + double-click on links to open them directly in your browser.
/home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:689:in block in send_request': [!] The request could not be completed because: (Spaceship::UnauthorizedAccessError) 	Unauthorized Access 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:519:in with_retry'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:683:in send_request' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:571:in request'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:473:in fetch_olympus_session' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:450:in send_shared_login_request'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/portal/portal_client.rb:27:in send_login_request' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:650:in do_login'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:355:in login' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/client.rb:71:in login'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/portal/spaceship.rb:25:in login' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/spaceship/lib/spaceship/portal/spaceship.rb:95:in login'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/cert/lib/cert/runner.rb:24:in login' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/cert/lib/cert/runner.rb:34:in run'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/cert/lib/cert/runner.rb:13:in launch' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/actions/get_certificates.rb:17:in run'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:253:in block (2 levels) in execute_action' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/actions/actions_helper.rb:50:in execute_action'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:231:in block in execute_action' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:227:in chdir'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:227:in execute_action' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:148:in trigger_action_by_name'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/fast_file.rb:148:in method_missing' 	from Fastfile:58:in block (2 levels) in parsing_binding'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/lane.rb:33:in call' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/lane.rb:33:in call'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:49:in block in execute' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:45:in chdir'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/runner.rb:45:in execute' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/lane_manager.rb:56:in cruise_lane'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/command_line_handler.rb:34:in handle' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/commands_generator.rb:107:in block (2 levels) in run'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/commander-fastlane-4.4.5/lib/commander/command.rb:178:in call' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/commander-fastlane-4.4.5/lib/commander/command.rb:178:in call'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/commander-fastlane-4.4.5/lib/commander/command.rb:153:in run' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/commander-fastlane-4.4.5/lib/commander/runner.rb:476:in run_active_command'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane_core/lib/fastlane_core/ui/fastlane_runner.rb:75:in run!' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/commander-fastlane-4.4.5/lib/commander/delegates.rb:15:in run!'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/commands_generator.rb:336:in run' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/commands_generator.rb:42:in start'
from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/fastlane/lib/fastlane/cli_tools_distributor.rb:94:in take_off' 	from /home_dir/.fastlane/bin/bundle/lib/ruby/gems/2.2.0/gems/fastlane-2.75.1/bin/fastlane:20:in <top (required)>'
from /home_dir/.fastlane/bin/bundle/bin/fastlane:22:in load' 	from /home_dir/.fastlane/bin/bundle/bin/fastlane:22:in '
Environment

buildNodeDSW:App jenkins$ fastlane env
[✔] 🚀
[12:20:28]: Generating fastlane environment output, this might take a few seconds...
✅ fastlane environment ✅
Stack

Key
Value

OS
10.13.2

Ruby
2.2.4

Bundler?
false

Git
git version 2.14.3 (Apple Git-98)

Installation Source
~/.fastlane/bin/bundle/bin/fastlane

Host
Mac OS X 10.13.2 (17C88)

Ruby Lib Dir
~/.fastlane/bin/bundle/lib

OpenSSL Version
OpenSSL 1.0.2g  1 Mar 2016

Is contained
true

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcodes/Xcode-9.2.app/Contents/Developer/

Xcode Version
9.2

System Locale

Variable
Value

LANG
en_US.UTF-8
✅

LC_ALL
en_US.UTF-8
✅

LANGUAGE
en_US.UTF-8
✅

fastlane files:
`./fastlane/Fastfile`
# Customize this file, documentation can be found here:
# https://docs.fastlane.tools/actions/
# All available actions: https://docs.fastlane.tools/actions
# can also be listed using the `fastlane actions` command

# Change the syntax highlighting to Ruby
# All lines starting with a # are ignored when running `fastlane`

# If you want to automatically update fastlane if a new version is available:
# update_fastlane

# This is the minimum version number required.
# Update this, if you use features of a newer version
min_fastlane_version("2.70.2")

default_platform(:ios)

platform :ios do
	before_all do
		unlock_keychain(
			path: "<PATH>",
			password: "hahahahahano",
			set_default: true,
		)
		puts ENV['BUILD_NUMBER']
	end

	desc "Create a CI build"
	lane :ciBuild do
        cert(
            keychain_path: "<PATH>",
            keychain_password: "hahahahahano",
            development: true
        )
        sigh(
            development: true,
            app_identifier: "com.mycompany.AppID",
            readonly: true,
            provisioning_name: "AppDev",
        )
        build_app(
			configuration: "Debug",
			include_symbols: true,
			build_path: "output/",
			export_method: "development",
		)
	end

	desc "Runs all the tests"
	lane :test do
		run_tests
	end

	desc "Submit a new Beta Build to Apple TestFlight"
	desc "This will also make sure the profile is up to date"
	lane :beta do
		# sync_code_signing(type: "appstore") # more information: https://codesigning.guide
        cert(
            keychain_path: "<PATH>",
            keychain_password: "hahahahahano",
            development: false,
        )
        sigh(
            development: false,
            app_identifier: "com.mycompany.AppID",
            provisioning_name: "AppTestflight",
        )
		increment_build_number(
			build_number: "8"
		)
        build_app(
			configuration: "Debug",
			include_symbols: true,
			build_path: "output/",
			export_method: "app-store",
		)
		upload_to_testflight(
			groups: "alpha",
		)

		# sh "your_script.sh"
		# You can also use other beta testing services here (run `fastlane actions`)
	end

	desc "Deploy a new version to the App Store"
	lane :release do
		# sync_code_signing(type: "appstore")
		capture_screenshots
		build_app # more options available
		upload_to_app_store(force: true)
		# frame_screenshots
	end

	# You can define as many lanes as you want

	after_all do |lane|
		# This block is called, only if the executed lane was successful

		# slack(
		#   message: "Successfully deployed new App Update."
		# )
	end

	error do |lane, exception|
		# slack(
		#   message: exception.message,
		#   success: false
		# )
	end
end

# More information about multiple platforms in fastlane: https://docs.fastlane.tools/advanced/#control-configuration-by-lane-and-by-platform
# All available actions: https://docs.fastlane.tools/actions

# fastlane reports which actions are used. No personal data is recorded.
# Learn more at https://docs.fastlane.tools/#metrics

`./fastlane/Appfile`
app_identifier "com.mycompany.AppID"
apple_id<EMAIL_ADDRESS>

fastlane gems

Gem
Version
Update-Status

fastlane
2.75.1
✅ Up-To-Date

Loaded fastlane plugins:
No plugins Loaded
Loaded gems

Gem
Version

CFPropertyList
2.3.5

addressable
2.5.1

babosa
1.0.2

bigdecimal
1.2.6

bundler
1.14.6

claide
1.0.2

colored
1.2

colored2
3.1.2

commander-fastlane
4.4.5

declarative
0.0.10

declarative-option
0.1.0

domain_name
0.5.20170404

dotenv
2.2.0

excon
0.55.0

faraday
0.12.1

faraday-cookie_jar
0.0.6

faraday_middleware
<IP_ADDRESS>

fastimage
2.1.0

gh_inspector
1.0.3

google-api-client
0.13.6

googleauth
0.5.1

highline
1.7.8

http-cookie
1.0.3

httpclient
2.8.3

hurley
0.2

io-console
0.4.3

json
1.8.1

jwt
1.5.6

libxml-ruby
3.0.0

little-plugger
1.1.4

logging
2.2.2

memoist
0.15.0

mime-types
3.1

mime-types-data
3.2016.0521

mini_magick
4.5.1

minitest
5.4.3

multi_json
1.12.1

multi_xml
0.6.0

multipart-post
2.0.0

nanaimo
0.2.3

nokogiri
1.5.6

os
0.9.6

plist
3.2.0

power_assert
0.2.2

psych
2.0.8

public_suffix
2.0.5

rake
10.4.2

rdoc
4.2.0

representable
3.0.4

retriable
2.1.0

rouge
1.11.1

rubygems-update
2.5.2

rubyzip
1.2.1

security
0.1.3

signet
0.7.3

slack-notifier
1.5.1

sqlite3
1.3.13

terminal-notifier
1.7.1

terminal-table
1.7.3

test-unit
3.0.8

tty-cursor
0.5.0

tty-screen
0.6.3

tty-spinner
0.7.0

uber
0.0.15

unf
0.1.4

unf_ext
<IP_ADDRESS>

unicode-display_width
1.1.3

word_wrap
1.0.0

xcodeproj
1.5.3

xcpretty
0.2.6

xcpretty-travis-formatter
0.0.4

generated on: 2018-01-15

[12:20:28]: Take notice that this output may contain sensitive information, or simply information that you don't want to make public.
[12:20:28]: 🙄  Wow, that's a lot of markdown text... should fastlane put it into your clipboard, so you can easily paste it on GitHub? (y/n)
n
[12:20:47]: Open https://github.com/fastlane/fastlane/issues/new to submit a new issue ✅

I am experiencing this issue today. I was able to build an app yesterday with the same login.

I have experience this issue today as well. Sometimes it seams like the fastlane can't find a build for hours and then when I log into iTC and navigate to it, it becomes available and fastlane continues. If I don't do the iTC step then after a longer period of time the Fastlane fails with the Unauthorized Access error.

Just ran into this issue.  Logging in on ITC on the affected account I see a 401 coming back on one of their requests.  Apple must've introduced some sort of bug in the recent update.

Also got this, seems like it started after WWDC.

I'm also experiencing this issue.

After some retries it works

I also see that in my builds. Before lunch (I had only one nonalcoholic beer) it was working fine for me. No I get this error.

I had a similar problem. When logging into iTunes Connect in a web browser, I had to accept Apple's new Terms & Conditions. This was likely what was causing an error for fastlane. It worked after I accepted the Terms & Conditions.

I'm also experiencing this issue since today. After some tries I could build the app this morning but now its back again.

I'm having the same problem when match tries to verify my profile on the dev portal. It seems to have just started a little while ago for me. I already accepted the new terms and conditions before this started. It seems to be intermittent. It fails most of the time but sometimes it randomly works.

I can not even log in IC.

https://developer.apple.com/system-status/ looks fine.

Same error. First time it has failed during checking if build has been processed and now it fails during Login to iTunes Connect. But I cannot see any pattern. It definitely ends up with fail but in different stages.

I still experience the issue with four different teams. I have accepted the terms.

I just saw this comment on a similar thread - https://github.com/fastlane/fastlane/issues/12675#issuecomment-395927914

RESOLVED. All members associated with the team must accept that privacy statement, regardless of which user's account is used to get the certificates.

Not sure if related but might be a thing to try / look into? 🤔 This would be not great if this was a solution


Frostwire Laptop Shutting Down

Frostwire on my laptop keeps shutting down after a few seconds saying launcher error. I have tried updating and removing and reinstalling and still the same result. Just started 2 days ago.

what operating system are you running on your laptop?
what version of it?


Update the dgrijalva/jwt-go library

PR #772 introduced a slightly outdated version of dgrijalva/jwt-go which might require some critical patches (for example this one). Unfortunately, the original author doesn't have time to maintain it so there is a conversation happening here to figure out a permanent home for this popular library. I have subscribed to this conversation and will submit a PR with the updated lib once the dust settles.

Just saying, hi! We are setting up a community effort clone of the aforementioned library at http://github.com/golang-jwt/jwt . We are still in the process of setting everything up, but we made great progress. https://github.com/golang-jwt/jwt/pull/12 will fix the mentioned issue.

Thank you very much @oxisto! I'll keep an eye on it.

We just released v3.2.1 with a fix. https://github.com/golang-jwt/jwt/releases/tag/v3.2.1


Update RunSAST.py

The content-length header is no longer returned by ASoC, so percentage based status messages has to be removed when downloading the SAClient.zip

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Tested on Bitbucket and scan completes successfully.


VS Code fails to open big files (60MB)

VSCode Version: 1.1.1
OS Version: Windows 10

Steps to Reproduce:

Try opening a, let's say, 60MB SQL file
VS Code says the file is either too large or in wrong format.

We have made some improvements to the way we read files / create buffers. To investigate if we can raise the max file size limit.

nope, no luck ... dies somehwere between 50 and 55Mb file size (still testing with UTF-8 sql dumps). I think you should reconsider the base of the file handling, so it does not depend on the file size at all (same goes for the lines length, btw. which are causing me troubles too).

Any updates on this one @alexandrudima

Same here. VSC version 1.4.0 fails to open my 53MB csv data file.
I have to use Sublime while this is not solved.
It would be nice to unlock this limitation.
Besides this issue, I like VSC more and more. Congrats !

@oscar6echo Yup, I must admit I also like VSC more, but sublime handles large files and long lines better, plus killer fast startup. So, If VSC needs urgent improvements, these are :

large files handling - can't open at all
long lines handling - can't handle very long lines at all, even with all wordwrapping and limitations off and syntax highlighters break with these too (probably due to not reading the whole line).
serach is dead slow when average long lines (but withing VSC handling limits) present
somewhat improve startup time

It's frozen when I load a 500K file. I'm using the latest VS code.

@zh-wowtv Please create a separate issue and also attach the file that causes the freeze or a file similar in shape or nature that reproduces the freeze you're experiencing. I'm very interested in such cases, but each one is unique.

I don't always open huge files.
But, when I do. I use neovim inside vscode terminal. Simple and fast.

I hit this with a 130MB log file. Vim and Notepad++ open this without any problem, and searches it just as fast as a shorter file.

more than 100m,vs code throws The file will not be displayed in the editor because it is either binary, very large or uses an unsupported text encoding. and notepad++/sublime open it is easy! @alexandrudima

this i18n message in the https://github.com/Microsoft/vscode/blob/c67ef57cda90b5f28499646f7cc94e8dcc5b0586/src/vs/base/browser/ui/resourceviewer/resourceViewer.ts#L156

look this https://github.com/Microsoft/vscode/blob/39d761bf1f4ce7ea605539bdba41a8727b3c7ce0/src/vs/platform/files/common/files.ts#L484
export const MAX_FILE_SIZE = 50 * 1024 * 1024;
so vs code only open <= 50M files

What are the downsides of simply increasing that constant?

@pfmoore  may be  the big txt file make the system slow . so declare the MAX_FILE_SIZE

OK, so it does need better coding internal to VS Code to support this :-(

The file size limit is somewhat arbitrary. Its sole purpose is to avoid OOM exceptions which lead to a crash. i.e. in our runtime (javascript) we cannot simply catch an OOM exception and stop loading a file, an OOM exception will crash the renderer process.
There are three possible "fixes":

continue to improve and fine-tune in how we represent a file. This is already today highly fine-tuned, but we do represent each line as an object, and we do need a couple of pointers in each one of those objects (i.e. a pointer to the text on the line, a pointer to the tokenization state, a pointer to the tokens, a pointer to markers living on the line). We can of course work towards improving here. Please also keep in mind that we are in a VM. So when we are "logically" consuming 200MB, Chromium/v8 ends up oftentimes consuming much more real RAM.

somehow compute up front how much memory opening a file would take. i.e. opening a 150MB file in a fresh instance of VSCode win 32 bit will probably succeed, but opening 20 50MB files will probably run out of memory. We would need to find out how much memory the process still has until it crashes and estimate how much memory opening the file would take. Then, we would refuse to open files until some other opened files are closed and some memory is freed.

ship a 64 bit build on Windows. Under a 64 bit build, the renderer process will not crash at ~ 1.8GB memory usage (~1GB "logical" memory usage)

Thanks for the explanation.

continue to improve and fine-tune in how we represent a file

I imagine you've already done a pretty good job with this, so improvements here are probably minimal.

somehow compute up front how much memory opening a file would take

A better heuristic here would be good. I appreciate that opening 20 50MB files is going to hit memory issues, but that shouldn't mean that I can't open one. Would it not be possible to track usage at some level, so that if I open a 150MB file, it's "charged" to me the same as if I opened 3 50MB files? Then the limit could be expressed in terms of how much memory I'm actually using, rather than worst-case predictions. With the current arrangement, it sounds like I could still provoke a crash by opening many medium-sized files.

ship a 64 bit build on Windows

That would be useful. While it's all very well saying that opening a 150MB file shouldn't hit memory limits, the reality is that it does, and going 64-bit seems like a way round that. (Again, I presume it's not quite as simple as that, though, or you'd have done it already!)
But I'm sure you've thought through this sort of thing in great detail, If you're faced with the potential for the runtime to just crash on hitting memory limits, that's something you have to take into account (losing data in an editor is bad :disappointed:). At the end of the day, this is a code editor and 50MB files are almost certainly not source code. But while I don't know about other people, I tend to use my editor for looking at all sorts of text files, so it gets stressed way out of its actual designed use case.
Hmm, one probably impractical thought - could there be a readonly mode for opening files that doesn't keep the file content in memory, as it's not going to be changed? That could be used by default for files over a given size. Of course having to read data from disk all the time might make things dreadfully slow, and the alternative data format would likely have massive effects on the rest of the code, which would have to be updated to work with the new format. I doubt this is a realistic option.
Anyway - sorry for rambling, and thanks for taking the time to explain the issues involved.

@alexandrudima Thanks for the explanation.

ship a 64 bit build on Windows. Under a 64 bit build, the renderer process will not crash at ~ 1.8GB memory usage (~1GB "logical" memory usage)

but wiki/How-to-Contribute#packaging seems only support win32 not win64?

@anjia0532 Yes, please upvote #507 -- there is no technical limitation AFAIK

64bit version is a good idea, but I'm still wodnering how SublimeText handles big files so easily.

When we're talking about huge files, there's no need to process it whole. Typically only the part visible to user (+- page) is processed.
I'd prefer that syntax highlighting or other parts depending on tokenization be disabled, but file will still be displayed and operable

@alexandrudima Creating one object per line is fine, as long as you don't try to do it for the whole file at once. Perhaps doing 1000 lines at a time, and then read the next 1000 and releasing the last 1000 once the user jumps/scrolls there? (With some overlap.)

Any to do list for this? Is there anyone consider make js work on top of OS rather than on chromium hehehe...

I'm on Windows 10. When I open a 27MB file it opens, but as soon as I try editing it, the window crashes.

VSCode associates itself with *.sql files right after been installed.
Not able to open most of them because of file size.
....

I'd give you a laugh emotji, If it wasn't so sad ...

Applause for VS code :clap: :clap: :v:

@faustinoaq Then what the Vs code for :v: :laughing:

I don't know anything about the inner workings of vscode, but the solution mentioned by @IvanFateev and @lionello seems most logical for me:
If the file reaches a certain size, go into a fallback mode, where not all the file is held in memory at the same time. Of course, this might be a pretty complicated change, if the whole codebase currently assumes that it has access to the whole file at all times.

Won't open a 51 939 KB file :(

The project is written in TS, so is there any way to open file in stream mode in TS? If there exits a function or a class that can do this, I think it can help to solve this problem.

Improvements have been made in #30180 , and our memory usage is now roughly:

file size on disk + 40-60 bytes (2 objects) per line + an array of pointers of length line count

We will look into removing or raising the limit on 64 bit platforms

The size has been raised for 64bit vscode on windows. You can try out the 64bit insiders version or wait until we release stable in the following week.
I tried opening a 77 mb csv file and everything worked fine -> verified.

I can confirm. Tried the 64bit insiders with a 177MB SQL file - opened and opened reasonably fast.
Scrolling not perfect - initially the scroll was not reflecting the real length of file, but overall - works just fine. Also no syntax highlighting, but that has to be expected, I think.

how to download 64bit? build by myself?

No just download from the official download page select the 64bit version.
Done.
Benyamin Limanto
sent from my ASUS FonePad 8
Pada tanggal 3 Agt 2017 15.22, "anjia0532"<EMAIL_ADDRESS>menulis:

how to download 64bit? build by myself?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode/issues/6474#issuecomment-319902241,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NEOdARpktdohRW3_2zZN_-foUay8ks5sUYNBgaJpZM4Ig_ra
.

@benyaminl thanks for your reply and  I tried opening a 283 mb json file . in this demo vscode use 6 process and more than 500M memory.

the nopad++ not open this file.

sublime  text 3 build 3126  use 1 process and more than 700M memory.

Errr. The 64bit now ha the stable version. I think you should check it.
Hehehe...
Benyamin Limanto
sent from my ASUS FonePad 8
Pada tanggal 3 Agt 2017 23.19, "anjia0532"<EMAIL_ADDRESS>menulis:
@benyaminl https://github.com/benyaminl thanks for your reply and I tried
opening a 283 mb json file . in this demo vscode use 6 process and more
than 500M memory.
[image: vscode]
https://user-images.githubusercontent.com/15098916/28931715-888fb086-78a9-11e7-88fb-0594d1978884.png
the nopad++ not open this file.
[image: nodepad]
https://user-images.githubusercontent.com/15098916/28931850-14adbfd6-78aa-11e7-95b7-dcb710b15580.png
sublime text 3 build 3126 use 1 process and more than 700M memory.
[image: sublime]
https://user-images.githubusercontent.com/15098916/28931971-810d8044-78aa-11e7-8b7f-9e3446fc8dfa.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode/issues/6474#issuecomment-320018560,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACr9NKvi9QmuZmk3EIpR0lq_Lw0Yi1Opks5sUfMngaJpZM4Ig_ra
.

if i use the stable vscode 64bit it's failed to read this json file.

but insider vscode 64bit can be read

Okay, it seems they still working on it.. Then Ok. Just wait for the new version released, Great!
Benyamin Limanto
sent from Windows Live Mail on Windows 7 (Asus A43SV)
From: anjia0532
Sent: Friday, August 04, 2017 8:10 AM
To: Microsoft/vscode
Cc: Benyamin Limanto ; Mention
Subject: Re: [Microsoft/vscode] VS Code fails to open big files (60MB) (#6474)
if i use the stable vscode 64bit it's failed to read this json file.
but insider vscode 64bit can be read
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.


editor chooser not working

looking for BIXLER-CORE-7009-01-N7K2-port-channel5-L3 Mustian-9300-01

will not show up any way I try it. Search for BIXLER-CORE-7009-01-N7K2-port-channel5 returns results shown in screenshot but no way to pick the graph I want

yeah the datasource selector does not work at all, even selecting a datasource that the selector does find enters a invalid rrd path like /8679/1234.rrd instead of /path/to/cacti/rra/8679/1234.rrd or it will just enter the graph title instead of an path to rrd at all.

@jdcoats, it's supposed to use the 'relative' path (to save space) and then find the realpath during collection.  I'm wondering how the leading slash ended up in the front of the path?  You should be able to use the full path, but the relative path, if it's Cacti should also be okay.

okay update, maybe I should have tested more thoroughly. If you copy+paste the datasource name precisely with no leading or trailing space it will find the datasource. If there is a leading or trailing space in copy+paste it will not find your search. additionally if you search for something like port-channel5 or po5 It would be a nice feature to find all datasources containing that. I concede last night when testing that after upgrading I let it re-install all packages and it changed my suggested names order and I didn't catch that the graph name and datasource name didn't always match :)  I fixed my own problem there and the datasource selector isn't completely broken and could simply use a little TLC, the search list when typing could be longer maybe 100 because port-channel5 above in my screenshot had to many results to display what i wanted to choose?

okay update, maybe I should have tested more thoroughly. If you copy+paste the datasource name precisely with no leading or trailing space it will find the datasource. If there is a leading or trailing space in copy+paste it will not find your search

Got it.

additionally if you search for something like port-channel5 or po5 It would be a nice feature to find all datasources containing that

I'll validate this.

I fixed my own problem there and the datasource selector isn't completely broken and could simply use a little TLC

Cool.

the search list when typing could be longer maybe 100 because port-channel5 above in my screenshot had to many results to display what i wanted to choose?

This is configurable: Console > Setting > Visual > Autocomplete Rows

The relative path does indeed work fine. To much wine on initial testing.

I'll drink to that.

wow, i like learning new stuff, Is autocomplete rows setting new?  :) Thanks!

So, does you discovering that "tweak" for Autocomplete address my second comment above?

@netniV want's infinite scroll, and that's possible with jQueryUI, but we are going to defer that to 1.2.x.  I'll take a brief look for this plugin as I think it's doing it's own thing and not using the Cacti API (at least I don't think so).

Confirming that I hacked (just for WeatherMap) what is currently a part of Cacti mainline (develop) for autocomplete.

yes the datasource selector and graph selector work fine when expanding autocomplete! :)

Cool.


Update Api.php

Use proper payment fields in the request according to the documentation https://docs.adyen.com/payment-methods/apple-pay/custom-integration-apple-pay-web/
Description
When place an order with ApplePay (Sandbox account) it shows and error
"Payment Not Complete"
and in logs adyen_exception.log i can see and error
Adyen_Payment_Exception: HTTP Status code 422 received, data {"status":422,"errorCode":"14_007","message":"Invalid payment method data","errorType":"validation"} in app/code/community/Adyen/Payment/Exception.php:43

Tested scenarios

use iPhone with touchID
log in to iCloud with sandbox account
got to web-shop
add product to card
got to checkout
fill in all required fields up to payment method step
choose ApplePay ad payment method and go to next step
tap on ApplePay button on review
in appeared popup select test card listed on https://developer.apple.com/apple-pay/sandbox-testing/
use passcode(or touchID)

Expected result: place and order
Fixed issue:  #1024

Hi @lykhouzov,
Thanks for the PR. We already have this fixed in release 2.15.0 and higher.
Regards,
Rik
Adyen


Disable director lookup when its not available (ie not on the jumpbox)

When working locally, or on a jumpbox different from the eventual target jumpbox, the bosh director could be missing.  This patch prevents having to wait the long timeout for 5 failures to reach the bosh director to determine its UUID.

Updated as per the review comments

Updated to remove remnants of previous changes.


Remove hardcoded noise scale from CDFtoGaussianTransform

Description
The CDFtoGaussianTransform used in gpvar and deepvar currently uses noise with a hard-coded scale of 0.2 as a regularizer for computation of the interpolated, empirical CDF.
This makes the CDFtoGaussianTransform unusable for any data which varies on a numerical scale << 0.2.
This is highly unexpected for the user and should either be removed completely or documented very clearly in the documentation of both gpvar and deepvar.

Hi @jaschau,
thank you for raising this issue. I agree that this could be configurable. Do you want to submit a pull request for that?
I would like to clarify the current code. The 0.2 scale gets multiplied with the standard deviation of the data when computing the noise scale (as implemented here). Thus, the 0.2 scale should be usable regardless the scale of the data. In other words, the spread of the noise will be proportional to the spread in the data. Thus, if you are experiencing issues with these models, I suspect it has a different root cause.

Hi @mbohlkeschneider,
thanks a lot for the quick response and the clarification of the code.
I indeed overlooked the multiplication by the std. Just for my understanding, is the std computed over the conditioning length or the context length?
As you correctly suspected, I was indeed puzzled by gpvar not performing well on my dataset when use_marginal_transformation=True and scaling=False. In fact, the behaviour was very similar to deepar with scaling=False which in my experience collapses to an essentially static noise scale with an inappropriate scale. Since my data was varying on a scale of ~ 0.1, I suspected the mentioned code to be the culprit. Sorry for the noise!
Since I suspect that the precise value does not really matter when the noise scale is essentially given by the std, I will close the issue.

Hi @jaschau,

I indeed overlooked the multiplication by the std. Just for my understanding, is the std computed over the conditioning length or the context length?

It is computed over the conditioning_length (or history_length, whatever is smaller).
If you need more help on getting the model to work, feel free to open another issue.


Jan Dvorak: Blacklist gain xt

Jan Dvorak requests the blacklist of the keyword gain xt. See the Metasmoke search here

Already caught by pattern-matching product name rule as long as it appears twice (which it will if used in the standard pharma spam)


if the order is in ASC or DESC the insert will be wrong.

https://github.com/howtographql/react-apollo/blob/7dc3f55f2f79313cc61dff7a91450b5bc18e131f/src/components/CreateLink.js#L49-L62
apollo will throw an error if you've sorted the results in a different order and the cache hasn't been hit with the set order.
how would this be solved?

Thanks @lifeiscontent for opening this issue 😄. I think what you're trying to do is an extra feature that's not contemplated to not over complicate the tutorial. Closing this for now!

@swcarlosrj sure, but at the same time I'm legitimately asking how is this done? Because currently there are no docs that cover this kind of case. I don't think this should be closed. 👎

@lifeiscontent thanks for getting back on this. I think this is not an issue, in fact it's a feature that's why I've closed it. Of course you could always ask how is being done but unfortunately I cannot answer to such core functionality. I'd def suggest you to contact with someone from the Apollo team, they'll probably help you on this!

@swcarlosrj not to be salty about it, but literally every issue I've ever opened on any apollo project goes unanswered. 👎

@lifeiscontent you're totally right, sorry about that! I guess that we could improve the existing tutorials but they'll eventually be outdated, idk if you've already seen this issue but in order to invest time on this, we should have a robust platform where we could freely & do an efficient work. Hopefully it'll happen soon 🤞, thanks for the feedback!


Update request.py

from django.http.multipartparser import MultiPartParser, MultiPartParserError
...
from .multipartparser import parse_header
I don't see the reason to have those separately

Thanks, however the project uses isort. You'll see that this patch causes the isort test to fail.

yeah , I've just noticed, sorry to bother you then
I haven't heard about isort before, now I wonder if it is the reason for the absolute vs relative imports, which I've noticed accross django
nvm tho, thanks and sorry


Update introduction.md

This fixes a few typos I found in the introduction.

+cc @mea

Changes should also be applied to the following files:

concept:characters-and-strings:about.md
exercise:characters-and-strings:introduction.md

Otherwise, the changes look good!

Alright, I've made the changes to the files you pointed out

+cc @exercism/swift


ValueError: 'rope_scaling' with meta-llama/Llama-3.2-1B

Hi, I would like to try meta-llama/Llama-3.2-1B in different scenario of IPEX-LLM solutions.
pip list:
bigdl-core-xe-21            2.6.0b20241001
intel-extension-for-pytorch 2.1.10+xpu
intel-openmp                2024.2.1
ipex-llm                    2.2.0b20241001
torch                       2.1.0a0+cxx11.abi
torchvision                 0.16.0a0+cxx11.abi
I encountered several errors:

All-in-one benchmark (INT4 and FP16):
raise ValueError(
ValueError: rope_scaling must be a dictionary with with two fields, type and factor, got {'factor': 32.0, 'high_freq_factor': 4.0, 'low_freq_factor': 1.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

example/GPU/HuggingFace/LLM/llama3.2
Following instructions

pip install transformers==4.45.0
pip install accelerate==0.33.0
pip install trl
python ./generate.py --repo-id-or-model-path meta-llama/Llama-3.2-1B works
What is AI?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
The following is a list of some of the things that we are working on. We are constantly working on new things.
But going back to all-in-one benchmark
ImportError: cannot import name 'BenchmarkWrapper' from 'ipex_llm.utils'
Please let us know if we can perform all-in-one benchmark. Thank you.

Hi Kpeacef,
We have looked into the issue and reproduced the problem. Here are solutions to this issue:

To the error ValueError: 'rope_scaling', refer to this issue and upgrade transformers to version 4.43.
To the error ImportError: cannot import name 'BenchmarkWrapper' from 'ipex_llm.utils',  please refer to  the readme of  /GPU/HuggingFace/LLM/llama3.2, and change your transformers version to 4.43, version 4.45 hasn't been fully supported by ipex-llm.


Missing boot_id

Build: 16296
To repro, install flatpak then:
federico@Bestia:~$ sudo flatpak update
error: Failed to open file '/proc/sys/kernel/random/boot_id': File o directory non esistente

federico@Bestia:/proc/sys/kernel/random$ ls -al
totale 0
dr-xr-xr-x 2 root root 0 set 23 12:35 .
dr-xr-xr-x 4 root root 0 set 23 12:35 ..
-r--r--r-- 1 root root 0 set 23 14:04 uuid

From the web (I'm not a great Linux user):

/proc/sys/kernel/random/boot_id: A random ID that is regenerated on each boot. As such it can be used to identify the local machine's current boot. It's universally available on any recent Linux kernel. It's a good and safe choice if you need to identify a specific boot on a specific booted kernel.

@benhillis is there any workaround available for this issue on fall creators build(1709).


After fixed the responsive footer

Fixed issue #302 :
Footer Not Responsive on Small Screens. The following changes were made to improve the responsiveness of the footer, ensuring a better user experience on mobile devices:

[x] Flexbox Adjustments: Updated the .row class to use flex-wrap: wrap and stack columns vertically on smaller screens.

[x] Responsive Column Widths: Modified the .footer-col class to use flexible units and set a minimum width to maintain readability.

[x] Margin Adjustments: Adjusted margins for better spacing and layout on mobile devices.

[x] Social Icons: Improved the styling of the .social-icons class to ensure icons are properly sized and spaced on small screens.

[x] Media Queries: Added media queries for widths up to 768px and 480px to handle layout adjustments for smaller screens effectively.

[x] Removed Unnecessary Inline CSS: Removed inline CSS styles and replaced them with common class names to maintain consistency and improve maintainability.

These changes make the footer more user-friendly and visually appealing on a variety of screen sizes, especially mobile devices.
Please review the changes and let me know if there are any further adjustments needed.
Screenshots

@codewithmurali this issue has been resolved


TCP_KEEPALIVE in uv__tcp_keepalive() under Windows Server 2003

setsockopt(socket, IPPROTO_TCP, TCP_KEEPALIVE....) always fails under Windows Server 2003.
Seems the socket option TCP_KEEPALIVE is not implemented in old Windows. So any libuv-based software wan't work there.
Libuv should ignore the error after this call.

Btw, WinSock error code 10042=WSAENOPROTOOPT (in this case of wrong setsockopt option) is not known by libuv. It translated into 'unknown'.

@piscisaureus @saghul Do we still consider Windows XP and Server 2003 to be supported platforms?  I don't think that silently ignoring setsockopt() errors is a good idea, by the way.

As far as I'm concerned we do support them on v1.x.
If setting TCP keepalive is not supported on 2k3 we should just return the
appropriate error, not ignore it, IMHO.
On Apr 4, 2015 17:22, "Ben Noordhuis"<EMAIL_ADDRESS>wrote:

@piscisaureus https://github.com/piscisaureus @saghul
https://github.com/saghul Do we still consider Windows XP and Server
2003 to be supported platforms? I don't think that silently ignoring
setsockopt() errors is a good idea, by the way.
—
Reply to this email directly or view it on GitHub
https://github.com/libuv/libuv/issues/302#issuecomment-89609012.

Is TCP_KEEPALIVE significantly different than SO_KEEPALIVE? Because all versions of Windows do support the latter.
https://msdn.microsoft.com/en-us/library/windows/desktop/ee470551(v=vs.85).aspx

SO_KEEPALIVE can only enable or disable keepalive, TCP_KEEPALIVE is used to set timers (tcp.c code)
SIO_KEEPALIVE_VALS IOCTL seems to be another way to set keepalive timers. Rust-lang use the latter (https://doc.rust-lang.org/net2-rs/net2/trait.TcpStreamExt.html#tymethod.set_keepalive_ms)


Add bitcoin forks config

Tao, can you add blocktime property to each chain too?

created an issue to track: https://github.com/trustwallet/blockatlas/issues/266


[261] add test cases for HabitFactController

Everything is fine, but you may want to remove this field: @mock private ModelMapper modelMapper; (since it was not used)

ModelMapper is used in save and update methods, so if i remove it, i get "jakarta.servlet.ServletException: Request processing failed: java.lang.NullPointerException: Cannot invoke "org.modelmapper.ModelMapper.map(Object, java.lang.Class)" because "this.mapper" is null".

It would be great to use some test naming conventions, e.g. MethodName_StateUnderTest_ExpectedBehaviour.
Also, remove any redundant code if possible: you are creating a variable of type ObjectMapper, but each time you use this class, you are creating a new instance with new ObjectMapper(). Either remove the variable or initialize it once.
It's better practice to use HabitFactService interface instead of HabitFactServiceImpl

Updated.


Improvement/updating-readme

Hey there 👋
When I first installed the app locally to start developing I found myself with a lot of questions and doubts about how I should proceed with everything. To make sure no one would face the same difficulties, I decided to improve the Readme a little bit. Take a look at it and feel free to suggest any changes or improvements :)

Thank you for the initiative, a fresh pair of eyes on the installation process is great!


Expose new cluster logging URL in namespace attributes

The new tenant and auth API supply the logging URL along with the console and metrics URLs.
This PR exposes the new URL in the user namespace attributes.
  "example-namespace-attributes": {
    "cluster-app-domain": "Aut consequatur unde earum nisi.",
    "cluster-console-url": "Debitis sit unde et.",
    "cluster-logging-url": "Nobis qui rerum sed.",
    "cluster-metrics-url": "Voluptatem tempora molestiae repellendus ad eius corporis.",
    "cluster-url": "Deserunt numquam officia dolorem minima voluptatem.",
    "created-at": "2016-11-29T23:18:14Z",
    "name": "Name for the tenant namespace",
    "state": "Deleniti et nemo.",
    "type": "run",
    "updated-at": "2016-11-29T23:18:14Z",
    "version": "Nulla aliquam."
  }

@stooke snapshot fabric8-wit image is available for testing.  docker pull docker.io/fabric8/fabric8-wit:SNAPSHOT-PR-1927-1

Codecov Report

Merging #1927 into master will increase coverage by 0.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #1927      +/-   ##
==========================================
+ Coverage   63.52%   63.54%   +0.01%
==========================================
  Files         144      144
  Lines       13388    13389       +1
==========================================
+ Hits         8505     8508       +3
  Misses       4072     4072
+ Partials      811      809       -2

Impacted Files
Coverage Δ

controller/user_service.go
0% <0%> (ø)
:arrow_up:

controller/workitem.go
82.56% <0%> (+0.72%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05bb68f...b6a18a1. Read the comment docs.


parse error - unexpected end of input

#include <cpprest/http_client.h>
#include <nlohmann/json.hpp>
#include <iostream>
int main() {
	 web::uri_builder builder("http://xkcd.com/614/info.0.json");   // first builder
     // web::uri_builder builder("http://i.jandan.net/?oxwlxojflwblxbsapi=jandan.get_duan_comments?page=25");   // second builder
        web::http::client::http_client client(builder.to_uri());
        nlohmann::json json = nlohmann::json::parse(client.request(web::http::methods::GET)
                              .get().extract_string().get());
        std::cout << json << std::endl;
}

Second builder can run.
But when run first builder, i will get:
terminate called after throwing an instance of 'std::invalid_argument'
what():  parse error - unexpected end of input
[1]    2056 abort (core dumped)  ./bin/xkcd_bot
Please help me! Thanks very much!

Can you make sure that the HTTP client is following redirections? I get the following output with httpie;
$ http http://xkcd.com/614/info.0.json
HTTP/1.1 301 Moved Permanently
Accept-Ranges: bytes
Connection: close
Content-Length: 0
Date: Wed, 19 Apr 2017 17:02:46 GMT
Location: https://xkcd.com/614/info.0.json
Retry-After: 0
Server: Varnish
Via: 1.1 varnish
X-Cache: HIT
X-Cache-Hits: 0
X-Served-By: cache-fra1228-FRA
X-Timer: S1492621366.171241,VS0,VE0

As you can see, the content-length is 0 bytes, so the error message "unexpected end of input" would make sense ;-)
The second URL returns proper JSON with a 200 status code.

OK, I know, thank you very much!

Was that the issue?

Nothing~


Fixes issues with Guidance selection and display

Placement of the new

 broke the connection between the tab menu and content. #944
Updated guidance selection to address issues reported in #943

will address issues with tests tomorrow


"all input arrays must have the same shape" error while running loFTR pipeline

Hello. Firstly congrats on this great work and thanks for sharing the code.
I have an indoor picture dataset consisting of 247 images each 9248x6936 pixels and taken with the same camera with the same settings.
I am trying to run a pipeline based on your Aachen_v1_1 pipeline using loFTR in order to increase the completeness of my point cloud (total number of 3d points). However, I keep getting the error given below in various different configurations while some configurations works smoothly.
For example:

When I use "loftr" as dense matcher with "netvlad" as feature extractor, I get the same array error below when I use "indoor" weights but it works fine when I set the weights as "outdoor" although it is an indoor picture set.
When I use "loftr_superpoint" as dense matcher with "superpoint_max_ as feature extractor, I get the same array error below if I set both "max_error" and "cell_size" to "1" or "2". However if I leave them both as "4" it works fine.
When I use "loftr_superpoint" as dense matcher with "superpoint_max" as feature extractor, setting both "max_error" and "cell_size" to 4 but changing "Number of image pairs for SfM" and
"Number of image pairs for loc" to 40 and 100, respectively gives me the same array error. Or just changing "Number of image pairs for SfM" from 20 to 40 causing the same error.

Since changes in different settings causing the same error, I cannot pinpoint what the actual reason can be. The only pattern I can see, whenever I change the settings in a way to increase the accuracy and completeness I start to get the same error.
The error is as follows:

[2022/12/01 23:18:34 hloc INFO] Importing matches into the database...
5%|                                                       | 250/4893 [00:00<00:02, 1578.48it/s]
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in
triangulation.main(
File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main
import_matches(image_ids, database, pairs, matches,
File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches
matches, scores = get_matches(matches_path, name0, name1)
File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 69, in get_matches
matches = np.stack([idx, matches[idx]], -1)
File "<array_function internals>", line 180, in stack
File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack
raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

Your help is much appreciated.

Can you replace
https://github.com/cvg/Hierarchical-Localization/blob/7e6551d78cbb8898520832851614a3438cb2cde7/hloc/utils/io.py#L69
with
try:
    matches = np.stack([idx, matches[idx]], -1)
except ValueError as e:
    print(name0, name1)
    print(idx.shape, matches[idx].shape, matches.shape)
    raise e

cc @Phil26AT in case you have any idea. Maybe related to https://github.com/cvg/Hierarchical-Localization/pull/241.

Thanks a lot for the prompt response. Below is the summary of the first configuration I mentioned above and output after modifying io.py as suggested. I will be running second and the third configs mentioned above and updating shortly.

(netvlad + loftr)
/pipeline.py
--Number of image pairs for SfM 	20
--Number of image pairs for loc	50
--max_kps				8192
--resize_max			512
/extract_features.py
--resize_max			512
--interpolation			pil_linear
/match_dense.py
--weights				outdoor(*)
--resize_max			512
--max_error			1
--cell_size			1

(*): When I change weights to 'indoor', it runs with no error in this config case.
python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 8,
                             'max_error': 4,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 2048}}}
[2022/12/07 03:06:36 hloc INFO] Reading the COLMAP model...
[2022/12/07 03:06:39 hloc INFO] Extracting image pairs from covisibility info...
100%|                                        | 247/247 [00:01<00:00, 151.08it/s]
[2022/12/07 03:06:40 hloc INFO] Found 4893 pairs.
[2022/12/07 03:06:40 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 1,
 'max_error': 1,
 'model': {'name': 'loftr', 'weights': 'indoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 512}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/07 03:06:44 hloc INFO] Performing dense matching...
100%|                                       | 3441/3441 [05:47<00:00,  9.91it/s]
[2022/12/07 03:12:32 hloc INFO] Assigning matches...
[2022/12/07 03:12:32 hloc INFO] Aggregating keypoints for 247 images.
100%|                                       | 3441/3441 [00:35<00:00, 97.49it/s]
[2022/12/07 03:13:07 hloc INFO] Finished assignment, found 7081.9 keypoints/image (avg.), total 1749234.
[2022/12/07 03:13:07 hloc INFO] Reassign matches with max_error=1.
100%|                                      | 3441/3441 [00:23<00:00, 144.45it/s]
[2022/12/07 03:13:31 hloc INFO] Importing features into the database...
100%|                                       | 247/247 [00:00<00:00, 1713.56it/s]
[2022/12/07 03:13:31 hloc INFO] Importing matches into the database...
  3%| \u258f                                    | 157/4893 [00:00<00:03, 1562.41it/s]20210115_111122.jpg 20210115_111211.jpg
(0,) (0, 2) (0, 2)
  5%| \u2589                                    | 250/4893 [00:00<00:02, 1571.75it/s]
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in <module>
    triangulation.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main
    import_matches(image_ids, database, pairs, matches,
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches
    matches, scores = get_matches(matches_path, name0, name1)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 74, in get_matches
    raise e
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 70, in get_matches
    matches = np.stack([idx, matches[idx]], -1)
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack
    raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

(superpoint+ loftr)
/pipeline.py
--Number of image pairs for SfM 20
--Number of image pairs for loc 50
--max_kps 8192
/extract_features.py
--resize_max 1024
--interpolation pil_linear
/match_dense.py
--weights outdoor
--resize_max 1024
--max_error 2 ()
--cell_size 2 ()
(*): When I set max_error and cell_size 2 and 2 or 1 and 1 respectively it fails with the same error but on a different image of the set. If I set them 4 and 4 it runs with no error in this config case.

python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 2,
                             'max_error': 2,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 1024}}}
[2022/12/07 03:34:21 hloc INFO] Reading the COLMAP model...
[2022/12/07 03:34:24 hloc INFO] Extracting image pairs from covisibility info...
100%|                                                                                | 247/247 [00:02<00:00, 121.82it/s]
[2022/12/07 03:34:26 hloc INFO] Found 4893 pairs.
[2022/12/07 03:34:26 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 2,
 'max_error': 2,
 'model': {'name': 'loftr', 'weights': 'outdoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 1024}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/07 03:34:28 hloc INFO] Performing dense matching...
100%|                                                                                | 3441/3441 [08:07<00:00,  7.06it/s]
[2022/12/07 03:42:35 hloc INFO] Assigning matches...
[2022/12/07 03:42:35 hloc INFO] Aggregating keypoints for 247 images.
100%|                                                                                | 3441/3441 [01:20<00:00, 42.66it/s]
[2022/12/07 03:43:56 hloc INFO] Finished assignment, found 7796.8 keypoints/image (avg.), total 1925816.
[2022/12/07 03:43:56 hloc INFO] Reassign matches with max_error=2.
100%|                                                                                | 3441/3441 [00:25<00:00, 133.21it/s]
[2022/12/07 03:44:22 hloc INFO] Importing features into the database...
100%|                                                                                | 247/247 [00:00<00:00, 1698.01it/s]
[2022/12/07 03:44:22 hloc INFO] Importing matches into the database...
  0%|                                                                                | 0/4893 [00:00<?, ?it/s]20210115_111053.jpg 20210115_112155.jpg
(0,) (0, 2) (0, 2)
  2%|    \u258b                                                                                | 95/4893 [00:00<00:03, 1537.08it/s]
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in <module>
    triangulation.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main
    import_matches(image_ids, database, pairs, matches,
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches
    matches, scores = get_matches(matches_path, name0, name1)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 74, in get_matches
    raise e
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 70, in get_matches
    matches = np.stack([idx, matches[idx]], -1)
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack
    raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

(superpoint+ loftr)
/pipeline.py
--Number of image pairs for SfM 40 ( * )
--Number of image pairs for loc 50
--max_kps 8192
/extract_features.py
--resize_max 1024
--interpolation pil_linear
/match_dense.py
--weights outdoor
--resize_max 1024
--max_error 4
--cell_size 4
( * ): When I set "Number of image pairs for SfM" to 40 it fails as below yet at another image. With 20 it runs normally. Also as another combination, I tried to keep "Number of image pairs for SfM" as 40 and increased "resize_max" from 1024 to 2048. This time it ran normally,too. I hope you can spot the problem and advise for a workaround.  Thanks in advance.

python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 4,
                             'max_error': 4,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 1024}}}
[2022/12/07 03:58:54 hloc INFO] Reading the COLMAP model...
[2022/12/07 03:58:57 hloc INFO] Extracting image pairs from covisibility info...
100%|                                                                                | 247/247 [00:01<00:00, 144.53it/s]
[2022/12/07 03:58:58 hloc INFO] Found 9344 pairs.
[2022/12/07 03:58:58 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 4,
 'max_error': 4,
 'model': {'name': 'loftr', 'weights': 'outdoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 1024}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/07 03:59:00 hloc INFO] Performing dense matching...
100%|                                                                                | 6014/6014 [13:58<00:00,  7.17it/s]
[2022/12/07 04:12:59 hloc INFO] Assigning matches...
[2022/12/07 04:12:59 hloc INFO] Aggregating keypoints for 247 images.
100%|                                                                                | 6014/6014 [01:58<00:00, 50.75it/s]
[2022/12/07 04:14:57 hloc INFO] Finished assignment, found 7913.6 keypoints/image (avg.), total 1954651.
[2022/12/07 04:14:57 hloc INFO] Reassign matches with max_error=4.
100%|                                                                                | 6014/6014 [00:41<00:00, 145.82it/s]
[2022/12/07 04:15:39 hloc INFO] Importing features into the database...
100%|                                                                                | 247/247 [00:00<00:00, 1653.35it/s]
[2022/12/07 04:15:39 hloc INFO] Importing matches into the database...
 14%|                                  \u258d                                                                                | 1345/9344 [00:00<00:04, 1724.69it/s]20210115_111302.jpg 20210115_112632.jpg
(0,) (0, 2) (0, 2)
 16%|                                     \u258a                                                                                | 1477/9344 [00:00<00:04, 1684.43it/s]
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 45, in <module>
    triangulation.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 224, in main
    import_matches(image_ids, database, pairs, matches,
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/triangulation.py", line 90, in import_matches
    matches, scores = get_matches(matches_path, name0, name1)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 74, in get_matches
    raise e
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/utils/io.py", line 70, in get_matches
    matches = np.stack([idx, matches[idx]], -1)
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 426, in stack
    raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape

Thanks for the detailed logs. Can you please try PR https://github.com/cvg/Hierarchical-Localization/pull/254? Please remove existing match and features files beforehand.

Hello,
I ran the first configuration again with "indoor" weights which was the failed case. It passed the previous stage. However, it failed again with a different type of array error at a different stage. Copying the output below.
Also I noticed that -as a side question- indoor weight produced far more less (only 1855) 3D points, while using outdoor weights produces 34013 points during 3D reconstruction. The image set belongs to an indoor area (a simple bedroom), so I would expect the number to be higher when "indoor" weights used. Is this an expected behavior?
python3 -m hloc.pipelines.NEWbedroom02.pipeline
Configs for dense feature matchers:
{'loftr': {'cell_size': 1,
           'max_error': 1,
           'model': {'name': 'loftr', 'weights': 'outdoor'},
           'output': 'matches-loftr',
           'preprocessing': {'dfactor': 8,
                             'grayscale': True,
                             'resize_max': 1024}},
 'loftr_aachen': {'cell_size': 8,
                  'max_error': 2,
                  'model': {'name': 'loftr', 'weights': 'outdoor'},
                  'output': 'matches-loftr_aachen',
                  'preprocessing': {'dfactor': 8,
                                    'grayscale': True,
                                    'resize_max': 1024}},
 'loftr_netvlad_indoor': {'cell_size': 1,
                          'max_error': 1,
                          'model': {'name': 'loftr', 'weights': 'indoor'},
                          'output': 'matches-loftr_NEWbedroom02',
                          'preprocessing': {'dfactor': 8,
                                            'grayscale': True,
                                            'resize_max': 512}},
 'loftr_superpoint': {'cell_size': 4,
                      'max_error': 4,
                      'model': {'name': 'loftr', 'weights': 'outdoor'},
                      'output': 'matches-loftr_aachen',
                      'preprocessing': {'dfactor': 8,
                                        'grayscale': True,
                                        'resize_max': 1024}},
 'loftr_superpoint_indoor': {'cell_size': 4,
                             'max_error': 4,
                             'model': {'name': 'loftr', 'weights': 'outdoor'},
                             'output': 'matches-loftr_NEWbedroom02',
                             'preprocessing': {'dfactor': 8,
                                               'grayscale': True,
                                               'resize_max': 2560}}}
[2022/12/07 23:50:36 hloc INFO] Reading the COLMAP model...
[2022/12/07 23:50:39 hloc INFO] Extracting image pairs from covisibility info...
100%|                                        | 247/247 [00:01<00:00, 127.88it/s]
[2022/12/07 23:50:41 hloc INFO] Found 4893 pairs.
[2022/12/07 23:50:41 hloc INFO] Extracting semi-dense features with configuration:
{'cell_size': 1,
 'max_error': 1,
 'model': {'name': 'loftr', 'weights': 'indoor'},
 'output': 'matches-loftr_NEWbedroom02',
 'preprocessing': {'dfactor': 8, 'grayscale': True, 'resize_max': 512}}
/home/paperspace/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py:554: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 12, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
[2022/12/07 23:50:45 hloc INFO] Performing dense matching...
100%|                                       | 3441/3441 [06:28<00:00,  8.87it/s]
[2022/12/07 23:57:14 hloc INFO] Assigning matches...
[2022/12/07 23:57:14 hloc INFO] Aggregating keypoints for 247 images.
100%|                                       | 3441/3441 [00:45<00:00, 76.04it/s]
[2022/12/07 23:57:59 hloc INFO] Finished assignment, found 7081.9 keypoints/image (avg.), total 1749234.
[2022/12/07 23:57:59 hloc INFO] Reassign matches with max_error=1.
100%|                                      | 3441/3441 [00:28<00:00, 121.12it/s]
[2022/12/07 23:58:28 hloc INFO] Importing features into the database...
100%|                                       | 247/247 [00:00<00:00, 1481.71it/s]
[2022/12/07 23:58:28 hloc INFO] Importing matches into the database...
100%|                                     | 4893/4893 [00:02<00:00, 1849.76it/s]
[2022/12/07 23:58:31 hloc INFO] Performing geometric verification of the matches...
100%|                                         | 247/247 [01:12<00:00,  3.43it/s]
[2022/12/07 23:59:43 hloc INFO] mean/med/min/max valid matches 3.93/2.77/0.00/33.15%.
[2022/12/07 23:59:43 hloc INFO] Running 3D triangulation...
[2022/12/08 00:01:14 hloc INFO] Finished the triangulation with statistics:
Reconstruction:
	num_reg_images = 247
	num_cameras = 1
	num_points3D = 1855
	num_observations = 4806
	mean_track_length = 2.59084
	mean_observations_per_image = 19.4575
	mean_reprojection_error = 1.44981
[2022/12/08 00:01:14 hloc INFO] Extracting local features with configuration:
{'model': {'name': 'netvlad'},
 'output': 'global-feats-netvlad',
 'preprocessing': {'resize_max': 512}}
[2022/12/08 00:01:14 hloc INFO] Found 248 images in root datasets/NEWbedroom02/images/images_upright.
100%|                                         | 248/248 [07:12<00:00,  1.74s/it]
[2022/12/08 00:08:34 hloc INFO] Finished exporting features.
[2022/12/08 00:08:34 hloc INFO] Extracting image pairs from a retrieval database.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pipelines/NEWbedroom02/pipeline.py", line 54, in <module>
    pairs_from_retrieval.main(
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_retrieval.py", line 94, in main
    query_desc = get_descriptors(query_names, descriptors)
  File "/home/paperspace/pixel-perfect-sfm/Hierarchical-Localization/hloc/pairs_from_retrieval.py", line 42, in get_descriptors
    return torch.from_numpy(np.stack(desc, 0)).float()
  File "<__array_function__ internals>", line 180, in stack
  File "/home/paperspace/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 422, in stack
    raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack

I recommend to always use outdoor weights, regardless of the type of environment. The indoor model likely overfits to the camera statistics of the ScanNet dataset.
Regarding your error: what are you trying to achieve? do you have query images to localize? If not, all steps after the triangulation/reconstruction are irrelevant and should be discarded from the original pipeline. This simply fails because it cannot find query images with the prefix query.

Thanks for the explanation. I have gone through it and understood it better. I have also run the other 2 config cases and they are working fine. I believe this issue can be closed.
I have 2 off-topic questions if you don't mind answering, though.

Given the photos taken by the same (consumer grade) camera under same lighting conditions, I am trying to maximize accuracy and completeness of indoor scenes. Going through the literature I think loftr+superpoint-->hloc-->pixelperfect is my best option for the time being. For hloc, assuming unlimited compute resources or/and time to compute, what would be the other parameters to tweak to achieve this? For example would changing 'dfactor' value or setting 'grayscale' false help? If you have a guideline out somewhere that I am missing, I appreciate if you could guide me to that.
Authors of LoFTR released another repo named QuadTreeAttention* with better performance** than LoFTR. Any plans to implement that into hloc?

( * ) https://github.com/Tangshitao/QuadTreeAttention
( ** ) https://www.kaggle.com/competitions/image-matching-challenge-2022/discussion/328805

Grayscale is a requirement of the feature extractor and cannot be tweaked. Increasing the size of the input images and matching all points (not only SuperPoint keypoints) can both increase the density of the 3D point cloud.
We do not have plan to integrate this in the near future.


Compiling FluxEngine on Raspberry Pi (Buster)

I've trying to compile on a Raspberry Pi (buster). I have installed the dependencies inthe documentation, but the make script is not recognising fmt.
pi@raspberrypi:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

pi@raspberrypi:~ $ cd fluxengine/
pi@raspberrypi:~/fluxengine $ make
Package fmt was not found in the pkg-config search path.
Perhaps you should add the directory containing `fmt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'fmt' found
Package fmt was not found in the pkg-config search path.
Perhaps you should add the directory containing `fmt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'fmt' found
lib/build.mk:81: *** Missing required pkg-config dependency:  fmt.  Stop.

pi@raspberrypi:~/fluxengine $ sudo apt-get install libfmt-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libfmt-dev is already the newest version (5.2.1+ds-2).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

I can't locate a fmt.pc file to place in the PKG_CONFIG_PATH environment variable.
Any pointers would be greatly appreciated.
Cheers,
Matt

Hmm. Could you do dpkg -L libfmt-dev? There should be a line there looking like /usr/lib/x86_64-linux-gnu/pkgconfig/fmt.pc (although different because you're on ARM).

Hi David,
Sure; here you go:
pi@raspberrypi:~/fluxengine $ dpkg -L libfmt-dev
/.
/usr
/usr/include
/usr/include/fmt
/usr/include/fmt/color.h
/usr/include/fmt/core.h
/usr/include/fmt/format-inl.h
/usr/include/fmt/format.h
/usr/include/fmt/ostream.h
/usr/include/fmt/posix.h
/usr/include/fmt/printf.h
/usr/include/fmt/ranges.h
/usr/include/fmt/time.h
/usr/lib
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/cmake
/usr/lib/arm-linux-gnueabihf/cmake/fmt
/usr/lib/arm-linux-gnueabihf/cmake/fmt/fmt-config-version.cmake
/usr/lib/arm-linux-gnueabihf/cmake/fmt/fmt-config.cmake
/usr/lib/arm-linux-gnueabihf/cmake/fmt/fmt-targets-none.cmake
/usr/lib/arm-linux-gnueabihf/cmake/fmt/fmt-targets.cmake
/usr/lib/arm-linux-gnueabihf/libfmt.a
/usr/share
/usr/share/doc
/usr/share/doc/libfmt-dev
/usr/share/doc/libfmt-dev/README.Debian
/usr/share/doc/libfmt-dev/changelog.Debian.gz
/usr/share/doc/libfmt-dev/changelog.gz
/usr/share/doc/libfmt-dev/copyright

I actually have now compiled this on Ubuntu 20.04, and it was fine, and am happily using fluxengine and Greaseweasel v4, but it would still be good to understand this issue.

Yeah, your version doesn't have the pkgconfig file in it. Which is wrong.
apt show libfmt-dev, please, and look at the version? Mine is 8.1.1+ds1-2.

My version appears to be older:
pi@raspberrypi:~ $ apt show libfmt-dev
Package: libfmt-dev
Version: 5.2.1+ds-2
Priority: optional
Section: libdevel
Source: fmtlib
Maintainer: Eugene V. Lyubimkin<EMAIL_ADDRESS>Installed-Size: 681 kB
Suggests: libfmt-doc
Conflicts: libcppformat1-dev, libcppformat2-dev, libfmt3-dev
Homepage: http://fmtlib.net/latest/index.html
Download-Size: 124 kB
APT-Manual-Installed: yes
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
Description: fast type-safe C++ formatting library -- development files
 This library provides fast, type-safe, small, C++11-aware replacement of
 (s)printf and related machinery. In some cases it's noticeably faster
 than boost::format, boost::lexical_cast and even sprintf itself.
 .
 This package contains the development files, include the static library.

In case it's of use to anyone else, I upgraded the Pi to Bullseye, and the libfmt-dev version is later, and has compiled without issue.

Good to know, thanks.
I'm afraid I'm limited as to just how many different library versions I can support --- I have to pick something which is reasonably sane...


[FEATURE] showing selected generator on sidebar

Description
when any of the generator gets selected, it shows it selected only when visitor hovering on the element, but it would be nice if we can make the generator as selected on click, or when a generator gets selected.
Just the attached screenshot, we can show the Box-shadow generator when it selected, (it's showing only when user hover on it)
Screenshots

Additional information
No response
👀 Have you checked if this issue has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

[X] I have read and understood the rules in the Contributing Guidelines

Are you willing to work on this issue ?
Yes I am willing to submit a PR!

Hi Rupali, thanks for raising the issue. I will assign you to this. Good luck

Hi @Dun-sin, i have been working on this, but i'm getting some weird changes, if i'm visibility: visible to selected element then the generator next to it is going to be disappeared.
can you give me some hints about fixing it?
screenshot:

Hi @Dun-sin, i have been working on this, but i'm getting some weird changes, if i'm visibility: visible to selected element then the generator next to it is going to be disappeared.
can you give me some hints about fixing it?
screenshot:

Sorry, I don't get what you are trying to say? Can you be more clear

i'm saying that, the next element or generator to text-gradient is box-shadow, and this text-gradient is overlapping that
okay, i'm pushing to u can see it

:tada: This issue has been resolved in version 2.49.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Web support

I changed the Webviewer with Easywebviewer, the only issue now is that I have no URL callback

@Heinrich-vanNieuwenhuizen  Please, can you share more information. I don't understand what the issue is.

Hi Wilburt/Rave_flutter,
If you compile for flutter web, the library webview is not compatible with
flutter web. I had to clone your library edit myself and use a webview that
works with flutter web. (I have an app and web app which I want to use rave
flutter payment). The issue with the library im using is there is no
callback function for the finished payment.
Another issue the subscription option is not working and there is no easy
way to tokenise the credit cards.
Kind Regards,
Heinrich van Nieuwenhuizen
FS Eng (TÜV Rheinland, #15376/ 17, HW/SW Design)
Director
Website: www.ecam.co.za
Cell :<PHONE_NUMBER>
Tel:<PHONE_NUMBER>/3107/3237
Fax:<PHONE_NUMBER>
No.134 Albertina Sisulu Road
Manufacta
Roodepoort
1724
**
Everything in this e-mail and attachments relating to the official business
of ECAM (Pty) Ltd. is proprietary to the company. It is confidential,
legally privileged and protected by law. ECAM (Pty) Ltd. does not own and
endorse any other content. Views and opinions are those of the sender
unless clearly stated as being that of ECAM (Pty) Ltd. The person addressed
in the e-mail is the sole authorized recipient. Please notify the sender
immediately if it has unintentionally reached you and do not read, disclose
or use the content in any way. Whilst all reasonable steps are taken to
ensure the accuracy and integrity of information and data transmitted
electronically and to preserve the confidentiality thereof, no liability or
responsibility whatsoever accepted if information or data is, for whatever
reason corrupted or does not reach its intended destination.
On Mon, Aug 31, 2020 at 3:22 AM Wilberforce Uwadiegwu <
<EMAIL_ADDRESS>wrote:

@Heinrich-vanNieuwenhuizen https://github.com/Heinrich-vanNieuwenhuizen
Please, can you share more information. I don't understand what the issue
is.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/wilburt/rave_flutter/issues/39#issuecomment-683504090,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABLUKUCLJ5AMNRZNXE5RXSDSDL3N3ANCNFSM4PX3KOCQ
.

I currently do not have plans for supporting the web for now. Please, feel free to make a PR if you can implement it successfully.


Login Error

after installation done when i'm trying to login then the verification failed status showing.
please fix this error

same issue

same issue bro
please solve login issue

same problem please fix it

same problem please fix it


src-cli: init at 4.0.0

Description of changes
src is a command line interface to Sourcegraph
https://github.com/sourcegraph/src-cli

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

#239021

Closed in favor of https://github.com/NixOS/nixpkgs/pull/239021


Move optional dependencies from dev_requirements to pyproject.toml

https://github.com/Hochfrequenz/python-generics/actions/runs/11047077960


Kidwise-App-Integration

KidWise revolutionizes childhood development by transforming the Omi into an AI-powered tracker. It provides real-time insights while creating a development diary of your child's growth journey.

Hi @deoxicit, we now allow submitting apps from within our Omi AI App. Please install our Omi AI app and submit your Kidwise app from within there as well. Thanks


Added isFirstSynchronization to options

Added isFirstSynchronizations to options
Made shouldDetectDeletes method protected

If source iModel had different external source aspects, they were imported to target. Then at the end of transformation detectElementsDeletes was called. It would detect that some elements should be deleted, because transformer would get random identifier values, example:

If source iModel had different external source aspects, they were imported to target. Then at the end of transformation detectElementsDeletes was called. It would detect that some elements should be deleted, because transformer would get random identifier values, example:

We shouldn't introduce this "new option" because the transformer is going to expose a higher-level Branching API soon, including an initBranch API which would supersede this. Also, it appears that you're adding this because you're not using wasSourceIModelCopiedToTarget in the branch initialization, which is not supported by the transformer.

Also adding, it seems the bug this was fixing was due to a default targetScopeElement 0x1 which conflicted with the scope of ExternalSourceAspect's added by others. The default of targetScopeElement is very bad and the default will be removed in 1.x


frontend: Add MatomoConsentPopup

Closes #1375
Stops Matomo tracking until the user accepts it in the pop up. Consent information is stored in Matomo.
@olympe-silve please check in the screenshot if the styling of the pop up is ok like this:

@manuelwedler the style of the pop up is ok, but I would rather it being completely centered, with the app in the background being blurred until the user makes a decision on the CTA. As soon as that's done, the app reappears normally and useable.

Can we also store the users selection inside localStorage ? That way the users that decline the consent will not have to decline again when they come to our website.

:100:
I am not sure about the legal aspect, but, as a user, if I consented/declined once, I would like my decision to be remembered.

Alright, will implement the suggestions :+1:
@olympe-silve do we want it also centered on the screen for the website?

Alright, will implement the suggestions 👍
@olympe-silve do we want it also centered on the screen for the website?

yes ! easiest option imo


Added Project Memory Game

Related Issue #168
Info about Issue or bug
Closes: #168
Changes

Added the Project Memory Game, a complete set of files.

Files added:

_index.html (driver)
css files (Normal styles, responsive design & behaviour)
js, jQuery files (consists the functionality of the game & Structured/DOM)_

any unusual behaviour of your code(Write NA if there isn't)

NA

Checklist:

[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have made corresponding changes to the documentation.
[x] New and existing unit tests pass locally with my changes.

Screenshot/Video Samples:
Here is the output of the project:
https://user-images.githubusercontent.com/26524467/109427275-99e9b980-7a17-11eb-8480-6d445ac48600.mp4

Ok, @Vishal-raj-1. I'll look forward.


Make print-media optional

For v4, it would be nice to be able to set $print-media = false or something similar.
That would enable us to simply @import bootstrap.scss, rather than having to import everything (by copying the contents of bootstrap.scss) and remove the print-styles partial.

Yeah, I could see us doing this one actually since I don't think most folks thinking about print styles for their sites these days anyway.

Fantastic, cheers - would play nicely with all the other optional functionality and variable-goodness.

Implemented in #19106.

Thanks! :+1:


Swedish å,ä, ö are treated as diacritics

The Scandinavian languages and the Finnish language, by contrast, treat the characters with diacritics å, ä, and ö as distinct letters of the alphabet, and sort them after z. Usually ä (a-umlaut) and ö (o-umlaut) [used in Swedish and Finnish] are sorted as equivalent to æ (ash) and ø (o-slash) [used in Danish and Norwegian]. Also, aa, when used as an alternative spelling to å, is sorted as such. Other letters modified by diacritics are treated as variants of the underlying letter, with the exception that ü is frequently sorted as y.
import Foundation
let symbol = "The Swedish letters Å, Ä, Ö"
let string = "a"
let symbolRange = symbol.range(of: string, options: [.caseInsensitive, .diacriticInsensitive])
if let range = symbolRange {
print("Found '(string)' in '(symbol)'") } else {
print("'(string)' not found in '(symbol)'")
}
Prints
Found 'a' in 'The Swedish letters Å, Ä, Ö'
Should print
'a' not found in 'The Swedish letters Å, Ä, Ö'
Replacing the string with "o" — same issue.

Right, as a Swedish native speaker the current behavior is very strange - if matching without diacritics we get what is the incorrect result really.

Also note that sorting will be incorrect. A, B, C, D, ...., X, Y, Z, Å, Ä, Ö is the correct sorting order for the Swedish alphabet.
In a Swedish dictionary é is sorted along with e and ü is sorted along with (both are true diacritics), but å and ä are not sorted along with a and ö is not sorted along with o.
As mentioned, this is a problem also for Norwegian and Danish.
It's peculiar that only Å and Æ are considered diacritics (Danish equivalent of Å and Ä) but Ø is not (Danish equivalent of Ö).

This API range(of: string, options:)  isn't locale/language aware. While these letters are distinct letters in Swedish, they are indeed diacritics in other languages, so it's challenging to make that distinction here.
That being said, I would definitely expect the localized version of this API, e.g. range(of: string, options: [.caseInsensitive, .diacriticInsensitive], locale: Locale(languageCode: .swedish)) to return what you described, but it isn't currently. Would you agree that we should track that issue instead?

It seems there are way more languages treating them as separate letters
See e.g.
https://en.wikipedia.org/wiki/Å#:~:text=It is a separate letter,Pamirian languages%2C and Greenlandic alphabets.
But as there is no single correct answer, moving this case to be for the Swedish locale would be ok I think. (Although I think the locale-unaware default is debatable, I guess it's been that way for some time...)