Compiling latest from git

Attempting to compile the last git repo using Visual Studio, I followed all the instructions on the wiki with regards to 'building from source' which includes the vcpkg steps.
When I compile however, I get the following error:
Error	MSB3073	The command "SETLOCAL ENABLEDELAYEDEXPANSION

IF NOT "G:\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\" == "" (
  IF "Release" == "Debug" (
    SET configDir="debug\"
  ) ELSE (
    SET configDir=""
  )

  SET sourceDir="G:\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\%configDir%bin\"

  xcopy /y /d "!sourceDir!ogg.dll" "C:\Users\ray\Source\Repos\OPHD\x64\Release\"
  xcopy /y /d "!sourceDir!vorbis.dll" "C:\Users\ray\Source\Repos\OPHD\x64\Release\"
  xcopy /y /d "!sourceDir!vorbisfile.dll" "C:\Users\ray\Source\Repos\OPHD\x64\Release\"
)
:VCEnd" exited with code 4.

Hi, thank you for the report.

The problem appears to be from the xcopy command from the Post-Build event script in the project file:
    <PostBuildEvent>
      <Command>SETLOCAL ENABLEDELAYEDEXPANSION

IF NOT "$(VcpkgCurrentInstalledDir)" == "" (
  IF "$(ConfigurationName)" == "Debug" (
    SET configDir="debug\"
  ) ELSE (
    SET configDir=""
  )

  SET sourceDir="$(VcpkgCurrentInstalledDir)%configDir%bin\"

  xcopy /y /d "!sourceDir!ogg.dll" "$(TargetDir)"
  xcopy /y /d "!sourceDir!vorbis.dll" "$(TargetDir)"
  xcopy /y /d "!sourceDir!vorbisfile.dll" "$(TargetDir)"
)</Command>
      <Message>Copy OGG dependencies if vckpg present</Message>
    </PostBuildEvent>

@Brett208: You may be more familiar with this code than I am.
Relevant documentation:
https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/xcopy
Exit code 4:

Initialization error occurred. There is not enough memory or disk space, or you entered an invalid drive name or invalid syntax on the command line.

Perhaps %sourceDir% was intended, rather than !sourceDir!? I've never seen that syntax before.

I tried changing it from !sourceDir! to $(sourceDir$) but it didn't make any difference.
And then if I search the cloned repo for 'ogg.dll', and 'vorbis.dll' and 'vorbisfile.dll', none of them are in the cloned repo at all, so i'm not sure where it is supposed to be getting these 3 files from.
I could just download the release build, but I wanted to play it with an up to date copy, the release build is about 18months old compared to the most recent git commit of only a few months ago.

I even checked the entire 'vcpkg' directory and that doesn't have those 3 files either.

I just tried changing !sourceDir! to %sourceDir%, but got the error you described. Curiously the builds are passing on the continuous integration servers using the old syntax.
Old syntax build log: https://ci.appveyor.com/project/OPU/ophd/builds/36817353/job/0fw45nhoau7qy1mm
New syntax build log: https://ci.appveyor.com/project/OPU/ophd/builds/37002619/job/8huh36c76s2k3fww

From the original pasted error text, the scripted line:
SET sourceDir="$(VcpkgCurrentInstalledDir)%configDir%bin\"

Expands to:
SET sourceDir="G:\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\%configDir%bin\"

That means the DLL files should be located in:
G:\vcpkg\installed\x64-windows\%configDir%bin\
Where %configDir% = "" for release builds, or "debug\" for debug builds.
If the files don't exist in there, it may be an issue running the vcpkg install steps. That is generally done by running the batch file:
nas2d-core/InstallVcpkgDeps.bat

(I haven't checked the Wiki in ages, so instructions in there may be old).

Edited nas2d-core\InstallVcpkgDeps.bat so that it pointed to where I have vcpkg installed (since vcpkg isn't in the PATH variable), I can see ogg.dll, etc... in x86-windows\bin, and now I can see it in x64-windows\bin too.
I don't remember that particular .bat file being on the wiki saying to run it.
Reset my postbuild stuff back to what it originally was.
Re-running the 'build solution', and now it has a whole stack more errors in it than it had before :(
Severity	Code	Description	Project	File	Line	Suppression State
Error	C1083	Cannot open include file: 'SDL2/SDL_mixer.h': No such file or directory	NAS2D	C:\Users\ray\Source\Repos\OPHD\nas2d-core\NAS2D\Resources\Sound.cpp	17
Error	C1083	Cannot open include file: 'SDL2/SDL_mixer.h': No such file or directory	NAS2D	C:\Users\ray\Source\Repos\OPHD\nas2d-core\NAS2D\Resources\Music.cpp	16
Error	C1083	Cannot open include file: 'SDL2/SDL_mixer.h': No such file or directory	NAS2D	C:\Users\ray\Source\Repos\OPHD\nas2d-core\NAS2D\Mixer\MixerSDL.cpp	22
Error	LNK2001	unresolved external symbol Mix_VolumeMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_FadeOutMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_HookMusicFinished	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_OpenAudio	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_Volume	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_PauseMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_Pause	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_ResumeMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_CloseAudio	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_PlayChannelTimed	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_FadeInMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_HaltChannel	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_PlayingMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_Resume	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_HaltMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(MixerSDL.obj)	1
Error	LNK2001	unresolved external symbol Mix_FreeMusic	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(Music.obj)	1
Error	LNK2001	unresolved external symbol Mix_LoadMUS_RW	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(Music.obj)	1
Error	LNK2001	unresolved external symbol Mix_FreeChunk	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(Sound.obj)	1
Error	LNK2001	unresolved external symbol Mix_LoadWAV_RW	OPHD	C:\Users\ray\Source\Repos\OPHD\OPHD\NAS2D.lib(Sound.obj)	1
Error	LNK1120	19 unresolved externals	OPHD	C:\Users\ray\Source\Repos\OPHD\x64\Release\OPHD.exe	1

I would just be happy for a new released version that isn't over a year old, then I wouldn't have to go down this path.

Looks like you have a vcpkg issue. I'm guessing vcpkg isn't integrated into your environment, hence why it wasn't available on the path.
The first step, which it seems you'd now accomplished, is to have vcpkg download and compile the dependencies. The second step, which is still a problem, is to compile the OPHD source against those dependencies.
The vcpkg tooling is used to find the headers for vcpkg installed dependencies when compiling, as well as to find the compiled libraries files for those dependencies when linking. The errors you listed are because vcpkg tooling isn't setup to find the headers or compiled libraries. You may want to consult the vcpkg reference about integrating it into your environment.
https://vcpkg.readthedocs.io/en/latest/examples/installing-and-using-packages/
See "Step 2", where they use vcpkg integrate install.

With that said, if all you want is the compiled binaries, every continuous integration build stores the executable file as one of the build artifacts. These are mainly for debugging, though should be publicly accessible even if there is no formal release for that build.
From the main GitHub code page, find the green check mark indicating a successful continuous integration build, click on it and scroll to the desired build (the Windows one has a blue icon) and follow the "details" link, select the platform you want (generally the 64-bit build these days), and then go to the artifacts tab. The download will be on that page. It'll be the smaller package, not the one with the debug symbols.
Here's the most recent build:
https://ci.appveyor.com/project/OPU/ophd/builds/36817353/job/0fw45nhoau7qy1mm/artifacts

@Brett208: This looks like it may have been a vcpkg setup issue. Our outdated Wiki instructions may have been to blame. Still curious about the !sourceDir! syntax, though that doesn't seem to have been the problem.
Maybe there is a way we can detect when vcpkg isn't integrated into the environment, and issue a meaningful error message. Right now the current failure mode doesn't provide any helpful hints as to what went wrong. It may be possible for the project file to detect some Vcpkg provided variable, and if not present, issue an error message to the build console, along with instructions on how to fix it.
Or would it be bad to tie the project file to a particular dependency manager?

I had already done the 'vcpkg integrate install' the other day before I opened this issue originally, so i'm not sure why it's not set on the path environment variable.
Basically followed the wiki to the letter with setting everything up, it said I could install vcpkg where-ever I wanted to, and I figured since my G drive has more space on it than my C drive, I would put it there.

I even did the optional build steps after the integration.

It's been a while since I've used a Windows environment, so I'm afraid I'm stuck speculating about the problem. The only Windows environment I really have access to these days are the continuous integration builds on AppVeyor, and their build images comes with vcpkg already installed and setup. If it's a vcpkg setup or integration issue, which it rather looks like, I'm not really in a good position to test.
As it sounds like you did indeed run the vcpkg integrate install command, my best guess might be if it was run without administrator access. I'm not sure if there is any error message in such a case, though I'm pretty sure it won't have lasting effect without administrator access. You mentioned running it yesterday, so perhaps you rebooted since you ran the command. Maybe it'll quietly only do session setup without administrator access, rather than permanent setup? Kind of a wild guess though.
I may need someone else to investigate if it's a vcpkg setup issue.

@DanRStevens or @ldicker83,
It sounds like @ray73864 is more interested in the most recent build from master to play or test as opposed to building up the infrastructure to compile from source.
It is common on other projects I've seen to have a link to pull the most recent build artifacts from the master tree for beta testing or other purposes.
Have we looked into maybe putting a link on the wiki or in the master readme for OPHD that links to the most recent build artifacts? Maybe just the Windows build to start with since it is the most common.

@DanRStevens,
If I recall correctly, the exclamation mark syntax !sourceDir! affects the order of when variables expand for Windows command line and batch script arguments. The default behavior without the ! sometimes causes unexpected results although I am not remembering the details without doing some research.
Since the automated build is working, I'm assuming the wiki instructions are either out of sync or maybe not clear enough.
@ray73864, I noticed you are using the G drive for vcpkg in the snippets above. Are you installing vcpkg on a different drive from where you are compiling OPHD? I haven't experimented with this before, although I suppose it should work?

I am, yes.
vcpkg is on the G drive, and OPHD is on the C drive.

OK, So I deleted the repo completely, came back here to github, clicked on 'code' then on the 'Open in Visual Studio', cloned it to the G drive too, opened it, tried to recompile, kept complaining about outdated pdb files, etc...
Closed Visual Studio, opened command prompt and did the vcpkg bootstrap thing again and then the integrate again.
Deleted the temporary folder from the OPHD directory, opened the solution, redid the 'build solution', it took a little bit but this time it has 'build succeeded'.

@Brett208, I think you're right about having a link to the latest build.
Though it does raise the question of where to host it. If we direct link to AppVeyor, and the project goes idle, AppVeyor will eventually delete old artifacts, and the link will be broken. Not sure where else to host it. Unless we created a GitHub release for the latest build, and marked it as such. That might give it more of an official look though, which we might want to avoid.
Of course Leeor was thinking of packaging a new release soon, so that might be at least a temporary solution.

Ahh, that clicks now. I found some info on EnableDelayedExpansion, which explains the !variable! syntax:
https://ss64.com/nt/delayedexpansion.html

@ray73864, sounds like we have a bit of work to make the experience a little more smooth. For some reason I thought we had designed it to work so data was accessible in either the deployment directory layout, or the development directory layout. It seems the development directory layout doesn't work though.
Glad you got it working though.

That sounds like a debugger setting which is usually in the user options. I thought I had that corrected.
I plan to have a release put together in the next day or so. Just tidying somethings up and then it'll be good to go.
Sorry I haven't been too involved -- this time of year is ridiculously busy at work so I tend to kind of fall off the radar until after the new year.

I looked into this after setting up a virtual machine with a clean install of Windows and Visual Studio. vcpkg really needs to be set up in the environment path or the whole build process breaks. I noted this in #737.
I'll be updating the build instructions and probably releasing a clean build later today. Keep an eye out!

I looked into this after setting up a virtual machine with a clean install of Windows and Visual Studio. vcpkg really needs to be set up in the environment path or the whole build process breaks. I noted this in #737.
I'll be updating the build instructions and probably releasing a clean build later today. Keep an eye out!

Have released a new version of OPHD just now: https://forum.outpost2.net/index.php/topic,6371.0.html
Can get from the releases tab. The 23MB zip archive -- that's the complete release with data assets.

Have released a new version of OPHD just now: https://forum.outpost2.net/index.php/topic,6371.0.html
Can get from the releases tab. The 23MB zip archive -- that's the complete release with data assets.

We should probably close this. I think the new release solved the immediate problem of getting an up-to-date executable. As for the compiling difficulties, there are two opened issues to address them:

#737
https://github.com/lairworks/nas2d-core/issues/373


Policy on let/const

I noticed this in the commit messages for switching over to let/const:
This takes a conservative approach for 'const' where
it will only be used for aliases and storing the "original" values in
tests.

I find this policy a bit confusing. In our code, let is an indication that that identifier is reassigned at some point. Everything else must be const. The Google style guide has the same policy:

5.1.1 Use const and let
Declare all local variables with either const or let. Use const by default, unless a variable needs to be reassigned. The var keyword must not be used.

Any reason for the deviation? I think not using const by default creates a lot of mental overhead.

No, you correctly understand the policy.  But when we were getting rid of "var" by using a massive "sed" script, "let" was safer.  So the times you see "const" today, those were done by hand.  Most of the "let" you see today were written by a regex.
We followed an outdated, pre-ES6 style guide until we started incorporating ES6 into our project.  We are still transitioning in terms of style.

Oh great, thanks for clarifying. So if I am contributing new code, I will follow the Google style guide.


WIP: Add a new convar for the MySQL connection string

MySqlAsync needs to be recompiled.

Merged in #9, Thanks !


Provide the ability to delete one or more stamps

Provide the ability to delete one or more stamps from the stamp list based on selection.
The prompt ideally should list stamps and provide the ability to deslect one or more stamps.

Deleting a single stamp was added to #53 work, but this uses a simple alert prompt.  Would like a nicer component for deletion here.

Basic design should be something like this:


Updates specific to Jurnal Sain Veteriner

[ ] Create page
[ ] Green area (or whatever alternative is decided upon)
[ ] Add papers
[ ] Share on socmed

Article is under review


Initial removal of Android instructions.

Removing Android instructions as we do not support Android SDK yet.

Jay - Approve


V2.2.0 No Job generation for manually spawned cars

Hey Faux,
I have noticed, that player spawned (Hopper) cars don't have jobs generated for them at the Steelmill.
I have made a couple of tests on this problem:
Test1:
I have F'd myself from the Steelmill to the Sawmill and shunted there for about 10 min. afterwards i F'd myself back to the Steelmill, to find that the cars sitting at the Steelmill had other jobs then before, but the player spawned hoppers were still without jobs.
Test2:
Using Teleport had the whole yard regenerated

log excerp of failure
[PersistentJobsMod] [Error] Exception thrown during UnusedTrainCarDeleter.InstantConditionalDeleteOfUnusedCars prefix patch:
System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
at System.Text.StringBuilder.AppendFormatHelper (System.IFormatProvider provider, System.String format, System.ParamsArray args) [0x000ff] in <567df3e0919241ba98db88bec4c6696f>:0
at System.String.FormatHelper (System.IFormatProvider provider, System.String format, System.ParamsArray args) [0x00023] in <567df3e0919241ba98db88bec4c6696f>:0
at System.String.Format (System.String format, System.Object arg0) [0x00008] in <567df3e0919241ba98db88bec4c6696f>:0
at PersistentJobsMod.Main+UnusedTrainCarDeleter_InstantConditionalDeleteOfUnusedCars_Patch.Prefix (UnusedTrainCarDeleter __instance, System.Collections.Generic.List`1[T] ___unusedTrainCarsMarkedForDelete, System.Collections.Generic.Dictionary`2[TKey,TValue] ___carVisitCheckersMap) [0x001c9] in <533eab387b5849239aaf47c91b80a7b5>:0
[PersistentJobsMod] Restored game coroutine.
[PersistentJobsMod] Inactive.
[PersistentJobsMod] [Critical] Deactivating mod PersistentJobs due to critical failure!
[PersistentJobsMod] [Warning] You can reactivate PersistentJobs by restarting the game, but this failure type likely indicates an incompatibility between the mod and a recent game update. Please search the mod's Github issue tracker for a relevant report. If none is found, please open one. Include the exception message printed above and your game's current build number.

Test3:
Loaded the Hoppers with the cargo load tool, moved to Sawmill, waited 5 min, moved back to Steelmill.
Result: one of two cuts of Hoppers has a Job for it

Thanks for the detailed bug report @JDTiger! It sounds like there's two bugs at play here:

logistical hauls are not being generated for jobless train cars
something in the fast travel code is throwing an exception and causing the mod to become deactivated

The mod becoming deactivated explains why you're seeing the train cars in the yard being regenerated. I'll investigate both of these bugs when I have some free time.


Error running npx wpgulp (Windows)

Hello,
I have been trying to get past the first step, but running npx wpgulp (on Windows) just results in a barrage of errors for me.
(I am more than happy to pay for the time to provide support / or cover the cost of an hour's maintenance).
Please find the first screenshot of running the command, followed by two screenshots of errors:

You can see various versions, of npx, node and npm below:

I've also attached below the error log from \AppData\Local\npm-cache\_logs:
2022-04-19T07_45_20_081Z-debug-0.log
I would very much appreciate some guidance.
Thanks again and kindest regards,
Elliott

I just tried uninstalling Node and reinstalling the LTS version 16.14.2, but it results in the same errors.

I have the same errors. I have uninstalled all node modules deleted all .json files and started fresh so:
npm init -y
npx wpgulp

Bur nothing apart of errors and lot of deprecated packages warnings is this tool still a good way to go in 2022?
I was using @wordpress/scripts before to bundle my files and all was working fine.
Anyone have solved this?
Below is my full error log showing all commands starting from clean slate:
C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite>npm init -y
Wrote to C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\package.json:

{
  "name": "testsite",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/Maryanos/testsite-simple.git"

Installing WPGulp in directory: C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite
This might take a couple of minutes.

√ DOWNLOADED WPGulp files
\ INSTALLING npm packages…
× UNHANDLED ERROR
× ERROR → Error
i REASON → Command failed: npm install
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>This package has been deprecated, please use @wordpress/eslint-plugin or @wordpress/scripts
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated<EMAIL_ADDRESS>this library is no longer supported
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
npm WARN deprecated<EMAIL_ADDRESS>https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated<EMAIL_ADDRESS>request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated<EMAIL_ADDRESS>gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
npm WARN deprecated<EMAIL_ADDRESS>This SVGO version is no longer supported. Upgrade to v2.x.x.
npm WARN deprecated<EMAIL_ADDRESS>This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! code 1
npm ERR! path C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-sass
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node scripts/build.js
npm ERR! Building: C:\Program Files\nodejs\node.exe C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\bin\node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp verb cli [
npm ERR! gyp verb cli   'C:\\Program Files\\nodejs\\node.exe',
npm ERR! gyp verb cli   'C:\\Users\\info\\Local Sites\\testsite\\app\\public\\wp-content\\themes\\testsite\\node_modules\\node-gyp\\bin\\node-gyp.js',
npm ERR! gyp verb cli   'rebuild',
npm ERR! gyp verb cli   '--verbose',
npm ERR! gyp verb cli   '--libsass_ext=',
npm ERR! gyp verb cli   '--libsass_cflags=',
npm ERR! gyp verb cli   '--libsass_ldflags=',
npm ERR! gyp verb cli   '--libsass_library='
npm ERR! gyp verb cli ]
npm ERR! gyp info using<EMAIL_ADDRESS>npm ERR! gyp info using<EMAIL_ADDRESS>| win32 | x64
npm ERR! gyp verb command rebuild []
npm ERR! gyp verb command clean []
npm ERR! gyp verb clean removing "build" directory
npm ERR! gyp verb command configure []
npm ERR! gyp verb check python checking for Python executable "python2" in the PATH
npm ERR! gyp verb `which` failed Error: not found: python2
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21)
npm ERR! gyp verb `which` failed  python2 Error: not found: python2
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21) {
npm ERR! gyp verb `which` failed   code: 'ENOENT'
npm ERR! gyp verb `which` failed }
npm ERR! gyp verb check python checking for Python executable "python" in the PATH
npm ERR! gyp verb `which` failed Error: not found: python
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21)
npm ERR! gyp verb `which` failed  python Error: not found: python
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21) {
npm ERR! gyp verb `which` failed   code: 'ENOENT'
npm ERR! gyp verb `which` failed }
npm ERR! gyp verb could not find "python". checking python launcher
npm ERR! gyp verb could not find "python". guessing location
npm ERR! gyp verb ensuring that file exists: C:\Python27\python.exe
npm ERR! gyp ERR! configure error
npm ERR! gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
npm ERR! gyp ERR! stack     at PythonFinder.failNoPython (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\lib\configure.js:484:19)
npm ERR! gyp ERR! stack     at PythonFinder.<anonymous> (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\lib\configure.js:509:16)
npm ERR! gyp ERR! stack     at callback (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\graceful-fs\polyfills.js:306:20)
npm ERR! gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:198:21)
npm ERR! gyp ERR! System Windows_NT 10.0.19044
npm ERR! gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\info\\Local Sites\\testsite\\app\\public\\wp-content\\themes\\testsite\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
npm ERR! gyp ERR! cwd C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-sass
npm ERR! gyp ERR! node -v v16.14.2
npm ERR! gyp ERR! node-gyp -v v3.8.0
npm ERR! gyp ERR! not ok
npm ERR! Build failed with error code: 1

npm ERR!      C:\Users\info\AppData\Local\npm-cache\_logs\2022-04-21T10_59_35_969Z-debug-0.log

i ERROR STACK ↓
 Error: Command failed: npm install
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>This package has been deprecated, please use @wordpress/eslint-plugin or @wordpress/scripts
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated<EMAIL_ADDRESS>this library is no longer supported
npm WARN deprecated<EMAIL_ADDRESS>See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
npm WARN deprecated<EMAIL_ADDRESS>https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated<EMAIL_ADDRESS>Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated<EMAIL_ADDRESS>request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated<EMAIL_ADDRESS>gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
npm WARN deprecated<EMAIL_ADDRESS>This SVGO version is no longer supported. Upgrade to v2.x.x.
npm WARN deprecated<EMAIL_ADDRESS>This version of tar is no longer supported, and will not receive security updates. Please upgrade asap.
npm ERR! code 1
npm ERR! path C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-sass
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node scripts/build.js
npm ERR! Building: C:\Program Files\nodejs\node.exe C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\bin\node-gyp.js rebuild --verbose --libsass_ext= --libsass_cflags= --libsass_ldflags= --libsass_library=
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp verb cli [
npm ERR! gyp verb cli   'C:\\Program Files\\nodejs\\node.exe',
npm ERR! gyp verb cli   'C:\\Users\\info\\Local Sites\\testsite\\app\\public\\wp-content\\themes\\testsite\\node_modules\\node-gyp\\bin\\node-gyp.js',
npm ERR! gyp verb cli   'rebuild',
npm ERR! gyp verb cli   '--verbose',
npm ERR! gyp verb cli   '--libsass_ext=',
npm ERR! gyp verb cli   '--libsass_cflags=',
npm ERR! gyp verb cli   '--libsass_ldflags=',
npm ERR! gyp verb cli   '--libsass_library='
npm ERR! gyp verb cli ]
npm ERR! gyp info using<EMAIL_ADDRESS>npm ERR! gyp info using<EMAIL_ADDRESS>| win32 | x64
npm ERR! gyp verb command rebuild []
npm ERR! gyp verb command clean []
npm ERR! gyp verb clean removing "build" directory
npm ERR! gyp verb command configure []
npm ERR! gyp verb check python checking for Python executable "python2" in the PATH
npm ERR! gyp verb `which` failed Error: not found: python2
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21)
npm ERR! gyp verb `which` failed  python2 Error: not found: python2
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21) {
npm ERR! gyp verb `which` failed   code: 'ENOENT'
npm ERR! gyp verb `which` failed }
npm ERR! gyp verb check python checking for Python executable "python" in the PATH
npm ERR! gyp verb `which` failed Error: not found: python
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21)
npm ERR! gyp verb `which` failed  python Error: not found: python
npm ERR! gyp verb `which` failed     at getNotFoundError (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:13:12)
npm ERR! gyp verb `which` failed     at F (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:68:19)
npm ERR! gyp verb `which` failed     at E (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:80:29)
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\node_modules\which\which.js:89:16
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\index.js:42:5
npm ERR! gyp verb `which` failed     at C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\isexe\windows.js:36:5
npm ERR! gyp verb `which` failed     at FSReqCallback.oncomplete (node:fs:198:21) {
npm ERR! gyp verb `which` failed   code: 'ENOENT'
npm ERR! gyp verb `which` failed }
npm ERR! gyp verb could not find "python". checking python launcher
npm ERR! gyp verb could not find "python". guessing location
npm ERR! gyp verb ensuring that file exists: C:\Python27\python.exe
npm ERR! gyp ERR! configure error
npm ERR! gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
npm ERR! gyp ERR! stack     at PythonFinder.failNoPython (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\lib\configure.js:484:19)
npm ERR! gyp ERR! stack     at PythonFinder.<anonymous> (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-gyp\lib\configure.js:509:16)
npm ERR! gyp ERR! stack     at callback (C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\graceful-fs\polyfills.js:306:20)
npm ERR! gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:198:21)
npm ERR! gyp ERR! System Windows_NT 10.0.19044
npm ERR! gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\info\\Local Sites\\testsite\\app\\public\\wp-content\\themes\\testsite\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--verbose" "--libsass_ext=" "--libsass_cflags=" "--libsass_ldflags=" "--libsass_library="
npm ERR! gyp ERR! cwd C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite\node_modules\node-sass
npm ERR! gyp ERR! node -v v16.14.2
npm ERR! gyp ERR! node-gyp -v v3.8.0
npm ERR! gyp ERR! not ok
npm ERR! Build failed with error code: 1

npm ERR!      C:\Users\info\AppData\Local\npm-cache\_logs\2022-04-21T10_59_35_969Z-debug-0.log

    at makeError (C:\Users\info\AppData\Local\npm-cache\_npx\5196e799d8ac98da\node_modules\execa\index.js:174:9)
    at C:\Users\info\AppData\Local\npm-cache\_npx\5196e799d8ac98da\node_modules\execa\index.js:278:16
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async C:\Users\info\AppData\Local\npm-cache\_npx\5196e799d8ac98da\node_modules\wpgulp\installer\index.js:58:3

C:\Users\info\Local Sites\testsite\app\public\wp-content\themes\testsite>

Same issue on MacOS.
First tried on 16.15.1 LTS then uninstalled it and installed 18.3.0 — no result.

I have both macOS and Windows and can't get it to work on either, just resorted to my own Gulp.js config for the time-being, but would've loved to switch over to WP Gulp.

Not sure, if it is related, but nobody mentioned it.
It's just a guess, but all the errors about missing Python (like: Error: Can't find Python executable "python", you can set the PYTHON env variable.) may suggest, that Python is missing.
IIRC, gyp is a python script, that helps NPM on Windows to work.
Have you python installed?

Have you python installed?
No actually I did not had Python installed that time and this could be the reason so thanks for pointing that out, but I have decided to not use the Gulp and invest my time in webpack.

I was able to get it to work by uninstalling gulp-sass and replacing it with gulp-dart-sass. (don't forget to update the requirement in the gulpfile too)

Have you python installed?
No actually I did not had Python installed that time and this could be the reason so thanks for pointing that out, but I have decided to not use the Gulp and invest my time in webpack.

Awais's build uses Python2, so you have to configure that first.
I'd prefer not to get my hands dirty with different versions of Python so I'm looking for simpler solutions.

Try using nvm, and node 14.15.3 (Or whatever the best version is there. Works for me!

Thank you @wkjones, glad you got it going. Hope your project goes well! 🎉

I got the same errors when installing, but had issues compiling after following @wkjones steps.
So I went back and ran

npm uninstall gulp-dart-sass
npm install gulp-sass --save-dev
npm install sass --save-dev
In gulpfile.babel.js changed require("gulp-dart-sass") to require("gulp-sass")(require("sass"))

Oddly, there are no issues going back after getting it installed in the first place.

An other solution for the issue. Use nvm to switch between node versions.  Use node 14.15.3  or similar and you will not get any errors.

To be honest, I wish I just rolled my own.
It took far too long to get this working.
It's good now and I've changed the config to compile any scss files rather than just one.
Not sure exactly how the vendor js is supposed to work but it doesn't.

use node version manager (nvm) switch node version 14 it's work for me


create-remix fails to download

What version of Remix are you using?
1.4.3
Steps to Reproduce
npx create-remix@latest
? Where would you like to create your app? ./remix-test1
? What type of app do you want to create? Just the basics
? Where do you want to deploy? Choose Remix if you're unsure; it's easy to change deployment targets. Remix App Server
? Do you want me to run `npm install`? Yes
⠹ Creating your app…🚨 There was a problem fetching the file from GitHub. The request responded with a 404 status. Please try again later.

with a bit of hacking, it's trying to download https://codeload.github.com/remix-run/remix/tar.gz/main and this is failing.
Every other config I've tries also fails.
Expected Behavior
It should create a remix app.
Actual Behavior
It fails at the first download and hasn't installed anything.

Closing this.
I had a GITHUB_TOKEN environment var from our internal github enterprise and it was (not too surprisingly) breaking the downloads.
The only thing that would have made this easier to diagnose would have been if there'd been a 401 in the error rather than a 404.  The 404 made it look like a path issue rather than an auth error.

Looks like this is happening here too: https://github.com/kentcdodds/onewheel-blog/issues/1

Ran into this myself right now:
D:\projects>npx create-remix@latest --debug --verbose
? Where would you like to create your app? ./my-remix-app
? What type of app do you want to create? Just the basics
? Where do you want to deploy? Choose Remix App Server if you're unsure; it's easy to change deployment targets. Remix App Server
? TypeScript or JavaScript? TypeScript
? Do you want me to run `npm install`? Yes
- Creating your app… 🔍  Using the remix template from the remix-run/remix repo
| Creating your app…Error: 🚨 There was a problem fetching the file from GitHub. The request responded with a 404 status. Please try again later.
    at downloadAndExtractTarball (C:\Users\u\AppData\Local\npm-cache\_npx\5164864a48bff686\node_modules\@remix-run\dev\cli\create.js:277:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async downloadAndExtractRepoTarball (C:\Users\u\AppData\Local\npm-cache\_npx\5164864a48bff686\node_modules\@remix-run\dev\cli\create.js:247:12)
    at async Object.createApp (C:\Users\u\AppData\Local\npm-cache\_npx\5164864a48bff686\node_modules\@remix-run\dev\cli\create.js:150:9)
    at async Object.create (C:\Users\u\AppData\Local\npm-cache\_npx\5164864a48bff686\node_modules\@remix-run\dev\cli\commands.js:76:3)
    at async Object.run (C:\Users\u\AppData\Local\npm-cache\_npx\5164864a48bff686\node_modules\@remix-run\dev\cli\run.js:415:9)

Workaround was indeed to remove GITHUB_TOKEN from my environment variables. 👍

@Svish  - how commands in the terminal did you use to remove  the GITHUB_TOKEN environment variable?

@cliffordfajardo I just used the Environment Variables settings dialog in Windows.


Forth does not start

Code compiles, loads but no "ok". This stopped after the addition of camera support. Booting begins:
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:8896
load:0x40080400,len:5816
entry 0x400806ac
.......
But a continuing full points across the screen

I'm assuming this is stale, please reopen if not.


[Live object detection] Added example project to showcase possibility of live object detection

Added example project to showcase possibility of live object detection
This project is based on old and not updated versions of live object detection, but uses latest flutter-tflite and other dependencies.

@VadymPinchuk I ran example from your fork here with custom model which I got here but I got an error

Why this happened?

@fepfitra Thanks for highlighting this. It is just different model which is trained in another way. I am not ML specialist, so I cant assist you here. While for the purpose of example I have used model and input/output configurations from the official sample, which works on a single input. I would suggest to check the model and what kind of input/output is expects.

Hey, added this to my review list. hopefully will find time tomorrow, otherwise early next week :)

@PaulTR  this sample is still not perfect. I would love to make it simpler, but don't want to import additional dependencies. Will be happy to receive feedback and improve it :) next week

@PaulTR  now I am happy of the quality of the code :) Can be reviewed. Thank you

So a couple things with this:
Did you have a chance to test it with iOS? I'm getting the camera feed on my iPad, but not getting the bounding boxes. It also kind of squishes the video feed.
Would it also be possible to support rotating (at least in Android; iOS rotates but doesn't display)?
Thanks!

@PaulTR I can try to run on my work iOS device to check how it works. As well - can improve state management for rotation. I am on it

@PaulTR  I cant run any apps on my work iPhone, so I am not able to test iOS version.
Regarding orientation, I would love to fix it, but till not faced camera rotation issue, which always gives 90 degrees. It will require additional work for covering all the things. If it is legal requirement, I will work on it during the week. Thanks

Thanks for the object detection example. What is the average prediction time running on a single image?

@MichaelHudgins  depends on the image quality and each specific frame. It jumps between 100 to 300 ms for medium quality. But it could be same for High quality as well.

Ok, Ive tried your code with yolov5n and the inference time is around 10 times higher. Regarding the input size of 640x640 it still is way to high. After that I validated my implementation by swapping out the models with the ssd_mobilnet and I got an inference time of 150 ms. Since the inference code is just _interpreter.run() idk where the problem could be. Do you have any idea what I can try or check @VadymPinchuk?

@MichaelRinger Do you have a link for the model? Out of curiosity I'd want to run it through the Android Object Detection sample for MediaPipe to see if that helps isolate it to this repo vs the model.

@MichaelRinger do you use custom model? Could you show me the way you train it?

@PaulTR It is the default pretrained model from the ultralytics repository. The model is in pytorch's format by default. I converted it via the in the repository available export.py script to tflite by running python export.py --weights yolov5n.pt --include tflite.
yolov5n.pt model weights:
https://github.com/ultralytics/yolov5/releases/download/v7.0/yolov5n.pt

@PaulTR, the inference of ayolov8s model also takes about 1.6 seconds, rendering it virtually unusable. Furthermore, due to the yolo models' demand for normalized pixel values, the preprocessing time is also extended.

@MichaelRinger Alright let's go ahead and move this into an independent issue that can be tracked. Model is missing metadata for other sample apps I have (tflite, mediapipe) so I'd need to put something custom together to test this model against an existing known platform baseline.

Hey @PaulTR

I cant run any apps on my work iPhone, so I am not able to test iOS version.
Regarding orientation, I would love to fix it, but till not faced camera rotation issue, which always gives 90 degrees.
It will require additional work for covering all the things.  (now I remembered why I was saving into a file before the detection (some time ago) because it gives ability to read rotation from exif headers)

If it is legal requirement, I will work on it during the week. Thanks

Sorry I definitely missed this response!

I think the iOS version is a major requirement. I don't see any border or classification UI coming up regardless of the orientation. Maybe someone else can jump in on this though?
I'm less worried about orientation. It's a nice to have and something we require from our official samples on MediaPipe, but this is a community project and not "hey let's make sure this is absolutely done" :P I think the image classification live stream sample that was just updated (PR is still open) might cover this FWIW.

Other than that I think this is awesome on Android! Looking forward to getting it merged in as an example for everyone.

I confirm on my iPhone 11 iOS 16.5.1 the rectangles are not showing. Another problems is that the camera view is not respecting the proportions, everything looks wider. Another issue is that if I try with android simulator a yolov5.tflite model the bottom stats disappear and the model does not detect anything. Thank you anyways, this look very promising, but I am a beginner and I am afraid I cannot help so much.

@PaulTR  I have asked my friend to look into it :) he is iOS eng initially. So definitely have iOS device. Thanks for flagging. Will look into it

Latest update on this:  WIP
Not very obvious thing, which happened in iOS is conversion of CameraImage to image.Image: convertBGRA8888ToImage - this method worked bad, and resulting image is of very poor quality. My friend is doing his best to make it right for iOS.

@PaulTR  we are happy to give an update - iOS version is fixed now. Please check and give a green/red light. Thank you

Nice! So I am seeing some weirdness with the camera->scaling for the model->rescaling for display in iOS.
In portrait I'm seeing a shift down and to the right:

I think that's the only thing that'd be worth fixing, otherwise I think we're golden :)

@PaulTR also fun thing to see - iOS prediction is faster. As everything else. Comparing to my Pixel 7.

Oh interesting, I didn't think about it with Android tablets, too. For native Android we have a scale ratio that we pull from the device and use for scaling down (to 224x224) and then back up. Using a 5th gen iPad Air running 16.5.1

@VadymPinchuk I ran example from your fork here with custom model which I got here but I got an error

I've changed the output shape by rearrange it to match to my model. But, I got negative value of bounding box.

@PaulTR I have checked on iPad, changed renderBox a bit. Now it works fine on Pixel, iPhone, iPad (what I was able to check). Please verify on yours :)

@PaulTR  we are ready :) when you have time

Thanks! I'll check it out Monday :)
On Fri, Jul 28, 2023, 11:11 AM Vadym Pinchuk @.***>
wrote:

@PaulTR https://github.com/PaulTR we are ready :) when you have time
—
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/flutter-tflite/pull/107#issuecomment-1656030546,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAR2C3ZZ2HRCYHY6QMDV55TXSPXCNANCNFSM6AAAAAA2GTNCRM
.
You are receiving this because you were mentioned.Message ID:
@.***>

@MichaelRinger any idea how to use yolov8 here? If I just change the model then I get no labels and no stats about times.

@codscino Here is a issue that has the test code for running yolov8. Unfortunatelly its really slow with this package and not really usable for real time applications.

@VadymPinchuk Any idea how to integrate yolo v8 in this amazing example? Here is a working repo https://github.com/codscino/yolo_tflite, but it works only with photo in assets folder and it is pretty slow.

@codscino sorry but I am not ML specialist, so from my perspective it is a black box, which takes some input and return some output :) I can spend some time with that example, but it would not be fast. While with static images almost every model should work, as we dont need any realtime update on the screen, and slow/fast models could be applied.

@VadymPinchuk thanks for the quick reply. I think it is more a problem of input e output formatting between ssd and yolo. I will try to investigate as well. I thought you had some insights by just seeing my super simple example. Btw thanks for this PR, it is super helpful.

@codscino

change model name in model path
change image resize to 640x640
check if input image pixels need to be normalized or not
change output shape to [1,84,8400] example:

final output = [List<List<double>>
          .filled(84, List<double>
          .filled(8400, 0.0))],

change interpretation of the output in the post process part to get bounding boxes cause mobile_net model returns [1, 1001] shape, so it have different data

and I think it should be it

@ArdeoDeo thanks a lot. Tomorrow I will try. Do you think is it possible to downsize to 320x320 for performance reasons?

@codscino Yolo model input is 640x640 so it need to be that size

LGTM! Thanks for getting this together. One small issue I ran into on iOS for delegates, but it can be addressed in a separate PR (I'm guessing we can just default to the CPU delegate instead of metal and call it good for an example use-case.) There's also a few errors in here about orientation and image format that can be addressed, but they're not show-stoppers. I just had to really wait for iOS to build and run.
2023-07-31 11:45:42.050505-0600 Runner[1579:440604] Metal GPU Frame Capture Enabled
2023-07-31 11:45:42.050768-0600 Runner[1579:440604] Metal API Validation Enabled
2023-07-31 11:45:42.269460-0600 Runner[1579:440604] [VERBOSE-2:FlutterDarwinContextMetalImpeller.mm(35)] Using the Impeller rendering backend.
2023-07-31 11:45:43.165071-0600 Runner[1579:441154] flutter: The Dart VM service is listening on http://<IP_ADDRESS>:61699/y8gue7ETImM=/
2023-07-31 11:45:43.307867-0600 Runner[1579:440604] Failed to change device orientation: Error Domain=UISceneErrorDomain Code=101 "The current windowing mode does not allow for programmatic changes to interface orientation." UserInfo={NSLocalizedDescription=The current windowing mode does not allow for programmatic changes to interface orientation.}
2023-07-31 11:45:44.110545-0600 Runner[1579:441126] Created TensorFlow Lite delegate for Metal.
INFO: Created TensorFlow Lite delegate for Metal.
2023-07-31 11:45:44.117229-0600 Runner[1579:441150] The selected imageFormatGroup is not supported by iOS. Defaulting to brga8888
2023-07-31 11:45:44.131268-0600 Runner[1579:441126] Initialized TensorFlow Lite runtime.
INFO: Initialized TensorFlow Lite runtime.
2023-07-31 11:45:44.133896-0600 Runner[1579:441126] Following operations are not supported by GPU delegate:
CUSTOM TFLite_Detection_PostProcess: TFLite_Detection_PostProcess
63 operations will run on the GPU, and the remaining 1 operations will run on the CPU.
ERROR: Following operations are not supported by GPU delegate:
CUSTOM TFLite_Detection_PostProcess: TFLite_Detection_PostProcess
63 operations will run on the GPU, and the remaining 1 operations will run on the CPU.

@PaulTR I can check these issues and make an update in a separate PR later today-tomorrow.


Metaclasses

Are there any plans to support metaclasses?
Codon gives syntax error when using class Klass(metaclass=...)

Can't speak for what codon does or will support.
This is general conversation about Python metaclass.
Recent change occurred, which depreciates metaclass in favor of init_subclass
So the codon devs can get away with not supporting metaclasses in favor of init_subclass
Metaclasses run into metaclass conflicts that are resolvable, but the syntax is awkward and non-intuitive.

Not yet supported; please see #606  for the current progress.


Remove --push-gateway from ghproxy deployment

We don't currently run an instance of pushgateway
If/when we want to:   https://github.com/kubernetes/test-infra/blob/master/config/prow/cluster/pushgateway_deployment.yaml
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Please update the version of the Chart.yaml

Done

Le thank you
/lgtm


Common Strings i18n

Summary
We need to move the CommonStrings service from @clr/angular to @clr/core to better support i18n. This will create a dependency between @clr/angular and @clr/core.
https://github.com/vmware/clarity/pull/3920

merged https://github.com/vmware/clarity/pull/3920


web: add LogPane component to HUD app

There's a bug here where it looks like the logs are scrolling to the right instead of top down, will investigate.

In the spirit of prototype and with the knowledge that this is exactly the problem that I am bad at solving I'm going to merge this with one outstanding problem that I'd like help with.

The logs are showing up and streaming just fine afaict, but they're streaming vertically instead of horizontally. Any idea what might be causing this? My visual diff of the CSS rules that got applied to those divs seemed identical but the problem might lie elsewhere.


How to send alert to the log file from other timezone?

I have aggregate logs from two different timezones to a single index with frequency rule,
but only the leading timezone will fire the alert.
When I test the rule, there are hits for the logs in different timezones.
How can I do to resolve the issue?

Solved!
I created multiple rules and use logstash and converted the timezone to local timezone.
Thanks~


Call between APIs with different scopes with shared access token

It is possible to have something like this? How to share the access token between two APIs with different scopes?

docs: https://identityserver4.readthedocs.io/en/release/topics/extension_grants.html


Concat problem

Bug in concatenation.
Buggy example with some comment is in test-integration/src/trim.cc.

This could be closed (fixed in #348 )

We will close this PR and create a test from tests-integration/src/trim.cc to check that the error is fixed.


Session Load/Save Hooks

Greetings @glepnir,
Thank you for work on this amazing plugin. It modernizes the neovim setup quite much. Is it possible to embed in session load/save's like pre and post commands?
I want to close out empty buffers whenever I open a session because I sometimes have leftover directory buffer or some random empty buffers on the session. Is this something possible or am I missing something that is already implemented in?

yup can support it.

Greetings @glepnir,
Thank you for work on this amazing plugin. It modernizes the neovim setup quite much. Is it possible to embed in session load/save's like pre and post commands?
I want to close out empty buffers whenever I open a session because I sometimes have leftover directory buffer or some random empty buffers on the session. Is this something possible or am I missing something that is already implemented in?

set sessionoptions-=blank will prevent blank buffers from being saved in sessions

You are back, cool! There were some messages including you and the hospital so hope you are doing fine!

Thanks ,Looks good so far, the surgery was a success, but I'm still on medication for a long time


Need to validate equivalence ratio in ignition delay cases

Need to use the given composition for the ignition delay data point to calculate the equivalence ratio and validate that the given value (if present) is correct.
To do so, need to convert InChI or SMILES to molecular formula. Can use a regex to get molecular formula from InChI, but SMILES will be a little harder. @rwest is thinking about this

Related to #23

This turns out to be not so easy... for instance, what is a fuel and what is an oxidizer? I don't want to force people to include specifications saying "This is a fuel, this is an oxidizer"


add new topic dwd (Deutscher Wetterdienst)

Please confirm this pull request meets the following requirements:

[x] I followed the contributing guidelines: https://github.com/github/explore/blob/master/CONTRIBUTING.md.
[x] I have no affiliation with the project I am suggesting (as a maintainer, creator, contractor, or employee).

Which change are you proposing?

[ ] Suggesting edits to an existing topic or collection
[x] Curating a new topic or collection
[ ] Something that does not neatly fit into the binary options above

Curating a new topic or collection

[x] I've formatted my changes as a new folder directory, named for the topic or collection as it appears in the URL on GitHub (e.g. https://github.com/topics/[NAME] or https://github.com/collections/[NAME])
[x] My folder contains a *.png image (if applicable) and index.md
[x] All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

I'd like to add a few details concerning the Deutscher Wetterdienst (@DeutscherWetterdienst) (DWD for short) to increase the visibility of itself and projects using it's data. The DWD is Germany's NOAA of sorts. I'd like to highlight that I'm not affliated with the DWD directly in any way. I'm only using their open data frequently.

I did not include the logo of DWD as an image for this topic as I'm not aware of the copyright policy of DWD concerning their trademarks.

would love to see this integrated!

@smashwilson thank you for your feedback 👍  I integrated it seconds ago

should be ready to merge by now


Test environment for authenticated-HTTP data acces

With @zonia3000, we prepared a testing environment for HTTP data access that requires BasicAuth (i.e. username and password) authentication.

Prepare the following Dockerfile:

FROM ghcr.io/fractal-analytics-platform/fractal-vizarr-viewer:stable

ENV CACHE_EXPIRATION_TIME=0
ENV FRACTAL_SERVER_URL=http://fake:8000
ENV ZARR_DATA_BASE_PATH=/zarr-files/
ENV AUTHORIZATION_SCHEME=testing-basic-auth

RUN apt install -y wget unzip

WORKDIR /zarr-files

RUN wget https://zenodo.org/records/10424292/files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1
RUN unzip 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1
RUN rm -rf __MACOSX 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1

WORKDIR /

Prepare the following build_and_run.sh script

#!/bin/bash

IMAGENAME=serve-zarr-behind-basic-auth

docker build . -t "$IMAGENAME"
docker run --network host -e TESTING_USERNAME=myusername -e TESTING_PASSWORD=mypassword "$IMAGENAME"

Run build_and_run.sh:

$ ./build_and_run.sh

[+] Building 0.2s (10/10) FINISHED                                                                                docker:default
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 616B                                                                                0.0s
 => [internal] load metadata for ghcr.io/fractal-analytics-platform/fractal-vizarr-viewer:stable                                                                                0.2s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => [1/7] FROM ghcr.io/fractal-analytics-platform/fractal-vizarr-viewer:stable@sha256:79f332cbc5f4e7a9cfa62dc28ab33a0a7026b60fdbcf0fc0f16e7e5a0c19cf3f                                                                                0.0s
 => CACHED [2/7] RUN apt install -y wget unzip                                                                                0.0s
 => CACHED [3/7] WORKDIR /zarr-files                                                                                0.0s
 => CACHED [4/7] RUN wget https://zenodo.org/records/10424292/files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1                                                                                0.0s
 => CACHED [5/7] RUN unzip 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1                                                                                0.0s
 => CACHED [6/7] RUN rm -rf __MACOSX 20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip?download=1                                                                                0.0s
 => exporting to image                                                                                0.0s
 => => exporting layers                                                                                0.0s
 => => writing image sha256:a88314ced7a342f62d00ace861b10fc9d66e6381307626c3a7de1d1116780d6e                                                                                0.0s
 => => naming to docker.io/library/serve-zarr-behind-basic-auth                                                                                0.0s
2024-11-06 15:06:40,154 - WARN - Authorization scheme is set to "testing-basic-auth". Do not use in production!
2024-11-06 15:06:40,156 - INFO - fractal-vizarr-viewer is listening at http://localhost:3000/vizarr/

At this point, you can access a file if you use the myusername/mypassword credentials (as an example, try accessing http://localhost:3000/vizarr/data/zarr-files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/0/.zarray in the browser).

Prepare the following test_http_access.py script

import fsspec.implementations.http
import zarr
import aiohttp

url = (
    "http://localhost:3000/vizarr/data/zarr-files/"
    "20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/0"
)
print(f"Zarr array available at {url=}\n")

print("WITH CREDENTIALS:")
fs = fsspec.implementations.http.HTTPFileSystem(
    client_kwargs=dict(auth=aiohttp.BasicAuth("myusername", "mypassword"))
)
store = fs.get_mapper(url)
array_zarr = zarr.open_array(store)
print(f"{array_zarr=}\n")

print("WITHOUT CREDENTIALS")
fs = fsspec.implementations.http.HTTPFileSystem()
store = fs.get_mapper(url)
array_zarr = zarr.open_array(store)
print(f"{array_zarr=}")

Run test_http_access.py in an appropriate virtual environment:

python test_http_access.py

Zarr array available at url='http://localhost:3000/vizarr/data/zarr-files/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/0'

WITH CREDENTIALS:
array_zarr=<zarr.core.Array (3, 1, 2160, 5120) uint16>

WITHOUT CREDENTIALS
Traceback (most recent call last):
  File "/home/tommaso/Fractal/http-zarr/open_remote_zarr_docker.py", line 24, in <module>
    array_zarr = zarr.open_array(store)
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/creation.py", line 649, in open_array
    init_array(
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/storage.py", line 455, in init_array
    _init_array_metadata(
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/storage.py", line 536, in _init_array_metadata
    shape = normalize_shape(shape) + dtype.shape
  File "/home/tommaso/Fractal/http-zarr/venv/lib/python3.10/site-packages/zarr/util.py", line 83, in normalize_shape
    raise TypeError("shape is None")
TypeError: shape is None

Very cool!
For the moment, I am using a public ome-zarr on github. But yeah, the tiny-http-server seems to be a better solution for the normal CIs (plus, I can also test authentication).
The docker CI might be a bit overkill for the current state of ngio, but at some point it would make so much sense to use it for larger integration runs (maybe weekly scheduled actions).


Basic authentication experiment

We are looking forward to collaborative authoring, and the first barrier toward that goal is user authentication. Once we can make user accounts with designated permissions, we can slowly build out commenting and editing capabilities on the site.
Experiment with:

[x] Investigate Amplify Authentication vs plain AWS Cognito user pool
[x] Make test user on new user pool
[x] Use Amplify Auth React component for login

Next steps:

Split the data access layer into two Lambda's, one for reading that's public and one for writing that's only accessible to authenticated users. This is the easiest way to break up access without implementing our own authentication checking within the GraphQL server code.
Hook up new login-only lambda to the front-end and try to make an edit.

After lots of investigation, the solution we can start with consists of the following components.

AWS Cognito User Pool, where users are registered along with their login information.
A single AWS Lambda function with two API Gateway endpoints, one at /api/graphql which allows unauthorized access and another at /api/graphql-edit which requires authorization.
Users login using the Amplify React components on the website, which will generate credentials that we pass along to /api/graphql-edit either with cookies or with the Authorization header.
The GraphQL server code running in the lambda reads the given user credentials (if any), and determines whether each GraphQL query can be executed based on the groups/tags/etc of the authenticated user.


Support resources waiting for finalizers to delete

Do something similar to  https://github.com/ibuildthecloud/finalizers

can I be assigned to this issue?

@Yuni-sa Take into account this https://github.com/yonahd/kor/issues/38
They might be slightly conflicting


Some geometry appearing in the wrong place

Hi Amphos,
thanks a lot for your great work on the Jupiter EX importer, it was really useful in learning more about the maps from FEAR!
I have one issue though: About 80% of the geometry imports properly, but around 20% is somehow "stuck" in a section around the map origin (0, 0, 0). It seems that it's mostly the dynamic objects, but also some sprites like graffiti. I checked the face/vertex coordinates in the .world00p file, they are really given like that, i.e. placed in the origin instead of at their proper location.
I also noticed that for some of those, only a single instance appears in the maps (both in the BSP and the RenderSurfaces section), although there should be many scattered around the level. For example, there is just 1 paint_can (per color) in the map, whereas they appear all over the place in shelves (in the "Intro" level). I would assume that there is some structure that "places" these mesh instances somewhere, but I couldn't find it.
In fact I spent the last couple of days searching the world file up and down with the help of your .bt template (thanks also for that!), but I wasn't successfull looking e.g. in the render tree or sectors. The only place I did find the individual instances is in the "Objects" section, where they are usually of type "WorldModel" with their proper position and rotation. BUT here the thing is that this WorldModel contains no link to a model, material or RenderSurface, so I'm kind of stuck again.
Since you appear to be the only expert around on Jupiter EX, I was hoping you might have encountered the issue too, or maybe have an idea how the heck the world files store the proper positioning info. :-)
Thanks a lot in advance!

Centering dynamic objects around 0,0,0 is necessary for correct transforms when assigned to a dynamic object instance. The plugin supports limited dynamic objects if you tick Import Objects; it will create empties at the correct positions and set them as parent to the relevant meshes to move them there.
I haven't gotten around to the next step of that where it would instance a shared mesh to multiple empties. You can see the relevant code in WorldObjects.py, but it's extremely bare at the moment.

The WorldObjects part I found, but ok now I see there can be multiple names per WorldModel/BSP, I just overwrote the name each time, so each BSP had just one name assigned. :-/
But still, now I have instances of the WorldModels, but not the RenderMeshes, the WorldModels just being collision geometry, you don't want to draw those. Do you by any chance know how to get from a BSP to the RenderMesh so I know which materials to use at those instance positions? The RenderMeshes don't seem to have names...

They're in sequential order and should be 1:1

Hm BSP and render faces count are quite different, but I think I'm getting the idea... I need to look at the RenderTree structure, right? Thanks, that helped a lot!! 😊

Yeah, there's still a lot left to solve about the render surfaces (even worse of in the Loki variant). I'm not focused on this right now, but if you figure anything out be sure to share it

I will :-) I've been working my way up from LT1, the objects haven't changed so much, but render structures did. Regarding sectors, there's a good article on lighting in the SDK docs (public tools v2) that explains a bit.
Will probably also look into Loki and possibly Yeti and some point.


Trouble using connector with Docker and AWS hosted ElasticSearch

Hello,
I'm trying to set up a search engine for my company's product with AWS hosted ElasticSearch. Our entire backend is containerized, so I intended to set up another container to run Mongo Connector. I'm basing my container off of yeasy's work here. I'm able to build the image and run the container, but I keep getting the following error:
/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py:70: UserWarning: Connecting to search-elastic-test-5d7q73lce7j53trxbq4fcsybyy.us-west-2.es.amazonaws.com using SSL with verify_certs=False is insecure.
'Connecting to %s using SSL with verify_certs=False is insecure.' % host)
From what I can gather, this happens when the urllib3 library does not have verify_certs set to true, but when I looked through the code for elastic2_doc_manager I saw that if aws is passed into the args parameter of the config file, it should set it to true. I've been looking through the source code but I haven't been able to find the exact source of the bug. All my code, including the Dockerfile and mongo-connector config, can be found here. Thanks!

Can you post the full stack trace in addition to the elastic2-doc-manager you're using?  If 'aws' is present it should indeed set verify_certs=True and the connection_class=RequestsHttpConnection but you're seeing verify_certs=False and getting an error from  Urllib3HttpConnection.

/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py:70: UserWarning: Connecting to search-elastic-test-5d7q73lce7j53trxbq4fcsybyy.us-west-2.es.amazonaws.com using SSL with verify_certs=False is insecure.
'Connecting to %s using SSL with verify_certs=False is insecure.' % host)
2016-10-31 22:19:31,015 [INFO] mongo_connector.connector:1062 - Beginning Mongo Connector
2016-10-31 22:19:31,017 [WARNING] mongo_connector.connector:116 - MongoConnector: Can't find oplog.timestamp, attempting to create an empty progress log
2016-10-31 22:19:31,019 [INFO] mongo_connector.connector:224 - MongoConnector: Empty oplog progress file.
2016-10-31 22:19:31,088 [INFO] mongo_connector.oplog_manager:92 - OplogThread: Initializing oplog thread
2016-10-31 22:19:31,119 [INFO] mongo_connector.connector:296 - MongoConnector: Starting connection thread MongoClient(host=['mongodb.dev.goalbook.local:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='singleNodeRepl')
2016-10-31 22:19:31,121 [DEBUG] mongo_connector.oplog_manager:196 - OplogThread: Run thread started
2016-10-31 22:19:31,121 [DEBUG] mongo_connector.oplog_manager:198 - OplogThread: Getting cursor
2016-10-31 22:19:31,121 [DEBUG] mongo_connector.oplog_manager:796 - OplogThread: reading last checkpoint as None
2016-10-31 22:19:31,212 [DEBUG] mongo_connector.oplog_manager:659 - OplogThread: Newest oplog entry has timestamp<PHONE_NUMBER>.
2016-10-31 22:19:31,212 [DEBUG] mongo_connector.oplog_manager:488 - OplogThread: Dumping set of collections ['atlas']
2016-10-31 22:19:31,212 [DEBUG] mongo_connector.oplog_manager:581 - OplogThread: Using bulk upsert function for collection dump
2016-10-31 22:19:31,212 [CRITICAL] mongo_connector.oplog_manager:630 - Exception during collection dump
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/mongo_connector/oplog_manager.py", line 583, in do_dump
upsert_all(dm)
File "/usr/local/lib/python3.6/site-packages/mongo_connector/oplog_manager.py", line 567, in upsert_all
dm.bulk_upsert(docs_to_dump(namespace), mapped_ns, long_ts)
File "/usr/local/lib/python3.6/site-packages/mongo_connector/util.py", line 32, in wrapped
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 229, in bulk_upsert
for ok, resp in responses:
File "/usr/local/lib/python3.6/site-packages/elasticsearch/helpers/init.py", line 161, in streaming_bulk
for bulk_actions in _chunk_actions(actions, chunk_size, max_chunk_bytes, client.transport.serializer):
File "/usr/local/lib/python3.6/site-packages/elasticsearch/helpers/init.py", line 55, in _chunk_actions
for action, data in actions:
File "/usr/local/lib/python3.6/site-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 195, in docs_to_upsert
for doc in docs:
File "/usr/local/lib/python3.6/site-packages/mongo_connector/oplog_manager.py", line 509, in docs_to_dump
database, coll = namespace.split('.', 1)
ValueError: not enough values to unpack (expected 2, got 1)
2016-10-31 22:19:31,226 [ERROR] mongo_connector.oplog_manager:638 - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient(host=['mongodb.dev.goalbook.local:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='singleNodeRepl'), 'local'), 'oplog.rs')
2016-10-31 22:19:31,226 [DEBUG] mongo_connector.oplog_manager:210 - OplogThread: Last entry is the one we already processed.  Up to date.  Sleeping.
2016-10-31 22:19:32,123 [ERROR] mongo_connector.connector:304 - MongoConnector: OplogThread <OplogThread(Thread-2, started<PHONE_NUMBER>66208)> unexpectedly stopped! Shutting down
2016-10-31 22:19:32,123 [INFO] mongo_connector.connector:362 - MongoConnector: Stopping all OplogThreads
2016-10-31 22:19:32,123 [DEBUG] mongo_connector.oplog_manager:377 - OplogThread: exiting due to join call.
I'm using the latest elastic2_doc_manager; the container runs pip install elastic2-doc-manager[aws] when building the image.

I still not sure about the insecure connection warning but your namespace of atlas is causing mongo-connector to fail. A namespace must be databasename.collectionname, see the namespaces include option.
Can you test if this issue is present in elastic2-doc-manager master? Install with:
pip install https://github.com/mongodb-labs/elastic2-doc-manager/archive/master.zip

I changed it to elastic2-doc-manager master. The warning is still appearing, but there's now an unauthorized error occurring. I included my AWS keys in the file, so I'm not sure what's happening there. Stack trace:
/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py:70: UserWarning: Connecting to search-elastic-test-5d7q73lce7j53trxbq4fcsybyy.us-west-2.es.amazonaws.com using SSL with verify_certs=False is insecure.
'Connecting to %s using SSL with verify_certs=False is insecure.' % host)
/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py:841: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
GET /_nodes/_all/clear [status:403 request:0.308s]
Fatal Exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/mongo_connector/util.py", line 90, in wrapped
func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/mongo_connector/connector.py", line 1059, in main
conf.parse_args()
File "/usr/local/lib/python3.6/site-packages/mongo_connector/config.py", line 118, in parse_args
option, dict((k, values.get(k)) for k in option.cli_names))
File "/usr/local/lib/python3.6/site-packages/mongo_connector/connector.py", line 854, in apply_doc_managers
dm_instances.append(DocManager(target_url, **kwargs))
File "/usr/local/lib/python3.6/site-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 114, in init
self.elastic = Elasticsearch(hosts=url, **client_options)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/init.py", line 169, in init
self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 126, in init
self.sniff_hosts(True)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 250, in sniff_hosts
node_info = self._get_sniff_data(initial)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 207, in _get_sniff_data
timeout=self.sniff_timeout if not initial else None)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 113, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.AuthorizationException: TransportError(403, '{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: elastic-test"}')
Traceback (most recent call last):
File "/usr/local/bin/mongo-connector", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/mongo_connector/util.py", line 90, in wrapped
func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/mongo_connector/connector.py", line 1059, in main
conf.parse_args()
File "/usr/local/lib/python3.6/site-packages/mongo_connector/config.py", line 118, in parse_args
option, dict((k, values.get(k)) for k in option.cli_names))
File "/usr/local/lib/python3.6/site-packages/mongo_connector/connector.py", line 854, in apply_doc_managers
dm_instances.append(DocManager(target_url, **kwargs))
File "/usr/local/lib/python3.6/site-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 114, in init
self.elastic = Elasticsearch(hosts=url, **client_options)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/client/init.py", line 169, in init
self.transport = transport_class(_normalize_hosts(hosts), **kwargs)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 126, in init
self.sniff_hosts(True)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 250, in sniff_hosts
node_info = self._get_sniff_data(initial)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/transport.py", line 207, in _get_sniff_data
timeout=self.sniff_timeout if not initial else None)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 109, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 113, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.AuthorizationException: TransportError(403, '{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: elastic-test"}')

Aha! You're running into this bug #516. Can you try out mongo-connector master:
pip install --upgrade https://github.com/mongodb-labs/mongo-connector/archive/master.zip
Can you also run this under Python 2.7? The library we're using requests-aws-sign is broken on Python 3: https://github.com/jmenga/requests-aws-sign/pull/1
Thanks for finding this!

I've made the changes you've suggested, but there seems to be an error caused by python 2.7. Stack trace:
Traceback (most recent call last):
File "/usr/local/bin/mongo-connector", line 11, in
load_entry_point('mongo-connector==2.5.0.dev0', 'console_scripts', 'mongo-connector')()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 564, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 2608, in load_entry_point
return ep.load()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 2268, in load
return self.resolve()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 2274, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python2.7/site-packages/mongo_connector/connector.py", line 28, in
from mongo_connector import config, constants, errors, util
File "/usr/local/lib/python2.7/site-packages/mongo_connector/config.py", line 19, in
from mongo_connector import compat, errors, version
ImportError: cannot import name version
I've updated the repo I linked to earlier to reflect my current image setup.

Looks like you're finding all of our bugs! I opened #582 to fix this.

Thanks for all the help Shane! Mongo-connector is working locally for me now. My image currently installs the mongo-connector master - do you know when the current version will be available from pip install mongo-connector?


Data is handled and output in a Tidy data format

All ingestion (raw) and processing (interim) functions can be developed as generic functions (read csv, load SQL table, check schema, and so on see NHSX data eng templates). Only the analysis/processed stage is then using custom code to keep things simple.

Much prefer DE terminology (ingest, transform, serve) as it more accurately describes what the code does at each stage, rather than just what state the data is in.


Support for new iTunes tags

[x] <itunes:type>

[x] db migration
[x] pull in importer
[x] new ui
[x] edit ui
[x] generate in feed

[x] <itunes:season>

[x] pull in importer
[x] new ui
[x] edit ui
[x] generate in feed

[x] <itunes:episode>

[x] pull in importer
[x] new ui
[x] edit ui
[x] generate in feed

[x] <itunes:title>

[x] pull instead of title in feed
[x] generate in feed
[x] set <title> to include generated name

[x] <itunes:episodeType>

[x] db migration
[x] pull in importer
[x] new ui
[x] edit ui
[x] generate in feed

[ ] <content:encoded>

[x] pull in importer in lieu of description
[ ] generate in feed when we can detect support
[ ] update sanitizer to use <p>, <ol>, <ul>, <a>, <em>, <i>, <b>

Skipping content:encoded for now, as it's unclear how it will regress existing content.


fix multiple assignment

Fix for Issue #12,
cannot assign *docker.DockerInfo to cd.dockerInfoData (type *docker.Env) in multiple assignment

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Create 3 Getting Started Manual bad Python Web Playground link

In the "Create 3 Getting Started Manual", on page 6 there is a link listed for Python Web Playground titled "Visit the iRobot Education Learning Library". However, when clicking on the link it produces a 404 Error:
Oops! We can't seem to find the page you're looking for.

Link that fails:
https://edu.irobot.com/learning-library/create-3-python-web-playground
Perhaps this is the correct link:
https://edu.irobot.com/learning-library/create-3-web-playground-for-python

@jomoengineer Thanks for the report! We'll fix it soon.


x/tools/gopls: add source test for inlay hints

Track progress of source tests for inlay hints.

Hey @suzmue 👋 , would like to help out with this issue. Would be great if you can please provide some more context regarding what needs to be done here. Thanks! 🙌

With the addition of features that are exposed through inlay hints, we may want to add source tests for these features (code pointer here).
These tests would need to call the source package's implementation here.
The lsp tests currently take the approach of getting the inlay hints for the whole file and then applying them to the document as if they were text edits. It will probably require some additional set up to be able to track what inlay hints we are expecting from these source tests.


⚠ Bofrost.be has degraded performance

In c39dd3d, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 2072 ms

Resolved: Bofrost.be performance has improved in 1238a82 after 13 minutes.


Run nvidia-gpu-exporter in k8s an error

1、use nvidia-gpu-exporter 0.3.0 images
this is error：
in nvidia-gpu-exporter container  input nvidia-smi  then output error：
/tmp/cuda-control/src/register.c: 66 can't register to manager, error No such file or directory
/tmp/cuda-control/src/register.c: 87 rpc client exit with 255

Please try the latest release and make sure you follow the steps described here: https://github.com/utkuozdemir/nvidia_gpu_exporter/blob/master/INSTALL.md#running-in-kubernetes
I'll close this issue since it is more about getting nvidia-smi to work in a k8s container - it is out of scope for the exporter, as every k8s setup/linux distro could potentially require different configurations.
My suggestion is to try to get nvidia-smi --query-gpu ... working in a pod in your k8s cluster, and when you get it working, looking into adapting it by taking the official helm chart as reference.
If you have any further findings, please feel free to share.


[Snyk] Security upgrade python from 3.9-slim to 3.13.0a6-slim

This PR was automatically created by Snyk using the credentials of a real user.Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Changes included in this PR

cmd/suggestion/chocolate/v1beta1/Dockerfile

We recommend upgrading to python:3.13.0a6-slim, as this image has only 50 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Some of the most important vulnerabilities in your base image include:

Severity
Priority Score / 1000
Issue
Exploit Maturity

****
CVE-2024-28085 SNYK-DEBIAN12-UTILLINUX-6508631
No Known Exploit

****
CVE-2024-28085 SNYK-DEBIAN12-UTILLINUX-6508631
No Known Exploit

****
CVE-2024-28085 SNYK-DEBIAN12-UTILLINUX-6508631
No Known Exploit

****
CVE-2024-28085 SNYK-DEBIAN12-UTILLINUX-6508631
No Known Exploit

500
Integer Overflow or Wraparound SNYK-DEBIAN12-ZLIB-6008963
No Known Exploit

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.09%) to 72.951%

Totals

Change from base Build<PHONE_NUMBER>:
0.09%

Covered Lines:
3115

Relevant Lines:
4270

💛  - Coveralls

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.09%) to 72.951%

Totals

Change from base Build<PHONE_NUMBER>:
0.09%

Covered Lines:
3115

Relevant Lines:
4270

💛  - Coveralls


qwik-solid integration (WIP)

What is it?

[x] Feature / enhancement
[ ] Bug
[ ] Docs / tests

Description
Qwikifying Solid.
Use cases and why

One use case

Another use case

Checklist:

[ ] My code follows the developer guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] Added new tests to cover the fix / functionality

Hello, is there any ETA for this integration? I am eagerly looking forward to it.

hi @voluntadpear thanks for your commitment and for this amazing piece of work.
The best option for this kind of integration is:

Have a community organization

Have a https://github.com/QwikDev organization that can publish to @qwikdev/____ The nice part of the org is that it still has the "seal" of Qwik without passing on the cost to the Qwik core maintainers.
This way, the core Qwik team is not on the hook to support the feature long time (as most of the time, they are not the experts)
Qwik core team is not involved in having to review the PR and make a judgment call if the quality is good enough (given that hey are not the experts in that area)

I will close this PR, thanks again for your awesome job.


Update docs with examples

Bug Report or Feature Request (mark with an x)
- [ x ] feature request

Command (mark with an x)
- [ x ] config

Desired functionality

Show how we can update build configurations using the terminal
something like:
ng config architect.@angular-devkit/build-angular:dev-server.namedChunks true

Thanks for reporting this issue. However, this issue is a duplicate of #11795. Please subscribe to that issue for future updates.