Index was outside the bounds of the array.

❯ /home/commonuserlol/Downloads/Cpp2IL --force-binary-path=libil2cpp.so --force-metadata-path=global-metadata.dat --force-unity-version=2021.3.29f1
===Cpp2IL by Samboy063===
A Tool to Reverse Unity's "il2cpp" Build Process.
Version 2022.1.0-development.1133+14b0f05.14b0f05ec257604abcfbd049707a4ed05d31a240

[Info] [Program] Running on Unix
[Info] [Plugins] Loading plugins from Plugins...
[Info] [Plugins] Using Plugin: Cpp2IL Built-In
[Warn] [Program] Using force options, I sure hope you know what you're doing!
[Info] [Library] Initializing Metadata...
[Info] [Library] 	Using actual IL2CPP Metadata version 29
[Info] [Library] Initialized Metadata in 689ms
[Info] [Library] Using binary type ELF (from LibCpp2IL)
[Info] [Library] Searching Binary for Required Data...
[Info] [Library] Got Binary codereg: 0x2E936C8, metareg: 0x2FB8398 in 1358ms.
[Info] [Library] Initializing Binary...
[Info] [Library] Initialized Binary in 245ms
[Info] [Library] Mapping pointers to Il2CppMethodDefinitions...

Cpp2IL.Core.Exceptions.LibCpp2ILInitializationException: Fatal Exception initializing LibCpp2IL!
 ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at LibCpp2IL.Il2CppBinary.GetMethodPointer(Int32, Int32, Int32, UInt32) in /home/runner/work/Cpp2IL/Cpp2IL/LibCpp2IL/Il2CppBinary.cs:line 425
   at LibCpp2IL.Metadata.Il2CppMethodDefinition.get_MethodPointer() in /home/runner/work/Cpp2IL/Cpp2IL/LibCpp2IL/Metadata/Il2CppMethodDefinition.cs:line 71
   at LibCpp2IL.LibCpp2IlMain.<>c.<Initialize>b__19_0(Il2CppMethodDefinition) in /home/runner/work/Cpp2IL/Cpp2IL/LibCpp2IL/LibCpp2IlMain.cs:line 179
   at System.Linq.Enumerable.SelectArrayIterator`2.MoveNext()
   at LibCpp2IL.LibCpp2IlMain.Initialize(Byte[], Byte[], UnityVersion) in /home/runner/work/Cpp2IL/Cpp2IL/LibCpp2IL/LibCpp2IlMain.cs:line 179
   at LibCpp2IL.LibCpp2IlMain.LoadFromFile(String, String, UnityVersion) in /home/runner/work/Cpp2IL/Cpp2IL/LibCpp2IL/LibCpp2IlMain.cs:line 208
   at Cpp2IL.Core.Cpp2IlApi.InitializeLibCpp2Il(String, String, UnityVersion, Boolean ) in /_/Cpp2IL.Core/Cpp2IlApi.cs:line 67
   --- End of inner exception stack trace ---
   at Cpp2IL.Core.Cpp2IlApi.InitializeLibCpp2Il(String, String, UnityVersion, Boolean ) in /_/Cpp2IL.Core/Cpp2IlApi.cs:line 73
   at Cpp2IL.Program.MainWithArgs(Cpp2IlRuntimeArgs runtimeArgs) in /home/runner/work/Cpp2IL/Cpp2IL/Cpp2IL/Program.cs:line 587
   at Cpp2IL.Program.Main(String[] args) in /home/runner/work/Cpp2IL/Cpp2IL/Cpp2IL/Program.cs:line 525

Waiting for you to press enter - feel free to copy the error..

files.zip

Verbose log? Might be grabbing the wrong code/meta reg

nah,
[Info] [Library] Initializing Metadata...

[Verb] [Library] 	Reading Elf File Ident...OK (4.7512 ms)
[Verb] [Library] 	Binary is 64-bit and little-endian.
[Verb] [Library] 	Reading and validating full ELF header...OK (2.7746 ms)
[Verb] [Library] 	Elf File contains instructions of type ArmV8
[Verb] [Library] 	Reading ELF program header table...Read 8 OK (3.0042 ms)
[Verb] [Library] 	Reading ELF section header table and names...
[Verb] [Library] 		-Name for section at 0x0 is
[Verb] [Library] 		-Name for section at 0x191E0 is
[Verb] [Library] 		-Name for section at 0x3008958 is
[Verb] [Library] 		-Name for section at 0x98C8 is
[Verb] [Library] 		-Name for section at 0x83E338 is
[Verb] [Library] 		-Name for section at 0x841860 is
[Verb] [Library] 		-Name for section at 0x228 is
[Verb] [Library] 		-Name for section at 0x32849BC is .shstrtab
[Verb] [Library] 		-Name for section at 0xADCA3C is il2cpp
[Verb] [Library] 	Read 9 OK (5.8093 ms)
[Verb] [Library] 	ELF global offset is 0x0
[Verb] [Library] 	Finding Relocations...
[Verb] [Library] 		-Got 567 from RELA section  at 0x8643384
[Verb] [Library] 		-Got 351095 from dynamic RELA section at 0x217104
[Verb] [Library] 	-Now Processing 351662 relocations...OK (1007.5971 ms)
[Verb] [Library] 	Processing Symbols...
[Verb] [Library] 		-Found .dynsym at 0x98C8
[Verb] [Library] 		-Found DT_SYMTAB at 0x98C8
[Verb] [Library] 		-Found 2657 symbols in table at 0x98C8
[Verb] [Library] 		-Found 2657 symbols in table at 0x98C8
[Verb] [Library] 	OK (60.5643 ms)
[Verb] [Library] 	Processing Initializers...Got 0 OK (7.4007 ms)
[Verb] [Library] 	Checking ELF Symbol Table for code and/or meta reg...Didn't find them, scanning binary...
[Verb] [Library] Searching for il2cpp structures in an ELF binary using non-arch-specific method...
[Verb] [Library] 	Looking for code reg (this might take a while)...
[Verb] [Library] 			Looking for bytes: 6d 73 63 6f 72 6c 69 62 2e 64 6c 6c 00
[Verb] [Library] 			Found 1 occurrences of mscorlib.dll: [25E9790]
[Verb] [Library] 			Looking for bytes: 90 97 5e 02 00 00 00 00
[Verb] [Library] 			Found 1 potential codegen modules for mscorlib: [2FB8248]
[Verb] [Library] 			Looking for bytes: 48 82 fb 02 00 00 00 00
[Verb] [Library] 			Found 1 address for potential codegen modules in potential codegen module lists: [309D9D0]
[Verb] [Library] 			Looking for bytes: 58 d7 09 03 00 00 00 00
[Verb] [Library] 			Looking for bytes: 50 d7 09 03 00 00 00 00
[Verb] [Library] 			Looking for bytes: 48 d7 09 03 00 00 00 00
[Verb] [Library] 			Looking for bytes: 40 d7 09 03 00 00 00 00
[Verb] [Library] 			Looking for bytes: 38 d7 09 03 00 00 00 00
[Verb] [Library] 			Looking for bytes: 30 d7 09 03 00 00 00 00
[Verb] [Library] 			Looking for bytes: 28 d7 09 03 00 00 00 00
[Verb] [Library] 			Looking for bytes: 20 d7 09 03 00 00 00 00
[Verb] [Library] 			Found valid address for pCodegenModules after a backtrack of 86, module count is 89
[Verb] [Library] 			Found 1 potential pCodegenModules addresses: [2E93738]
[Verb] [Library] 			pCodegenModules is the second-to-last field of the codereg struct. Therefore on this version and architecture, we need to subtract 112 bytes from its address to get pCodeReg
[Verb] [Library] 			Only found one codegen module pointer, so assuming it's correct and returning pCodeReg = 0x2E936C8
[Verb] [Library] 	Got code reg 0x2E936C8
[Verb] [Library] 	Looking for meta reg (post-27)...
[Verb] [Library] 			Looking for the number of type definitions, 0x31B2
[Verb] [Library] 			Looking for bytes: b2 31 00 00 00 00 00 00
[Verb] [Library] 			Found 2 instances of the number of type definitions: [2FB83E8, 2FB83F8]
[Verb] [Library] 			Found 2 potential metadata registrations: [2FB8388, 2FB8398]
[Verb] [Library] 			Rejecting metadata registration 0x2FB8388 because the pointer at index 1 is 0.
[Verb] [Library] 			Accepting metadata reg as VA 0x2FB8398
[Verb] [Library] 	Got meta reg 0x2fb8398
[Info] [Library] Got Binary codereg: 0x2E936C8, metareg: 0x2FB8398 in 1461ms.
[Info] [Library] Initializing Binary...

[Verb] [Library] 		-Read module data for Assembly-CSharp-firstpass.dll, contains 867 method pointers starting at 0x30F2518
[Verb] [Library] 			-Read 867 method pointers.
[Verb] [Library] 			-Read 8 RGCTX ranges.
[Verb] [Library] 			-Read 23 RGCTXs.
[Verb] [Library] 		-Read module data for Assembly-CSharp.dll, contains 19740 method pointers starting at 0x3258F08
[Verb] [Library] 			-Read 19740 method pointers.
[Verb] [Library] 			-Read 100 RGCTX ranges.
[Verb] [Library] 			-Read 471 RGCTXs.
[Verb] [Library] 		-Read module data for BestHTTP.dll, contains 2533 method pointers starting at 0x30EC6B0
[Verb] [Library] 			-Read 2533 method pointers.
[Verb] [Library] 			-Read 85 RGCTX ranges.
[Verb] [Library] 			-Read 407 RGCTXs.
[Verb] [Library] 		-Read module data for CsvHelper.dll, contains 307 method pointers starting at 0x3233090
[Verb] [Library] 			-Read 307 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 2 RGCTXs.
[Verb] [Library] 		-Read module data for DOTweenPro.dll, contains 55 method pointers starting at 0x30C4B68
[Verb] [Library] 			-Read 55 method pointers.
[Verb] [Library] 		-Read module data for DOTween.dll, contains 1180 method pointers starting at 0x30C5F70
[Verb] [Library] 			-Read 1180 method pointers.
[Verb] [Library] 			-Read 59 RGCTX ranges.
[Verb] [Library] 			-Read 110 RGCTXs.
[Verb] [Library] 		-Read module data for DiresuUnity.Android.dll, contains 14 method pointers starting at 0x3258468
[Verb] [Library] 			-Read 14 method pointers.
[Verb] [Library] 		-Read module data for DiresuUnity.Base.dll, contains 25 method pointers starting at 0x30C56B0
[Verb] [Library] 			-Read 25 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 3 RGCTXs.
[Verb] [Library] 		-Read module data for DiresuUnity.dll, contains 5 method pointers starting at 0x30F1CC8
[Verb] [Library] 			-Read 5 method pointers.
[Verb] [Library] 		-Read module data for DiresuUnity.Settings.dll, contains 40 method pointers starting at 0x30E3380
[Verb] [Library] 			-Read 40 method pointers.
[Verb] [Library] 		-Read module data for Firebase.Analytics.dll, contains 41 method pointers starting at 0x30CDD08
[Verb] [Library] 			-Read 41 method pointers.
[Verb] [Library] 		-Read module data for Firebase.App.dll, contains 226 method pointers starting at 0x3249F98
[Verb] [Library] 			-Read 226 method pointers.
[Verb] [Library] 		-Read module data for Firebase.Crashlytics.dll, contains 144 method pointers starting at 0x30F2098
[Verb] [Library] 			-Read 144 method pointers.
[Verb] [Library] 		-Read module data for Firebase.Platform.dll, contains 131 method pointers starting at 0x3280E60
[Verb] [Library] 			-Read 131 method pointers.
[Verb] [Library] 			-Read 8 RGCTX ranges.
[Verb] [Library] 			-Read 45 RGCTXs.
[Verb] [Library] 		-Read module data for GamePotUnity.SimpleJSON.dll, contains 67 method pointers starting at 0x3258250
[Verb] [Library] 			-Read 67 method pointers.
[Verb] [Library] 		-Read module data for GoogleMobileAds.Android.dll, contains 298 method pointers starting at 0x327FB08
[Verb] [Library] 			-Read 298 method pointers.
[Verb] [Library] 		-Read module data for GoogleMobileAds.dll, contains 59 method pointers starting at 0x30C54D8
[Verb] [Library] 			-Read 59 method pointers.
[Verb] [Library] 		-Read module data for GoogleMobileAds.Common.dll, contains 54 method pointers starting at 0x30E26C0
[Verb] [Library] 			-Read 54 method pointers.
[Verb] [Library] 		-Read module data for GoogleMobileAds.Core.dll, contains 66 method pointers starting at 0x30C4D30
[Verb] [Library] 			-Read 66 method pointers.
[Verb] [Library] 		-Read module data for GoogleMobileAds.Ump.Android.dll, contains 38 method pointers starting at 0x31BDF80
[Verb] [Library] 			-Read 38 method pointers.
[Verb] [Library] 		-Read module data for GoogleMobileAds.Ump.dll, contains 27 method pointers starting at 0x30CDE50
[Verb] [Library] 			-Read 27 method pointers.
[Verb] [Library] 		-Read module data for Google.Play.Common.dll, contains 61 method pointers starting at 0x30B6DA8
[Verb] [Library] 			-Read 61 method pointers.
[Verb] [Library] 			-Read 3 RGCTX ranges.
[Verb] [Library] 			-Read 25 RGCTXs.
[Verb] [Library] 		-Read module data for Google.Play.Core.dll, contains 52 method pointers starting at 0x30F96E8
[Verb] [Library] 			-Read 52 method pointers.
[Verb] [Library] 			-Read 9 RGCTX ranges.
[Verb] [Library] 			-Read 38 RGCTXs.
[Verb] [Library] 		-Read module data for Google.Play.Review.dll, contains 19 method pointers starting at 0x319F1E0
[Verb] [Library] 			-Read 19 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 5 RGCTXs.
[Verb] [Library] 		-Read module data for ICSharpCode.SharpZipLib.dll, contains 198 method pointers starting at 0x319EBB0
[Verb] [Library] 			-Read 198 method pointers.
[Verb] [Library] 		-Read module data for Microsoft.Bcl.HashCode.dll, contains 28 method pointers starting at 0x30F1F78
[Verb] [Library] 			-Read 28 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 2 RGCTXs.
[Verb] [Library] 		-Read module data for Microsoft.CSharp.dll, contains 1482 method pointers starting at 0x324A6A8
[Verb] [Library] 			-Read 1482 method pointers.
[Verb] [Library] 			-Read 9 RGCTX ranges.
[Verb] [Library] 			-Read 33 RGCTXs.
[Verb] [Library] 		-Read module data for Mono.Security.dll, contains 515 method pointers starting at 0x3233A88
[Verb] [Library] 			-Read 515 method pointers.
[Verb] [Library] 		-Read module data for Newtonsoft.Json.dll, contains 2387 method pointers starting at 0x30F4A80
[Verb] [Library] 			-Read 2387 method pointers.
[Verb] [Library] 			-Read 57 RGCTX ranges.
[Verb] [Library] 			-Read 346 RGCTXs.
[Verb] [Library] 		-Read module data for ProCamera2D.Runtime.dll, contains 882 method pointers starting at 0x31BC3A0
[Verb] [Library] 			-Read 882 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 4 RGCTXs.
[Verb] [Library] 		-Read module data for System.dll, contains 6095 method pointers starting at 0x31924E8
[Verb] [Library] 			-Read 6095 method pointers.
[Verb] [Library] 			-Read 20 RGCTX ranges.
[Verb] [Library] 			-Read 222 RGCTXs.
[Verb] [Library] 		-Read module data for System.Configuration.dll, contains 35 method pointers starting at 0x31C0A70
[Verb] [Library] 			-Read 35 method pointers.
[Verb] [Library] 		-Read module data for System.Core.dll, contains 3587 method pointers starting at 0x3250F78
[Verb] [Library] 			-Read 3587 method pointers.
[Verb] [Library] 			-Read 242 RGCTX ranges.
[Verb] [Library] 			-Read 1301 RGCTXs.
[Verb] [Library] 		-Read module data for System.Data.dll, contains 3427 method pointers starting at 0x30DAC40
[Verb] [Library] 			-Read 3427 method pointers.
[Verb] [Library] 			-Read 17 RGCTX ranges.
[Verb] [Library] 			-Read 131 RGCTXs.
[Verb] [Library] 		-Read module data for System.IO.Compression.dll, contains 80 method pointers starting at 0x31C0B88
[Verb] [Library] 			-Read 80 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 4 RGCTXs.
[Verb] [Library] 		-Read module data for System.Numerics.dll, contains 191 method pointers starting at 0x3280458
[Verb] [Library] 			-Read 191 method pointers.
[Verb] [Library] 		-Read module data for System.Runtime.Serialization.dll, contains 6 method pointers starting at 0x31C06D8
[Verb] [Library] 			-Read 6 method pointers.
[Verb] [Library] 		-Read module data for System.Xml.dll, contains 6664 method pointers starting at 0x30B6F98
[Verb] [Library] 			-Read 6664 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 7 RGCTXs.
[Verb] [Library] 		-Read module data for System.Xml.Linq.dll, contains 228 method pointers starting at 0x3281278
[Verb] [Library] 			-Read 228 method pointers.
[Verb] [Library] 			-Read 4 RGCTX ranges.
[Verb] [Library] 			-Read 16 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.AIModule.dll, contains 3 method pointers starting at 0x31C0708
[Verb] [Library] 			-Read 3 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.AndroidJNIModule.dll, contains 408 method pointers starting at 0x30C90F0
[Verb] [Library] 			-Read 408 method pointers.
[Verb] [Library] 			-Read 23 RGCTX ranges.
[Verb] [Library] 			-Read 41 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.AnimationModule.dll, contains 87 method pointers starting at 0x30E9E28
[Verb] [Library] 			-Read 87 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.AssetBundleModule.dll, contains 14 method pointers starting at 0x3234BF8
[Verb] [Library] 			-Read 14 method pointers.
[Verb] [Library] 			-Read 2 RGCTX ranges.
[Verb] [Library] 			-Read 3 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.AudioModule.dll, contains 50 method pointers starting at 0x3280C90
[Verb] [Library] 			-Read 50 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.dll, contains 0 method pointers starting at 0x0
[Verb] [Library] 		-Read module data for UnityEngine.CoreModule.dll, contains 2634 method pointers starting at 0x30E34C0
[Verb] [Library] 			-Read 2634 method pointers.
[Verb] [Library] 			-Read 112 RGCTX ranges.
[Verb] [Library] 			-Read 330 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.DirectorModule.dll, contains 3 method pointers starting at 0x30DAC28
[Verb] [Library] 			-Read 3 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.GridModule.dll, contains 1 method pointers starting at 0x3234C68
[Verb] [Library] 			-Read 1 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.IMGUIModule.dll, contains 475 method pointers starting at 0x30F9898
[Verb] [Library] 			-Read 475 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.ImageConversionModule.dll, contains 1 method pointers starting at 0x30FA800
[Verb] [Library] 			-Read 1 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.InputLegacyModule.dll, contains 62 method pointers starting at 0x32584E8
[Verb] [Library] 			-Read 62 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.InputModule.dll, contains 8 method pointers starting at 0x30F4788
[Verb] [Library] 			-Read 8 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.JSONSerializeModule.dll, contains 6 method pointers starting at 0x30C5390
[Verb] [Library] 			-Read 6 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 2 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.ParticleSystemModule.dll, contains 114 method pointers starting at 0x32587F8
[Verb] [Library] 			-Read 114 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.Physics2DModule.dll, contains 104 method pointers starting at 0x30C4F40
[Verb] [Library] 			-Read 104 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.PhysicsModule.dll, contains 67 method pointers starting at 0x30C8E28
[Verb] [Library] 			-Read 67 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.ScreenCaptureModule.dll, contains 2 method pointers starting at 0x32584D8
[Verb] [Library] 			-Read 2 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.SharedInternalsModule.dll, contains 66 method pointers starting at 0x319F278
[Verb] [Library] 			-Read 66 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.SpriteShapeModule.dll, contains 0 method pointers starting at 0x0
[Verb] [Library] 		-Read module data for UnityEngine.SubsystemsModule.dll, contains 28 method pointers starting at 0x31C1BC8
[Verb] [Library] 			-Read 28 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 9 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.TerrainModule.dll, contains 41 method pointers starting at 0x3234AA0
[Verb] [Library] 			-Read 41 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.TextCoreFontEngineModule.dll, contains 121 method pointers starting at 0x30C85E0
[Verb] [Library] 			-Read 121 method pointers.
[Verb] [Library] 			-Read 2 RGCTX ranges.
[Verb] [Library] 			-Read 7 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.TextCoreTextEngineModule.dll, contains 370 method pointers starting at 0x31C0F48
[Verb] [Library] 			-Read 370 method pointers.
[Verb] [Library] 			-Read 5 RGCTX ranges.
[Verb] [Library] 			-Read 10 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.TextRenderingModule.dll, contains 48 method pointers starting at 0x30EA3D0
[Verb] [Library] 			-Read 48 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.TilemapModule.dll, contains 42 method pointers starting at 0x32580A0
[Verb] [Library] 			-Read 42 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.UIElementsModule.dll, contains 5364 method pointers starting at 0x30CDF28
[Verb] [Library] 			-Read 5364 method pointers.
[Verb] [Library] 			-Read 87 RGCTX ranges.
[Verb] [Library] 			-Read 835 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.UIElementsNativeModule.dll, contains 245 method pointers starting at 0x30C5778
[Verb] [Library] 			-Read 245 method pointers.
[Verb] [Library] 			-Read 2 RGCTX ranges.
[Verb] [Library] 			-Read 5 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.UIModule.dll, contains 106 method pointers starting at 0x31C0720
[Verb] [Library] 			-Read 106 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.UI.dll, contains 1880 method pointers starting at 0x30C9DB0
[Verb] [Library] 			-Read 1880 method pointers.
[Verb] [Library] 			-Read 14 RGCTX ranges.
[Verb] [Library] 			-Read 62 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.UnityAnalyticsModule.dll, contains 12 method pointers starting at 0x324DE58
[Verb] [Library] 			-Read 12 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.UnityWebRequestAssetBundleModule.dll, contains 9 method pointers starting at 0x30E2678
[Verb] [Library] 			-Read 9 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.UnityWebRequestModule.dll, contains 87 method pointers starting at 0x30F47C8
[Verb] [Library] 			-Read 87 method pointers.
[Verb] [Library] 			-Read 1 RGCTX ranges.
[Verb] [Library] 			-Read 1 RGCTXs.
[Verb] [Library] 		-Read module data for UnityEngine.VFXModule.dll, contains 33 method pointers starting at 0x30C53C0
[Verb] [Library] 			-Read 33 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.VRModule.dll, contains 1 method pointers starting at 0x30B6F90
[Verb] [Library] 			-Read 1 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.VideoModule.dll, contains 24 method pointers starting at 0x30EA2F0
[Verb] [Library] 			-Read 24 method pointers.
[Verb] [Library] 		-Read module data for UnityEngine.XRModule.dll, contains 67 method pointers starting at 0x30F42B0
[Verb] [Library] 			-Read 67 method pointers.
[Verb] [Library] 		-Read module data for Unity.2D.Animation.Runtime.dll, contains 334 method pointers starting at 0x30E2870
[Verb] [Library] 			-Read 334 method pointers.
[Verb] [Library] 			-Read 9 RGCTX ranges.
[Verb] [Library] 			-Read 42 RGCTXs.
[Verb] [Library] 		-Read module data for Unity.Burst.dll, contains 77 method pointers starting at 0x30F1CF0
[Verb] [Library] 			-Read 77 method pointers.
[Verb] [Library] 			-Read 5 RGCTX ranges.
[Verb] [Library] 			-Read 9 RGCTXs.
[Verb] [Library] 		-Read module data for Unity.Burst.Unsafe.dll, contains 2 method pointers starting at 0x30F9888
[Verb] [Library] 			-Read 2 method pointers.
[Verb] [Library] 		-Read module data for Unity.Collections.dll, contains 254 method pointers starting at 0x32505C8
[Verb] [Library] 			-Read 254 method pointers.
[Verb] [Library] 			-Read 17 RGCTX ranges.
[Verb] [Library] 			-Read 208 RGCTXs.
[Verb] [Library] 		-Read module data for Unity.InternalAPIEngineBridge.001.dll, contains 4 method pointers starting at 0x3258080
[Verb] [Library] 			-Read 4 method pointers.
[Verb] [Library] 		-Read module data for Unity.Mathematics.dll, contains 113 method pointers starting at 0x30EC030
[Verb] [Library] 			-Read 113 method pointers.
[Verb] [Library] 		-Read module data for Unity.TextMeshPro.dll, contains 1618 method pointers starting at 0x3281A78
[Verb] [Library] 			-Read 1618 method pointers.
[Verb] [Library] 			-Read 19 RGCTX ranges.
[Verb] [Library] 			-Read 114 RGCTXs.
[Verb] [Library] 		-Read module data for Unity.VisualScripting.Core.dll, contains 1155 method pointers starting at 0x31BE0B0
[Verb] [Library] 			-Read 1155 method pointers.
[Verb] [Library] 			-Read 74 RGCTX ranges.
[Verb] [Library] 			-Read 596 RGCTXs.
[Verb] [Library] 		-Read module data for Unity.VisualScripting.Flow.dll, contains 232 method pointers starting at 0x3232950
[Verb] [Library] 			-Read 232 method pointers.
[Verb] [Library] 			-Read 3 RGCTX ranges.
[Verb] [Library] 			-Read 62 RGCTXs.
[Verb] [Library] 		-Read module data for __Generated, contains 3 method pointers starting at 0x30CDCF0
[Verb] [Library] 			-Read 3 method pointers.
[Verb] [Library] 		-Read module data for mscorlib.dll, contains 13021 method pointers starting at 0x319F488
[Verb] [Library] 			-Read 13021 method pointers.
[Verb] [Library] 			-Read 263 RGCTX ranges.
[Verb] [Library] 			-Read 1740 RGCTXs.
[Verb] [Library] 		-Read module data for spine-csharp.dll, contains 1234 method pointers starting at 0x324DEB8
[Verb] [Library] 			-Read 1234 method pointers.
[Verb] [Library] 			-Read 5 RGCTX ranges.
[Verb] [Library] 			-Read 72 RGCTXs.
[Verb] [Library] 		-Read module data for spine-unity.dll, contains 830 method pointers starting at 0x30EA580
[Verb] [Library] 			-Read 830 method pointers.
[Verb] [Library] 			-Read 4 RGCTX ranges.
[Verb] [Library] 			-Read 5 RGCTXs.
[Verb] [Library] 	OK (20.9576 ms)
[Verb] [Library] 	Reading generic method tables...OK (21.6077 ms)
[Verb] [Library] 	Reading method specifications...OK (12.8288 ms)
[Verb] [Library] 	Reading generic methods...OK (125.089 ms)
[Info] [Library] Initialized Binary in 315ms
[Info] [Library] Mapping pointers to Il2CppMethodDefinitions...

Something is weird with this binary, the sections outside of the il2cpp and .shstrtab ones don't have names?

Looks like some packer/obfuscator?
❯ readelf -S libil2cpp.so
There are 9 section headers, starting at offset 0x3284ad0:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL<PHONE_NUMBER>000000  00000000
<PHONE_NUMBER>000000<PHONE_NUMBER>000000           0     0     0
  [ 1]                   STRTAB<PHONE_NUMBER>0191e0  000191e0
<PHONE_NUMBER>01a908<PHONE_NUMBER>000000   A       0     0     1
  [ 2]                   DYNAMIC<PHONE_NUMBER>009958  03008958
<PHONE_NUMBER>000210<PHONE_NUMBER>000010  WA       1     0     8
  [ 3]                   DYNSYM<PHONE_NUMBER>0098c8  000098c8
<PHONE_NUMBER>00f918<PHONE_NUMBER>000018   A       1     3     8
  [ 4]                   RELA<PHONE_NUMBER>83e338  0083e338
<PHONE_NUMBER>003528<PHONE_NUMBER>000018  AI       3     5     8
  [ 5]                   PROGBITS<PHONE_NUMBER>841860  00841860
<PHONE_NUMBER>002390<PHONE_NUMBER>000010  AX       0     0     16
readelf: Warning: [ 6]: Link field (4) should index a symtab section.
  [ 6]                   HASH<PHONE_NUMBER>000228  00000228
<PHONE_NUMBER>0049a0<PHONE_NUMBER>000004   A       4     0     8
  [ 7] .shstrtab         NULL<PHONE_NUMBER>000000  032849bc
<PHONE_NUMBER>000111<PHONE_NUMBER>000000           0     0     1
  [ 8] il2cpp            PROGBITS<PHONE_NUMBER>adca3c  00adca3c
<PHONE_NUMBER>971e30<PHONE_NUMBER>000000  AX       0     0     4
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  D (mbind), p (processor specific)
readelf: Error: no .dynamic section in the dynamic segment

binary have protection


planning: open space fix dual warm up fail

@deidaraho Better to move the handling of dual variable warm up failure out to https://github.com/ApolloAuto/apollo/blob/master/modules/planning/open_space/trajectory_smoother/dual_variable_warm_start_problem.cc , otherwise you may need to handle this failure twice (In both OSQP/IPOPT interfaces, and potentially more).


Virtual router timezone always set to UTC instead of local timezone

ISSUE TYPE

Improvement Request

COMPONENT NAME

VR

CLOUDSTACK VERSION

4.11.3

CONFIGURATION

Local timezone: CET/CEST
OS / ENVIRONMENT

N/A
SUMMARY

The Virtual Router has its timezone set to UTC implicitly. While this provides a sane default, it's not necessarily the best choice in every environment.
In particular, when debugging issues, the UTC timestamps can make it difficult to track events across multiple systems.
STEPS TO REPRODUCE

The management server would propagate its local time zone into the host config of each VR.
Alternatively, a new global time zone setting would be introduced, that is used to set the VR time zone.

EXPECTED RESULTS

Each CloudStack component has a consistent local time zone set.
ACTUAL RESULTS

The virtual router always logs events with UTC timestamps.

if you expect the MS local time-zone to be on every VR this is a straightforward feature to implement. I don't think this is very useful for most clouds as they span time-zones (even corporate ones do at times) this would make it a per-zone setting.

@onitake are you planning any implementation (for next release)?


full sticky prediction

About the PR
move all the sticky code to shared and do general cleanup
Why / Balance
prediction good + big prank needs it in shared
Technical details
nothing fancy
Media
https://github.com/user-attachments/assets/85f924a9-7907-4870-83f7-adea8bce2961

[X] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Breaking changes
StickyComponent and StickySystem are now in shared
attempt and stuck/unstuck events are now in shared
stuck/unstuck events are now by ref structs
Changelog
no cl no fun

I feel like there is a pretty big prank coming up soon....

Mom come pick me up I'm scared


Bump up GOLANG_VERSION for CAPX to 1.19 once builder-base has it

Upstream CAPX is built with Go 1.19. However, builder-base doesn't support Go 1.19.
This PR should add Go 1.19 https://github.com/aws/eks-distro-build-tooling/pull/484
Once that is merged, we can update the GOLANG_VERSION for CAPX to 1.19

/close


Terminal command error decoration hover does not show exit code

Testing #143433

Run a command
Interrupt the command
Command gets error decoration
:bug: hover over decoration, exit code is not displayed

I also see no exit code reported for e.g. when running yarn compile with an obvious compile error.

working for me many months later 😅


In dropdown menu, catbot is not an option

I went over the install instructions twice, just to make sure I wasn't doing anything incorrectly. Have both overlayplugins up, dps modules work. But when I go to make a new overlay, when selecting type, catbot is not an option.
Any ideas on what I'm doing wrong?

Go to the OverlayPlugin/addons/ folder, right-click the cactbot.dll and check if it is blocked.
If so, unblock it.

I was going to be pretty disappointed in myself if that was the issue. I unblocked the zip file after downloading it, and just checked the .dll and it is not blocked.

Same issue here, just tried to install cactbot, followed the instructions to the letter, restarted multiple times, made sure everything is unblocked and the DLL is present. No option for Cactbot in the drop down menu for creating a new overlay.

What does your folder structure look like on disk? Is there an ACT directory/OverlayPlugin/addons/CactbotOverlay.dll file at all?
What do you mean by "both" OverlayPlugins?

Is this still happening?

This is likely similar to #590.  Please let me know if this is still an issue.


Accession Workflow states

Refresh Status
Not Started
In Process
Needs QA
Ready for transfer
transferred
Button to mark all child record entries as transferred if is_refreshed equals true
move note into panel header

Updated in: https://github.com/NYU-ACM/medialog/commit/5af79359f95d74c85d520825e7d1a6d59d4c266e
moving button for updating transferred material to new Issue


What are some reasons alignItems may not work?

Report

[x] I have searched existing issues and this is not a duplicate

Issues and Steps to Reproduce
See Playground Example
Try changing alignItems to center or flex-end and notice it is not working the same as with CSS.
Expected Behavior
Expecting it to work like in CSS. Video:
https://github.com/user-attachments/assets/5d4663d1-6b7f-44aa-b8e1-3919abdc45d6
Actual Behavior
In the yoga example, the items do not shift like in the CSS video
Link to Code
see above

I tried setting container.setAlignContent(Align.Center), but that breaks it all, makes the computed height of container go to zero.
Is there an order of operations I might need to be aware of for Yoga that we normally don't have to think about with CSS flexbox?

I discovered that if I do not set alignItems, and set only alignContent, then alignContent works.
In CSS, both can be set, and it seems that align-content takes precendence over align-items, but in Yoga if I apply both, the computed height of root goes to zero instead of alignContent effectively overriding alignItems.

In yogalayout.dev, I am not able to reproduce the issue that happens by using both alignItems and alignContent together. Paste the following into yoga playground and alignContent does its job:
<Layout config={{useWebDefaults: true}}>
  <Node
    style={{
      width: 250,
      height: 250,
      padding: 10,
      gap: 2,
      justifyContent: 'flex-end',
      alignItems: 'flex-end',
      alignContent: 'center',
      flexWrap: 'wrap',
    }}>
    <Node style={{margin: 5, height: 50, width: 50}} />
    <Node style={{margin: 5, height: 50, width: 50}} />
    <Node style={{margin: 5, height: 50, width: 50}} />
    <Node style={{margin: 5, height: 50, width: 50}} />
  </Node>
</Layout>

In my case adding alignContent just breaks. Seems like there are pecularities in how I'm calling Yoga (order of operations? calculateLayout args?) that are affecting my case.

I deleted all my comments. There are multiple issues, but I fimplified the OP to show only one issue with reproduction.

Yoga vs web for reference: https://jsfiddle.net/xkrfp2do/1/
This looks like a bug.


[ingress/controllers/nginx] Use Service Virtual IP instead of maintaining Pod list

Is there a way to tell the NGINX Ingress controller to use the Service Virtual IP Address instead of maintaining the Pods IP addresses in the upstream configuration ?
I couldn't find it. If not, I think it would be good. Because with the current situation, when we scale down a service. the Ingress controller does not work in harmony with the Replication Controller of the service.
That means, some requests to the Ingress Controller will fail while waiting for the Ingress Controller to be updated.
If we use the Service Virtual IP address, we can let kube-proxy do its job in harmony with the replication controller and we have a seamless down scaling.

when we scale down a service. the Ingress controller does not work in harmony with the Replication Controller of the service.

What do you mean?
After a change in the number of replicas in a rc it takes a couple of seconds to receive the update from the api server.
In the case of scaling down the number of replicas you need to tune the upstream check defaults docs
Besides this I'm testing the module lua-upstream-nginx-module to avoid reloads and be able to add/remove servers in an upstream

Ok, I'll try to explain with another example:
When you update a deployment resource (like changing the docker image), depending your configuration (rollingUpdate strategy, max surge, max unavailable), the deployment controller will bring down some pods, and create new one. All of this, in a fashion way where there is no downtime if you use the Service VIP to communicate with the pods.
Because first, when it wants to kill a pod, it removes the pod IP address from the service to avoid any new connection, and it follow the termination grace period of the pod to drain the existing connections. Meanwhile, it also creates a new pod, with the new docker image, and wait for the pod to be ready, and add the pod behind the service VIP.
By maintaining the pod list yourself in the Ingress Controller, at a certain point, during a deployment resource update, some requests will be redirected to pods which are shutting down. Because the Ingress Controller, does not know a RollingUpdate Deployment is happening. It will know maybe 1 second later. But for services, with a lots of connection/sec, it's potentially a lots of requests failing.
I personally don't want to tune the upstream to handle this scenario. Kubernetes is already doing an amazing job to update pods with no downtime. That only if you use the Service VIP.
Did I miss something ? If it's still not clear, or there is something I'm clearly not understanding, please don't hesitate.

The NGINX Ingress Controller (https://github.com/nginxinc/kubernetes-ingress) used to use the service VIP. But they changed recently to a system like yours (pod list in the upstream).
Before they changed this behaviour, I did some test. I was continuously spamming requests to the Ingress Controller (5/sec). Meanwhile, I updated the Deployment resource related to those requests (new docker images):

Kubernetes/Contrib: You can clearly see some requests failing at the time of the update
NGINX/Controller: It looks like nothing happened behind the scene, perfect deployment, with no downtime (all because of Kubernetes doing a great job behind the scene)

@edouardKaiser how are you testing this? The request are GET or POST? Can you provide a description of the testing scenario?

I personally don't want to tune the upstream to handle this scenario.

I understand that but your request is the contradicts what other users requested (control over the upstream checks). Is hard to find a balance in the configuration that satisfies all the user scenarios.

I understand some people might want to tweak the upstream configuration, but on the other side Kubernetes is doing a better job at managing deployment without downtime with the concept of communicating with pods through the Service VIP.
To reproduce, I just used the Chrome App Postman, and their Runner feature (you can specify some requests to run to a particular endpoint, with a number of iteration, delay....). And while the runner was running, I just updated the Deployment resource and watched the behaviour of the runner.
When the request is GET and it fails, Nginx automatically passes the request to the next server. But for non-idempotent method like POST, it does not (and I think it's the right behavior), and then we have failure.

But for non-idempotent method like POST, it does no

This is documented scenario https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#retries-in-no-idempotent-methods
NGINX changed this behavior.
Please add the option retry-non-idempotent=true in the nginx configmap to restore the old behavior

But it doesn't change the root of the problem: Ingress Controller and Deployment Controller don't work together.
Your pod might have accepted the connection and started to process it, but what the Ingress Controller does not know, is that this pod is gonna get killed the next second by the Deployment Controller.

For the Deployment Controller it's fine, the Deployment Controller did its job, removed the pod from the service and waited the termination grace period.
On the Ingress Controller: it's not fine, your connection will suddenly be aborted because the pod died. If you're lucky, the pod got removed by the Ingress Controller before any request gets in. If you're not, you'll experiment some failure. If it's a POST request, most of the time you really don't want to retry.

I know this is not a perfect world, and we need to embrace failure. Here, we have a way to potentially avoid that failure by using Service VIP.
I'm not saying it should be the default behaviour, but an option to use Service VIP instead of Pod endpoint would be awesome.

I'm with @edouardKaiser because:

You (as devops or operations guy) cannot guarantee that the final developer will follow the best practices regarding to keeping, dropping connections whenever it gets a SIGTERM for example. However, if services were used the responsibility for site reliability falls completely in concrete person/team.
Although retry-no-idempotent can mitigate some problems others could arise, choosing to retry those is not an option in much cases.

IMO the controller should expose a parameter or something to choose between final endpoints or services, that would cover all the use cases.

I couldn't have explained it better. Thanks @glerchundi

If you go through the service VIP you can't ever do session affinity.  It
also incurs some overhead, such as conntrack entries for iptables DNAT.  I
think this is not ideal.
To answer the questions about "coordination" this is what readiness and
grace period are for.  What is supposed to happen is:
RC creates 5 pods A, B, C, D, E
all 5 pods become ready
endpoints controller adds all 5 pods to the Endpoints structure
ingress controller sees the Endpoints update
... serving ...
RC deletes 2 pods (scaling down)
pods D and E are marked unready
kubelet notifies pods D and E
endpoints controller sees readiness change, removes D and E from endpoints
ingress controller sees the Endpoints update and removes D and E
termination grace period ends
kubelet kills pods D and E
It is possible that your ingress controller falls so far behind that the
grace expires before it has a chance to remove endpoints, but that is the
nature of distrubited systems.  It's equally possible that kube-proxy falls
behind - they literally use the same API.
On Mon, Jun 27, 2016 at 6:51 PM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

I couldn't have explained it better. Thanks @glerchundi
https://github.com/glerchundi
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-228926948,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVLVBHdUQdCIRD6uVDhs8Icwg5k8tks5qQH4hgaJpZM4IuliB
.

I do understand this is not idea for everyone, this is why I was talking about an option for this behaviour.

But I don't see how it is better for anyone?  It buys you literally
nothing, but you lose the potential for affinity and incur performance hit
for no actual increase in stability.
On Mon, Jun 27, 2016 at 8:50 PM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

I do understand this is not idea for everyone, this is why I was talking
about an option for this behaviour.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-228941142,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVEExdHf-NncMom53MS7914scnzJjks5qQJnrgaJpZM4IuliB
.

Correct me if I'm wrong, I probably misunderstood something in the termination of pods flow:
When scaling down, the pod is removed from endpoints list for service and, at the same time, a TERM signal is sent.
So, for me, at this exact moment, there is an opened window. Potentially, this pod (which is shutting down gracefully), might still get some requests forwarded by the nginx ingress-controller (just the time it needs for the ingress-controller to notify the changes, update and reload the conf).

On Mon, Jun 27, 2016 at 9:53 PM, Edouard Kaiser
<EMAIL_ADDRESS>wrote:

Correct me if I'm wrong, I probably misunderstood something in the termination of pods flow:
When scaling down, the pod is removed from endpoints list for service and, at the same time, a TERM signal is sent.

Pedanticly, "at the same time" has no real meaning.  It happens
asynchronously.  It might happen before or after or at the same time.

So, for me, at this exact moment, there is an opened window. Potentially, this pod (which is shutting down gracefully), might still get some requests forwarded by the nginx ingress-controller (just the time it needs for the ingress-controller to notify the changes, update and reload the conf).

The pod can take as long as it needs to shut down.  Typically
O(seconds) is sufficient time to finish or cleanly terminate open
connections and ensure no new connections arrive.  So, for example,
you could request 1 minute grace, keep accepting connections for max(5
seconds since last connection, 30 seconds), drain any open
connections, and then terminate.
Note that the exact same thing can happen with the service VIP.
kube-proxy is just an API watcher.  It could happen that kube-proxy
sees the pod delete after kubelet does, in which case it would still
be routing service VIP traffic  to th epod that had already been
signalled.  There's literally no difference.  That's my main point :)

True, "at the same time" doesn't mean that much here, it's more like those operations are triggered in parallel.
I wanted to point out that possibility because I ran some tests before opening this issue (continuously sending requests to an endpoint backed by multiple-pods while scaling down). And when ingress-controller was using VIP, the down-scaling was happening more smoothly (no failure, no request passed to the next server by nginx), contrary to when the ingress-controller is maintaining the endpoint list (I noticed some requests were failing for that short time-window, and passed to the next server for the GET, PUT type...).
I'm surprised the same thing can happen with the service VIP. I supposed that Kubelet would start the shutdown only once the pod was removed from the iptable entries, but I was wrong.
So your point is, I got lucky during my tests, because depending the timing, I might have ended up with the same situation even with Service VIP.

On Mon, Jun 27, 2016 at 10:33 PM, Edouard Kaiser
<EMAIL_ADDRESS>wrote:

I'm surprised the same thing can happen with the service VIP. I supposed that Kubelet would start the shutdown only once the pod was removed from the iptable entries, but I was wrong.

Nope.  kube-proxy is replaceable, so we can't really couple except to the API.

So your point is, I got lucky during my tests, because depending the timing, I might have ended up with the same situation even with Service VIP.

I'd say you got UNlucky - it's always better to see the errors :)
If termination doesn't work as I described (roughly, I may get some
details wrong) we should figure out why

Thanks for the explanation Tim, I guess I can close this one.

Not to impose too much, but since this is a rather frequent topic, I wonder
if you want to write a doc or an example or something?  A way to
demonstrate the end-to-end config for this?  I've been meaning to do it,
but it means so much more when non-core-team people document stuff (less
bad assumptions :).
I'll send you a tshirt...
On Mon, Jun 27, 2016 at 11:29 PM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

Thanks for the explanation Tim, I guess I can close this one.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-228960154,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVL1tS9yhMYvUh8MM6UKypMtCJRYEks5qQL9YgaJpZM4IuliB
.

Happy to write something.
Were you think about updating the README of the Ingress Controllers (https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx)?
We could add a new paragraph about the choice of using endpoint list instead of service VIP (advantages like upstream tuning, session affinity..) and showing that there is no guarantee of synchronisation even by using the service VIP.

@thockin thanks for the explanation, it's very water clear now.

I'm glad I have a better understanding on how it works, it makes sense if you think about the kube-proxy as just an API watcher.
But to be honest, now I'm kind of stuck. Some of our applications don't handle very well the SIGTERM (no graceful stop..). Even if the application is in the middle of a request, a SIGTERM would shutdown the app immediately.
Using Kubernetes, I'm not sure how to deploy without downtime now. My initial understanding was this flow when scaling down/deploying new version:

Remove the pod from the endpoint list
Wait for the terminationGracePeriod (to wait for any request in progress to finish)
Then shutdown with SIGTERM

We need to rethink about how to deploy or see if we can adapt our application to handle SIGTERM.

wrt writing something, I was thinking a doc or a blog post or even an
example with yaml and a README
On Mon, Jun 27, 2016 at 11:45 PM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

Happy to write something.
Were you think about updating the README of the Ingress Controllers (
https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx
)?
We could add a new paragraph about the choice of using endpoint list
instead of service VIP (advantages like upstream tuning, session
affinity..) and showing that there is no guarantee of synchronisation even
by using the service VIP.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-228964234,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVIOlLSwqcb-Gq7h7JcXUutoVQjmzks5qQMMCgaJpZM4IuliB
.

You also have labels at your disposal.
If you make your Service select app=myapp,active=true, then you can start
all your pods with that set of labels.  When you want to do your own
termination, you can remove the active=true label from the pod, which
will update the Endpoints object, and that will stop sending traffic.  Wait
however long you think you need, then delete the pod.
Or you could teach your apps to handle SIGTERM.
Or you could make an argument for a configurable signal rather than SIGTERM
(if you can make a good argument)
Or ... ? other ideas welcome
On Tue, Jun 28, 2016 at 4:47 AM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

I'm glad I have a better understanding on how it works, it makes sense if
you think about the kube-proxy as just an API watcher.
But to be honest, now I'm kind of stuck. Some of our applications don't
handle very well the SIGTERM (no graceful stop..). Even if the application
is in the middle of a request, a SIGTERM would shutdown the app immediately.
Using Kubernetes, I'm not sure how to deploy without downtime now. My
initial understanding was this flow when scaling down/deploying new version:

Remove the pod from the endpoint list
Wait for the terminationGracePeriod (to wait for any request in
progress to finish)
Then shutdown with SIGTERM

We need to rethink about how to deploy or see if we can adapt our
application to handle SIGTERM.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-229026299,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVMo0zaGLSpVIPgp_E8TwYDsNX-Rcks5qQQnUgaJpZM4IuliB
.

Thanks for the advice, I tend to forget how powerful labels can be.
Regarding writing something, I can write a blog to explain why using and endpoint list is better. But I'm not sure what kind of example (YAML) you are talking about.

I guess there's not much YAML to write up. :)  I just want to see something
that I can point the next person who asks this at and say "read this"
On Tue, Jun 28, 2016 at 8:48 PM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

Thanks for the advice, I tend to forget how powerful labels can be.
Regarding writing something, I can write a blog to explain why using and
endpoint list is better. But I'm not sure what kind of example (YAML) you
are talking about.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-229249421,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVPjK5UBgJeyzfyR0VczUdDbpIw6Dks5qQesegaJpZM4IuliB
.

No worries Tim, I keep you posted.

Fantastic!!
On Tue, Jun 28, 2016 at 11:05 PM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

No worries Tim, I keep you posted.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-229264761,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVDx-gvq8eGKsrHoOrAgMbN0R2WXZks5qQgsXgaJpZM4IuliB
.

I just created this blog entry:
http://onelineatatime.io/ingress-controller-forget-about-the-service/
I hope it will help some people. Feel free to tell me if there is anything wrong, anything that I could do to improve this entry.
Cheers,

Great post!!
Small nit:
1. Replication Controller deletes 1 pod
2. Pod is marked unready and shows up as Terminating
3. TERM signal is sent
4. Pod is removed from endpoints
5. Pod receives SIGKILL after grace period
6. Kube-proxy detects the change of the endpoints and update iptables

should probably be:
1. Replication Controller deletes 1 pod
2. Pod is marked as Terminating
3. Kubelet observes that change and sends SIGTERM
4. Endpoint controller observes the change and removes the pod from
Endpoints
5. Kube-proxy observes the Endpoints change and updates iptables
6. Pod receives SIGKILL after grace period

3 and 4 happen roughly in parallel.  3 and 5 are async to each other, so
it's just as likely that 5 happens first as vice-versa.  Your Ingress
controller would be 4.1. Ingress controller observes the change and updates the proxy :)
On Wed, Jul 6, 2016 at 10:45 PM, Edouard Kaiser<EMAIL_ADDRESS>wrote:

I just created this blog entry:
http://onelineatatime.io/ingress-controller-forget-about-the-service/
I hope it will help some people. Feel free to tell me if there is anything
wrong, anything that I could do to improve this entry.
Cheers,
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-230984547,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVGMuCT6-ArM6-SodThc8wrsEFfSDks5qTJKRgaJpZM4IuliB
.

Thanks Tim, I will update it !

If you make your Service select app=myapp,active=true, then you can start all your pods with that set of labels.  When you want to do your own termination, you can remove the active=true label from the pod, which will update the Endpoints object, and that will stop sending traffic.  Wait
however long you think you need, then delete the pod.

I was wondering if the above approach could potentially be built into Kubernetes directly. The benefit I see is that people won't need to create custom routines which effectively bypass all standard tooling (e.g., kubectl scale / delete).
If labels aren't the right thing for this case, I could also think of a more low-levellish approach: Introduce a new state called Deactivating that precedes Terminating and serves as a trigger for the Endpoint controller to remove a pod from rotation. After (yet another) configurable grace period, the state would switch to Terminating and cause kubelet to SIGTERM the pod as usual.
@thockin would that be something worth pursuing or rather be out of question?

I'm very wary of adding another way of doing the same thing as a core
feature.  For the most part, graceful termination should do the right thing
for most people.
I could maybe see extending DeploymentStrategy to offer blue-green rather
than rolling, but that's not really this.
On Sat, Jul 9, 2016 at 5:53 AM, Timo Reimann<EMAIL_ADDRESS>wrote:

If you make your Service select app=myapp,active=true, then you can start
all your pods with that set of labels. When you want to do your own
termination, you can remove the active=true label from the pod, which
will update the Endpoints object, and that will stop sending traffic. Wait
however long you think you need, then delete the pod.
I was wondering if the above approach could potentially be built into
Kubernetes directly. The benefit I see is that people won't need to create
custom routines which effectively bypass all standard tooling (e.g.,
kubectl scale / delete).
If labels aren't the right thing for this case, I could also think of a
more low-levellish approach: Introduce a new state called Deactivating that
precedes Terminating and serves as a trigger for the Endpoint controller to
remove a pod from rotation. After (yet another) configurable grace period,
the state would switch to Terminating and cause kubelet to SIGTERM the pod
as usual.
@thockin https://github.com/thockin would that be something worth
pursuing or rather be out of question?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-231533093,
or mute the thread
https://github.com/notifications/unsubscribe/AFVgVABppSzvMyBPqIy2Xuyw38h0L3poks5qT5m0gaJpZM4IuliB
.

@thockin If I understand correctly, the way to allow for a non-interruptive transition using graceful termination is to have a SIGTERM handler that (in the most simplistic use case) just delays termination for a safe amount of time.
Is there a way to reuse such a handler across various applications, possibly through a sidecar container? Otherwise, I see the problem that the handler must be implemented and integrated for each and every application (at least per language/technology) over and over again. For third-party applications, it may even be impossible to embed a handler directly.

On Sun, Jul 10, 2016 at 3:01 PM, Timo Reimann<EMAIL_ADDRESS>wrote:

@thockin If I understand correctly, the way to allow for a non-interruptive transition using graceful termination is to have a SIGTERM handler that (in the most simplistic use case) just delays termination for a safe amount of time.

There's no use to handle SIGTERM from a sidecar, if the main app dies
upon receiving it.  It doesn't "just" delay - it notifies the app that
its end-of-life is near, and that it should wrap up and exit soon, or
otherwise be prepared.

Is there a way to reuse such a handler across various applications, possibly through a sidecar container? Otherwise, I see the problem that the handler must be implemented and integrated for each and every application (at least per language/technology) over and over again. For third-party applications, it may even be impossible to embed a handler directly.

The problem is that "handling" SIGTERM is really app-specific.  Even
if you just catch it and ignore it, that's a decision we shouldn't
make for you.
Now, we have a proposal in flight for more more generalized
notifications, including HTTP, so maybe we can eventually say that,
rather than SIGTERM being hardcoded, that is merely the default
handler, but someone could override that.  But that spec is not fully
formed yet, and I don't want to build on it just yet.
I'm open to ideas, but I don't see a clean way to handle this.  Maybe
a pod-level field that says "don't send me SIGTERM, but pretend you
did" ?  That's sort of ugly..

What I meant by delaying termination is that a custom SIGTERM handler could keep the container alive (i.e., time.Sleep(reasonablePeriod)) long enough until the point in time where it's safe to believe that the endpoint controller has taken the pod out of rotation so that requests won't hit an already dead pod. I don't think this is an ideal approach for reasons I have mentioned -- my assumption was that this is what you meant when you said "graceful termination [as a core feature] should do the right thing for most people". Maybe I misunderstood you; if so, I'd be glad for some clarification.
To repeat my intention: I'm looking for the best way to prevent request drops when scaling events / rolling-upgrades occur (as the OP described) without straying too far away from what standard tooling (namely kubectl) gives. My (naive) assessment is that the Kubernetes control plane is best suited for doing the necessary coordinative effort.
Do you have any issue/PR numbers to share as far as that generalized notification proposal is concerned?

you should fail your readiness probe when you receive a sigterm. The nginx controller will health check endpoint readiness every 1s and avoid sending requests. Set termination grace to something high and keep nginx alive till existing connections drain. Is this enough? (I haven't read through previous conversation, so apologies if this was already rejected as a solution).
It sounds like what you're really asking for is to use the service vip in the nginx config and cut out the racecondition that springs from going: kubelt readiness -> apiserver -> endpoints -> kubeproxy, we've discussed various ways to achieve this (https://github.com/kubernetes/kubernetes/issues/28442), but right now the easiest way is to health check endpoints from the ingress controller.

On Sun, Jul 10, 2016 at 3:57 PM, Timo Reimann<EMAIL_ADDRESS>wrote:

What I meant by delaying termination is that a custom SIGTERM handler could keep the container alive (i.e., time.Sleep(reasonablePeriod)) long enough until the point in time where it's safe to believe that the endpoint controller has taken the pod out of rotation so that requests won't hit an already dead pod. I don't think this is an ideal approach for reasons I have mentioned -- my assumption was that this is what you meant when you said "graceful termination [as a core feature] should do the right thing for most people". Maybe I misunderstood you; if so, I'd be glad for some clarification.

I'm a little confused, I guess.  What you're describing IS the core
functionality.  When a pod is deleted, we notify it and wait at least
grace-period seconds before killing it.  During that time window (30
seconds by default), the Pod is considered "terminating" and will be
removed from any Services by a controller (async).  The Pod itself
only needs to catch the SIGTERM, and start failing any readiness
probes.  Assuming nothing is totally borked in the cluster, the
load-balancers should stop sending traffic and the pod will be OK to
terminate within the grace period.  This is, truthfully, a race and a
bit of wishful thinking.  If something is borked in the cluster, it
is possible that load-balancers don't remove pods "in time" and when
the pod dies it kills live connections.
The alternative is that we never kill a pod while any service has the
pod in it's LB set.  In the event of brokenness we trade a hung
rolling update for the above-described early termination.  Checking
which Services a pod is in is hard, and we just don't do that today.
Besides that, it's an unbounded problem.  Today it is Services, but it
is also Ingresses.  But ingresses are somewhat opaque in this regard,
so we can't actually check.  And there may be arbitrary other
frontends to a Pod.  I don't think the problem is solvable this way.
So you're saying that waiting some amount of time is not "ideal", and
I am agreeing.  But I think it is less bad than the other solutions.
The conversation turned to "but my app doesn't handle SIGTERM", to
which I proposed a hacky labels-based solution.  It probably works,
but it is just mimicing graceful termination

To repeat my intention: I'm looking for the best way to prevent request drops when scaling events / rolling-upgrades occur (as the OP described) without straying too far away from what standard tooling (namely kubectl) gives. My (naive) assessment is that the Kubernetes control plane is best suited for doing the necessary coordinative effort.

Graceful termination.  This is the kubernetes control plane doing the
coordinative effort.  "wait some time" is never a satisfying answer,
but in practice it is often sufficient.  In this case, the failures
that would cause "wait" to misbehave probably cause worse failures if
you try to close the loop.

Do you have any issue/PR numbers to share as far as that generalized notification proposal is concerned?

https://github.com/kubernetes/kubernetes/pull/26884

@thockin sorry for not getting back on this one earlier. I did intend to follow up on your response.
First, thanks for providing more details to the matter.
I'm fine with accepting the fact that graceful termination involves some timely behavior which also provides a means to set upper bounds in case things start to break. My concerns are more about the happy path and the circumstance that presumably a lot of applications running on Kubernetes will have no particular graceful termination needs but want the necessary coordination between the shutdown procedure and load-balancing adjustments to take place. As discussed, these applications need to go through the process of implementing a signal handler to switch off the readiness probe deliberately.
To add a bit of motivation on my end: We plan to migrate a number of applications into Kubernetes where the vast majority of them serves short-lived requests only. When we want to take instances down in our infrastructure, we just remove them from LB rotation and make sure in-flight requests are given enough time to finish. Moving to Kubernetes, we'll have to ask every application owner to implement and test a custom signal handler, and in the case of closed third-party applications resort to complicating workarounds with workflows/tooling separate from the existing standards. My impression is that this represents an undesirable coupling between the applications running on Kubernetes and an implementation detail on the load balancing routing part of the cluster manager.
That's why I think having a separate mechanism exclusively implemented in the control plane could significantly simplify running applications on Kubernetes by removing some of the lifecycle management boilerplate. To elaborate a bit on my previous idea: Instead of making each application fail its readiness probe, make Kubernetes do that "externally" and forcefully once it has decided to take a pod down, and add another grace period (possibly specified with the readiness probe) to give the system sufficient time for the change in readiness to propagate. This way, custom signal handlers for the purpose of graceful termination become limited in scope to just that: Giving an application the opportunity to execute any application-specific logic necessary to shut down cleanly, while all the load balancing coordination happens outside and up front. I'm naively hopeful that by reusing existing primitives of Kubernetes like readiness probes, timeouts, and adjustments to load-balancing, we can avoid dealing with the kind of hard problems that you have mentioned (checking which services a pod is in, unbound number of service frontends).
I'm wondering if it might be helpful to open a separate proposal issue and discuss some of these things in more detail. Please let me know if you think it's worthwhile carrying on.
Thanks.

Sorry for re-opening an old issue but i'm struggling a tad to find concrete answers in the core docs on this and this is the best thread i've found so far which explains what's going on... so can I clarify a few things in relation to termination? If this should be posted somewhere else, please let me know.
A: The termination cycle:

Before a pod is terminated, it enters a "terminating" phase that lasts the duration of a configurable grace period (which by default is 30 seconds). Once this period concludes, it enters a "terminated" phase while resources are cleaned up and is then eventually deleted (or does it just get deleted after the terminating phase?).
As soon as the pod enters the "terminating" phase, each container is sent a SIGTERM or a custom command (if the preStop lifecycle hook is configured on a container) and at the same time the pod immediately automatically advertises an "unready" state.
The rules in 1 and 2 are followed regardless of the reason for termination i.e. if the pod is exceeding max memory / cpu usage, node is OOM etc. In no case will the Pod be SIGKILL'd without first entering the "terminating" phase with the grace period.
Services, ingress etc. will see the change of the pod to the "unready" state via a subscription to the state store and start removing the pod from their own configurations. Because this entire process is async, the pod may still get some additional traffic for a few seconds after it's received a SIGTERM.

B: Handling the termination

If you want 100% throughput, with no dropped traffic it is not recommended that containers actually terminate themselves after being sent a SIGTERM command. Instead, they should clean up what they can and stick around to handle any remaining requests that may trickle through until the grace period expires or at least for the period of time that you guess it would take for services / ingress etc. to update their configurations.
If you're happy to drop some traffic, you can instead actually terminate your processes / containers when you get your SIGTERM signal or have them start failing their liveness probes. Even if marked as "essential", containers will not be restarted if the pod is in a terminating phase. If all containers stop prior to the grace period expiring, the pod immediately enters the "terminated" phase.

@ababkov From what I can see, your description is pretty much correct. (Others are needed to fully judge though.)
Two remarks:
Re: A.3.: I'd expect an OOM'ing container to receive a SIGKILL right away in the conventional Linux sense. For sure it exits with a 137 code, which traditionally represents a fatal error yielding signal n where n = 137 - 128 = 9 = SIGKILL.
Re: A.5.: Ingress sends traffic to the endpoints directly, which means that in principle it's possible for it to retry requests to other pods. Whether that's an actual default or something that can be configured depends on the concrete Ingress controller used. AFAIK, both Nginx and Traefik support retrying.
As far as Services are concerned, you need to distinguish between the two available modes: In userspace mode, requests can be retried. In iptables mode (the current default), they cannot. (There's a ticket to try out IPVS as a third option which, as far as I understood, would bring the benefits of the two modes together: high performance while also being able to retry requests.)
Here's a recommendable read to better understand Ingresses and Services: http://containerops.org/2017/01/30/kubernetes-services-and-ingress-under-x-ray/

Thanks very much for your reply @timoreimann - re A.5 will watch the IPVS item, also the post you linked is really good - helped me to get a better understanding of kube-proxy - had i not spent days gradually coming to the realisation how services and ingress work it probably would have helped for that as well.
Re A.3 - Is your explanation based on a pod that's gone over its memory allocation or a node that is out of memory and killing pods so it can continue running? An immediate sigkill might be a little frustrating if you're trying to ensure your apps have a healthy shut down phase.
If i could get a few more opinions on my post above from one or two others and / or some links to relevant documentation where you think these scenarios are covered in detail (understanding I have done quite a lot of research before coming here) that would be great.
I know I can just experiment with this myself and "see what happens", but if there's someway to shortcut that and learn from others and / or the core team, that would be awesome.

@ababkov re: A.3: Both: There's an OOM killer for the global case (node running out of memory) and the local one (memory cgroup exceeding its limit). See also this comment by @thockin on a Docker issue.
I think that if you run into a situation where the OOM killer has selected a target process, it's already too late for a graceful termination: After all, the (global or local) system took this step in order to avoid failure on a greater scale. If "memory-offending" processes were given an additional opportunity to lengthen this measure arbitrarily, the consequences could be far worse.
I'd argue that it's better to prevent an OOM situation from occurring in the first place by means of monitoring/alerting on memory consumption continuously such that you still have enough time to react.
While doing a bit of googling, I ran across kubelet soft eviction thresholds. With these, you can define an upper threshold and tell kubelet to shut down pods gracefully in time before a hard limit is reached. From what I can tell though, the eviction policies operate on a node level, so it won't help in the case where a single pod exceeds its memory limit.
Again, chances are there's something I'm missing. Since this issue has been closed quite a while ago, my suggestion to hear a few more voices would be to create a new issue / mailing list thread / StackOverflow question. I'd be curious to hear what others have to say, so if you decide to follow my advice please leave a reference behind. :-)

@timoreimann the soft eviction thresholds add another piece to the puzzle - thanks for linking.
Happy to open another topic - i'm still new to the project but presuming i'd open this in this repo in particular?
Topic would be along the lines of trying to get some clarity in place around the nature of the termination lifecycle in every possible scenario that a pod can be terminated.

@ababkov I'd say that if the final goal is to contribute the information you will gain back to Kubernetes project (supposedly in form of better documentation), an issue in the main kubernetes/kubernetes repo seems in order.
OTOH, if this is "only" about getting your questions answered, StackOverflow is probably the better place to ask.
Up to you. :-)

@timoreimann more than happy to contribute - thanks for your help mate.

It is very odd to me that in order to not drop any traffic, SIGTERM should not actually terminate my app, but instead let it hang around for a bit? (Until Ingress updates its configurations). If I wanted actual 100% uptime during this time period, it's not possible with the default k8s? I really would rather not drop traffic if I can help it, and testing with ab definitely shows 502s with an nginx ingress controller.
I think this kind of issue should be prioritized. Otherwise I can try something like that label-based solution mentioned earlier, but then it feels like re-inventing the wheel and seems quite complex.

On Thu, Mar 30, 2017 at 9:39 PM, César Del Solar
<EMAIL_ADDRESS>wrote:

It is very odd to me that in order to not drop any traffic, SIGTERM should not actually terminate my app, but instead let it hang around for a bit? (Until Ingress updates its configurations). If I wanted actual 100% uptime during this time period, it's not possible with the default k8s? I really would rather not drop traffic if I can help it, and testing with ab definitely shows 502s with an nginx ingress controller.

I'm not sure what you're expressing here?  Are you saying that the
SIGTERM/SIGKILL sequence is distasteful?  Or are you saying it doesn't
work in some case?

I think this kind of issue should be prioritized. Otherwise I can try something like that label-based solution mentioned earlier, but then it feels like re-inventing the wheel and seems quite complex.

This was prioritized, and what resulted was the SIGTERM/SIGKILL,
readiness flip, remove from LB pattern.  Can you please clarify the
problem you are experiencing?

@thockin What I am saying is that after SIGTERM is sent, the Ingress still sends traffic to my dying pods for a few seconds, which then causes the end users to see 502 Gateway errors (using for example an Nginx ingress controller). A few people in this thread have mentioned something similar. I don't know of any workarounds, or how to implement that "labels" hack mentioned earlier.
How do I get a zero-downtime deploy?

On Fri, Mar 31, 2017 at 2:25 PM, César Del Solar
<EMAIL_ADDRESS>wrote:

@thockin What I am saying is that after SIGTERM is sent, the Ingress still sends traffic to my dying pods for a few seconds, which then causes the end users to see 502 Gateway errors (using for example an Nginx ingress controller). A few people in this thread have mentioned something similar. I don't know of any workarounds, or how to implement that "labels" hack mentioned earlier.

I'm not sure I understand, still, sorry.    It sends a DELETE call.

The pod is marked as "being deleted".

Now you enter distributed-systems land.  In parallel and totally
asynchronously:

3a) kubelet sends SIGTERM to the Pod, Pod ignores or starts counting
connections, or something
3b) a controller removes the Pod from Endpoints
3c) kube-proxy removes the Pod from virtual LBs
3d) possibly external LBs are updated

Now time passes, generally O(10-seconds)

Kubelet sends SIGKILL if the pod is still alive

The pod goes away.

Where in there are you getting a 502?

How do I get a zero-downtime deploy?

The above should be zero-downtime, at the cost of non-zero realtime.
You have to simply wait - wait for everyone upstream to shut their
traffic flows off, wait for connections to die, etc.

Step 3. Kubelet sends SIGTERM to the Pod. Let's say the pod is running Gunicorn, which, upon receiving a SIGTERM, stops receiving new connections, and gracefully stops; finishing its current requests up until a 30-second timeout.
During this time, since all of 3 is async, Ingress is still sending some traffic to this Gunicorn, which is now refusing all connections. The nginx Ingress controller I am using then returns a 502 Bad Gateway.

On Fri, Mar 31, 2017 at 3:42 PM, César Del Solar
<EMAIL_ADDRESS>wrote:

Step 3. Kubelet sends SIGTERM to the Pod. Let's say the pod is running Gunicorn, which, upon receiving a SIGTERM, stops receiving new connections, and gracefully stops; finishing its current requests up until a 30-second timeout.

That's the wrong behavior.  It's a distributed system.  It should not
stop accepting, it should just consider itself on notice.  Or you can
use a different signal - build your image with a different STOPSIGNAL,
and simply trap-and-ignore it.

During this time, since all of 3 is async, Ingress is still sending some traffic to this Gunicorn, which is now refusing all connections. The nginx Ingress controller I am using then returns a 502 Bad Gateway.

Yeah, as soon as you said the above, I got it.  Closing the socket is
the wrong behavior.
To do what I think you really want would require a different state of
existence for pods - where we have received the DELETE command but
haven't told the pod about it yet, but HAVE told everyone else about
it...

So then, what is the point of the SIGTERM? My preStop could literally just have a bash -c sleep 60?
I'm not sure how to make it so that gunicorn basically ignores the SIGTERM. I set up a hack with a readinessProbe that just tries to cat a file, and in the preStop, I delete the file and sleep 20 seconds. I can see that the pod sticks around for a while, but I still drop some requests. Should I sleep longer? Would it have to do with an Http Keep-Alive? I'm kind of lost as to the best way to implement this.

I added a simple preStop hook:
        lifecycle:
          preStop:
            exec:
              # A hack to try to get 0% downtime during deploys. This should
              # help ensure k8s eventually stops giving this node traffic.
              command: ["sh", "-c", "sleep 75"]

I watch my pods when I do an apply and deploy a new container version. Since I have maxUnavailable: 0 my pod doesn't start terminating until another pod is ready. Then, it starts terminating and the countdown timer above starts.  My container didn't get a sigterm until 75 seconds later (I saw that being logged), which was way after these 502s.

By the way, I think I figured out how to do this; thanks for your help in showing me the rough sequence of events. I think what was happening is that between the new container coming up and it being ready to accept requests, a small amount of time passes, and nginx was directing requests to the new container too early. I added a hack. My readinessProbe now looks like:
        readinessProbe:
          exec:
            command:
            - cat
            - /opt/myapp/test_requirements.txt
          successThreshold: 1
          failureThreshold: 2
          periodSeconds: 5
          initialDelaySeconds: 5   # Important so that the container doesn't get traffic too soon

and the preStop looks like:
        lifecycle:
          preStop:
            exec:
              # A hack to try to get 0% downtime during deploys. This should
              # help ensure k8s eventually stops giving this node traffic.
              command: ["sh", "-c", "rm /opt/myapp/test_requirements.txt && sleep 20"]

This can probably be tweaked a bit; for example, I should be using a real readinessProbe that hits my API, and then on the preStop, run a script to tell my app to start failing the probe, rather than deleting a file. The sleep is required too, so that the old container doesn't die too quickly before the Ingress stops directing traffic to it. The numbers are probably too high but this works, I ran a bunch of tests, and no dropped requests or "connection refused" logs in nginx, which means I think it's doing the right thing. Thanks!

What's wrong with the current process:
The main issue is that SIGTERM basically means... finish what you're doing, don't start any new work and gracefully exit.
At the moment it's instead, finish what you're doing, continue accepting and processing new work for 0 - 10 seconds? ... or however long the load balancers take to update... and somehow work out when an appropriate time might be for you to gracefully exit before you're sent a SIGKILL.
I think the ideal situation here would be to:

modify the termination flow to first put the containers into an unready status.
have each load balancing service register itself as listening to a container.
after each load balancing service see's the unready state of the containers and removes the containers from their pool, they should also remove their "i'm listening to this container" marker.
When all markers are gone (or after some max configurable amount of time), continue with the termination cycle as per usual.

All of that being said, the above is probably overly intricate and not a great idea...
Perhaps the best we can do is:

Educate people better about how this works in the documentation (it really needs a page that diagrams all the different ways a container can be terminated, how termination state flows and potential repercussions).
Add something to the config to allow for the equivalent of the what @domino14 is doing above to delay the sigterm + subsequent sigkill commands by a load balancer grace period. Eg. go into an immediate unready state 10 seconds before attempted SIGTERM and then continue the lifecycle as expected.

@ababkov I think your proposed ideal is very similar to what I described above a few months ago. Effectively, it's request draining, and I believe it's not too uncommon to have that with other networking components. I'd still think that an implementation could primarily reuse already existing concepts (state transitions/observations, LB rotation changes, timers, etc.) and be fairly loosely coupled. It would be "yet another waiting layer" on top (or in place) of the existing SIGTERM logic.
We are currently transitioning 100+ services over to Kubernetes. Having to ask every service owner to provide the kind of SIGTERM procedure needed by Kubernetes is quite an effort, especially for 3rd party software which we need to wrap in custom scripts running pre-stop hooks.
At this stage, opening a separate feature request to discuss the matter seems worthwhile to me.

@timoreimann I agree though I'd ask that you take the lead on that given that you're in the process of transitioning and can probably more clearly explain the use case. I fully understand your current situation (we're about to be in the same position) and am happy to contribute.
The only thing i'm not fully sure of from a proposal standpoint is whether the feature should / can try and address anything beyond a preliminary "draining" status that occurs for a fixed configurable period of time. The alternative of course being implementing a solution where things that route traffic to the container register themselves as listeners (via state updates) and acknowledge (via state updates) when a container had gone into draining status.

@timoreimann we have the same issue, we have to ask every service owner to implement a proper SIGTERM handler to make sure deployment is transparent to the users.
It's true that it would make things easier if the pod was just flagged as not-ready anymore. Give time to remove it behind the service, draining requests, and then SIGTERM....

@ababkov I'd be happy to be the one who files the feature request.
My understanding is that any solution requiring some means of extended coordination will be much harder to push through. @thockin and other members of the community have expressed in the past that too much coupling is rather dangerous in a distributed system like Kubernetes, and it's the reason why Kubernetes was designed differently. Personally, I think that does make a lot of sense.
I think we will time to delve into these and other implementation alternatives on the new ticket. Will drop pointers here once I can find some time to file it (after I made sure no one else had put forward a similar request yet).

@timoreimann based on that i'd suggest that the request be logged as the simpler alternative of allowing a configurable amount of time where the container sits in an unready / draining state prior to the remainder of the termination flow taking place (maybe in reference to this conversation).
That alone would make everything 10 times clearer and more straight forward.

I think at the very least a few notes should be added in the documentation, as it's not clear that a special set of procedures is needed to get actual zero downtime.

@domino14 did you @edouardKaiser's blog post he referenced before?

@ababkov sounds good to me!

Yep, I did see the post and it was helpful in showing me what was going on behind the scenes, but it's still a bit tough to go from that to knowing we need delays in several places, etc, to get zero downtime deploys. That's why I suggested notes in the documentation.
In any case I'm a newcomer to K8S and I appreciate greatly the work done here. I'd submit a documentation patch if I knew the terminology better.

On Mar 31, 2017 11:52 PM, "César Del Solar"<EMAIL_ADDRESS>wrote:

So then, what is the point of the SIGTERM? My preStop could literally just have a bash -c sleep 60?

SIGTERM is notification that your grace period has started.  There are
LOTS of kinds of apps out there - more than we can predict.

I'm not sure how to make it so that gunicorn basically ignores the SIGTERM. I set up a hack with a readinessProbe that
just tries to cat a file, and in the preStop, I delete the file and sleep 20 seconds. I can see that the pod sticks around for
a while, but I still drop some requests. Should I sleep longer? Would it have to do with an Http Keep-Alive? I'm kind of
lost as to the best way to implement this.

There's a way to change the STOPSIGNAL in Dockerfile, but we don't
re-expose that in a pod.  Not sure if that is configurable to Docker
outside of Dockerfile.
A preStop hook is maybe better, as you seem to have found, though
still a little clumsy..
I'm open to exploring more intricate solutions, but I don't have
bandwidth to drive the design right now.  The proposed back-and-forth
between LB providers would be quite a challenge to debug.  I bet we
can find something simpler...

With regards to the original issue, could it be related to differences in connection strategies between the nginx ingress and kube-proxy. At least for the userspace kube-proxy it has a retry if dialling an endpoint fails https://sourcegraph.com/github.com/kubernetes/kubernetes@master/-/blob/pkg/proxy/userspace/proxysocket.go#L94:1-95:1
I'm not sure if nginx ingress or the other kube-proxy modes have a similar strategy.
When gunicorn receives a SIGTERM, does it stop listening on the socket but continue to drain the open requests? In that case it should gracefully drain with kube-proxy userspace since kube-proxy will move on to the next endpoint if gunicorn does not accept the connection.

There's a race still, even with graceful termination.
t0: client deletes pod
t1: apiserver sets pod termination time
t2: kubelet wakes up and marks the pod not-ready
t3:

kubelet sends SIGTERM to pod
controller wakes up and removes pod from endpoints

t4: kube-proxies wake up and remove pod from iptables
in between t3 and t4, an external LB could route traffic.  The pod could
have stopped accept()ing, which will make the external LB unhappy.  Pods
should continue to accept() traffic, but be ready to terminate when
connections reach 0 (or whatever the heuristic is).
The alternative would be to force a sequencing between LBs and pod
signalling, which we don't have today.
On Mon, Apr 17, 2017 at 5:58 PM, Keegan Carruthers-Smith <
<EMAIL_ADDRESS>wrote:

With regards to the original issue, could it be related to differences in
connection strategies between the nginx ingress and kube-proxy. At least
for the userspace kube-proxy it has a retry if dialling an endpoint fails
https://sourcegraph.com/github.com/kubernetes/
kubernetes@master/-/blob/pkg/proxy/userspace/proxysocket.go#L94:1-95:1
I'm not sure if nginx ingress or the other kube-proxy modes have a similar
strategy.
When gunicorn receives a SIGTERM, does it stop listening on the socket but
continue to drain the open requests? In that case it should gracefully
drain with kube-proxy userspace since kube-proxy will move on to the next
endpoint if gunicorn does not accept the connection.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/contrib/issues/1140#issuecomment-294639964,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVOREboHbpuCgk20pHyrwASjbcDcWks5rxAqkgaJpZM4IuliB
.

Even if servers like (g)unicorn are doing the wrong thing theres the no guarantee which is faster a) endpoint update b) SIGTERM on process. I think the endpoint update should be sync and only after its finished we continue with SIGTERM (as discussed above). Any other solution has this timing problem.
Some notes about our testing and workarounds for now:

You can workaround this by using the sleep hack in case of services. Magic sleeps are never really "real" patterns, hacks at best
This problem is mitigated by nginx  for GETs and such because it retries 50x.
-Theres another issue in nginx where full upstream flip (eg. pool [a,b] -> [c,d) causes the proxy_next_upstream to not work. Testable with eg. rollingupdate, replicas 3, maxSurge 25%, maxUnavailable 50% where it ends up splitting from 4 pods (2 old in pool, 2 new getting ready)  to 2 new in pool
I haven't been able to reproduce this on GLBC. Any ideas what it does differently since simply using a service vip doesn't fix it. Does it do the same as proxy_next_upstream but better ?

Theres another issue in nginx where full upstream flip (eg. pool [a,b] -> [c,d) causes the proxy_next_upstream to not work. Testable with eg. rollingupdate, replicas 3, maxSurge 25%, maxUnavailable 50% where it ends up splitting from 4 pods (2 old in pool, 2 new getting ready) to 2 new in pool

The focus after the next nginx controller release (0.9) will be avoid nginx reloads for upstream updates.

👍

Just ran into this issue - at the very least this should be better documented, I'm sure the vast majority of people using rolling updates believe that kubernetes waits until a pod is out of rotation before sending SIGTERM, and this is not the case!
I also think that even if not implemented right away, sequencing SIGTERM to happen after a pod has been taken out of rotation should be the eventual goal: I don't think saying "gunicorn is doing the wrong thing" is a useful response, that's just kubernetes redefining what it thinks these signals mean, gunicorn and other servers behave in a fairly standard way wrt signals, and kubernetes should aim to be compatible with them rather than offloading the problem onto users.

@thockin you mentioned in an earlier comment:

During that time window (30 seconds by default), the Pod is considered “terminating” and will be removed from any Services by a controller (async). The Pod itself only needs to catch the SIGTERM, and start failing any readiness probes.
Why do you need to start failing the readiness probes?
When a Pod is marked as Terminating, the Endpoint controller observes the change and removes the pod from Endpoints.
Since this is already done as a result of a terminating pod, why do we also need to start failing the readiness probe, which (after it fails a certain amount of times) will also result in the prod being removed as the endpoint of the service.

@simonkotwicz you do not need to start failing the readiness probe (these days anymore?). I'm not sure if it used to be like that, but it's certainly not the case these days. Things work fine without that part.


BibLaTeX always exports title case

If I generate a .bib file with BetterBibLaTeX, I always get title casing, whereas using the native BibTeX exporter, text is formatted correctly.
With BetterBibLaTeX:
@article{wright_genetics_1946,
title = {Genetics of Natural Populations. {{XII}}. Experimental Reproduction of Some of the Changes Caused by Natural Selection in Certain Populations of {{{\emph{Drosophila}}}}{\emph{ Pseudoobscura}}},
volume = {31},
issn = {0016-6731},
url = {http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1209320/},
timestamp = {2017-03-16T11:03:56Z},
number = {2},
journaltitle = {Genetics},
shortjournal = {Genetics},
author = {Wright, Sewall and Dobzhansky, Theodosius},
urldate = {2017-02-15},
date = {1946-03},
pages = {125--156},
file = {...},
eprinttype = {pmid},
eprint = {21021044},
pmcid = {PMC1209320}
}

With BibTeX:
@article{wright_genetics_1946,
title = {Genetics of natural populations. {XII}. experimental reproduction of some of the changes caused by natural selection in certain populations of \textit{{Drosophila} pseudoobscura}},
volume = {31},
issn = {0016-6731},
url = {http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1209320/},
number = {2},
urldate = {2017-02-15},
journal = {Genetics},
author = {Wright, Sewall and Dobzhansky, Theodosius},
month = mar,
year = {1946},
pmid = {21021044},
pmcid = {PMC1209320},
pages = {125--156},
file = {PubMed Central Full Text PDF:/Users/Mark/Library/Application Support/Zotero/Profiles/h74ggxun.default/zotero/storage/2GJX2NSH/Wright and Dobzhansky - 1946 - Genetics of Natural Populations. Xii. Experimental.pdf:application/pdf}
}

The title is formatted in Zotero as Genetics of natural populations. XII. experimental reproduction of some of the changes caused by natural selection in certain populations of Drosophila melanogaster
I will feel silly if this is just an option somewhere, but I couldn't find anything.
PCJ3BWK9
Steps to reproduce:
export using BibTeX
export using BetterBibLaTeX
compare
Thanks!

Not meaning to be pedantic about it, but it is BetterBibLaTeX which exports correctly, not the standard BibLaTeX export. Citeproc (which Zotero uses to format references) expects titles to be in sentence case, but biblatex processors expect titles of English (or language-unspecified) references to be in Title Case, so in order to preserve intent, BBT title-cases those references. You can go off-spec by setting the hidden preference (they talk about macs but the instruction works on all platforms) extensions.zotero.translators.better-bibtex.suppressTitleCase to true, but then references may render wrong; I don't know how biblatex behaves on sentence-cased references.

The life sciences are a tough place to be for bibliographies, I guess. It seems to work alright with the vanilla BibTeX file.
I'm guessing this trick won't work for standalone Zotero?

Again, no intention to lecture you on your field, which I know nothing about, or about the pragmatics of getting your references out, which must trump any ivory tower discussion about this, but if your references work in sentence case, that is likely just a coincidence of the bibliography style in use.
Fortunately, the case suppression works exactly the same in standalone. But are you sure the title cased references render incorrectly?

I haven't seen any issues so far. I would have been happy to leave them in title case for this project, but for those pesky species names. For future reference, is there a way to suppress individual words/characters without changing that option?

Ah, I see. Uh, yes, there is, but you may not like it: you can surround any part of the reference in ... and anything between those will not be case-meddled with by either Zotero or BBT.
This is, of course, hideous, and not listed on the Zotero page that details their "rich" markup, but it is supported by virtue of the fact that they use citeproc to produce bibliographies, and citeproc supports it; as a matter of fact, if you want to have references that work both in things like the Word/LibreOffice plugins and BibTeX (and render to the same output), the nocase spans are the only way which will get that done.
The references you have now will not render correctly in some styles available in Zotero itself, and BBT aims to produce BibTeX that respects the specced intent of the references... and that means BBT must do titlecasing by default. But if you're just using Zotero as a fancy BibTeX manager, suppressTitleCase will do the job in the majority of cases.

That's good to know, even if it is a bit of a pain. Thanks.

My pleasure. For completeness: the nocase stuff isn't my design, I'm just implementing a counterpart to zotero functionality. I wouldn't half mind seeing  a more elegant solution.


Android Hardware Back Button Doesn't Allow Modal Re-Open

I'm running into an issue on Android.

Open Modal
Press hardware back button
The modal is now closed
Tap to open the modal again
Modal is not visible. I'm able to see the render and console log the bool I'm using to show the modal (which is true).

Any ideas why this is happening?

+1


Bump discord-haskell to 1.12.4

Todo:

[x] Check the diff between 1.12.0 and 1.12.4, and handle changes appropriately. If the patch doesn't necessitate a change in code, include both in the version bounds.
[x] Bump version bounds in stack file, autogenerate cabal file
[x] Update versions in the README
[x] Increase patch version or minor version, whichever is appropriate

No changes required to relax the version bounds to include 1.12.1, 1.12.3, and 1.12.4. Updated metadata on Hackage, README, and stack/cabal files.


ci: Implement build and release

This PR creates a github workflow that builds bgfx using bgfx.cmake.
If successful, and pushed to master, the revision and sha1 in version.h will be updated accordingly.
A release will then be created.
This was in support of trying to get a bgfx formula into homebrew, see:
https://github.com/Homebrew/homebrew-core/pull/85145

Can't rely on CMake in main repo. Can you make this part of bgfx.cmake repo?

Can't rely on CMake in main repo. Can you make this part of bgfx.cmake repo?

Thank you. I figured this would be an issue.
Maybe I can make the formula bgfx.cmake instead since Homebrew wants to download release tags from the main repo.
Even if that works, I still have version.h to worry about.  Maybe I can adapt the formula to update it before it makes the bottle.

I think bgfx.cmake could be the main repo for homebrew.


triage survey feedback

[ ] spin up tickets summarising the feedback
[ ] schedule meeting with dev team / interest group
[ ] MSCW tickets
[ ] schedule & assign ownership

duplicate.