[BUG] Quirk Issue / Question

Bug description
I've recently bought a smart plug from Tuya which is labeled as: "_TZ3000_lotmgthb TS011F", it loads the standard "zhaquirks.tuya.ts011f_plug.Plug", I've noticed that the ENDPOINT and INPUT_CLUSTERS are pretty similar, except that the one in the quirk includes an INPUT as [9] but reading the remainder of the quirks within the same file I couldn't figure out what it stands for, I can write and load custom quirks, but writing my own custom quirk by using others as an example I land at the standard quirk mentioned above.
 I re-wrote the quirk using the skeleton of a different one that works at home, but honestly it looks exactly the same as the quirk it's already loading on it.
Steps to reproduce
Adding the plug and going through the device settings page trying to change params on it doesnt work, but it doesnt throw errors on the log, it just doesnt affect the unit.
Expected behavior
Led turn-off/turn-on, last state setup or child lock doesn't work on it.
Screenshots/Video
Screenshots/Video
[Paste/upload your media here]

Device signature
Device signature
"signature": {
      "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.FullFunctionDevice|MainsPowered|RxOnWhenIdle|AllocateAddress: 142>, manufacturer_code=4417, maximum_buffer_size=66, maximum_incoming_transfer_size=66, server_mask=10752, maximum_outgoing_transfer_size=66, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)",
      "endpoints": {
        "1": {
          "profile_id": "0x0104",
          "device_type": "0x010a",
          "input_clusters": [
            "0x0000",
            "0x0003",
            "0x0004",
            "0x0005",
            "0x0006",
            "0x0702",
            "0x0b04",
            "0xe000",
            "0xe001"
          ],
          "output_clusters": [
            "0x000a",
            "0x0019"
          ]
        }
      },
      "manufacturer": "_TZ3000_lotmgthb",
      "model": "TS011F"
    },

Diagnostic information
Diagnostic information
{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2023.10.3",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.11.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "America/Sao_Paulo",
    "os_name": "Linux",
    "os_version": "6.1.21-v8",
    "supervisor": "2023.10.0",
    "host_os": "Home Assistant OS 11.0",
    "docker_version": "24.0.6",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "sonoff": {
      "version": "3.5.2",
      "requirements": [
        "pycryptodome>=3.6.6"
      ]
    },
    "watchman": {
      "version": "0.5.1",
      "requirements": [
        "prettytable==3.0.0"
      ]
    },
    "hacs": {
      "version": "1.33.0",
      "requirements": [
        "aiogithubapi>=22.10.1"
      ]
    }
  },
  "integration_manifest": {
    "domain": "zha",
    "name": "Zigbee Home Automation",
    "after_dependencies": [
      "onboarding",
      "usb"
    ],
    "codeowners": [
      "@dmulcahey",
      "@adminiuga",
      "@puddly"
    ],
    "config_flow": true,
    "dependencies": [
      "file_upload"
    ],
    "documentation": "https://www.home-assistant.io/integrations/zha",
    "iot_class": "local_polling",
    "loggers": [
      "aiosqlite",
      "bellows",
      "crccheck",
      "pure_pcapy3",
      "zhaquirks",
      "zigpy",
      "zigpy_deconz",
      "zigpy_xbee",
      "zigpy_zigate",
      "zigpy_znp",
      "universal_silabs_flasher"
    ],
    "requirements": [
      "bellows==0.36.5",
      "pyserial==3.5",
      "pyserial-asyncio==0.6",
      "zha-quirks==0.0.104",
      "zigpy-deconz==0.21.1",
      "zigpy==0.57.2",
      "zigpy-xbee==0.18.3",
      "zigpy-zigate==0.11.0",
      "zigpy-znp==0.11.5",
      "universal-silabs-flasher==0.0.14",
      "pyserial-asyncio-fast==0.11"
    ],
    "usb": [
      {
        "vid": "10C4",
        "pid": "EA60",
        "description": "*2652*",
        "known_devices": [
          "slae.sh cc2652rb stick"
        ]
      },
      {
        "vid": "1A86",
        "pid": "55D4",
        "description": "*sonoff*plus*",
        "known_devices": [
          "sonoff zigbee dongle plus v2"
        ]
      },
      {
        "vid": "10C4",
        "pid": "EA60",
        "description": "*sonoff*plus*",
        "known_devices": [
          "sonoff zigbee dongle plus"
        ]
      },
      {
        "vid": "10C4",
        "pid": "EA60",
        "description": "*tubeszb*",
        "known_devices": [
          "TubesZB Coordinator"
        ]
      },
      {
        "vid": "1A86",
        "pid": "7523",
        "description": "*tubeszb*",
        "known_devices": [
          "TubesZB Coordinator"
        ]
      },
      {
        "vid": "1A86",
        "pid": "7523",
        "description": "*zigstar*",
        "known_devices": [
          "ZigStar Coordinators"
        ]
      },
      {
        "vid": "1CF1",
        "pid": "0030",
        "description": "*conbee*",
        "known_devices": [
          "Conbee II"
        ]
      },
      {
        "vid": "10C4",
        "pid": "8A2A",
        "description": "*zigbee*",
        "known_devices": [
          "Nortek HUSBZB-1"
        ]
      },
      {
        "vid": "0403",
        "pid": "6015",
        "description": "*zigate*",
        "known_devices": [
          "ZiGate+"
        ]
      },
      {
        "vid": "10C4",
        "pid": "EA60",
        "description": "*zigate*",
        "known_devices": [
          "ZiGate"
        ]
      },
      {
        "vid": "10C4",
        "pid": "8B34",
        "description": "*bv 2010/10*",
        "known_devices": [
          "Bitron Video AV2010/10"
        ]
      }
    ],
    "zeroconf": [
      {
        "type": "_esphomelib._tcp.local.",
        "name": "tube*"
      },
      {
        "type": "_zigate-zigbee-gateway._tcp.local.",
        "name": "*zigate*"
      },
      {
        "type": "_zigstar_gw._tcp.local.",
        "name": "*zigstar*"
      },
      {
        "type": "_uzg-01._tcp.local.",
        "name": "uzg-01*"
      },
      {
        "type": "_slzb-06._tcp.local.",
        "name": "slzb-06*"
      }
    ],
    "is_built_in": true
  },
  "data": {
    "ieee": "**REDACTED**",
    "nwk": 58737,
    "manufacturer": "_TZ3000_lotmgthb",
    "model": "TS011F",
    "name": "_TZ3000_lotmgthb TS011F",
    "quirk_applied": true,
    "quirk_class": "zhaquirks.tuya.ts011f_plug.Plug",
    "manufacturer_code": 4417,
    "power_source": "Mains",
    "lqi": 144,
    "rssi": -64,
    "last_seen": "2023-10-14T19:23:55",
    "available": true,
    "device_type": "Router",
    "signature": {
      "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.Router: 1>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.FullFunctionDevice|MainsPowered|RxOnWhenIdle|AllocateAddress: 142>, manufacturer_code=4417, maximum_buffer_size=66, maximum_incoming_transfer_size=66, server_mask=10752, maximum_outgoing_transfer_size=66, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=False, *is_full_function_device=True, *is_mains_powered=True, *is_receiver_on_when_idle=True, *is_router=True, *is_security_capable=False)",
      "endpoints": {
        "1": {
          "profile_id": "0x0104",
          "device_type": "0x010a",
          "input_clusters": [
            "0x0000",
            "0x0003",
            "0x0004",
            "0x0005",
            "0x0006",
            "0x0702",
            "0x0b04",
            "0xe000",
            "0xe001"
          ],
          "output_clusters": [
            "0x000a",
            "0x0019"
          ]
        }
      },
      "manufacturer": "_TZ3000_lotmgthb",
      "model": "TS011F"
    },
    "active_coordinator": false,
    "entities": [
      {
        "entity_id": "button.tomada_hall_entrada_identify",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "select.tomada_hall_entrada_power_on_state",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "select.tomada_hall_entrada_backlight_mode",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "sensor.tomada_hall_entrada_rms_current",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "sensor.tomada_hall_entrada_rms_voltage",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "sensor.tomada_hall_entrada_power_factor",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "sensor.tomada_hall_entrada_active_power",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "sensor.tomada_hall_entrada_summation_delivered",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "switch.tomada_hall_entrada_switch",
        "name": "_TZ3000_lotmgthb TS011F"
      },
      {
        "entity_id": "switch.tomada_hall_entrada_child_lock",
        "name": "_TZ3000_lotmgthb TS011F"
      }
    ],
    "neighbors": [],
    "routes": [],
    "endpoint_names": [
      {
        "name": "ON_OFF_PLUG_IN_UNIT"
      }
    ],
    "user_given_name": "Tomada Hall Entrada",
    "device_reg_id": "44660936b737cd5b8edf9b32d4346904",
    "area_id": "cozinha",
    "cluster_details": {
      "1": {
        "device_type": {
          "name": "ON_OFF_PLUG_IN_UNIT",
          "id": 266
        },
        "profile_id": 260,
        "in_clusters": {
          "0x0000": {
            "endpoint_attribute": "basic",
            "attributes": {
              "0x0001": {
                "attribute_name": "app_version",
                "value": 192
              },
              "0x0004": {
                "attribute_name": "manufacturer",
                "value": "_TZ3000_lotmgthb"
              },
              "0x0005": {
                "attribute_name": "model",
                "value": "TS011F"
              },
              "0x0007": {
                "attribute_name": "power_source",
                "value": 1
              },
              "0xfffe": {
                "attribute_name": "reporting_status",
                "value": 0
              },
              "0x0000": {
                "attribute_name": "zcl_version",
                "value": 3
              }
            },
            "unsupported_attributes": {}
          },
          "0x0003": {
            "endpoint_attribute": "identify",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x0004": {
            "endpoint_attribute": "groups",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x0005": {
            "endpoint_attribute": "scenes",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x0006": {
            "endpoint_attribute": "on_off",
            "attributes": {
              "0x8001": {
                "attribute_name": "backlight_mode",
                "value": 0
              },
              "0x8000": {
                "attribute_name": "child_lock",
                "value": 1
              },
              "0x4002": {
                "attribute_name": "off_wait_time",
                "value": 0
              },
              "0x0000": {
                "attribute_name": "on_off",
                "value": 1
              },
              "0x4001": {
                "attribute_name": "on_time",
                "value": 0
              },
              "0x8002": {
                "attribute_name": "power_on_state",
                "value": 2
              }
            },
            "unsupported_attributes": {
              "0x4003": {
                "attribute_name": "start_up_on_off"
              }
            }
          },
          "0x0702": {
            "endpoint_attribute": "smartenergy_metering",
            "attributes": {
              "0x0000": {
                "attribute_name": "current_summ_delivered",
                "value": 71
              },
              "0x0302": {
                "attribute_name": "divisor",
                "value": 100
              },
              "0x0306": {
                "attribute_name": "metering_device_type",
                "value": 0
              },
              "0x0301": {
                "attribute_name": "multiplier",
                "value": 1
              },
              "0x0200": {
                "attribute_name": "status",
                "value": 0
              },
              "0x0303": {
                "attribute_name": "summation_formatting",
                "value": 0
              },
              "0x0300": {
                "attribute_name": "unit_of_measure",
                "value": 0
              }
            },
            "unsupported_attributes": {
              "0x0400": {
                "attribute_name": "instantaneous_demand"
              },
              "0x0100": {
                "attribute_name": "current_tier1_summ_delivered"
              },
              "0x0102": {
                "attribute_name": "current_tier2_summ_delivered"
              },
              "0x0304": {
                "attribute_name": "demand_formatting"
              },
              "0x0104": {
                "attribute_name": "current_tier3_summ_delivered"
              },
              "0x0106": {
                "attribute_name": "current_tier4_summ_delivered"
              },
              "0x0108": {
                "attribute_name": "current_tier5_summ_delivered"
              },
              "0x010a": {
                "attribute_name": "current_tier6_summ_delivered"
              }
            }
          },
          "0x0b04": {
            "endpoint_attribute": "electrical_measurement",
            "attributes": {
              "0x0603": {
                "attribute_name": "ac_current_divisor",
                "value": 1000
              },
              "0x0602": {
                "attribute_name": "ac_current_multiplier",
                "value": 1
              },
              "0x050b": {
                "attribute_name": "active_power",
                "value": 0
              },
              "0x0508": {
                "attribute_name": "rms_current",
                "value": 0
              },
              "0x0505": {
                "attribute_name": "rms_voltage",
                "value": 125
              }
            },
            "unsupported_attributes": {
              "0x0300": {
                "attribute_name": "ac_frequency"
              },
              "0x0601": {
                "attribute_name": "ac_voltage_divisor"
              },
              "0x0302": {
                "attribute_name": "ac_frequency_max"
              },
              "0x0600": {
                "attribute_name": "ac_voltage_multiplier"
              },
              "0x0604": {
                "attribute_name": "ac_power_multiplier"
              },
              "0x050d": {
                "attribute_name": "active_power_max"
              },
              "0x0605": {
                "attribute_name": "ac_power_divisor"
              },
              "0x0507": {
                "attribute_name": "rms_voltage_max"
              },
              "0x0401": {
                "attribute_name": "ac_frequency_divisor"
              },
              "0x050a": {
                "attribute_name": "rms_current_max"
              },
              "0x0000": {
                "attribute_name": "measurement_type"
              },
              "0x0403": {
                "attribute_name": "power_divisor"
              },
              "0x050f": {
                "attribute_name": "apparent_power"
              },
              "0x0402": {
                "attribute_name": "power_multiplier"
              },
              "0x0400": {
                "attribute_name": "ac_frequency_multiplier"
              }
            }
          },
          "0xe000": {
            "endpoint_attribute": "tuya_manufacturer_specific_57344",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0xe001": {
            "endpoint_attribute": "tuya_external_switch_type",
            "attributes": {
              "0xd030": {
                "attribute_name": "external_switch_type",
                "value": 0
              }
            },
            "unsupported_attributes": {}
          }
        },
        "out_clusters": {
          "0x000a": {
            "endpoint_attribute": "time",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x0019": {
            "endpoint_attribute": "ota",
            "attributes": {},
            "unsupported_attributes": {}
          }
        }
      }
    }
  }
}

Logs
Logs
[Paste the logs here]

Additional information
This is exactly what I had written:
class Plug_v2_Mod(CustomDevice):
    """Another TS011F Tuya plug. First one using this definition is _TZ3000_lotmgthb."""

    signature = {
		MODELS_INFO: [("_TZ3000_lotmgthb", "TS011F")],
        ENDPOINTS: {
            # "profile_id": 260,
            # "device_type": "0x010a",
            # "in_clusters": ["0x0000", "0x0003", "0x0004", "0x0005", "0x0006", "0x0702", "0x0b04", "0xe000", "0xe001"],
            # "out_clusters": ["0x000a", "0x0019"]
            1: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.ON_OFF_PLUG_IN_UNIT,
                INPUT_CLUSTERS: [
                    Basic.cluster_id,
                    Identify.cluster_id,
                    Groups.cluster_id,
                    Scenes.cluster_id,
                    OnOff.cluster_id,
                    Metering.cluster_id,
                    ElectricalMeasurement.cluster_id,
                    TuyaZBE000Cluster.cluster_id,
                    TuyaZBExternalSwitchTypeCluster.cluster_id,
                ],
                OUTPUT_CLUSTERS: [Time.cluster_id, Ota.cluster_id],
            },
        },
    }
    replacement = {
        ENDPOINTS: {
            1: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.ON_OFF_PLUG_IN_UNIT,
                INPUT_CLUSTERS: [
                    Basic.cluster_id,
                    Identify.cluster_id,
                    Groups.cluster_id,
                    Scenes.cluster_id,
                    TuyaZBOnOffAttributeCluster,
                    TuyaZBMeteringCluster,
                    TuyaZBElectricalMeasurement,
                    TuyaZBE000Cluster,
                    TuyaZBExternalSwitchTypeCluster,
                ],
                OUTPUT_CLUSTERS: [Time.cluster_id, Ota.cluster_id],
            },
        },
    }

Change CustomDevice to EnchantedDevice (might need to import that). Then, restart HA with the new custom quirk and reconfigure the plug using the UI (or re-pair it).
Then, check if it works. Otherwise, I think your plug doesn't support those attributes for some reason.

Out of curiosity, what's the difference between Custom and Enchanted?
I just did what you recommended, I can see the quirk loaded into the system on the log:
Logger: zhaquirks
Source: components/zha/init.py:122
First occurred: 20:35:32 (1 occurrences)
Last logged: 20:35:32
Loaded custom quirks. Please contribute them to https://github.com/zigpy/zha-device-handlers
But removing/adding the plug still defaults to the standard Plug implementation, so looking at the re-added plug lists:
Quirk: zhaquirks.tuya.ts011f_plug.Plug
Instead of the one I wrote-up. Did I miss anything? Thanks!

It was missing Endpoint 242, I got it re-added using the custom quirk, but still couldnt get it to work with the Led Status / Last State / Child lock.

Custom and Enchanted?

Enchanted "casts a Tuya spell" when reconfiguring clusters (so during pairing) that Tuya devices need in order to work properly. (They try to detect if they're connected to a Tuya hub)

But removing/adding the plug still defaults to the standard Plug implementation

Well, then you don't need a custom quirk, because an existing one already matches.
If child lock and so on don't work, then it's possible that the plug just doesn't support them in the first place.
How are you trying to configure those options? Using the clusters menu (Manage Zigbee device)?
If so, make sure to only change the upper text field. Always leave the lower text field completely empty.
Can you read the attributes on the Tuya OnOff clusters using the clusters UI?


TypeScript compilation with --no-babel

I am trying the sleep.js example with TypeScript and avoid Babel Transpilation since TypeScript performs Transpilation.
The output I get from TypeScript is the following:
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, Promise, generator) {
    return new Promise(function (resolve, reject) {
        generator = generator.call(thisArg, _arguments);
        function cast(value) { return value instanceof Promise && value.constructor === Promise ? value : new Promise(function (resolve) { resolve(value); }); }
        function onfulfill(value) { try { step("next", value); } catch (e) { reject(e); } }
        function onreject(value) { try { step("throw", value); } catch (e) { reject(e); } }
        function step(verb, value) {
            var result = generator[verb](value);
            result.done ? resolve(result.value) : cast(result.value).then(onfulfill, onreject);
        }
        step("next", void 0);
    });
};
var micro_1 = require('micro');
var then_sleep_1 = require('then-sleep');
exports.default = function (req, res) {
    return __awaiter(this, void 0, Promise, function* () {
        yield then_sleep_1.default(500);
        micro_1.send(res, 200, 'Ready!');
    });
};

It appears to start fine however visiting localhost:3000 has the following error
TypeError: then_sleep_1.default is not a function

I suspect this might require changes to then-sleep module however it would be nice to see how the same example without the async/await.
What is the expected ES6 code (without async/await) for this sleep example?

Are you running it with the --no-babel flag?

@rauchg Correct I am running with --no-babel as the title says. I'm not entirely convinced that it is avoiding babel because startup time is still 1 or 2 seconds.
https://github.com/styfle/micro-ts-example/blob/master/package.json

I believe the problem comes from the fact that all of the babel modules (except babel-register) get imported unconditionally.
:+1: for fixing this because at the moment I'm not using micro CLI, partly because of this startup time issue.

You can use micro without the CLI this way:
import micro from 'micro';

async function requestHandler(req, res) { ... }

micro(requestHandler).listen(process.env.PORT || 3000);

I don't know TS, but basically, the idea is that if you import micro you can use it to create a HTTP server passing your requestHandler (your function) as an argument and then using .listen(port).
This way you can just run node index.js, of course, you should need to transpile your code from ES* or TS to JS.
You can read about it in the docs.

@sergiodxa I think this is really the correct solution. Thanks!
Here is the same thing in typescript:
import * as micro from 'micro';
import {IncomingMessage, ServerResponse} from 'http';

async function requestHandler(req: IncomingMessage, res: ServerResponse) {
    res.writeHead(200);
    res.end('hello world');
}

micro(requestHandler).listen(process.env.PORT || 3000);

Moving forward, we're making micro-serve(1) take the exported file that's been pre-compiled, which should be much better for production.


2x scale photo without downscale

Hi @jmiller656, thank you for your great work, i have some photo want to upscale it by 2x , but after i train the network and try to use it to scale my own photo to 2x , it downscaled and croped, could you please explan to me how can to train the network to upscal 2x dirctly , then how to use test.py to this target .

Hey, sorry about that! Please stay tuned, I plan on having this fixed within the next week.

wow, that is really great news، i will be waiting for this update until you get some time to it, thanks you 👍

@naseeresmael just wanted to notify you, I've submitted a fix for this issue. If you run into any more issues, feel free to reopen this issue or create a new one. Thanks!

ok, thank you @jmiller656 For your efforts 👍👏

hi @jmiller656, when i try to trining i found this error
Done building!
Begin training...
0%|
Traceback (most recent call last):
File "train.py", line 18, in
network.train(args.iterations,args.savedir)
File "F:\EDSR\model.py", line 212, in train
x,y = self.data(*self.args)
File "F:\EDSR\data.py", line 66, in get_batch
img = crop_center(img,original_size,original_size)
File "F:\EDSR\data.py", line 81, in crop_center
y,x,_ = img.shape
ValueError: not enough values to unpack (expected 3, got 2)
how can i fix that, sorry for wasting your time.


solvers types names

The current type system for the solvers (including the changes in the latest PRs)

LinearSolver

DirectSolver

GaussElimination
HansenBliekRohn
LinearOettliPrager

IterativeSolver

GaussSeidel
Jacobi
Krawczyk

OettliPrager (solves nonlinear equalities using ICP.jl)

I think there's room for improvement, here's a proposal

AbstractIntervalLinearSolver

AbstractDirectIntervalLinearSolver

GaussElimination
HansenBliekRohn
LinearOettliPrager

AbstractIterativeIntervalLinearSolver

GaussSeidel
Jacobi
LinearKrawczyk (to distinguish from Krawczyk exported in IntervalRootFining.jl)
NonLinearOettliPrager

what do you think? Feel free to comment other proposals or suggestions

similarly for Preconditioning the current Precondition could be renamed AbstractPrecondition

I have mixed feelings about the verbose abstract names, on the one hand the package is called IntervalLinearAlgebra so maybe the IntervalLinear is redundant, but on the other hand they are exact and unambiguous, Otherwise DirectSolver could be expected to solve also non-interval non-linear problems.

Gauss -> Gaussian

the advantage of haivng Abstract... is that it's clear they are abstract types. at least for someone new to the library.
i would only suggest to remove the "Interval" word for the direct subtypes of AbstractIntervalSolver, like this:

AbstractIntervalSolver

AbstractDirectSolver

GaussElimination
HansenBliekRohn
LinearOettliPrager

AbstractIterativeSolver

GaussSeidel
Jacobi
LinearKrawczyk (to distinguish from Krawczyk exported in IntervalRootFining.jl)
NonLinearOettliPrager

just double checking, the most abstract would be AbstractSolver `?

the abstract types are generally not exported from the library. hence you could pick the name you prefer, since eg. AbstractSolver  (which exists in other libraries for sure), is going to be used only internally.


TinyUSB support for ESP32-C6 (IDFGH-10682)

Is your feature request related to a problem?
No response
Describe the solution you'd like.
I wanted to use some ESP32-C6's (that I already have) for some new projects that involve USB (possibly both HID and serial). I had seen some USB examples in the IDF repository and had assumed they'd work for all modern ESP32 boards.
However, now I see that the examples (and tinyusb components) only have ESP32-S2/ESP32-S3 as the target and trying to use them on a C6 results in errors like:
CMake Error at /Users/danny/Dev/Microcontrollers/ESP-IDF-stable/esp-idf/tools/cmake/build.cmake:540 (message):
  HINT: Component "espressif/esp_tinyusb" has suitable versions for other
  targets: "esp32s2", "esp32s3".  Is your current target "esp32c6" set
  correctly?

I understand these modules have different CPUs, though I have no idea of the scale of work to support the C6 (it seems like tinyusb does support some RISC-V chips already?).
Describe alternatives you've considered.
No response
Additional context.
No response

ESP32C6 doesn't have the USB-OTG IP, what's why TinyUSB is not supported.

Oh, I see.  Is it possible to use C6 as a USB device at all (eg. without using TinyUSB)?
If not, are the S2/S3 the only boards that have the hardware to support this? I was trying to limit the number of different boards I use and had picked the C6 because of Zigbee/Thread support (and I had naively assumed the C6 being new was an improved version of the earlier boards 😄).
Thanks!

Is it possible to use C6 as a USB device at all (eg. without using TinyUSB)?

C6 can work as a USB CDC Device thanks to another USB-related peripheral called "USB-Serial-JTAG". https://www.espressif.com/sites/default/files/documentation/esp32-c6_technical_reference_manual_en.pdf#usj
You may want to  check this example for the usage: https://github.com/espressif/esp-idf/blob/master/examples/system/console/basic/main/console_example_main.c#L110

Thanks - I'll take a look, but it sounds like it won't support being a HID device?
Just to ensure I understand correctly - the S3 has specific hardware that is required to be a USB device (like HID or MSC) that isn't present on the C6? I had assumed that as long as USB is connected to the chip that any kind of use would be possible from software, but it sounds like that was a bad assumption (I'm a newbie in this area) :-)
I suspect the best thing will be to pick up some S3 boards. I presume ESP32-S3 is the newest available board that will support this?
Thank you!

I picked up some S2 boards and will experiment with them. Thanks!


download_geoip.py does not respect CAs in System: Trust: Authorities

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:
[-] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[-] I have searched the existing issues and I'm convinced that mine is new.
Is your feature request related to a problem? Please describe.
Since the addition of the GeoIP setting tab, I have been unable to download GeoIP data from our server. From the command line if I run download_geoip.py I get:

OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

We are downloading the GeoIP data from an internal server whose certificate is signed by an internal CA. This CA is listed in "System: Trust: Authorities", but download_geoip.py does not seem to honor that. I have temporarily replaced the certificate with one generated by Let's Encrypt and verified that the data then successfully downloads, so I am confident that this is the only problem. Unfortunately, keeping a Let's Encrypt certificate on this server is not a viable long-term solution for us.
Describe the solution you'd like
I would like to be able to download GeoIP data from a server whose certificate is signed with an internal signing certificate.
Describe alternatives you've considered
I verified that our internal CA is being populated in /etc/ssl/cert.perm. I would like either see download_geoip.py honor /etc/ssl/cert.perm or "System: Trust: Authorities" also populate CAs whereever download_geoip.py is looking for them.
Additional context

Long story short: https://github.com/certifi/python-certifi/issues/22
We use Python to download the files.

would it be hard to use system to invoke curl to download the file and then have the python script read the file from disk instead?

Exiting python to download a file using curl will get hackery pretty quickly unfortunately, I don't think the python requests library supports other options than certifi at the moment and we don't want to solve the same kind of problem in multiple areas.
We're open for suggestions, as long as they can be implemented generically and don't turn into a lot of magic.

What about using pycurl instead of certifi? It would allow you to specify CA locations.

When using requests, things are easy: CA Certs Handling

Looking at our forum, I was under the impression that people already tried that https://forum.opnsense.org/index.php?topic=6679.0 , we do use python requests indeed (and don't intend switching to something else currently).
If someone wants to try this, the following should be enough to add REQUESTS_CA_BUNDLE to the environment:
set REQUESTS_CA_BUNDLE="/usr/local/openssl/cert.pem"

I would prefer /etc/ssl/cert.pem for future compatibility (this is where FreeBSD looks for certificates as well)

There is no simple / single answer, just lists: https://serverfault.com/a/722646

so things aren't easy.... (when using requests)

Depends on what hard things you've seen in the wild – setting an env var counts as easy in my book. ;)

@jhermann just repeating your own words, first it's simple, then there's no simple answer. It would be more meaningful if one just tried something out on OPNsense, just my two cents....

should likely work since https://github.com/opnsense/core/commit/ae3650587394f0fb6d3c129ae00a7f5d0b079dc7


How to get top-5 or top-3 accuracy by ConfusionMatrix?

Hello all,
Can I use ConfusionMatrix to compute top-n accuracy for my classifier or not?
If yes, how can I do that? Is there any sample code to do that?
Thanks

hi,
I think top-k acc. need to hack the ConfusionMatrix.lua file.
And there're two ways to calculate top-k acc., hope the following helps.

please kindly check the resnet training script L107.

Torchnet's utils for topk.


AL Namespaces

Since names and ID's should be unique across all APPs it would make sense to introduce a namespace functionality.
You should be able to assign a namespace for an app. for example, in the app.json like the ID range. Thus, all objects in the database would always receive the namespace as a prefix and it would lead to fewer conflicts between apps.
My attitude towards number ranges is well known.

I would support this request because we are limited with 30 character names and we already have situation where we need to extend same object from different apps and we want to use meaningful names in our objects, so I would ask you guys to definitely push this one to top of your requirements. I have to say that number ranges does not help here. Namespaces would solve this issue efficently.
Thanks

Once again, prefix system is total mess. We have our own codeunit that extends Dimension Management functionality, and the object is called "XXX Dimension Management". Now I know we could call it differently, but we want to stay inline with naming conventions and we want names to be as self sufficient as possible.
Here's the catch: If you name all of your objects inside an app with prefix, logical thing would be to name your object variables starting with the prefix. Now that gives you a spectacular code mess...

This prefix solution is just a plain band aid, but it doesn't hold well. Come on guys, introduce simple namespace to app config files that would be read when app is being loaded and simply added like prefix to existing object name. It can not be so hard to change internal nav service code to have object names with namespace longer then 30 characters?!

I am closing this suggestion as ideas are tracked on http://aka.ms/bcideas. This topic already has an entry
https://experience.dynamics.com/ideas/idea/?ideaid=57b8d9d1-e28b-ea11-99e5-0003ff68f606


Build paths

I set up a fresh generator.
Doing gulp build creates the files in my /dist folder.
The paths for the CSS + JS in the index.html file are absolute, so do not load correctly when I try to open the html file in the browser.
Why not make them relative by default?
How do I change this behavior so the build make them relative, making the files 'portable' to wherever I drop them ?
Thanks,
=: s

I tried changing this in the config.js
from
// Wiredep task config
module.exports.wiredep = {
  sass: {
    src: path.join(app, styles, '/*.scss'),
    dest: path.join(app, styles),
    cfg: {
      ignorePath: '',
      overides: {}
    }
  },
  jade: {
    src: path.join(app, views, '/layouts/*.jade'),
    dest: path.join(app, views, '/layouts'),
    cfg: {
      exclude: ['modernizr'],
      ignorePath: '../../',
      overides: {}
    }
  }
}

to
module.exports.wiredep = {
  sass: {
    src: path.join(app, styles, '/*.scss'),
    dest: path.join(app, styles),
    cfg: {
      ignorePath: '',
      overides: {},
      relative: true
    }
  },
  jade: {
    src: path.join(app, views, '/layouts/*.jade'),
    dest: path.join(app, views, '/layouts'),
    cfg: {
      exclude: ['modernizr'],
      ignorePath: '../../',
      overides: {},
      relative: true
    }
  }
}

as per https://github.com/klei/gulp-inject
but doesnt seem to be having any effect.

Hello @lifeinchords,
we are using https://github.com/taptapship/wiredep for handling css & js.
In our use case, we are using files in folders with same layout.jade so, it's easier to maintain correct paths with absolute uri. Maybe it can by done other way, I'm not sure right now.
If you want relative paths and you don't care about files in subfolders, you can change paths in app/views/layouts/_default.jade in like this (removing leading slash):
...
//- build:css({.tmp,app}) /styles/main.min.css -->
// bower:css
// endbower
link(rel="stylesheet", href="/styles/main.css")
//- endbuild -->
...

to:

...
//- build:css({.tmp,app}) styles/main.min.css -->
// bower:css
// endbower
link(rel="stylesheet", href="styles/main.css")
//- endbuild -->
...

Secondly, we are using hrefs without '.html' extension, so it's not possible do it without gulp serve.

Hi @adammockor
Thanks for the detailed reply. That was helpful, and I got it working for my needs. Thanks for open sourcing this generator.
=: s


Add refinery for honeycomb sampling

Adds refinery to do honeycomb tail sampling for us.
Our configuration is set up to send all events that have an error property, and throttle all other events so that we send a max of about 30/second. That should keep us below our Honeycomb plan limits.
The refinery container sits next to the server container and we redirect all of our honeycomb API calls to the refinery container. Then the refinery container makes a decision on what to send based on the rules we defined in server/refinery/rules.yaml. More docs on those here and here.
We use a EMAThroughputSampler to set a target throughput for non-errors. It uses the name field to group spans for throttling, so that it doesn't completely drop certain spans. If we want to throttle certain spans more or with more granularity, we can add additional rules to the yaml file.

Woohoo!!


Frontend layout changes for mobile/tablet

From @vimode's comment:

[ ] Small screen: Make navbar collapsible for smaller screens.
[ ] Small screen: Adjust the alignment of the items to all follow the same space from the margin (logo, github logo and nav items)
[ ] Small screen: Header logo shrinks to a very small size for mobile view. The custom width of 38% does not scale properly for small screen - fix this.
[ ] Accessibility: Better color contrast, the current color(s) and some font-sizes do not pass the WCAG standard for accessbile text. But this can be fixed later with a higher contrast color palette or simply increasing some font-sizes.
[ ] Accessibility: popover tooltip is good for general UI but is not an accessible UX.
Proposed solution: Add an aria-label to all items which have popover tooltip.
[ ] Large screens: Text heavy pages like About, Community etc  have centered text with left align title.
Proposed solution: Align all text to left and only center for small screens.
[ ] Expand the header to full viewport width as for bigger screens but keep the content in the center container as it is right now. With the current layout the page looks like a snapshot in the center of the page
[ ] Small screen: expand the header to the full viewport width instead and make sure that the inner container has some padding so the content does not stick to the width and has enough whitespace to still make it easy to touch.

Perhaps further discussion required (but PRs welcome):

Consistent usage of bootstrap throughout for all the layout and components
Currently the cards use a custom column layout which can be changed to use the Card component instead which follows the CSS grid markup.
Proposed solution: Avoid usage of custom CSS as currently being used through App.css and heavily rely on Bootstrap for consistent theme, maintainable & upgradable style.

Add a footer not only its a great secondary nav especially for pages with more content Models and Data but also a great way to display some persistent information about the project.

Original issue for basic responsive layout, closed 2022-11-28:

Already use bootstrap
Layout responsive to screen size/aspect ratio
Consider putting navigation elements at the top of the page rather than in a sidebar (sometimes, always?)

I was going to open a new issue about making a responsive navbar but noticed this closed issue so bumping it if its something that still needs to be improved for better mobile view.
Its a quick fix to make the navbar responsive and make it expand only at larger breakpoints.

Improvements to make the layout more responsive would be very useful, please do open a new issue/reopen this one if you like.

Thank you @ots22 Please reopen this issue.
I can use this issue to compile a list of layout & responsive improvements so we can continue discussing which are necessary or needed to improve the UI/UX experience.

Hello @ots22
Here are some improvements, which I think will aid the current layout for improved UI/UX.

Small screen: Make navbar collapsible for smaller screens.
Small screen: Adjust the alignment of the items to all follow the same space from the margin (logo, github logo and nav items)
Small screen: Header logo shrinks to a very small size for mobile view. The custom width of 38% does not scale properly for small screen.
With the current number of nav items, the navbar for tablet view can still say in the expanded view.
Consistent usage of bootstrap throughout for all the layout and components
Currently the cards use a custom column layout which can be changed to use the Card component instead which follows the CSS grid markup.
Proposed solution: Avoid usage of custom CSS as currently being used through App.css and heavily rely on Bootstrap for consistent theme, maintainable & upgradable style.
Accessibility: Better color contrast, the current color(s) and some font-sizes do not pass the WCAG standard for accessbile text. But this can be fixed later with a higher contrast color palette or simply increasing some font-sizes.
Accessibility: popover tooltip is good for general UI but is not an accessible UX.
Proposed solution: Add an aria-label to all items which have popover tooltip.
Large screens: Text heavy pages like About, Community etc  have centered text with left align title.
Proposed solution: Align all text to left and only center for small screens.
Expand the header to full viewport width as for bigger screens but keep the content in the center container as it is right now. With the current layout the page looks like a snapshot in the center of the page
Similarly for small screens. the header can expand to the full viewport width instead, we can make sure that the inner container has some padding so the content does not stick to the width and has enough whitespace to still make it easy to touch.
Model and Datasource pages can be improved but I guess that will depend on what kind of data will be fed to it later as its currently only using from the mock json data, I suppose?
Add a footer not only its a great secondary nav especially for pages with more content Models and Data but also a great way to display some persistent information about the project.

Let me know which you think are necessary and important to work on and I could start working on those.
If you are open to some design and structural changes I can (later) slowly work on proposing some ideas as I understand more about the project while I work on these improvements.
Thank you.

Hi @vimode,
Thanks for compiling this list - I'll add some to-do items to the top comment, any/all of these seem useful to work on.
If some thoughts on the priority order are helpful, I'd say the 'small screen' and 'accessibility' ones are probably the highest priority, but all are interesting.

Consistent usage of bootstrap throughout for all the layout and components
Currently the cards use a custom column layout which can be changed to use the Card component instead which follows the CSS grid markup.
Proposed solution: Avoid usage of custom CSS as currently being used through App.css and heavily rely on Bootstrap for consistent theme, maintainable & upgradable style.

The custom CSS for the card layout was to allow the number of columns to expand dynamically (from memory I think the Card component had a fixed, small, number of columns), but a better solution here would be nice, especially if it could make more standard use of bootstrap.  I think what we really want is some kind of 'image grid' component, rather than misusing card columns as I ended up doing!

Model and Datasource pages can be improved but I guess that will depend on what kind of data will be fed to it later as its currently only using from the mock json data, I suppose?

Yes this is a bit of a work in progress at the moment (there are a few PRs/issues open at the moment with some ideas).
I think keeping this data as static json files has certain advantages, so I imaging we'll keep doing that unless/until we really outgrow it, but largely independent from the frontend proposals.  The schema is still open for changes (wherever the data ends up being hosted) so we can add things that we would like to include on these pages.

Add a footer not only its a great secondary nav especially for pages with more content Models and Data but also a great way to display some persistent information about the project.

A PR adding this is definitely welcome, it would be interesting to think about the design of this and what we want to include.

Thanks for adding them up and adding some priority to those. I will get started on those accordingly.

The custom CSS for the card layout was to allow the number of columns to expand dynamically ...

I think bootstrap's Card component (if not the react-bootstrap component itself) does follow the grid markup so wrap should work. I will look into it and work accordingly.
You brought up an interesting point about it being an image grid than a card, I am not sure though as it looks like an image but its a info card in some ways. Definitely will consider that when I look into testing its layout.

The schema is still open for changes (wherever the data ends up being hosted) so we can add things that we would like to include on these pages.

As I go through the project I will get the sense of what data would be helpful to be displayed but for now I would highly recommend adding some sort of an id to the data files, React requires a unique key for each list item as it uses this key to match the array item to the component, it also uses this key in its diffing algorithm to update it correctly in the DOM tree if sorted or deleted or inserted. (Hence, the current React warnings that list item requires a key prop)

That would be brilliant, thanks.

I would highly recommend adding some sort of an id to the data files,

The 'name' field is intended to have this role (the name field must be unique, although this isn't clear from the json schema - it's actually enforced by a test in the python library, which just loads it and triggers some validation).

Hence, the current React warnings that list item requires a key prop

Ah yes, I'd been meaning to look at that - thanks for the link.  For the catalog data, 'name' should do the job. I think there may be some other arrays where we need to make a decision (or just use a random value as you say).

The 'name' field is intended to have this role (it must be unique...

Correct me I am wrong, but doesn't this check for name only within its own Catalog data? If so, the DOM tree created by react (like on the homepage) which has Models, Datasources, Projects, the name as a key could be repeated especially since its a user entered string. I might be over thinking with edge cases.
I also noticed, linting errors with the usage of == instead of === in many places of the frontend. Was this on purpose or just that python equality operator slipped in? If so, will replace wherever necessary.
In JavaScript, == will do the type conversion and then compare, so will also accepts falsy statements as true.  0 == '0' is true
===(strict equality) is abstract equality operator, checks without type conversion 0 === '0' is false.
So generally === is safe to use than == as an equality operator.

For the catalog data, 'name' should do the job. I think there may be some other arrays where we need to make a decision (or just use a random value as you say).

Once we come to a decision, just add it to the list of tasks.

Correct me if I am wrong, but doesn't this check for name only within its own Catalog data? If so, the DOM tree created by react (like on the homepage) which has Models, Datasources, Projects, the name as a key could be repeated especially since its a user entered string. I might be over thinking with edge cases.

I'd understood a key only had to be unique within a given array, rather than over the whole DOM (e.g. https://reactjs.org/docs/lists-and-keys.html#keys-must-only-be-unique-among-siblings). It might be the case that we end up mixing different kinds of entity within one array though, then we need to think about that. One approach could be to distinguish these keys with a prefix.
If I've misunderstood something and we do in fact need a globally unique id, I'd have a preference to only add this to the json data files as a last resort - but no issues with any other approach.

I also noticed, linting errors with the usage of == instead of === in many places of the frontend. Was this on purpose or just that python equality operator slipped in? If so, will replace wherever necessary.

Yes, almost certainly not intentional - most probably a pythonism!
Some linting as a CI step would seem useful to catch this kind of thing now, I'll make a PR to turn that on.

If I've misunderstood something and we do in fact need a globally unique id...

I stand corrected. Thank you for the link. I got confused with the Recursing on children problem at Reconciliation page as an example throughout the DOM tree.

...One approach could be to distinguish these keys with a prefix.

Yes, that should do it.

Closing as #521 fixed all but one of these, and adding a footer is now #527

Hey @ots22
Sorry, I been away for a while.
I will resume working on the frontend to streamline the usage of react bootstrap components as much as possible over using html tags as discussed in the my PR #521.
Since I been away for a while, please let me know if things have changed and those changes are not required anymore.
Meanwhile I will finish the necessary accessibility upgrades to make sure it passes most if not all accessibility checks.

Hi @vimode, that's great, those changes would still be very welcome, thank you!


Upgrade pitest-maven-plugin 1.15.1 -> 1.15.2

This PR contains the following updates:

Package
Type
Update
Change

pitest-maven-plugin (source)
build
patch
1.15.1 -> 1.15.2

Release Notes

hcoles/pitest (pitest-maven-plugin)
v1.15.2
Compare Source

#1268 - Filter Lombok NonNull checks
#1267 - Expand test api to allow reporting of errors during test scanning

#1267 will ensure that previously hidden errors encountered by the JUnit5 plugin will now be reported. This is a backwards compatible api change. It will have no effect until the JUnit5 plugin is updated to use the expanded api. The updated plugin will not be compatible with earlier versions of pitest.

[ ] If you want to rebase/retry this PR, check this box

Suggested commit message:
Upgrade pitest-maven-plugin 1.15.1 -> 1.15.2

See:
- https://github.com/hcoles/pitest/releases/tag/1.15.2
- https://github.com/hcoles/pitest/compare/1.15.1...1.15.2


Decoding token in controller.

I'm trying to find it in documentation if there is a way to decode token into user id. Does this bundle provide such option/service? Thanks, this is a great bundle.

I ended up doing this, is that the right way?
https://stackoverflow.com/questions/50803095/lexikjwtauthenticationbundle-decoding-token-in-controller?answertab=active#tab-top


Need help: How to modify user context variables from external task

Hi, I'm working to integrate aiogram-dialog with IoT project. The basic idea is to read temperature from device and show into Telegram bot (It is a MQTT/Telegram integration).
I would like if someone could give me an example to understand how to change an specific user context variable from a task periodic added by me. In my working example I have did it but using global variables, which is not the best solution.
Also I have readed about middleware class but I didn't understand if it has something to do here.
I will appreciate any kind of help.
import asyncio
import logging
import random

from aiogram import Bot, Dispatcher
from aiogram.contrib.fsm_storage.memory import MemoryStorage
from aiogram.dispatcher.filters.state import StatesGroup, State
from aiogram.types import Message, CallbackQuery

from aiogram_dialog import Dialog, DialogManager, Window, DialogRegistry, BaseDialogManager, \
    StartMode
from aiogram_dialog.widgets.kbd import Button
from aiogram_dialog.widgets.text import Const, Multi, Progress, Format

API_TOKEN = "YOUR TOKEN"

temperature = 0

# ---------------------------------------------------------------------------------

# States
class MainSG(StatesGroup):
    get_temp = State()

# Getter
async def get_bg_data(dialog_manager: DialogManager, **kwargs):
    return {
        "temp": dialog_manager.current_context().dialog_data.get("temp", 0),
    }

async def refresh_temp(c: CallbackQuery, button: Button, manager: DialogManager):
    await manager.update({
            "temp": temperature ,
        })

main_menu = Dialog(
    Window(
        Const("Clik to obtain temperature"),
        Button(Format("Your temp:  {temp} °C"), id="b1", on_click=refresh_temp ),
        state=MainSG.get_temp,
        getter=get_bg_data
    ),
)

async def start(m: Message, dialog_manager: DialogManager):
    await dialog_manager.start(MainSG.get_temp, mode=StartMode.RESET_STACK)

logging.basicConfig(level=logging.INFO)
logging.getLogger("aiogram_dialog").setLevel(logging.DEBUG)
storage = MemoryStorage()
bot = Bot(token=API_TOKEN)
dp = Dispatcher(bot, storage=storage)
registry = DialogRegistry(dp)
registry.register(main_menu)
dp.register_message_handler(start, text="/start", state="*")

async def main():
    await dp.start_polling()

# MyTask Function that receive temperature and pass to context variables
async def periodic(sleep_for):
    global temperature
    while True:
        ID = 111111111  # Simulated ID from know user linked to device
        # -----------------------------------
        # Here I thin we must add the magic but at the moment I'm using
        # a global variable
        temperature = random.randint(20,35)   # simulate temperature reading
        # -----------------------------------
        await asyncio.sleep(sleep_for)

async def runner():
    await asyncio.gather(main(),periodic(2))

if __name__ == '__main__':
    asyncio.run(runner())

You can create bg manager in the same way as it is done in ManagerImpl class:
        return BgManager(
            user=user,
            chat=chat,
            bot=bot,
            registry=registry,
            stack_id=stack_id,
        )

For stack you can use default stack, user and chat you can just create on place like User(id=user_id)


[Bug]: Unable to authorize new GSheets on an environment which has been upgraded from last version

Is there an existing issue for this?

[X] I have searched the existing issues

Description
On the upgraded release environment, Unable to authorize new or existing GSheets
Steps To Reproduce

Sign into https://release-ee-https.appsmith.com/applications
Create a new workspace, and create a new gsheets datasource
Click on save and authorize.
Note user is taken to the error page -
{"responseMeta" : {"status": 500, "success": false, "error": {"code":5000, "message" : "Internal server error while processing request"}}}

Public Sample App
No response
Environment
Release
Issue video log
No response
Version
Cloud

Looks like the API server is using release-cs now but when it started up the first time it used cs.appsmith and registered itself there. So it does not seem like a code issue but a setup issue.

@sumitsum - Just tried creating a new Gsheets DS and tried to authorize the same on release-ee - it throws the same error.

Closing it as evidence points it towards a setup issue. details


⚠ Bofrost.fr has degraded performance

In 0bd4874, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 826 ms

Resolved: Bofrost.fr performance has improved in 2c90cbc after 8 minutes.


Give Prometheus test classes meaningful names

TestPrometheusIntegrationTests1, TestPrometheusIntegrationTests2, TestPrometheusIntegrationTests3  are not the best names possible.

Thanks for your comment @GeorgePanaretos
within Presto we have these conventions:

test classes start with Test... (i know it's not a common convention in general, but it's also something that we grow very used to)
test classes are "meaningful" without looking at the package name. it simply helps communicate and work in the IDE

consider example of all subclasses of AbstractTestIntegrationSmokeTest class. They all would look very much the same if we removed "PostgreSql", "Hive" or "MySql" from their name

thus, the new names here should start with TestPrometheus.... The rest is to be determined.
Since the classes are named as they are, i do not know what they are supposed to test.
Someone needs to look at them, guess their intended meaning and propose some new names.

Test...1 -> TestPrometheusIntegrationMetrics / TestPrometheusIntegrationURI

startup and basic test functions being ran
metrics checked
URI checked

Test...2 -> TestPrometheusIntegrationSchema / TestPrometheusIntegrationSplit

Table values types checked
Tests correct number of splits that were created.

Test...3 -> TestPrometheusIntegrationHealth / TestPrometheusIntegrationStatus

Runs a re-check on metrics
Checks to see that Prometheus is up and running

Sounds good to me.


Navigation tutorial: amcl problem ?

Hi there !

Which TurtleBot3 you have?

[X] Burger
[ ] Waffle
[ ] Waffle Pi

Which SBC(Single Board Computer) is working on TurtleBot3?

[X] Raspberry Pi 3
[ ] Intel Joule 570x
[ ] etc (PLEASE, WRITE DOWN YOUR SBC HERE)

Which OS you installed in SBC?

[X] Ubuntu MATE 16.04 or later
[ ] Raspbian
[ ] etc (PLEASE, WRITE DOWN YOUR OS)

Which OS you installed in Remote PC?

[ ] Ubuntu 16.04 LTS (Xenial Xerus)
[ ] Ubuntu 18.04 LTS (Bionic Beaver)
[ ] Linux Mint 18.x
[X] Xenial 16.04 + Kinetic, native on the TB3, inside a docker container (FROM osrf/ros:kinetic-desktop-full) on my Fedora 27 laptop

Write down software version and firmware version

Software version: [1.12.13 on both the TB3 and the docker container on my laptop]
Firmware version: [x.x.x]

Write down the commands you used in order

following the robotis tutorials:
[laptop]
roscore
[tb3]
roslaunch turtlebot3_bringup turtlebot3_robot.launch
[laptop]
roslaunch turtlebot3_slam turtlebot3_slam.launch slam_methods:=gmapping
I move the robot using the android app and when the map is complete I run:
rosrun map_server map_saver -f ~/map
Then I kill stop the former gmapping task, move the robot by hand to a reference place in the room, and launch the navigation task:
roslaunch turtlebot3_navigation turtlebot3_navigation.launch map_file:=/home/ros/map.yaml

Copy and Paste your error message on terminal
I get the following warnings from the navigation task:

"could not get robot pose, cancelling reconfiguration"
"Costmap2DROS transform timeout. Current time: 153253596.3332, global_pose_stamp:<PHONE_NUMBER>.8062, tolerance: 0.5000"

Please, describe detailedly what difficulty you are in

Then in RViz I click to set the 2D Pose Estimate, I iterate to align laserscan dots with the occupancy map. The problem is that when I set a nav goal or move the robot by teleoperation, the robot does not move on the map.
And I get warning over and over in the console.

I'm pretty new to ROS but as the robot does not move on the map, I feel like the amcl is not working for some reason.
How to troubleshoot and fix ? I'm starting to feel like a turtle without salad !
DoNcK

Some screenshots and extra info:
Screenshot of RViz after manual aligning:

Screenshot of RViz after waiting 1 minute (no visible changes except warning in console) and then moving the robot:

Strange stuffs I noticed:
rostopic hz /amcl_pose
=> no new messages
and on roswtf:
WARNING The following nodes are unexpectedly connected:
 * /robot_state_publisher->/rqt_gui_py_node_13192 (/tf_static)

Found 1 error(s).

ERROR The following nodes should be connected but aren't:
 * /move_base->/move_base (/move_base/global_costmap/footprint)
 * /move_base->/move_base (/move_base/local_costmap/footprint)

Hi @DoNck
I think you are sending two cmd_vel topics. In the last picture of the post you uploaded, you are sending cmd_vel on each of the move_base and turtlebot3_teleop_keyboard nodes. In particular, the turtlebot3_teleop_keyboard node sends a cmd_vel value of zero. Shutting down the turtlebot3_teleop_keyboard node seems to solve your problem.
Maybe you have not chekc it yet, we explained why we should terminate turtlebot3_teleop_keyboard in the tip of the manual below.
http://emanual.robotis.com/docs/en/platform/turtlebot3/navigation/#estimate-initial-pose

As you suggested, I terminated the teleop node.
$ rostopic info /cmd_vel
Type: geometry_msgs/Twist

Publishers:
 * /move_base (http://<IP_ADDRESS>:33955/)

Subscribers:
 * /turtlebot3_core (http://<IP_ADDRESS>:46539/)

Then I restarted the navigation task (at that point one initial message was published on /amcl_pose), set the 2D pose estimate (a second message is published, then after some time I get the warning).
Still I have the warnings and I can't see any other improvement: map is not getting auto-aligned, as soon as I set a goal, the robot starts to move forward, the lidar and map aligment gets worst and worst...
Regarding the manual, that is the one I've been trying to follow (my only "deviation" is that I'm using Docker on he Remote PC). Could that be a problem ?
From my understanding of the guide:

Run roscore on Remote PC
Run turtlebot3_robot.launch from turtlebot3_bringup on the TB3
Run turtlebot3_slam on Remote PC and save the map
Terminate the turtlebot3_slam and run turtlebot3_navigation instead (using the saved map).
Set the 2D pose estimate in RViz
Run teleop on Remote PC to move back and forth using until the amcl magick works
(I guessed it should improve the lidar/map aligment in RViz and send regular position updates to /amcl_pose)
As soon as amcl is working, quit teleop (tip), set a 2D nav goals and have some fun

Is my understanding correct ? Or what did I do wrong ?
Did I forget to run others nodes ?

Hi @DoNck
I think you are perfect. I have never seen SLAM work without problems but Navigation does not work. Both must be good or not both. Can you send me your bag data? I think we should look into what is wrong.

Hello :)
Did you reset your turtlebot3 before you launch navigation?
If not, you are supposed to reset odometry and imu data to use /reset topic
or re-launch bringup.

OK well I even restarted the rosrecore on remote PC and rebooted the raspberry pi on the robot, it did not improve.
I also tried what you said: not better either.
And then I saw this big reset button on the OpenCR: I terminated the bringup, unplug/plug the OpenCR, pressed reset, restarted the bringup: everything is now working as exepected.
Thanks a lot for your support !
I let you close the issue unless you want some more information about my setup.

Thank you for sharing your experiences. If you have any further issues, please feel free to open issue.
Thanks

Hello :)
I have a short question. Im exactly in the same situation.  Why the reset is so important? does the robot kind of save the odom before?

Hi I wanna ask, in initial pose set. You said you move the robot forward and backward until the AMCL do the magic? what's the indicator the AMCL works and we're good to set the 2d nav goal?
Please help, me i have exact problem with you, but still the robot doesn;t move

Hi, my two cents:
The indicator that AMCL works is that the green particle cloud around the robot should get closer and closer around the robot actual position.
The cloud should update if you move the robot. If it does not, try moving the robot slower and along a longer path.
Hint: In my setup, I noticed that I can get good results faster, by slowly turning the robot right/left instead of moving back/forth.
Also, since I press the reset buton of the OpenCR board before running the bringup, and reset+restart bringup after battery hotpswaps, I got rid of the "robot not moving correctly on the map" problem.

Hi DoNock
I really appreciate your fast response, i still don't know if my AMCL is working or not, if you see my picture in my first comment. Do you think from the green points that line up around the wall is good enough to start 2d nav goal? Does it even already have green particle cloud?
Could you kindly show me some pics or even a video that show how the good initial pose when AMCL is working by making the progress in green particle cloud?
I tried to the reset button on OpenCR, restart the Turtlebot and relaunch bringup. But still after 2d nav goal set the robot do nothing. I'll try the battery hotswap too.

Hi timdori, thanks for the response
I tried the android teleop when initial pose setting, the command velocity is good and the robot move. Then i kill the teleop node, send the 2d nav goal, but the robot doesn't move.
I checked the rostopic info /cmd_vel is published by node (move_base) , the topic  /move_base_simple/goal is updated every time i set the 2d-nav-goal , but still like the command velocity not updated.

youtube video with particle cloud  (in red) updating and converging to robot position

youtube video with particle cloud (in red) updating and converging to robot position

Hi DoNck
the trick to rotate slowly and AMCL did it magic :D ,

I tried to hotswap and reset openCR , but still the 2d nav goal set doesn't seem to update the cmd_vel :(

Hi DoNcK and timdori. I've solved my problem by timesync the turtlebot and my pc with timedatectl on same timezone, the robot move when i set the 2d nav goal. The problem was Costmap always timeout , because the dfferent time set, once it set right the costmap defined and the robot can move :D
Thank you very much

Good that you spotted the time issue, they are not obvious when they occur...
In case your clock was not synced because you work on a network that filters out UDP, you can update your clock using this simple HTTP based python script I wrote.

Furthermore when you use the rviz to set an inital pose check if your Fixed Frame setting in rviz is set to "map", otherwise the inital pose message send out by rviz will be wrong


Logging in sidecar should focus starting on most recent logs

Use kubectl to pull some pod logging, the sidecar opens with the logs.  The view starts at the oldest set of logs, so the user has to scroll all the way down to get to the newest set of logging.

i believe that this is improved in the new code base https://github.com/kui-shell/plugin-kubeui


[HPRO-1537] Added and converted NPH toolbar/index to bootstrap 5

Q
A

Target Release?
Next available

Bug fix?
❌

New feature?
✅

Database migration?
❌

New configuration?
❌

Composer updates?
❌

NPM updates?
❌

Jira Ticket(s)
HPRO-1537

Summary

Instructions for testing
Screenshots

@Ehaic I have updated the NPH home page route to nph_home. Could you also modify the new bootstrap 5 template accordingly? You may need to switch to the latest develop branch and then compare the changes I made to the NPH navbar.html.twig, updating it as needed.

@Ehaic Looks like NPH nav brand is missing when the user belongs to only one program.

@Ehaic Noticed that when you click on any sub category menu item (ex: Lookup)  the text color is changing. We may want to keep the same color.


Add contributing guidelines

Closing this issue as I'm deprecating the project due to the fact that the upstream Cartographer project will not receive any new development and it's currently in a maintenance phase.


Thread-safe, functionally relevant and async map tile memory cache

Issue Type
[v] Improvement
[v] Performance
Description and/or steps/code to reproduce the problem
As experienced in bugs #776 and #812, the map tile cache management could be improved:

bullet-proof thread-safe access - in order to avoid #776
async garbage collecting - in order to avoid #812, slow Activity detach or ANR
fine grained functional relevancy of the tiles stored in the cache and possible deprecation of LRU - depending on each app's expressed needs

All the new stuff must be explicitly unit tested as it's a core feature.
Version of osmdroid the issue relates to:
6.0.0

Following points were fixed by #908:

bullet-proof thread-safe access - in order to avoid #776
async garbage collecting - in order to avoid #812, slow Activity detach or ANR

Now the focus is on fine grained functional relevancy of the tiles stored in the cache.

I am on the master as of 2/25 and I observe the below code going into an infinite loop, when I switch between the tile sources:
    public void clear() {
        while(!this.isEmpty()) {
            try {
                this.remove(this.keySet().iterator().next());
            } catch (NoSuchElementException var2) {
                ;
            }
        }

        super.clear();
    }

@pasniak In which class do you see this code, which is an old version? Obviously there's confusion: what I see in MapTileCache is
public void clear() {
	// remove them all individually so that they get recycled
	final MapTileList list = new MapTileList(mCachedTiles.size());
	populateSyncCachedTiles(list);
	for (int i = 0; i < list.getSize() ; i ++) {
		final long index = list.get(i);
		remove(index);
	}

	// and then clear
	mCachedTiles.clear();
}

My code was for decompiled LRUMapTileCache.

@pasniak LRUMapTileCache is now deprecated and not used anymore. Its code is more or less embedded in MapTileCache now.

@pasniak I'm definitely not an expert in git and I don't know what's in master, but the code I mentioned is part of the current Code version of MapTileCache

Thanks, this was another case of AS 3.0 not updating my projects's dependencies... Argh! I built osmdroid with:
./gradlew clean publishToMavenLocal

Yet my projects linked with stale osmdroid...  I had to manually delete /home/pasniak/.android/build-cache

Solve by #956


[supply] Checking for interactive mode before user's input when creating from config

Checklist

[x] I've run bundle exec rspec from the root directory to see all new and existing tests pass
[x] I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
[x] I've read the Contribution Guidelines
[x] I've updated the documentation if necessary.

Motivation and Context
These changes are being made to fix #14128. This issue is an unexpected behaviour in CI with supply action.
Description
When any AbstractGoogleServiceClient is being created from configuration, it searches for authentication parameters for authenticating with Google. I've moved this logic to an independent method, which can be overloaded from any of its child.
PlaycustomappClient will still be using just the non deprecated way, so if you don't provide 'json_key' or 'json_key_data' params using CreateAppOnManagedPlayStoreAction, Fastlane will crash, as specified in this PR.
In the other hand, the old Client will still support both of the authentication types, but printing an important message on UI, making an advice on using the JSON files.
The issue was fixed by wrapping the input line into a condition, which asks whether the UI is interactive or not. If this checks fails, Fastlane will crash.
Existing tests were fixed in order to make them work with these changes, and also added a new case into the "without :json_key or :json_key_data" set.

@joshdholtz it appears that the suggested change feature from Github changed CLA status leaving it in an inconsistent state :( Is there anything I can do to fix this?

@joshdholtz fixed :D

@joshdholtz I was missing the param name in super(...) call. Made some tests with your setup and it looks fine. Should be ready to go!


arm64 (qemu) "make check" fails

hello,
I'm interested in running gperftools on ARM64, so ...
# make check
rm -f debugallocation_test.sh
cp -p ./src/tests/debugallocation_test.sh debugallocation_test.sh
rm -f tcmalloc_unittest.sh
cp -p ./src/tests/tcmalloc_unittest.sh tcmalloc_unittest.sh
rm -f sampling_test.sh
cp -p ./src/tests/sampling_test.sh sampling_test.sh
rm -f heap-profiler_unittest.sh
cp -p ./src/tests/heap-profiler_unittest.sh heap-profiler_unittest.sh
rm -f heap-checker_unittest.sh
cp -p ./src/tests/heap-checker_unittest.sh heap-checker_unittest.sh
rm -f heap-checker-death_unittest.sh
cp -p ./src/tests/heap-checker-death_unittest.sh heap-checker-death_unittest.sh
rm -f sampling_debug_test.sh
cp -p ./src/tests/sampling_test.sh sampling_debug_test.sh
rm -f heap-profiler_debug_unittest.sh
cp -p ./src/tests/heap-profiler_unittest.sh heap-profiler_debug_unittest.sh
rm -f heap-checker_debug_unittest.sh
cp -p ./src/tests/heap-checker_unittest.sh heap-checker_debug_unittest.sh
rm -f profiler_unittest.sh
cp -p ./src/tests/profiler_unittest.sh profiler_unittest.sh
for la in libtcmalloc_minimal.la libtcmalloc_minimal_debug.la libtcmalloc.la libtcmalloc_debug.la libtcmalloc_and_profiler.la; do lib=".libs/`basename $la .la`.a"; [ ! -f "$lib" ] || : "$lib"; done
make  pprof_unittest
make[1]: Entering directory '/home/root/gperftools'
./src/pprof -test
AddressAdd 32-bit tests: 5 passes, 0 failures
AddressAdd 64-bit tests: 10 passes, 0 failures
AddressSub 32-bit tests: 5 passes, 0 failures
AddressSub 64-bit tests: 10 passes, 0 failures
AddressInc 32-bit tests: 5 passes, 0 failures
AddressInc 64-bit tests: 10 passes, 0 failures
PASS
make[1]: Leaving directory '/home/root/gperftools'
make  check-TESTS
make[1]: Entering directory '/home/root/gperftools'
./src/pprof -test
AddressAdd 32-bit tests: 5 passes, 0 failures
AddressAdd 64-bit tests: 10 passes, 0 failures
AddressSub 32-bit tests: 5 passes, 0 failures
AddressSub 64-bit tests: 10 passes, 0 failures
AddressInc 32-bit tests: 5 passes, 0 failures
AddressInc 64-bit tests: 10 passes, 0 failures
PASS
make[2]: Entering directory '/home/root/gperftools'
PASS: low_level_alloc_unittest
PASS: atomicops_unittest
PASS: stacktrace_unittest
PASS: tcmalloc_minimal_unittest
PASS: tcm_min_asserts_unittest
PASS: tcmalloc_minimal_large_unittest
PASS: tcmalloc_minimal_large_heap_fragmentation_unittest
PASS: addressmap_unittest
PASS: system_alloc_unittest
PASS: packed_cache_test
PASS: frag_unittest
PASS: markidle_unittest
PASS: current_allocated_bytes_test
PASS: malloc_hook_test
PASS: malloc_extension_test
PASS: malloc_extension_c_test
PASS: memalign_unittest
PASS: page_heap_test
PASS: pagemap_unittest
PASS: realloc_unittest
PASS: stack_trace_table_test
PASS: thread_dealloc_unittest
PASS: tcmalloc_minimal_debug_unittest
PASS: malloc_extension_debug_test
PASS: memalign_debug_unittest
PASS: realloc_debug_unittest
rm -f debugallocation_test.sh
cp -p ./src/tests/debugallocation_test.sh debugallocation_test.sh
PASS: debugallocation_test.sh
rm -f tcmalloc_unittest.sh
cp -p ./src/tests/tcmalloc_unittest.sh tcmalloc_unittest.sh
PASS: tcmalloc_unittest.sh
./test-driver: line 107:  2944 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: tcm_asserts_unittest
PASS: tcmalloc_both_unittest
PASS: tcmalloc_large_unittest
PASS: tcmalloc_large_heap_fragmentation_unittest
PASS: raw_printer_test
PASS: sampler_test
rm -f sampling_test.sh
cp -p ./src/tests/sampling_test.sh sampling_test.sh
PASS: sampling_test.sh
rm -f heap-profiler_unittest.sh
cp -p ./src/tests/heap-profiler_unittest.sh heap-profiler_unittest.sh
PASS: heap-profiler_unittest.sh
PASS: simple_compat_test
rm -f heap-checker_unittest.sh
cp -p ./src/tests/heap-checker_unittest.sh heap-checker_unittest.sh
FAIL: heap-checker_unittest.sh
rm -f heap-checker-death_unittest.sh
cp -p ./src/tests/heap-checker-death_unittest.sh heap-checker-death_unittest.sh
FAIL: heap-checker-death_unittest.sh
PASS: tcmalloc_debug_unittest
PASS: sampler_debug_test
rm -f sampling_debug_test.sh
cp -p ./src/tests/sampling_test.sh sampling_debug_test.sh
PASS: sampling_debug_test.sh
rm -f heap-profiler_debug_unittest.sh
cp -p ./src/tests/heap-profiler_unittest.sh heap-profiler_debug_unittest.sh
PASS: heap-profiler_debug_unittest.sh
rm -f heap-checker_debug_unittest.sh
cp -p ./src/tests/heap-checker_unittest.sh heap-checker_debug_unittest.sh
FAIL: heap-checker_debug_unittest.sh
PASS: getpc_test
PASS: profiledata_unittest
PASS: profile_handler_unittest
rm -f profiler_unittest.sh
cp -p ./src/tests/profiler_unittest.sh profiler_unittest.sh
PASS: profiler_unittest.sh
PASS: tcmalloc_and_profiler_unittest
============================================================================
Testsuite summary for gperftools 2.9.1
============================================================================
# TOTAL: 49
# PASS:  45
# SKIP:  0
# XFAIL: 0
# FAIL:  4
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to<EMAIL_ADDRESS>============================================================================
make[2]: *** [Makefile:6952: test-suite.log] Error 1
make[2]: Leaving directory '/home/root/gperftools'
make[1]: *** [Makefile:7060: check-TESTS] Error 2
make[1]: Leaving directory '/home/root/gperftools'
make: *** [Makefile:7612: check-am] Error 2

cat test-suite.log
========================================
   gperftools 2.9.1: ./test-suite.log
========================================

# TOTAL: 49
# PASS:  45
# SKIP:  0
# XFAIL: 0
# FAIL:  4
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: tcm_asserts_unittest
==========================

FAIL tcm_asserts_unittest (exit status: 139)

FAIL: heap-checker_unittest.sh
==============================

Testing ./heap-checker_unittest with HEAPCHECK= ... OK
Testing ./heap-checker_unittest with HEAPCHECK=local ... ./heap-checker_unittest.sh: line 59:  3728 Aborted                 (core dumped) $HEAP_CHECKER > $TMPDIR/output 2>&1
FAILED
Output from the failed run:
----
WARNING: Perftools heap leak checker is active -- Performance may suffer

----
FAIL heap-checker_unittest.sh (exit status: 4)

FAIL: heap-checker-death_unittest.sh
====================================

Testing ./heap-checker_unittest with HEAPCHECK= ... FAIL
Test was taking unexpectedly long time to run and so we aborted it.
Try the test case manually or raise the timeout from 120
to distinguish test slowness from a real problem.
Output from failed run:
---
Turning perftools heap leak checking off
HeapLeakChecker got turned off; we won't test much...

Adding pthread-specifics for thread<PHONE_NUMBER>23072 pid 3764
Creating extra thread 1
Creating extra thread 2
Creating extra thread 3
A new HeapBusyThread 1
Adding pthread-specifics for thread<PHONE_NUMBER>12064 pid 3764
Creating extra thread 4
Creating extra thread 5
Creating extra thread 6
Creating extra thread 7
Creating extra thread 8
A new HeapBusyThread 0
Adding pthread-specifics for thread<PHONE_NUMBER>04768 pid 3764
A new HeapBusyThread 3
Adding pthread-specifics for thread<PHONE_NUMBER>33440 pid 3764
A new HeapBusyThread 2
Adding pthread-specifics for thread<PHONE_NUMBER>19360 pid 3764
A new HeapBusyThread 7
Adding pthread-specifics for thread<PHONE_NUMBER>62624 pid 3764
A new HeapBusyThread 4
Adding pthread-specifics for thread<PHONE_NUMBER>40736 pid 3764
A new HeapBusyThread 6
Adding pthread-specifics for thread<PHONE_NUMBER>55328 pid 3764
A new HeapBusyThread 5
Adding pthread-specifics for thread<PHONE_NUMBER>48032 pid 3764
Creating extra thread 9
Creating extra thread 10
Creating extra thread 11
Creating extra thread 12
A new HeapBusyThread 8
Adding pthread-specifics for thread<PHONE_NUMBER>17408 pid 3764
A new HeapBusyThread 10
Adding pthread-specifics for thread<PHONE_NUMBER>32000 pid 3764
Creating extra thread 13
Creating extra thread 14
Creating extra thread 15
A new HeapBusyThread 9
Adding pthread-specifics for thread<PHONE_NUMBER>24704 pid 3764
A new HeapBusyThread 11
Adding pthread-specifics for thread<PHONE_NUMBER>39296 pid 3764
A new HeapBusyThread 12
Adding pthread-specifics for thread<PHONE_NUMBER>46592 pid 3764
A new HeapBusyThread 13
Adding pthread-specifics for thread<PHONE_NUMBER>53888 pid 3764
A new HeapBusyThread 14
Adding pthread-specifics for thread<PHONE_NUMBER>61184 pid 3764
Creating extra thread 16
Creating extra thread 17
A new HeapBusyThread 15
Adding pthread-specifics for thread<PHONE_NUMBER>09184 pid 3764
A new HeapBusyThread 16
Adding pthread-specifics for thread<PHONE_NUMBER>16480 pid 3764
Adding pthread-specifics for thread<PHONE_NUMBER>23072 pid 3764
In main(): heap_check=
Heap checker is not active, hence checker "all" will do nothing!
To activate set the HEAPCHECK environment variable.
Heap checker is not active, hence checker "trivial" will do nothing!
To activate set the HEAPCHECK environment variable.
Heap checker is not active, hence checker "simple" will do nothing!
To activate set the HEAPCHECK environment variable.

Pre leaking : 0xf03aeb77289c4f7b ^ 0xf03a5f7bf03a5f7b

Pre leaking : 0xf03aeb772893bf7b ^ 0xf03a5f7bf03a5f7b
Heap checker is not active, hence checker "trick" will do nothing!
To activate set the HEAPCHECK environment variable.

Leaking : 0xf03aeb7728905f7b ^ 0xf03a5f7bf03a5f7b

Leaking : 0xf03aeb77289fbe9b ^ 0xf03a5f7bf03a5f7b
Expected leaks not found: Some liveness flood must be too optimistic
Heap checker is not active, hence checker "death_simple" will do nothing!
To activate set the HEAPCHECK environment variable.

Leaking : 0xf03aeb7728933f7b ^ 0xf03a5f7bf03a5f7b

Leaking : 0xf03aeb77289cca3b ^ 0xf03a5f7bf03a5f7b
Expected leaks not found: Some liveness flood must be too optimistic
Heap checker is not active, hence checker "death_loop" will do nothing!
To activate set the HEAPCHECK environment variable.

Leaking : 0xf03aeb77289edcbb ^ 0xf03a5f7bf03a5f7b

Leaking : 0xf03aeb77289edcab ^ 0xf03a5f7bf03a5f7b
Expected leaks not found: Some liveness flood must be too optimistic

Pre leaking : 0xf03aeb77289c4f7b ^ 0xf03a5f7bf03a5f7b
Heap checker is not active, hence checker "death_inverse" will do nothing!
To activate set the HEAPCHECK environment variable.

Leaking : 0xf03aeb7728933f7b ^ 0xf03a5f7bf03a5f7b
Expected leaks not found: Some liveness flood must be too optimistic
Heap checker is not active, hence checker "death_noleaks" will do nothing!
To activate set the HEAPCHECK environment variable.

Pre leaking : 0xf03aeb772890bf7b ^ 0xf03a5f7bf03a5f7b

Pre leaking : 0xf03aeb772890bfab ^ 0xf03a5f7bf03a5f7b
Heap checker is not active, hence checker "death_count_less" will do nothing!
To activate set the HEAPCHECK environment variable.

Leaking : 0xf03aeb7728933f7b ^ 0xf03a5f7bf03a5f7b
Expected leaks not found: Some liveness flood must be too optimistic

Pre leaking : 0xf03aeb7728933f7b ^ 0xf03a5f7bf03a5f7b
Heap checker is not active, hence checker "death_count_more" will do nothing!
To activate set the HEAPCHECK environment variable.

Leaking : 0xf03aeb772890bfab ^ 0xf03a5f7bf03a5f7b

Leaking : 0xf03aeb772890bf7b ^ 0xf03a5f7bf03a5f7b
---
FAIL heap-checker-death_unittest.sh (exit status: 1)

FAIL: heap-checker_debug_unittest.sh
====================================

Testing ./heap-checker_unittest with HEAPCHECK= ... OK
Testing ./heap-checker_unittest with HEAPCHECK=local ... ./heap-checker_debug_unittest.sh: line 59:  4502 Aborted                 (core dumped) $HEAP_CHECKER > $TMPDIR/output 2>&1
FAILED
Output from the failed run:
----
WARNING: Perftools heap leak checker is active -- Performance may suffer

----
FAIL heap-checker_debug_unittest.sh (exit status: 4)

coredumpctl gdb
           PID: 4502 (lt-heap-checker)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Sat 2021-03-06 09:33:44 UTC (2h 22min ago)
  Command Line: /home/root/gperftools/.libs/lt-heap-checker_unittest
    Executable: /home/root/gperftools/.libs/lt-heap-checker_unittest
 Control Group: /user.slice/user-0.slice/session-c1.scope
          Unit: session-c1.scope
         Slice: user-0.slice
       Session: c1
     Owner UID: 0 (root)
       Boot ID: d80841f5d6794b3096b508abb314b482
    Machine ID: b25da0a60ae64f3182fbf7c53ce11357
      Hostname: moderated
       Storage: /var/lib/systemd/coredump/core.lt-heap-checker.0.d80841f5d6794b3096b508abb314b482.4502.1615023224000000.lz4
       Message: Process 4502 (lt-heap-checker) of user 0 dumped core.

GNU gdb (GDB) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-msft-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/root/gperftools/.libs/lt-heap-checker_unittest...
[New LWP 4502]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/home/root/gperftools/.libs/lt-heap-checker_unittest'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt full
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
        set = {__val = {0, 0, 1,<PHONE_NUMBER>25888,<PHONE_NUMBER>13440,<PHONE_NUMBER>76452,<PHONE_NUMBER>81480,<PHONE_NUMBER>30976,<PHONE_NUMBER>33024,<PHONE_NUMBER>79908,<PHONE_NUMBER>81704,<PHONE_NUMBER>94944,
<PHONE_NUMBER>99704, 0, 0, 0}}
        pid = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
#1  0x0000f00e70afedd4 in __GI_abort () at abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0xfffff2b24958, sa_sigaction = 0xfffff2b24958}, sa_mask = {__val = {2676586395008836901,<PHONE_NUMBER>008836901,<PHONE_NUMBER>904676419,
<PHONE_NUMBER>664567401,<PHONE_NUMBER>507808, 10, 0, 0,<PHONE_NUMBER>9671623680,<PHONE_NUMBER>3709551360, 0,<PHONE_NUMBER>370496,<PHONE_NUMBER>722356224,<PHONE_NUMBER>471028224,
<PHONE_NUMBER>471028225,<PHONE_NUMBER>471028225}}, sa_flags = -298790392, sa_restorer = 0xb8caee2f8584}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x0000b8caee2f8584 in LogPrintf (severity=-4, pat=<optimized out>, ap=<error reading variable: Cannot access memory at address 0x0>) at ./src/base/logging.h:209
        buf = "\n\000\000\000\000\000\000\000H\250\321p\016\360\000\000\001\000\000\000\000\000\000\000p\177\031q\016\360\000\000hJ\262\362\377\377\000\000dJ\262\362\377\377\000\000d\357\367p\016\360\000\000\220J\262\362\377\377\000\000\320K\027q\016\360\000\000\260K\262\362\377\377\000\000\000\000\000\000\000\000\000\000\360W\304p\016\360\000\000\001\000\000\000\000\000\000\000\300\321\031q\016\360\000\000HK\262\362\377\377\000\000\070K\262\362\377\377\000\000\242\326\327\355\000\000\000\000\235\002/\356\312\270\000\000X\325\031q\016\360\000\000dJ\262\362\377\377\000\000hJ\262\362\377\377\000\000Z_\267\003\000\000\000\000H\367.\356\312\270\000\000HK\262\362\377\377\000\000\070d\030q\016\360\000\000"...
        __PRETTY_FUNCTION__ = "void LogPrintf(int, const char*, va_list)"
#3  0x0000b8caee2f860c in LOG (lvl=<optimized out>, pat=<optimized out>) at ./src/base/logging.h:230
        ap = {__stack = 0xfffff2b24cb0, __gr_top = 0xfffff2b24cb0, __vr_top = 0xfffff2b24c80, __gr_offs = -48, __vr_offs = -128}
#4  0x0000b8caee2f27b4 in VerifyMemoryRegionMapStackGet () at src/tests/heap-checker_unittest.cc:1326
        caller_addr_limit =<PHONE_NUMBER>70556
        addr = 0xf00e71194000
        caller =<PHONE_NUMBER>79908
        caller_addr_limit = <optimized out>
        addr = <optimized out>
        caller = <optimized out>
        l = <optimized out>
        i = <optimized out>
#5  main (argc=<optimized out>, argv=<optimized out>) at src/tests/heap-checker_unittest.cc:1387
        heap_check = {lock_ = 0xb8cb23ccc5a0, name_ = 0xfffff2b24d90 "", start_snapshot_ = 0xb8caee2f3fd8 <__static_initialization_and_destruction_0(int, int)+2088>, has_checked_ = 40,
          inuse_bytes_increase_ =<PHONE_NUMBER>15000, inuse_allocs_increase_ =<PHONE_NUMBER>81736, keep_profiles_ = 248}
(gdb)

heap checker tests are expected to fail on arm64 (it lacks thread listing and ptrace-ing support currently; but I plan to address it sometimes soon).
That tcm_asserts is failing is definitely unexpected. Thanks for report. I'll take a look soon. (malloc should still be usable so go ahead and use it)

Crash in tcm_asserts_unittest happens on my system too. Ends up being yet another problem with libunwind. So just disable it. Or use "minimal" configuration that doesn't ever collect any backtraces.

shall we report it to libunwind?

@alk Hi, I also encountered the same problem on arm.
These three use cases failed.
heap-checker_unittest.sh  heap-checker_unittest.sh heap-checker-death_unittest.sh
Is there any progress now?

heap checker tests are expected to fail on arm64 (it lacks thread listing and ptrace-ing support currently; but I plan to address it sometimes soon).

any progress? thx~

1、failed testcases
heap-checker_unittest.sh heap-checker_unittest.sh heap-checker-death_unittest.sh
2、rpm versions are:
glibc 2.34
gperftools 2.9.1
gcc 10.3
kernel 5.10.0
arch aarch64
3、some report are following that:
it is found that three failed testcases actually invoke the .libs/lt-heap-checker_unittest that generated by shell.
A. gain backtrace by gdb binfile corefile

the lock value has problem in apperance
B. add print due to heapchecker is invalidate using gdb
https://gperftools.github.io/gperftools/heap_checker.html
Invoking process is following that:
TCMalloc_ListAllProcessThreads  the lock variable inits here
-> local_clone
clone_pid = local_clone((int (*)(void *))ListerThread, &args);
-> sys_clone
return sys_clone(fn, child_stack,
CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_UNTRACED, arg, 0, 0, 0);
-> LSS_NAME(clone)
lock value is same between  start of LSS_NAME(clone) and end of LSS_NAME(clone)
-> ListerThread
i add print before if(sem_wait(args->lock) < 0), the lock value is 0X2020202020...
it feels bad, but i do not know where lock value changes
C. compare with x86 architecture
In ListerThread, the lock value does not change
anyone has ideas? I will keep analyzing.

Is there any progress now?

linux/aarch64 fully works now.


Python Agent: continuity issues

Empire Version
newest dev branch, 6/22/17
OS Information (Linux flavor, Python version)
ubuntu on the server, newest macos on victim
Expected behavior and description of the error, including any actions taken immediately prior to the error. The more detail the better.
The mechanism by which agents identify themselves to empire should include a routine which takes into consideration the empire installation may have been restarted, empire "removed an agent", and should recreate the files for itself. Ideally, if an orphaned shell comes back to life, empire should be able to handle it. This process is currently broken, this is what happens instead:

This was a domain fronting agent attempting to reconnect through cloudfront.
It does this forever until you find and kill the errant agent. There is no way to 'recover' from this, as the errant agent will continue to try contacting the c2, but the listener can't seem to reconnect, so this flood is perpetual. Empire should understand what to do when a shell that's been 'removed' comes back to life and tries to reconnect.
To recreate this issue, all you have to do is shell a mac, then type 'remove all' in the agents context in empire. The errant shell will attempt to reconnect like this forever.

I'll take a look. I think I know where this needs to be fixed.

The powershell agent is fixed and will re-negetiate if it reports its self orphaned. Unfortunately on the python side,  any attempts to execute the re-negotiate function will fail as it is i a different namespace???