SVG export contains black layer in background of gene and chromosome tracks

Commit:

Master branch of higlass
Compiled version of Danielle's higlass-multivec
Compiled version of master branch of higlass-website
Master branch of higlass-server

Steps to reproduce

Add hg38 gene annotations and chromosome tracks
Scroll to any spot
Export views as SVG

Observed behavior

Gene and chromosome tracks contain a layer with a black background

Here is a snapshot of what the SVG looks like in Adobe Illustrator CC 22.1:

Notice the selected layer in the Layers palette and the selected rect (or g? — perhaps not as it is labeled as a Path element in Adobe...) element in the SVG.
To view this in a browser, here is a test URL:
http://explorer.altius.org/app/?config=ObkyhPvST7KdrO9J9PNk-g
This test viewconf contains viewport projections; however, closing those viewports to remove them does not change how the SVG is rendered (the track background layers are still rendered black in the SVG).
Expected behavior

Tracks contain the configured background (white or transparent, if default or not specified in JSON view object)

Here's a stripped-down viewconf:
http://explorer.altius.org/app/?config=UC-0VxJgSFyvBYbJDnmRJQ
The resulting SVG uses a transparent fill on the two rect elements used for the chromosome and gene annotations.
For example, here's the first rect:
<svg
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xmlns="http://www.w3.org/2000/svg">
    <g>
        <g class="g-labels" style="clip-path:url(#MDG-JZS-Sn2oHrLzO1p7cA);">
            <rect x="15" y="77" width="1187" height="55" fill="transparent"></rect>
            ...
</svg>

Manually changing this from transparent to white fixes the rendering in Illustrator and Inkscape.
As to why this happens, according to the SVG specification, the default fill color assignment is black (https://www.w3.org/TR/SVG11/painting.html#SpecifyingPaint). If transparent is not a valid color keyword (https://www.w3.org/TR/SVG11/types.html#ColorKeywords), then that might explain why the fill color is black.
This doesn't appear to affect stroke properties (https://www.w3.org/TR/SVG11/painting.html#StrokeProperties).
If the keyword transparent is used as a fill color for other elements, this may happen there, when rendering to SVG.
Another property for specifying transparency or alpha that might instead be useful is fill-opacity (https://www.w3.org/TR/SVG11/painting.html#FillOpacityProperty).

@alexpreynolds is this still an issue? I can't seem to reproduce the issue in Affinity Designer or Gapplin (both running on macOS 10.12.6).

It is for me, albeit with a locally built server/client/multivec plug setup that may not be in sync with whatever is current on higlass.io. You might try using Inkscape (free, requires separate Xquartz install) or a demo copy of Adobe Illustrator, if you don't have a license. If you have a link to your viewconf, I can test it out too.
I suspect the issue is that transparent should not be used as a fill color in the SVG output, ultimately.  It may not be treated as a valid color keyword, and so may get ignored by SVG renderers that follow spec and replace it with the default (black).

I downloaded it from the link you provided so it's unrelated to higlass.io but you're right fill="transparent" is nonsense. I'll try to fix it

@alexpreynolds Fixed in develop. Will be part of v1.2.4

Cool! Do you know when that might be coming down the pipe?

Should be released soonish. Unless @pkerpedjiev has some more things we could just publish v1.2.4 of the client. Regarding the whole suite, I think we're just trying to harden the new v1.2 branch which comes with quite a few new additions but it shouldn't take much longer.


Added Longest Common Subsequence.cpp

Implemented Longest Common Subsequence in cpp using Dynamic Programming

PR for issue #3

@neeladriOO7 This branch has conflicts that must be resolved in .DS_Store file


Add govuk notification warning class and update titles

https://design-system.service.gov.uk/components/notification-banner/
Changes in this PR

As per guidance update notification banner titles
Add warning class

Review app deployed to https://teaching-vacancies-review-pr-3153.london.cloudapps.digital


Update ghc-head

I have cherry-picked everything from ghc-8.4 onto ghc-head but haven't taken care of the tests so far.
In addition, 5193e44 is a small adjustment for DerivingVia.
Using c7bc067, I get this test output.
Apart from html-test the testsuites pass, although there are warnings that I hadn't seen before:
No HTML directory found for base test dependency: array
No HTML directory found for base test dependency: base
No HTML directory found for base test dependency: ghc-prim
No HTML directory found for base test dependency: process
No HTML directory found for base test dependency: template-haskell

Can someone familiar with the recent changes please look at the diffs for html-test and figure out if there are any problems?

@sjakobi Thanks for your work. I will take a look at the tests.

I think the output you get with c7bc067 is legit. It just means that now you are able to link stuff which you couldn't before.

Apart from html-test the testsuites pass, although there are warnings that I hadn't seen before:
No HTML directory found for base test dependency: array
No HTML directory found for base test dependency: base
No HTML directory found for base test dependency: ghc-prim
No HTML directory found for base test dependency: process
No HTML directory found for base test dependency: template-haskell

That's new from c7bc06743bcb9e88cd05edfae4b7269a8015adce.

@alexbiehl:

I think the output you get with c7bc067 is legit.

So I can simply accept the new output!?

So I can simply accept the new output!?
Yes.

In order to incorporate @hvr's wip/ghc-head in this update I tried to rebase ghc-8.4 on wip/ghc-head.
When building that branch I ran into the following error:
src/Haddock/Types.hs:678:15: error:
    Not in scope: type constructor or class ‘XPArrTy’
    Perhaps you meant ‘XParTy’ (imported from GHC)
    |
678 | type instance XPArrTy          DocNameI = NoExt

I haven't looked into this further.

Parallel arrays were recently removed (see f77c9c5cc8bb669f584d36494630589ea80eb799), so that instance no longer makes sense.

Parallel arrays were recently removed (see f77c9c5), so that instance no longer makes sense.

Ah thanks. I forgot to include the recent changes on ghc-head.

Closed in favor of #857.


Challenge Description Wording

Challenge http://freecodecamp.com/challenges/waypoint-iterate-with-javascript-while-loops has an issue.
Text reads, "Let's try getting a for loop to work by pushing values to an array." It should instead read, "Let's try getting a while loop to work by pushing values to an array."

@alexbrotman thanks for posting this issue!
@BerkeleyTrue seems this issue is solved and can be closed.
Please review the Guidelines for Contributing, thank you!.


nushell: as login shell via module

Description of changes
Pls ignore for now

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Is this stale? @cafkafk

Is this stale? @cafkafk

Lowkey yea, I mean, this is enough to have nu as a login shell IIRC, but I was planning to add nushell completion support but just got very very busy with other stuff.
If you just need this for login I can mark it as ready for review.

Is this stale? @cafkafk

Lowkey yea, I mean, this is enough to have nu as a login shell IIRC, but I was planning to add nushell completion support but just got very very busy with other stuff.
If you just need this for login I can mark it as ready for review.

Yea you can make the other pr later :3

This wouldn't work correctly as is according to K900, since it ignores stuff like environment.variables, it needs a full on module.

For loading env vars (sourcing bash's), I uses this plugin https://github.com/tesujimath/nu_plugin_bash_env.

what could work would be to launch bash/another POSIX shell and immediately exec into nu?

I'm not gonna work on this for the foreseeable future, so I'll close this for now.


remote home assistant slave devices not connected automatically after WAN outage

Hi,
I have this setup at two locations, one haster home assistant at a stable fiber WAN link and a "slave" home assistant at with a 4G mobile WAN link. Sometimes the 4G WAN links drops for some time and I think that is normal with mobile WAN links. Sometimes everything works when 4G WAN link is back but sometimes not.
Another case is that if I restart the slave home assistant device after for example changed some configuration the "remote entites" in the haster home assistant are not updated and the connection between them are lost.
The solution to get this connection up and running aging is to restart the master home assistant device and then everything is ok.
I am wondering if it is possible to implement some kind of heartbeat in this plugin so the master home assistant talk to the "slaves" regularly even if no event is sent between them. Then if that fail maybe try to reestablish the connection between them.

what you are describing is actually what should already be happening - when the websocket connection is anyhow interrupted, the remote entities should disappear automatically. Then every 10 seconds, a connection should be attempted again. If successful, entities should appear again.
This is working in my setup, no restarts whatsoever needed, but how you are describing it there seems to be a bug somewhere in that logic.

what you are describing is actually what should already be happening - when the websocket connection is anyhow interrupted, the remote entities should disappear automatically. Then every 10 seconds, a connection should be attempted again. If successful, entities should appear again.
This is working in my setup, no restarts whatsoever needed, but how you are describing it there seems to be a bug somewhere in that logic.

Ok is it possible for me to find some logs at my setups that can help you find this?

Now it happened again, 2 days without updates from the remote HA in the garage

Same for me, my slave is on a 3G modem and when it change WAN IP it take some time for the duckdns.org to change the redirect and under that time the connection is lost and don't get back up until i reboot the master. As said before, would be good to be able to get som kind of logs.

Is it possible that this issue only appear when using a Dynamic DNS?
Maybe home-assistant-remote can't recognize cause something is cached (like some .js in a browser). I have the same issue.
I access my remote instance over https://mysuper.subdomain.de:8125.
When i change the Port in my Router to block it, for example to 8126,
the master instance has no clue that it is gone. Entitys don't disappear and
no error logs. After a while (10-15min) this appears in the log
(not sure if related, could be some other problem, maybe someone here check if he had the same logs) :
2020-06-15 15:47:28 ERROR (MainThread) [homeassistant.components.rest.switch] Error while fetching data: [Errno 104] Connection reset by peer Traceback (most recent call last):   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 198, in async_update     await self.get_device_state(self.hass)   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 210, in get_device_state     self._resource, auth=self._auth, headers=self._headers   File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 504, in _request     await resp.start(conn)   File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start     message, payload = await self._protocol.read()  # type: ignore  # noqa   File "/usr/local/lib/python3.7/site-packages/aiohttp/streams.py", line 591, in read     await self._waiter aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer 2020-06-15 15:53:08 ERROR (MainThread) [homeassistant.components.rest.switch] Error while fetching data: [Errno 104] Connection reset by peer Traceback (most recent call last):   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 198, in async_update     await self.get_device_state(self.hass)   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 210, in get_device_state     self._resource, auth=self._auth, headers=self._headers   File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 504, in _request     await resp.start(conn)   File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start     message, payload = await self._protocol.read()  # type: ignore  # noqa   File "/usr/local/lib/python3.7/site-packages/aiohttp/streams.py", line 591, in read     await self._waiter aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer 2020-06-15 15:53:38 ERROR (MainThread) [homeassistant.components.rest.switch] Error while fetching data: [Errno 104] Connection reset by peer Traceback (most recent call last):   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 198, in async_update     await self.get_device_state(self.hass)   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 210, in get_device_state     self._resource, auth=self._auth, headers=self._headers   File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 504, in _request     await resp.start(conn)   File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start     message, payload = await self._protocol.read()  # type: ignore  # noqa   File "/usr/local/lib/python3.7/site-packages/aiohttp/streams.py", line 591, in read     await self._waiter aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer 2020-06-15 15:56:13 ERROR (MainThread) [homeassistant.components.rest.switch] Error while fetching data: [Errno 104] Connection reset by peer Traceback (most recent call last):   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 198, in async_update     await self.get_device_state(self.hass)   File "/usr/src/homeassistant/homeassistant/components/rest/switch.py", line 210, in get_device_state     self._resource, auth=self._auth, headers=self._headers   File "/usr/local/lib/python3.7/site-packages/aiohttp/client.py", line 504, in _request     await resp.start(conn)   File "/usr/local/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 847, in start     message, payload = await self._protocol.read()  # type: ignore  # noqa   File "/usr/local/lib/python3.7/site-packages/aiohttp/streams.py", line 591, in read     await self._waiter aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
After i change the port back and verify connection is ok, the master instance still don't recognize it and doesn't update. Only a restart could solve this.

I also have the same problem over a 3g/4g router. I connecting using nabu casa.
I see this on the host trying to connect to.
Logger: homeassistant.components.websocket_api.http.connection.1810656624
Source: components/websocket_api/http.py:47
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 24 juni 2020 2:53:50 (1 occurrences)
Last logged: 24 juni 2020 2:53:50
Disconnected: Did not receive auth message within 10 seconds.

I have the same problem but master and slaves are on the same local LAN… If I reboot the master it will work again for 1-2 days.

Same Problem here :-(

Thanx for the update, it is looking very prommising!

Hade been running version 2.2 with extra features for handling connection problems but tonight when my home assistant with 4G in my garage list internet for 3-4 minutes the connection at my master HA was not resumed. How do I collect all logs you need to troubleshoot this? A reboot of master HA solved the problem

8 days of testing and 0 problem for me, it usually lost connection within 48 hrs or less.
It handles re connections great. my network is LAN and Wifi based no 3g/4g etc.

8 days of testing and 0 problem for me, it usually lost connection within 48 hrs or less.
It handles re connections great. my network is LAN and Wifi based no 3g/4g etc.

This is my experience as well in a LAN configuration. Having 3 different HA instances in a hub-spoke configuration.

This is how I fixed it...  starting with line 153 thru "async def _init(self):"
It stays connected for short interruptions and reconnects without fail including on network disconnects...
It has been extremely stable and reliable and eliminates logs every 10 seconds while disconnected.
This component can also be used bi-directionally with multiple systems being both masters and slaves...
This should be done as a pull request if I had the time to learn how to do that...
    self._connection = None
    self._entities = set()
    self._handlers = {}
    self._remove_listener = None
    self._remove_entities = True

    self._instance_attrs = {
        'entity_prefix': self._entity_prefix,
        'host': self._host,
        'port': self._port,
        'secure': self._secure,
        'verify_ssl': self._verify_ssl
    }

    self._entities.add(self._connection_state_entity)
    self._hass.states.async_set(self._connection_state_entity, STATE_CONNECTING, self._instance_attrs)

    self.__id = 1

def _prefixed_entity_id(self, entity_id):
    if self._entity_prefix:
        domain, object_id = split_entity_id(entity_id)
        object_id = self._entity_prefix + object_id
        entity_id = domain + '.' + object_id
        return entity_id
    return entity_id

@callback
def _full_url(self):
    """Full url for connection."""
    return '%s://%s:%s/api/websocket' % (
        'wss' if self._secure else 'ws', self._host, self._port)

@callback
def _short_url(self):
    """Short url for display."""
    return '%s://%s:%s' % (
        'wss' if self._secure else 'ws', self._host, self._port)

async def async_connect(self):
    """Connect to remote HA websocket..."""
    url = self._full_url()
    urld = self._short_url()

    session = async_get_clientsession(self._hass, self._verify_ssl)
    conn_err = False
    self._hass.states.async_set(self._connection_state_entity, STATE_CONNECTING, self._instance_attrs)

    while True:
        try:
            _LOGGER.info('Connecting to %s', urld)
            self._connection = await session.ws_connect(url)
        except aiohttp.client_exceptions.ClientError as err:
            if not conn_err:
                _LOGGER.error(
                    'Could not connect to %s, retry every 10 seconds...', urld)
            conn_err = True
            self._hass.states.async_set(self._connection_state_entity, STATE_RECONNECTING, self._instance_attrs)
            await asyncio.sleep(10)
        else:
            _LOGGER.info(
                'Connected to HA websocket at %s', urld)
            self._hass.states.async_set(self._connection_state_entity, STATE_CONNECTED, self._instance_attrs)
            break

    async def stop():
        """Close connection."""
        if self._connection is not None:
            _LOGGER.error('Websocket connection at %s closing...', urld)
            await self._connection.close()
        self._hass.states.async_set(self._connection_state_entity, STATE_DISCONNECTED)

    self._hass.bus.async_listen_once(EVENT_HOMEASSISTANT_STOP, stop)

    asyncio.ensure_future(self._recv())

def _next_id(self):
    _id = self.__id
    self.__id += 1
    return _id

async def _call(self, callback, message_type, **extra_args):
    _id = self._next_id()
    self._handlers[_id] = callback
    try:
        await self._connection.send_json(
            {'id': _id, 'type': message_type, **extra_args})
    except aiohttp.client_exceptions.ClientError as err:
        _LOGGER.error('Remote websocket connection closed: %s', err)
        await self._disconnected()

async def _disconnected(self):
    if self._remove_entities:
        # Remove all published entries
        for entity in self._entities:
            self._hass.states.async_remove(entity)
    if self._remove_listener is not None:
        self._remove_listener()
    self._remove_listener = None
    if self._remove_entities:
        self._hass.states.async_set(self._connection_state_entity, STATE_DISCONNECTED)
        self._entities = set()
    self._remove_entities = True
    asyncio.ensure_future(self.async_connect())

async def _recv(self):
    """Retreive data from remote instance."""

    while not self._connection.closed:
        try:
            data = await asyncio.wait_for(self._connection.receive(), 30)
        except aiohttp.client_exceptions.ClientError as err:
            _LOGGER.error('Remote websocket connection closed: %s', err)
            break
        except:
            _LOGGER.error('Websocket connection lost...')
            await self._connection.close()
            break

        if not data:
            break

        if data.type in (aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSED):
            _LOGGER.error('Websocket connection is closing...')
            break

        if data.type == aiohttp.WSMsgType.ERROR:
            _LOGGER.error('Websocket connection had an error...')
            break

        try:
            message = data.json()
        except TypeError as err:
            _LOGGER.error('Could not decode data (%s) as json: %s', data, err)
            break

        if message is None:
            break

        _LOGGER.debug('Received: %s', message)

        if message['type'] == api.TYPE_AUTH_OK:
            await self._init()

        elif message['type'] == api.TYPE_AUTH_REQUIRED:
            if not (self._access_token or self._password):
                _LOGGER.error('Access token or api password required, but not provided')
                return
            if self._access_token:
               data = {'type': api.TYPE_AUTH, 'access_token': self._access_token}
            else:
               data = {'type': api.TYPE_AUTH, 'api_password': self._password}
            try:
               await self._connection.send_json(data)
            except Exception as err:
               _LOGGER.error('Could not send data to remote connection: %s', err)
               break

        elif message['type'] == api.TYPE_AUTH_INVALID:
            _LOGGER.error('Auth invalid, check your access token or API password')
            await self._connection.close()
            return

        else:
            callback = self._handlers.get(message['id'])
            if callback is not None:
                try:
                   callback(message)
                except:
                   await self._connection.close()
                   self._remove_entities = False
                   break

    await self._disconnected()

async def _init(self):

I believe I have fixed most re-connect issues in the latest release. If possible, please give it a try if you are experiencing problems and let me know if something isn't working.

I believe I have fixed most re-connect issues in the latest release. If possible, please give it a try if you are experiencing problems and let me know if something isn't working.

Yes at least with version 3.0 and 3.1 it get a lot better, great job! Will close this now


fix(searchPlugin): ensure search updates are unique to collection

Description
Our app is using the searchPlugin with two different collections. We were having issues with search results disappearing randomly when we saved certain records.
After digging in, we found that the syncWithSearch hook is only searching for matching search records by id, not relationTo.
This resulted in the syncWithSearch hook incorrectly overwriting records in the wrong collection if they share the same ID. This patch resolves the issue.

[x] I have read and understand the CONTRIBUTING.md document in this repository.

Type of change

[x] Bug fix (non-breaking change which fixes an issue)

Checklist:

[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Existing test suite passes locally with my changes

I did run the test suite locally but it seems to be failing for other unrelated reasons

@lynndylanhurley I think your fix makes sense. I don't think the failing tests are related.


Add new release notes for Workers

Release notes for releases:

2024.3.0
2024.3.100
2024.3.200
2024.3.300
2024.3.400

Thank you!


Result Count

Would really love if after searching, right next to the search input it showed a number of results found.

I added the result count to the scrolling header in order to indicate the results are filtered if you're scrolled down the list.

Awesome, when's the next release coming? and is it possible for you to open an issue or use projects to indicate what updates are coming in the next release? I'd love to know what's in the works without having to comment on an issue

Oh sure, I've never used Projects before but I can take a look. The releases happen whenever I feel like I've sufficiently tested the new features, which is an arbitrary amount of time right now.


WIP - add analysis server features

On hold until we:

[ ] figure out how to deploy a recent version of the DAS to users

superseded by #500


Add models to CI

@ahsan-ca  add the list of models to the ticket here
Models to add :
ResNet50-int8
GPT model either small/distill or GPT-2
Yolo-v5.
Models to remove :
slim-vgg_1
slim-resnet50_v2_64
slim-mobilenet_1
torchvision-inceptionv3_1

Draft PR put up for yolov5s: https://github.com/ROCmSoftwarePlatform/migraphx-benchmark-utils/pull/31

Link from where GPT2 model was downloaded from: https://github.com/onnx/models/blob/main/text/machine_comprehension/gpt-2/model/gpt2-10.onnx

PR updated with tuning info for yolov5s and resnet50-int8 models. Also updated the PR to remove the models in the description.


Create role gpgsync_keylist

Create a role to generate the gpgsync keylist

This should be part of the overall gpgsync role


PLRU related updates

There is are couple changes
1. In the case of a read miss, plru is updated when the refill response is received wihtout an error
2. In the case of CMO_PREFETCH hit, plru is updated (correction after an RTL bug fix)

Thanks
Tanuj

OK @MikeOpenHWGroup
Thanks for this one.
Regards
Tanuj


How do i fail a build from a plugin?

Is there a way to get sdm to fail from a plugin? I’m doing everything I can think of and it’s still just chugging along assuming the toucan plugin succeeded. I’m explicitly writing to stderr and exiting with a non 0 exit code
if [ $? -ne 0 ]; then >&2 echo "Failed to get claim.pem.crt from AWS" && exit 1; fi

sdm output
Failed to get claim.pem.crt from AWS
* Phase 0 Completed
* Enter image '../2023-05-03-raspios-bullseye-arm64-lite.img' for Phase 1

This is problematic because sdm will build an image that doesn't have what we require. Worse, it's burying the error message in the massive amount of output.

Similarly it would be nice to be able to have the sdm execution fail if the image runs out of space or apt fails to install a package. If i'm not staring at stdout or looking through the logs afterwards, it's incredibly easy to miss these errors.

@justin-robinson No, there is currently no way to fail from within a plugin. If a plugin has an error, things just keep chugging along.
I originally thought that it was best to let things go, because that would expose the most errors in a run. But I can understand that in certain environments that may not be desirable.
I can think of at least two possible solutions. What are your thoughts on these?

Change your failure path to provide a better, perhaps coded error message. By "coded" I mean something that you can grep the log for (e.g, FAIL or AWS-KEY-FAIL, or whatever) to determine if there are any problems.
It's possible that some things currently characterized as warnings should be errors, but I think I've been careful about this.

Implement a way for a plugin to signal to the upper levels of sdm that something has gone seriously awry, and have the upper levels notice this and stop sdm. The signal could either be a real signal (invoked via a function that the plugin calls) or a flag file in /etc/sdm.

A real signal is perhaps cleaner, but would likely be much more definitive, in that the trap handler wouldn't do anything besides clean up (unmount disk, delete loop device), and exit. The code raising the signal could also provide some text to be output immediately before exiting.
Flag file existence would indicate an error and cause sdm to cease processing. This check would likely only be done in the loops that run the plugins in phase0, phase1, and post-install. The contents of the file could include additional information to be displayed as sdm stops processing and exits.

I'm not wed to either of those, and open to other ideas, as long as they aren't too complex/costly.
Also, in case you weren't aware, messages output by sdm and plugins are preceded with a single character:

* Start of a new 'section' (e.g., * Start Phase 1 image customization)
> Informing you of something that sdm is doing
% Warning you that something isn't quite right, but forging ahead
? An error was identified
This idea was that you grep the log for '%' and '?' to decide if things worked or not. I use this all the time.

@wreedamz are you aware that the end of the log can have information in it such as
%%%
% IMG '2023-10-10-raspios-bookworm-arm64-lite.img' is 100% full (0 1K-blocks free)
% Review /etc/sdm/apt.log in the image for insufficient disk space errors
% If needed use --extend --xmb nnnn with --customize to create more space
%%%
? apt reported errors; review /etc/sdm/apt.log
* Enter Shell Command Prompt
  'exit' to exit back to host system
...
sdm~#

As an aside, sdm checks for apt errors by
    grep -q "^E:" /etc/sdm/apt.log && logtoboth "? apt reported errors; review /etc/sdm/apt.log"

which you can obviously do yourself as well. If this information is not sufficient for you, what would you propose?

After further thought on this, I think the best way is to implement this via real signals. There are still a few nuances to work through, so premature to predict when this will pop out, but I think it's important so it is at the top of the ToDo list.

Ignore previous comment. Here's sketch of what I'm thinking. I'd appreciate your thoughts on it.

New functions for plugins to use:

Take a note: Write a line of text to a file that is displayed at the end of a customization.
Abort: print provided text and abort the customization

sdm will check for an abort request every time a plugin returns from being called for a phase. It will also abort via this mechanism if out of disk space or if apt logs a message starting with 'E:' in /etc/sdm/apt.log
sdm will print the contents of the take a note file at the end of the customization, if it exists. Content and format are up to you.
Will this work for you?  Thx

I really appreciate the thought you put into this. I think your idea with "real signals" was the best. It's the expected behavior from a bash script and requires less documentation from you and learning from us. Principle of least surprise would support failing if a plugin exiting with a non zero exit code.

I also understand this is a breaking change so gating this with a cli flag or env variable would be best

I really appreciate the thought you put into this. I think your idea with "real signals" was the best. It's the expected behavior from a bash script and requires less documentation from you and learning from us. Principle of least surprise would support failing if a plugin exiting with a non zero exit code.

I agree with you, and that was my first inclination. Until I thought about how it would work, and it won't.
When sdm is running phase 1 and post-install code it's running in a systemd-nspawn container so the plugin can't signal the sdm code running in the parent.

I understand and that sounds like a hard stop. If you think that's the best way then I'l defer to your judgement.

I understand and that sounds like a hard stop. If you think that's the best way then I'l defer to your judgement.

I wish it wasn't, but as often happens in software development it's trading off "purity of programming" vs "keeping focused".
It will seem natural for people writing sdm plugins (using one new call), and it will be definitive, in that calling abort will not return, keeping the plugin code simple.
And lastly, it solves your problem sooner 😎

I understand and that sounds like a hard stop. If you think that's the best way then I'l defer to your judgement.

I reconsidered this during my short San Juans trip and am now moving down the path to properly handle failure codes from plugins. This is not "real signals"; those are a complexity I'd prefer to avoid for now.
Rather, if a plugin exits any of the phases with a non-zero status the sdm customization will be terminated.

My main concern with honoring plugin return status was that the sdm world as we know it would be broken. That was not the case (none of the sdm-provided plugins broke), so this will be in the next release which will likely arrive the week after Thanksgiving.
So much for my "hard stop" 🤣
You may find this new command line switch useful as well: The --plugilist switch. It takes a list of files (e.g., --pluglist file1,file2,file3), and can also be invoked on the command line multiple times. Each file is a list of plugins. For example:
user:deluser=pi
user:adduser=bls
system:fstab=fstab.extension|eeprom=stable|sysctl=/path/to/some-sysctl-config.conf
network:nmconn=/path/to/eth0.nmconnection,/path/to/wlan0.nmconnection
trim-enable
quiet:consoleblank=300|noquiet|nosplash
vnc:tigervnc=2540x1350,1880x960,1700x1200,1880x1100
apps:name=myapps|apps=@myappslist

This is complete in V10.0. Please give it a go and LMK of any issues. Thx

Thank you! It'll be a couple months before we touch this again but I'll reply back when I have feedback.

Thank you! It'll be a couple months before we touch this again but I'll reply back when I have feedback.

Thanks for inspiring this change. Looking forward to hearing how it works for you when you are back at it.

Closing this issue. if/when you get around to testing it, please feel free to re-open this issue or start a new one for any problems you run into. Thx

I'm getting around to testing this now and I'm having some issues.

If a plugin fails with an error, the error code that it fails with is not logged. Perhaps Plugin '<name>' exited with failure could be modified to include this
If an app installation fails with apt, sdm execution exits successfully rather than with an error code

Also looks like I don't have permission to re-open this issue @gitbls, let me know if you would like me to start a new issue

I'm getting around to testing this now and I'm having some issues.

If a plugin fails with an error, the error code that it fails with is not logged. Perhaps Plugin '<name>' exited with failure could be modified to include this
If an app installation fails with apt, sdm execution exits successfully rather than with an error code

is reasonable. Will look into it for next release
I don't understand. Exactly how is apt invoked in this failing scenario? Is it in your plugin or something else?

Also looks like I don't have permission to re-open this issue @gitbls, let me know if you would like me to start a new issue

should be open now

In the scenario I was testing, one of the apps specified no longer exists in a particular repository. You can reproduce this issue with something like --plugin apps:"apps=fakeappdoesntexist"

Ah, so the failure is that the apps plugin doesn't check status. Yes, indeed, it doesn't now, but it will for the next release, which I'm planning for the week of March 18th due to some impending out-of-pocketness.

In the scenario I was testing, one of the apps specified no longer exists in a particular repository. You can reproduce this issue with something like --plugin apps:"apps=fakeappdoesntexist"

Doing:
 sdm --customize --poptions noupdate,noupgrade,noautoremove --plugin apps:apps=foobaz virgin-image.img

on 11.4 I get:
* Start [default] Application Installs
> [default] Application Installs: foobaz
? apt returned an error; review /etc/sdm/apt.log
? Plugin 'apps' exited with failure
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted

So I'm a little confused. Is the scenario more complex than this?
In spite of my confusion, I've corrected the involved code to correctly handle status, as well as providing the status code per your suggestion:
* Start [default] Application Installs
> [default] Application Installs: foobaz
? apt returned an error; review /etc/sdm/apt.log
? Plugin 'apps' exited with failure status '1'
umount: /mnt/sdm/boot/firmware unmounted
umount: /mnt/sdm unmounted

Thx

In the scenario I was testing, one of the apps specified no longer exists in a particular repository. You can reproduce this issue with something like --plugin apps:"apps=fakeappdoesntexist"

For this issue, the final return/exit code of SDM was not being set correctly. The error messages were printed as appropriate and the final return code was 0 despite the error. I'm not sure if that's been corrected in the console output you shared

Ah! Thanks for the clarification. Will address.

Fixed in V11.6. Please test/verify. Thx

Closing due to no activity. Please re-open or start a new issue if required.


chore: bdd-ghfjxy-import-1583256199 to 0.0.1

chore: Promote bdd-ghfjxy-import-1583256199 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


systemvm: have flags to check x86_64 to install specifics for amd64 arch

This flips conditional checks to check specifically for x86_64/amd64 to install dependencies in systemvm.

@rohityadavcloud , how do we test s390x? do we know a guy ;) ?

@DaanHoogland kicked build using Jenkins, to see arm64 & x86_64 template building okay: https://build.yadav.cloud/systemvm/
Build logs:
build-arm64.txt
build-x86.txt

@DaanHoogland kicked build using Jenkins, to see arm64 & x86_64 template building okay which should be enough to validate this PR: https://build.yadav.cloud/systemvm/
Build logs: build-arm64.txt build-x86.txt

agree on those two @rohityadavcloud . I was just wondering about the s390x arch; is it used anywhere and do we know anybody willing to test that one?


Status: not cached on normal url when option to cache url with query strings is enabled

Hello. I think I've found another bug with caching exceptions. When the option to cache url with query strings is disabled, everything is ok. URLs without query strings are cached, ones with query strings are not:
Page without query:
Caching:            enabled
Status:             cached

Page with query:
Caching:            disabled
Reject reason:      Requested URI contains query
Status:             not cached

However, with the option to cache urls with query strings enabled, the results are:
Page without query:
Caching:            enabled
Status:             not cached

Page with query:
Caching:            enabled
Status:             cached

Note that caching is enabled, but status is not cached, which is strange. I don't know if it is just a debug error - pages are loading fast even when status is not cached and they seem to get cached. But I'm not sure if everything is ok.
By the way, in the second case, "page with query" is not just any page with a query. The query must have an effect on rendered content. But in the first case, any query will do to be treated as a page with query and be rejected from caching.

It is something acpu related. I switched to disk enhanced, but then the option to cache pages with queries is disabled. Anyway it's not that necessary for me to cache pages with queries and disk enhanced seems to perform better than acpu on my vps.

Sorry for the delay.  Have you moved on to using the official 0.9.5 release (on WP.org)?  Delayed looking into this because Frederick (w3tc author) has finally risen from the dead.  But, for the sake of consistency ill have a look into this issue later tonight.  Thanks again for the heads up!

So to be clear...this all works fine when in Disk: Basic mode (since cache queries checkbox is available but disabled under Disk Enhanced).  However, in APCu mode only the problem happens?

Update
I did some tests with APCu and it seems this is not a bug....let me explain:
When you see:
Caching:    enabled
Status:     not cached

it indicates that a cache entry doesnt currently exist and one is going to be generated (to be used for subsequent requests to that page).  In other words, someone is calling a page for the first time (or it has an old cache file) and a new cache needs to be generated and written out.  So "not cached" is an indication that this page doesn't currently have a cache entry but since there is a "caching:enabled", a new cache file is being generated.  This is explains why when you looked, despite seeing "not cached", that you saw it was indeed cached.
And so, that leads us to this:
Caching:    enabled
Status:     cached

This indicates that a cache entry does exist (and hasn't expired yet) and is currently being served back to the client browser.
Not sure if your cached entries are repeatedly showing "not cached" for pages that were cached moments earlier.  If so then i wonder about your APCu settings in that i know there is a feature that flushes the entire memory after a certain interval and if that interval time is set very low.
So for now i will Close this thread, but if you think i am wrong please feel free to re-open it.  More than happy to assist.


implement class bmpl::BMPImagesList<>

in file bmp_image.h.

Completed. Complies. Still to be tested.


Q API

See also PR on StopWatch: https://github.com/logrhythm/StopWatch/pull/28
Q API + tests borrowed and modified from https://github.com/KjellKod/Q
The template powered Q API supports

Sender API:   push

Receiver API: pop, pop_and_wait
if pop_and_wait is not implemented in the Q Type then it will be using a template implementation approach utilizing the pop

Base Q API:

  usage() -> % of queue that is filled up
  size() -> used size of the queue
  capacity() -> max size possible
  capacity_free() -> unused capacity
  full() -> if queue is full
  empty() -> if queue is empty

As long as the Q Type implements the base API and the Sender Receiver API any queue can be used in the templated API.
It is recommended that using or typedef is used in the client code to reduce template arguments and to keep the queue specifics changeable in one code location.
With this setup the creating location in the code will have the template arguments specifics but the actual client code using the queues does not need to know the details and the queue type can change without any client code changes.

@josh-cogdill  @craig-cogdill  Please review this on Monday and go ahead and merge it if it looks good. We did one pair review on it already.


scoretxt never updates unless you did hit/miss a note

can prevent memory leaking due to that text being updated every frame
example video:
https://user-images.githubusercontent.com/45212377/176214243-31123860-2a89-4d38-8b10-a8c4e8b4a57d.mp4

waletr., whit e

waletr., whit e

https://user-images.githubusercontent.com/45212377/176217100-e89192b3-15dc-4dce-af3e-bf89bea87f9d.mp4
!!!

Hwalter hwite... Also wouldn't updating score under recalculateRating() also work? I have it like that on a personal build and I'm wondering if your way would be better instead

hmmm you're right, gonna test this

oh yeah, for custom score shits, you need to use onRecalculateRating
example:
function onRecalculateRating()
    if ratingFC == '' then -- if the FC is nothing
        setProperty('scoreTxt.text',
        'Score: ' .. score ..
        ' • Misses: ' .. misses ..
        ' • Accuracy: ' ..round(rating * 100, 2).. '%')
    else
        setProperty('scoreTxt.text',
        'Score: ' .. score ..
        ' • Misses: ' .. misses ..
        ' • Accuracy: ' ..round(rating * 100, 2)..
        '% [' ..ratingFC..']')
    end
end

right now


Down URL lagging releases

README
kibana-enhanced-table-X.Y.Z_A.B.C.zip
Releases
enhanced-table-0.6.0_6.1.1.zip

Hi @janekdb
You're right => good catch !
I just fixed README file.


ractive.hasListener() example/usage

Description:
Currently, ractive.hasListener('foo') does not behave as I expected:
new Ractive({
  el: 'body',
  template: `
    <foo on-bar="baz">hello</foo>       <!-- has listener -->
    <foo on-bar="nonexist">hello</foo>  <!-- has listener -->
    <foo>hello</foo>                    <!-- does not have listener -->
  `,
  components:{
    foo: Ractive.extend({
      template: `<button on-click="bar">{{>content}}</button>`,
      onrender: function(){
        if (this.hasListener('bar')){
          alert('component has on-bar=... listener!')
        }
      }
    })
  },
  on:{
    baz: function(ctx){
      alert('hello from baz!')
    }
  }
})

...gives the following exception:
Uncaught TypeError: this.hasListener is not a function
    at Child.onrender (script.js:12)
    at Hook.fire (ractive.js:6923)
    at Array.<anonymous> (ractive.js:17905)
    at flushChanges (ractive.js:5792)
    at Object.end (ractive.js:5689)
    at render$1 (ractive.js:17938)
    at Ractive$render [as render] (ractive.js:17969)
    at initialise (ractive.js:17866)
    at new Ractive (ractive.js:18571)
    at script.js:1

Versions affected:
Edge

("expected" link is updated, this comment will be deleted after its notfication task has been executed)

hasListener is only available on contexts, so you'd need to use the context passed into the render event. That context still won't look at the template for the component, but it probably should.

I'm using my hasEvent method to decide to add some other abilities to the component (or remove some other) onrender. For example, if my specific component has sync-topic="foo.bar", then there is an actor is created to communicated with the application network in realtime. That's why hasListener has to be available in the instance, not only the context. Would you consider this change, or should I keep hasEvent method for this purpose?

My only issue with adding a hasListener to the ractive prototype would be that I would expect it to look at events subscribed with ractive.on and not/not just the component template event listeners.
As of edge, I've fixed context.hasListener to work with ractive contexts (as opposed to just element contexts), so my example above should work as soon as travis is done doing what travis does. I've also made ractive.getContext() with no args return a ractive context, so you can use it outside of lifecycle events e.g. ractive.getContext().hasListener('myevent'). Does that get you closer to where you need to be?

As of now, ctx.hasListener() works well: Playground.
Thanks.


Prodigy Optimizer

Feature request
Want to know if it is possible to implement Prodigy optimizer into bnb with 8bit support.
Motivation
Prodigy is now widely used in FT since it is more user friendly especially for not-expert users. Who are also most not likely to have lot of knowledge about NN and good GPUS.
And since Prodigy have 4 full size state, means it will consume a lot of vram. I think it is crucial to implement some kind of space optimization for prodigy. And I think bnb's 8bit optimizers are good  way for this goal.
Your contribution
None, sry
I have read the source code and I think we will need to add a new "4state"(or even 5state) optimizer class. And since the thing need to be done is related to some CUDA things which is out of my ability.
If we can have some general template for optimizer which let us to just fill the logic inside of it. It may be ok for me to make PR for Prodigy or other optimizers.

Sure would be nice to get this as 8bit for sure as it is greatly needed.

Yes please!

+1 I'd find this immensely useful seeing as Prodigy is already slower than AdamW.

+1


Slates should allow users to make aspect ratio changes, or reflow content.

Overview
Slates should allow users to make aspect ratio changes, or reflow content by adjusting edge affordances. Currently the documentation describes this, but it isn't how MRTK prefabs or scenes actually behave. Ideally, developers could choose between the rotate and resize behavior of the edge affordance and visual feedback would communicate to end users which behavior to expect (with developer documentation that describes how to implement this).
Expected behavior
Expected behavior is described in the on-line documentation:

By "grab and drag" manipulation affordances, users can do uniform scaling through the corner affordances, and can reflow the slate via the edge affordances

On the HoloLens 2, the Microsoft Store is an example of an application that behaves as expected.
Actual behavior
Edge affordances rotate the slate instead of changing the dimensions of the slate. See the hand interaction example scene and the slate prefabs for examples.
Steps to reproduce

Create an empty MRTK scene
Add one of the Slate prefabs to the scene.
Run scene and note that edge affordances are mapped to rotate, not a resize behavior.
Check documentation to see if there are instructions for changing this behavior... Documents do not reflect the actual behavior of the prefabs.

Unity editor version
2019.3.13f1
Mixed Reality Toolkit release version
2.5.1

Hi @CDiaz-MS this sounds like what you having been working on!

We appreciate your feedback and thank you for reporting this issue.
Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.
Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.
Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.
Thank you for your continued support of the Mixed Reality Toolkit!


[master] - Update OSA SHA

It's that time again!

Closing in favor of: https://github.com/rcbops/rpc-openstack/pull/1957


inconsistencies with filter syntax

There are inconsistencies with filter syntax between the Azure AD Powershell module and the Azure AD dynamic group syntax.  If I'm creating a filter with powershell using the Azure AD module (The Graph API version of the module), the syntax is required to be written in oData 3.0 standards.  The oData 3.0 standards have their own limitations, but then if I'm trying to keep things just a little consistent between that and dynamic group membership rules, that immediately fails.  The operator for oData does not use a prefixed dash, where as the operator for dynamic group membership rules DO use a prefixed dash.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5f7a4070-1be8-5f1f-0092-68fb6b9e2305
Version Independent ID: cd34312f-b507-affe-555e-1e0c63e7d494
Content: Dynamic automatic group membership rules reference in Azure Active Directory
Content Source: articles/active-directory/users-groups-roles/groups-dynamic-membership.md
Service: active-directory
GitHub Login: @curtand
Microsoft Alias: curtand

@poortom1004 Thanks for your feedback! We will investigate and update as appropriate.

@poortom1004 I get your point.  Please post this as a Product feedback at https://feedback.azure.com/forums/169401-azure-active-directory. This will allow the community to upvote and for the product team to include into their plans. We also have the "Give Product Feedback" button in our documents now to take you directly to the appropriate feedback page.

@poortom1004 We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


OpenAI timeout

OpenAI APIs are quite unstable recently and time out often. Training a prompt requires quite a number of calls which means 99% you will experience a timeout. The current max_time is hard-coded as part of the Backoff decorator and set to 1000 seconds. It would be good to be able to change this value so that the compilation would not take forever.

Same problem

@raquette @andreapiso provide stack trace?

@darinkishore  what stack trace do you need? The program does not crash, it hangs because openAI does not response until it prints that it's retrying with exponential backoff because api.openai.com did not respond.

@andreapiso I tested with gpt-3.5-turbo-1106 (new version), it is much better.

Ah, I think this issue is (in the short term) less severe now?

I tested it today with 1106 and the OpenAI API still gets stuck fairly often, which makes dspy take a almost two hours to complete a generation of 8 candidate programs with 24 examples.

In particular, I am starting to get:
Error for example in dev set: 		 HTTP code 502 from API (
<title>502 Bad Gateway</title>

502 Bad Gateway

cloudflare

)

I recognise the issue is on the OpenAI side but not being able to change the behaviour and the timeout from dspy makes it quite a bit worse.

It is taking me more than two hours to train a chain-of-thought based question-answer predictor having 400 pairs, and often I am facing timeout problems. It would be good to resolve this issue.

Can you help me understand the issue? I run large experiments with OpenAI (the latest turbo model at all times) and, except when they note on https://status.openai.com/ that there are issues, it's always very fast and stable.
I understand sometimes they're unreliable. In that case, I don't see how DSPy can help though.

max_time is hard-coded as part of the Backoff decorator and set to 1000 seconds

We can definitely set this to less than 1000, but how can that help? Won't this just timeout on the next call still?
Basically if OpenAI is inherently unstable at a given hour, how can the library deal with that? The main thing that comes to mind is make sure you set the number of threads to be low enough depending on your rate limit.
Happy to address this if I'm missing some context @dasjyotishka @andreapiso @raquette @darinkishore

This may explain the issue from @dasjyotishka ?

Can you help me understand the issue? I run large experiments with OpenAI (the latest turbo model at all times) and, except when they note on https://status.openai.com/ that there are issues, it's always very fast and stable.
I understand sometimes they're unreliable. In that case, I don't see how DSPy can help though.

max_time is hard-coded as part of the Backoff decorator and set to 1000 seconds

We can definitely set this to less than 1000, but how can that help? Won't this just timeout on the next call still?
Basically if OpenAI is inherently unstable at a given hour, how can the library deal with that? The main thing that comes to mind is make sure you set the number of threads to be low enough depending on your rate limit.
Happy to address this if I'm missing some context @dasjyotishka @andreapiso @raquette @darinkishore

Not sure if the experience is different depending on the geographic region you are in (I am in  South East Asia), but my experience with OpenAI API has been very different, and it consistently drops about 10% of requests (not an exaggeration) especially when submitting multiple requests in parallel (still being very far from the rate cap). The requests being dropped happen at seemingly random so re-submitting the request most of the time works. This is has been experienced by multiple users, on different wifi networks using different machines (and different clients outside of DSPy, we get the same timeouts when calling OpenAI from javascript).
So, when optimising a prompt in dspy with something like random search, every 10-15 requests effectively dspy gets stuck for 15 minutes, unless I am there to stop the notebook manually and re-run it since the previous iterations of the optimisations get cached.

It happens consistently with single thread too, which is actually the worst case scenario, because at least with multiple requests in parallel other requests in the optimisation set can stil proceed while one gets stuck, instead the 15 minute break paralyses the whole process every time.

Okay, a couple more questions. Does this happen with the latest gpt-3.5-turbo (at whatever time you faced the issue)? It’s possible OpenAI downgrades older models.
Does your code include generation of many completions at once (like n=10) or something like that?

Basically, with a single thread, during the “15 minute” gap, the code is retrying the openai call with exponential backoff. If every repeated call keeps failing, that is completely outside our control. I think during 15 minutes, it basically has to retry around 10 times and fail every time. Is that true? In that case, what behavior do you expect from the library?

You may benefit from requesting a rate limit increase by the way. The default rate limit (tokens per minute) is absurdly small.

I am definitely nowhere near hitting the rate limit, this happens even just sending 20-30 calls over the span of the optimisation process.
Basically, with a single thread, during the “15 minute” gap, the code is retrying the openai call with exponential backoff. If every repeated call keeps failing, that is completely outside our control. I think during 15 minutes, it basically has to retry around 10 times and fail every time. Is that true? In that case, what behavior do you expect from the library?

The point is that if this was happening, I would see the "backoff retry" message being printed out right? Instead, nothing gets printed out for an absurd amount of time. The expectation would be to allow users to pass the timeout parameter for dspy to set accordingly? So that if I know my calls should last  time, I could set a timeout of <x + buffer>.
I think failing 10 times in 15 mins with backoff would happen for a 60 seconds timeout, but are you sure the timeout is 60 seconds, and not 10 minutes?  (e.g. see here: https://github.com/openai/openai-python/issues/762 ) the experience right now is that every time a call is dropped everything in DSPy hangs for 10+ minutes with no backoff or anything else being printed.

The default rate limit can be triggered by as little as 20 calls. Isn’t it something like 90k tokens per minute?
That said, I understand now your concern. It’s not directly related the 15 minute gap. You basically need a timeout for each individual request, not a smaller timeout for the whole backoff logic.
I think that’s reasonable. We can make sure each individual request times out in 40 seconds (though I would have preferred that OpenAI drops the request properly on their end rather than an arbitrary small timeout). We still want to keep the 1000 second logic as-is. Does this sound right to you?

Facing the same issue on self histed LLM which works when token is set to 1000 but timeout when max_token is 2000.
When using openai api SDK directly with the prompt generated by checking inspect_history it works fine. Something wrong with OpenAI gpt3.py in dspy.
Error is same as above 502 Bad Gateway.


Using the native select in the mobile context

I'm submitting a ...  (check one with "x")
[ ] bug report => search github for a similar issue or PR before submitting
[ ] feature request
[X] support request

Current behavior

The mat-select doesn't work properly in the mobile context is there a way to use the native select experience in the mobile context ?
Expected behavior

Being able to set the native select in mobile context
Minimal reproduction of the problem with instructions

What is the motivation / use case for changing the behavior?

The default select in the mobile context in a superior UX
Please tell us about your environment:

Angular version: 2.0.X

Browser: [all | Chrome XX | Firefox XX | IE XX | Safari XX | Mobile Chrome XX | Android X.X Web Browser | iOS XX Safari | iOS XX UIWebView | iOS XX WKWebView ]

Language: [all | TypeScript X.X | ES6/7 | ES5]

Node (for AoT issues): node --version =

in that you case can create you own select type (named native-select) see https://formly-js.github.io/ngx-formly/guide/custom-formly-field (check the bootstrap select type which use the native one https://github.com/formly-js/ngx-formly/blob/master/src/bootstrap/src/lib/types/select.ts and how to create custom type for material ui https://github.com/formly-js/ngx-formly/tree/master/src/material/select/src)
then in your formly config you may check which type to use:```ts
{
type: 'city',
type: isMobile ? 'select':'native-select'
...
}


Cannot import Bugsnag from 'bugsnag' in test environment

I'm using the Bugsnag.notify API and therfore have to
import Bugsnag from 'bugsnag';

However, this breaks my tests:

Chrome 43.0 - Global error: Uncaught Error: Could not find module bugsnag imported from

Any suggestions here? I'm curious why index.js explicitly does not include bugsnag in test environments.
// index.js
this._includeBugsnag =
      this.isDevelopingAddon() || process.env.EMBER_ENV !== 'test';

Any chance we could import just the bugsnag/shim in test environments?

Hi @flexyford, thanks for reporting. Can you show me an example test that reproduces the problem and a bit more information about where you're doing the import in your app. e.g. is it in an initializer? Any example files you can provide are helpful

Hi @binhums thanks for the quick reply. I want to throw an exception, log it to bugsnag, and catch it, without terminating the program execution. Here's a dummy example of what I'm trying to accomplish:
// my-app/components/some-component.js
import Ember from 'ember';
import Bugsnag from 'bugsnag';

export default Ember.Component.extend({
  didInsertElement() {
    this._super(...arguments);

    // Bugsnag Telemetry
    try {
        throw new BugsnagException(
          "Expect this to log a Bugsnag Exception without terminating program execution"
        );
      }

    } catch(e) {
      Bugsnag.notify(e.name, e.message);
    }
  }
}

function BugsnagException(message) {
   this.name = 'Bugsnag Exception';
   this.message = message;
}

and to clarify what type of test would this be in? I want to recreate an example where we achieve this, possibly in the dummy test folder for this addon.

In my instance, it's an acceptance test which uses some-component.js.
I've come up with a solution which I hope still models the intended behavior.

In my instance, it's an acceptance test which uses some-component.js.
I've come up with a solution which I hope still models the intended behavior.
// ember-cli-bugsnag/index.js
// ...
config: function(env, config) {
    const { bugsnag } = config;
    const { releaseStage = env } = bugsnag;

    // Determine if we should include bugsnag:
    // That way if `env = test` it can still include `bugsnag`
    // if `bugsnag.releaseState !== 'test'`
    this._includeBugsnag = this.isDevelopingAddon() || releaseStage !== 'test';

    return {
      bugsnag: readEnvironmentConfig(process.env)
    };
  },

I'm confused why isn't the code in https://github.com/binhums/ember-cli-bugsnag/blob/master/index.js#L45 working in your situation.

It would seem to me that for whatever reason we're explicitly not loading bugsnag in test environments?

Because I'm in a test environment:
isDevelopingAddon() // => false and process.env.EMBER_ENV === 'test'
So this._includeBugsnag = this.isDevelopingAddon() || process.env.EMBER_ENV !== 'test' // => false

I'm not so good with the pattern matching syntax bear with me. Just so we're clear: so your code snippet overrides this line at the config step based on the release stage rather than the ember environment?
I'd definitely be open to supporting test environments of course, but I think a solution would be more likely focussed around mocking the Bugsnag library. You wouldn't actually want errors to be reported to bugsnag every time you want your test suite.
My open questions are:

why is it explicitly disabled in test mode currently, e.g. performance reasons?
the releaseStage is probably not what we're looking for, that is what gets recorded in bugsnag. I.e. this error happened in staging rather than production.
does the config hook always get called after the included hook? Is there a way we can make this relationship more verbose?
perhaps if test environment is listed in the notifyReleaseStages in the config then we should override the exclusion, but again I don't think you actually want to be hitting bugsnag.

There are examples of what you're describing in the acceptance tests for this addon https://github.com/binhums/ember-cli-bugsnag/blob/master/tests/acceptance/custom-release-stage-test.js. I'm assuming that these work because of the isDevelopingAddon line right?

why is it explicitly disabled in test mode currently, e.g. performance reasons?

to be clear I don't expect you to have an answer to this one :)

My hunch is it's disabled because we do not want to log errors in tests to bugsnag. However, if we exclude test from notifyReleaseStages then this should not be an issue.
Okay, that makes more sense. The above code is not the right solution then.
In my experience so far, the config hook always gets called BEFORE the included hook. So this._includeBugsnag would already be set.
I don't think test should be listed in notifyReleaseStages either.

The examples in acceptance tests work because this addon adds bugsnag as a dependency in package.json.
I think the real mystery is #1: Why is it explicitly disabled?

🕵  time to pull out the blame hammer
Would you be averse to specifying bugsnag as a dependency in your package.json as a workaround? Not trying to halt this conversation, just trying to help you achieve your immediate goals.

Looks like #52 (they made blame so much better in Github now!):

Approach:
Detect when the host app is in testing and basically make the addon enter stealth mode. Nothing is included in the app/vendor (bugsnag, initializers... nothing)

OK, it doesn't seem like an attempt to stop bugsnag reporting by accident at all costs because surely it isn't reported if it's not in notifyReleaseStages. Could it be performance?

🤔 I'm sure there are some performance gains by not including it, but it means tests fail when Reporting Handled Bugsnag Exceptions.
The correct solution here might be exposing all Bugnag. from ember-cli-bugsnag? That way the Custom Diagnostics such as getMetaData are still envoked even when calling Bugsnag.notify or Bugsnag.notifyException.

Ok I finally figured out what my actual issue was: Reporting Handled Bugsnag Exceptions.
Using the following pattern, I can handle exceptions, pass them to bugnsag using ember-cli-bugsnag and do not have to import Bugnsag from 'bugnsag'. It models the usage to Bugsnag.notifyException(exception[, name])
// app/utils/bugsnag-exception.js
export default function bugsnagNotifyException(e, name) {
  const onerror = Ember.onerror || function() {};
  e.name = name || e.name;
  onerror(e);
};

// app/components/some-component.js
import bugsnagNotifyException from '../utils/bugsnag-exception';
try {
  // Some code which might throw an exception
} catch (exception) {
  bugsnagNotifyException(exception, 'customErrorName');
}

I think ember-cli-bugsnag should expose more functionality for reporting error handling, but this is a good workaround that works for my use case.

Hey Alex. Really glad you were able to come to a solution in the end.
It's definitely not what that interface was originally designed for but I'm glad it works.
Are you happy for me to close the issue? I'm looking back to hearing from @cibernox on the specific motivation for the other PR.

I think ember-cli-bugsnag should expose more functionality for reporting error handling

Any chance you could give a concrete example? I often find myself being tempted to implement features that wrap around features that bugnsnag-js implements directly and I'm relatively convinced that the right thing there is to let the user work that out for themselves. E.g. see #53.

Woops, closed this issue by merging my branch, did not know github did that when referencing different repos? Yes I'm fine closing this issue, but will let @binhums decide

I often find myself being tempted to implement features that wrap around features that bugnsnag-js implements directly and I'm relatively convinced that the right thing there is to let the user work that out for themselves.

IMO a bugsnag wrapper should be exposed so that ember-cli-bugsnag users can utilize all the great things bugsnag-js has to offer without doubling up on the bugsnag-js dependency (Once for ember-cli-bugsnag and once again for the bugsnag methods which are not exposed by the addon).
Regardless, I think the README could benefit from an explanation of "reporting handled exceptions"
Reporting Handled Exceptions
To handle exceptions but still log them in Bugnag we have to call Ember.onerror directly:
try {
  // Some code which might throw an exception
} catch (exception) {
  exception.name = 'customErrorName';    // Default Name: 'Error'
  if (Ember.onerror) { Ember.onerror(e) };
}

@flexyford sorry for letting things hang so long, it didn't feel so actionable at the time.

exposed so that ember-cli-bugsnag users can utilize all the great things bugsnag-js has to offer without doubling up on the bugsnag-js dependency

Agreed

Regardless, I think the README could benefit from an explanation of "reporting handled exceptions"

Agreed, I know Bugsnag is quite limited on what you can do here. One thing that I'm not sure on is the distinction between marking a project as a JS project vs an Ember.js project. Perhaps they do a better job of grouping errors that are specific to the Ember.js ecosystem.

if (Ember.onerror) { Ember.onerror(e) };

It probably makes more sense to rethrow a wrapped exception. I wouldn't want to make any assumptions about the consuming app's setup. Otherwise looks great. Can you add a PR for the docs?
Sorry I've let this sit open a while. @flexyford correct me if I'm wrong but the two actions are:

Add README docs for renaming exceptions and
Get clarification from @cibernox on excluding in test instances.

I agree with @flexyford's (at least initial) assessment that we should NOT prevent the bugsnag library from being included, but should rather than the notifyReleaseStages config determine which bugsnags should actually be reported.
I talked with @cibernox about this; it would be bad if we suddenly started importing Bugsnag if people aren't prepared for it, since it might mean they'll start getting real bugsnags from test environments if they haven't spent time tweaking their notifyReleaseStages config. Perhaps the best way to go is to make this configurable?

@machty I think that a config option that overrides the default behavior and allows people to opt in to inporting bugsnag in test environment makes sense, it would be backwards-compatible and easy to implement.


Find optimal threshold with ROC analysis

I hate thresholds... but I have to live with them.. oh well..
At least let's try to find one that is not completely subjective.. with a ROC analysis. That's what is done in this pull request.
Done
At training time:

new flag in the CLI: --roc_increment --> controls the increment of thresholds between 0 and 1 that are considered during the ROC analysis. For instance: --roc_increment 0.1 --> 0, 0.1, 0.2, 0.3 ... 0.9, 1.0
If this flag is used in the command, then a ROC analysis is performed after the training using the validation sub dataset: for each batch, we run the model and aggregate the sensititivy and specificity computed on the predictions that have been thresholds with different values (0, 0.1, 0.2 etc). For each threshold, we compute the mean sensititivy (TPR) and 1-Specificity (FPR) to find the optimal threshold by maximising the TPR-FPR difference and plot this curve (saved in log_directory/roc.png).
We update log_directory/config_file.json with the optimal threshold (binarize_predictions) which could be further used at testing time.

At testing time:

Validation metrics are computed on data binarized with binarize_predictions. In case no ROC has been performed and binarize_predictions=-1, the metrics are computed on soft predictions (ie no binarization).
Output niftis are binarized using binarize_predictions (which is the optimal threshold if the ROC analysis has ben run) if binarize_nifitis=true. This allows us to potentially compute metrics on hard predictions while keeping soft nifitis.

When automate_training:

At testing time, we use log_directory/config_file.json in case binarize_predictions has been updated after training.

TODO:

Update config file if the change binarize_predictions & binarize_niftis is approved.

Fixes #368.

ivadomed -c config_roc.json -r 0.05

Gives:

THe green dot is related to the optimal threshold.

awesome! do i interpret correctly that the optimal thr from the above figure is 0.95?
what is the -r ?

awesome! do i interpret correctly that the optimal thr from the above figure is 0.95?

You can't read the correct threshold from the figure. Each dot is a threshold (eg 0.1, 0.6) that has been tested: for each tested tresholds, we binarize the predictions and measure the mean TPR & FDR --> new dot on the plot. The dot which minimizes TPR-FDR turns green (ie optimal threshold).
The optimal threshold is indicated in the terminal output:
Running ROC analysis to find optimal threshold
	Optimal threshold: 0.35
	Saving plot: spineGeneric/roc.png

and used to update log_directory/config_file.json with the optimal threshold (binarize_predictions param) which could be further used at testing time.

what is the -r ?

usage: ivadomed -c CONFIG [-g GIF] [-r ROC_INCREMENT] [-h]

MANDATORY ARGUMENTS:
  -c CONFIG, --config CONFIG
                        Path to configuration file.

OPTIONAL ARGUMENTS:
  -g GIF, --gif GIF     Generates a GIF of during training, one frame per
                        epoch for a given slice. The parameter indicates the
                        number of 2D slices used to generate GIFs, one GIF per
                        slice. A GIF shows predictions of a given slice from
                        the validation sub-dataset. They are saved within the
                        log directory.
  -r ROC_INCREMENT, --roc_increment ROC_INCREMENT
                        A ROC analysis is performed at the end of the training
                        using the trained model and the validation sub-dataset
                        to find the optimal binarization threshold. The
                        specified value indicates the increment between 0 and
                        1 used during the ROC analysis (e.g. 0.1). ROC plot is
                        saved under "log_directory/roc.png" and the optimal
                        threshold in "log_directory/config_file.json as
                        "binarize_predictions" parameter.
  -h, --help            Shows function documentation.

awesome! do i interpret correctly that the optimal thr from the above figure is 0.95?

You can't read the correct threshold from the figure. Each dot is a threshold (eg 0.1, 0.6) that has been tested: for each tested tresholds, we binarize the predictions and measure the mean TPR & FDR --> new dot on the plot. The dot which minimizes TPR-FDR turns green (ie optimal threshold).
The optimal threshold is indicated in the terminal output:
Running ROC analysis to find optimal threshold
	Optimal threshold: 0.35
	Saving plot: spineGeneric/roc.png

and used to update log_directory/config_file.json with the optimal threshold (binarize_predictions param) which could be further used at testing time.

ok I see. So this is somewhat different than our other approach for finding a threshold. I guess there is no single way to find "the best threshold" (whatever that means), but we should keep this in mind, in case at some point we want to change the method. In the present case, the "best threshold" is biased towards the ground truth (which has been generated mostly using older models, which were been generated using propseg, which is biased towards contrast). Just something to keep in mind.

ok I see. So this is somewhat different than our other approach for finding a threshold. I guess there is no single way to find "the best threshold" (whatever that means), but we should keep this in mind, in case at some point we want to change the method.

Yes absolutely! I made quite clear (I think) in the documentation how this "golden" threshold is computed so that the used method is clearly identified.
This other method would be super great for SC segmentation with multi contrast. But not appropriated for GM, lesions or tumor tasks. That's why I first implemented this other method (ROC).

In the present case, the "best threshold" is biased towards the ground truth (which has been generated mostly using older models, which were been generated using propseg, which is biased towards contrast). Just something to keep in mind.

:+1:


Infrabox project should enable sync permission from Github repo automatically

It is frustrating to give permissions to new collaborators from a Github repository manually to the related Infrabox project. New collaborators are also frustrated that they don't have the permissions they need when they need them, needing to ask someone to give them to him manually. Infrabox should use the Github hook to recover the list of collaborators in the Github repository and automatically give them Developer permissions in the related Infrabox project.

@yz21606948

TEMPLATE CONFIRMED


It's a little weird that the iOS icon preview is bigger than the macOS icon preview

Yes, it is. But definitely Future.

Since switching to the new multi-icon layout, this no longer seems so strange. Won't fix.


Add a docker file

Expected behaviour
The setup for Linux users should be easy and have one authoritative source. A docker container would be the ideal format.
Actual behaviour
Setup is a manual process. In some cases, there are scripts to automate aspects of it, but none are conformant.
Any other details
We have to run our analysis on a wide array of systems (even in the Linux world). We're currently using Singularity containers to package analysis but it'd be good to have an authoritative source that we could use as a base build.

Closed in 6fafa446eafc9f2bf6698fe97749b6366038d857


occs data

Check where 9's in individual level occs data are coming from

and in general, check min/max values in all data sets for potential issues


Strange error handling of constraints

Describe the bug
The behavior when accidentally passing in an expression in the list of constraints (as opposed to an equality/inequality), is very odd. I'm not sure what is happening.
To Reproduce
I encountered the bug while solving a portfolio optimization problem like below:
import cvxpy as cp
import numpy as np

n = 3
np.random.seed(1)
mu = np.random.randn(n)
Sigma = np.random.randn(n, n)
Sigma = Sigma.T @ Sigma

w = cp.Variable(n)
obj = cp.Maximize(mu.T@w - cp.quad_form(w, Sigma))
constr = [cp.sum(w), w >= 0]
prob = cp.Problem(obj, constr)
prob.solve(verbose=True)

Expected behavior
Some sort of error should be raised in this case. Definitely the problem shouldn't solve!
Output
If applicable, include program output. If reporting a program crash, please include the entire stack trace.
Version

OS: Mac
CVXPY Version: 1.1.15

Additional context
Add any other context about the problem here.

Good catch.
The Expression masquerading as a constraint is propagated forward in Dcp2Cone --- canonicalization doesn't meaningfully distinguish between constraints and expressions in this reduction. https://github.com/cvxpy/cvxpy/blob/master/cvxpy/reductions/canonicalization.py#L59-L70
Next, in ConeMatrixStuffing, the expression masquerading as a constraint is silently dropped.
https://github.com/cvxpy/cvxpy/blob/master/cvxpy/reductions/dcp2cone/cone_matrix_stuffing.py#L317-L331

A simple fix is to ban the construction of Problems with invalid constraint lists. In particular just check that each entry is in fact a Constraint object. I can make a PR.


Move documentation into the repository.

Changes
Moves API documentation into the repository. This will allow us to update documentation alongside code
changes in pull requests & hopefully keep things a bit more up-to-date! :books:
I've also fixed any outdated information I came across on older endpoints and added documentation for the new profile, signup, and reportback endpoints.
What's it look like?
You can poke around on this branch.

For review: @angaither or @weerd
/cc @aaronschachter @jonuy

trés fancy! :+1:


ci: add release workflow

Create a release workflow to build and upload assets on every tag pushed on the repository.
How to test: run the test workflow (to be removed before merge) (dry-run: true), and assess results.
Fixes #13

@santhosh-tekuri I'd need your help to undesertand which bin name I can use for the action. Thx

it seems this action is using cargo build --bin boon to build binary.
but it should be built using cargo build --example boon to build binary.
I use example rather than bin, because I did not want additional dependencies required by boon to be added to dependencies section in cargo.toml. If I use example then I can add additional dependencies required by boon to dev-dependencies section
can you check some other GitHub action which allows to manually do cargo build, and upload artefact to release

Release will be created only once for the total job, not 3 times (because of the matrix)

I guess long names for binary looks odd.
I created new branch bin-release-artifact in which I moved the example into its own cli crate
now you can use  https://github.com/marketplace/actions/build-and-upload-rust-binary-to-github-releases on this crate
use manifest-path in GitHub action to point to cli crate
I think this should make the life easier

It might be easier if you open another PR from your branch then. Thank you. I'll close this one.

I have already pushed the branch.
You can pull the branch into your repo and raise new pull request.


⚠ Bofrost.at has degraded performance

In 116f4d9, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1679 ms

Resolved: Bofrost.at performance has improved in e0accd8 after 6 minutes.


[RFC] Need to export all HybridBlocks in a Gluon model

Description
Gluon can have any hierarchy of Blocks where a top-level Block is not a HybridBlock, and lower-level blocks are HybridBlocks. These HybridBlocks can be dispersed throughout the model architecture. Users can optimize the parts of their model that support it by calling hybridize on the top level block to trigger a recursive call throughout all child blocks. But there is currently no way to export all HybridBlocks without going through and manually calling export on each one. Further, theres no way to reload those exported symbol files back without changing the model design and swapping those HybridBlocks for SymbolBlocks and than one-by-one calling imports to reload.
I would like to ask for suggestions from the community and have an active discussion on the different ways to address this problem. To kick things off, here is a proposal to start the discussion around:
def save_cached_graphs(block):
    def _save_cached_graphs(blk, index):
        if isinstance(blk, mx.gluon.nn.HybridBlock):
            blk.export(blk.name + str(index[0]))
        for child in blk._children.values():
            index[0] += 1
            _save_cached_graphs(child, index)
    #save top-level block
    index = [0]
    _save_cached_graphs(block, index)

def load_cached_graphs(block):
    def _load_cached_graphs(blk, index):
        if isinstance(blk, mx.gluon.nn.HybridBlock):
            sym = symbol.load(blk.name + str(index[0]) + '-symbol.json')
            blk._cached_graph = sym
        for child in blk._children.values():
            index[0] += 1
            _load_cached_graphs(child, index)
    #load top-level block
    index = [0]
    _load_cached_graphs(block, index)

With these two functions, we can recursively export each hybrid block and then reload the symbols. Obviously the code is not complete or even functional (_cached_graph is actual a tuple of symbols and sym.var inputs). But should serve as a point of reference.
Items on v1.x

Since a Block's children are stored in a dictionary, need to save/restore their unique names
Since parameters are mapped to their block's name, need to synchronize names after reloading model architecture to match save params

Items on master

Since parameters have a UUID, need to save/restore mapping of a Block's params to UUID

General Approach

Recursively create a dictionary structure of blocks mimicking the model architecture
Be able to uniquely identify the block in the model
For HybridBlocks, save/restore the cached graph (symbol + inputs) and in/out formats
Save the model architecture dictionary & parameters
Restore the model architecture with unique identifiers to synchronize with parameter naming/IDs

Heres a silly example of a hierarchical Block/HybridBlock to play around with:
class MyBlock(mx.gluon.nn.Block):
    def __init__(self, **kwargs):
        super(MyBlock, self).__init__(**kwargs)
    def add(self, block):
        self._children[block.name + str(len(self._children))] = block
    def forward(self, x, *args):
        out = (x,) + args
        for block in self._children.values():
            out = block(*out)
        return out

# create the Model
inside = MyBlock()
inside.add(mx.gluon.nn.Dense(10))
net = MyBlock()
net.add(inside)
net.add(mx.gluon.nn.Dense(10))
net.initialize()
x = mx.nd.empty((1,10))
out = net(x)

#hybridize and create cached_graphs
net.hybridize()
out = net(x)

#save cached_graphs
save_cached_graphs(net)

Heres a complete, working solution on v1.8.x branch. Notice the new save and load functions perform the full model export/reload of both model architecture and params.
import mxnet as mx
import json

class MyBlock(mx.gluon.nn.Block):
    def __init__(self, **kwargs):
        super(MyBlock, self).__init__(**kwargs)
    def add(self, block):
        self._children[block.name + str(len(self._children))] = block
    def forward(self, x, *args):
        out = (x,) + args
        for block in self._children.values():
            out = block(*out)
        return out
    def save(self, prefix):
        # create empty model structure
        model = {}
        def _save_cached_graphs(blk, index, structure):
            # create new entry for this block
            mdl = {'orig_name': blk.name}
            # encode unique name based on block type and ID
            name = type(blk).__name__.lower()
            structure[name+str(index[0])] = mdl
            if isinstance(blk, mx.gluon.nn.HybridBlock):
                # save in/out formats
                mdl['in_format'] = blk._in_format
                mdl['out_format'] = blk._out_format
                # save cached graph & input symbols
                syms, out = blk._cached_graph
                mdl_syms = []
                for sym in syms:
                    mdl_syms.append(sym.tojson())
                mdl['inputs'] = mdl_syms
                mdl['symbol'] = out.tojson()
            children = dict()
            mdl['children'] = children
            # recursively save children
            for ch_name, child in blk._children.items():
                index[0] += 1
                # save child's original name in this block's map
                children[child.name] = ch_name
                _save_cached_graphs(child, index, mdl)
        # save top-level block
        index = [0]
        _save_cached_graphs(self, index, model)
        # save model
        fp = open(prefix+'-model.json','w')
        json.dump(model, fp)
        fp.close()
        # save params
        self.save_parameters(prefix+'-model.params')

    def load(self, prefix):
        # load model json from file
        fp = open(prefix+'-model.json')
        model = json.load(fp)
        fp.close()
        def _load_cached_graphs(blk, index, log):
            # get block name
            name = type(blk).__name__.lower()
            # lookup previous encoded name based on block type and ID
            mdl = log[name+str(index[0])]
            # rename block to what it was when saved
            blk._name = mdl['orig_name']
            if isinstance(blk, mx.gluon.nn.HybridBlock):
                # restore in/out formats
                blk._in_format = mdl['in_format']
                blk._out_format = mdl['out_format']
                # get saved symbol
                out = mx.sym.load_json(mdl['symbol'])
                syms = []
                # recreate inputs for this symbol
                for inp in mdl['inputs']:
                    syms.append(mx.sym.load_json(inp))
                # reset cached_graph and active status
                blk._cached_graph = (syms, out)
                blk._active = True
            # rename params with updated block name
            pnames = list(blk.params.keys())
            for p in pnames:
                param = blk.params._params[p]
                new_name = blk.name +'_'+ p[len(blk.params._prefix):]
                blk.params._params.pop(p)
                blk.params._params[new_name] = param
            # recursively reload children
            for ch_name, child in blk._children.items():
                index[0] += 1
                _load_cached_graphs(child, index, mdl)
            # current set of child names
            ch_names = list(blk._children.keys())
            # original child names
            children = mdl['children']
            # loop and remap children with original names
            for ch_name in ch_names:
                child = blk._children[ch_name]
                blk._children.pop(ch_name)
                orig_name = children[child.name]
                blk._children[orig_name] = child
        # load top-level block
        index = [0]
        _load_cached_graphs(self, index, model)
        # load params
        self.load_parameters(prefix+'-model.params')

def createNet():
    inside = MyBlock()
    dense = mx.gluon.nn.Dense(10)
    inside.add(dense)
    net = MyBlock()
    net.add(inside)
    net.add(mx.gluon.nn.Dense(10))
    return net

# create and initialize model
net = createNet()
net.initialize()
# run first inference to test
x = mx.nd.empty((1,10))
out = net(x)
# hybridize (the hybridizeable blocks, ie. the Dense layers)
net.hybridize()
out = net(x)

# save hybridized model
net.save('MyModel')

# create a new model, uninitialized
net = createNet()
# reload hybridized model
net.load('MyModel')
# run inference again
out = net(x)

Thanks @samskalicky, this would sound like an interesting feature for Sockeye as well. In Sockeye, we use a mix of Hybrid and non-Hybrid blocks, both during training and inference. We currently do not use the export mechanism of Gluon, but keep our own configuration file to build the model graph at inference time, according to its specifications, we therefore tie code and model artifacts closely together. An export functionality that would allow us to export the inference computation after training sounds useful.
My main question about this though would be how exported HybridBlocks (i.e. SymbolBlocks) behave w.r.t changing input shapes? Before Gluon, we had to make heavy use of the BucketingModule in MXNet to account for varying shapes at inference time.

Thanks @fhieber is your approach for Sockeye's configuration file generalizable to something we could build in to mxnet? Can you point me to an example where you do this?

My main question about this though would be how exported HybridBlocks (i.e. SymbolBlocks) behave w.r.t changing input shapes? Before Gluon, we had to make heavy use of the BucketingModule in MXNet to account for varying shapes at inference time.

Nothing would change for that problem with exporting models. The restrictions around input shapes is specific to the MXNet engine memory allocation scheme where reshaping causes re-allocation. Its much lower level in the stack. If you wanna discuss that lets start a separate GitHub issue.

Created PRs on v1.x (#19565) and master (#19564) branches. Closing this RFC issue and migrating to PRs for further discussion.


[Summer of code] Add design and quick_start document for controller mode

Make sure set the target branch to develop
What is the purpose of the change
tracking issue: https://github.com/apache/rocketmq/issues/4330
Brief changelog
Add design and quick_start document for controller mode
Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

@RongtongJin please help review the documents (Design and quick start), and after the review, I will add the English version of the documents.
Also, I will add a deploy_guide document.

Coverage increased (+0.1%) to 47.486% when pulling 1355e2590b36463940e7ab964f51c0927f55ff1b on hzh0425:feature/add-document into 8515a8d0915d1d2e2e41a0fa3ea32ba6efc1595f on apache:5.0.0-beta-dledger-controller.

Codecov Report

Merging #4413 (f76adde) into 5.0.0-beta-dledger-controller (8515a8d) will decrease coverage by 0.04%.
The diff coverage is 0.00%.

@@                         Coverage Diff                         @@
##             5.0.0-beta-dledger-controller    #4413      +/-   ##
===================================================================
- Coverage                            43.27%   43.23%   -0.05%
+ Complexity                            6340     6331       -9
===================================================================
  Files                                  840      840
  Lines                                59699    59702       +3
  Branches                              8138     8139       +1
===================================================================
- Hits                                 25833    25810      -23
- Misses                               30515    30539      +24
- Partials                              3351     3353       +2

Impacted Files
Coverage Δ

.../rocketmq/broker/hacontroller/ReplicasManager.java
0.00% <0.00%> (ø)

...ketmq/store/ha/autoswitch/AutoSwitchHAService.java
55.31% <0.00%> (ø)

...q/namesrv/routeinfo/BrokerHousekeepingService.java
72.72% <0.00%> (-9.10%)
:arrow_down:

...ava/org/apache/rocketmq/filter/util/BitsArray.java
59.82% <0.00%> (-2.57%)
:arrow_down:

...he/rocketmq/client/trace/AsyncTraceDispatcher.java
77.22% <0.00%> (-1.49%)
:arrow_down:

.../apache/rocketmq/logging/inner/LoggingBuilder.java
63.44% <0.00%> (-1.27%)
:arrow_down:

...cketmq/broker/schedule/ScheduleMessageService.java
56.00% <0.00%> (-1.18%)
:arrow_down:

...che/rocketmq/namesrv/kvconfig/KVConfigManager.java
59.18% <0.00%> (-1.03%)
:arrow_down:

.../org/apache/rocketmq/store/ha/DefaultHAClient.java
58.46% <0.00%> (-0.52%)
:arrow_down:

...che/rocketmq/acl/plain/PlainPermissionManager.java
75.19% <0.00%> (-0.52%)
:arrow_down:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8515a8d...f76adde. Read the comment docs.


CA1859: Use concrete types when possible for improved performance

Type of issue
Typo
Description
Hello
I get this a lot in my unit tests when I am testing a service that explicitly implements an interface.
In this case, I must return the interface or the method I am testing cannot be called
Can you improve the checks to stop this case?
I know I can supress it
Many thanks
Page URL
https://learn.microsoft.com/en-gb/dotnet/fundamentals/code-analysis/quality-rules/ca1859
Content source URL
https://github.com/dotnet/docs/blob/main/docs/fundamentals/code-analysis/quality-rules/ca1859.md
Document Version Independent Id
c8f7f102-2398-9acc-71b8-b0f848a05b92
Article author
@gewarren
Metadata

ID: 0e2d460f-c6f3-1e26-d932-b7e69746e513
Service: dotnet-fundamentals

@jaredpar I think this needs to be transferred to the roslyn-analyzers repo, but I don't have permissions. Can you transfer it?


Pilot abilities are not in ascending order

After Han Solo, the following pilot abilities start at 3XP and climb up again.
Using the version hosted on netlify.
http://hotac-ship-builder.netlify.com/

Ah yes, I have noticed this also. It seems to be browser dependent as well.