Is there a way to forward declare nlohmann::json?

It's preferable not to have to include large header files in other header files, but in my current project I've ended up with json.hpp included in a lot of headers because I can't find a way to forward declare the nlohmann::json type. It's made complicated by all the templating going on. I was wondering if there is actually a way that I just haven't found yet and, if there is, if it could maybe be included as part of the documentation/readme somewhere?

Looks like this isn't possible for a user to do due to the default template parameters.  The library would need to be updated to provide a forward declaration header with the default template parameters, and remove the defaults from the definition.

You could try adding a #pragma once to the json.hpp right above #ifndef NLOHMANN_JSON_HPP. That pragma is supposed to ensure the file isn't even opened on subsequent inclusions. My experience is that compilers are pretty smart about the #ifndef.

Unless you're building your whole project each time, it shouldn't take much time. You could also use ccache to help speed up your build process.

I think it would be quite cumbersome to realize a forward declaration. All I can think of right now would be a new header with a forward declaration defining a preprocessor symbol and adjusting the current json.hpp to use that preprocessor symbol to decide whether the default template parameters should be defined.

I added a file src/json_fwd.hpp in my forked repo. This header file is minimal and provides a forward declaration of nlohmann::basic_json and adds a typedef nlohmann::json. I've tested it with MinGW and it works. I've also made a pull request to the main repo, so I hope the new header file will be visible to the public there soon.

@ralphtandetzky Thanks! Just out of curiosity: Does the forward declaration header improve your compilation times? Or maybe @rachelnertia could also have a look at this?

Would it be possible to do something along these lines?
    struct json : basic_json<...> { using basic_json::basic_json; };
I know this is not what std::string does, but it would solve the forward declare problem, and it would solve another annoyance I have. Whenever I try to use a debugger, I see a lot of ugly C++ symbols around json. It would be nice that you only have to "pay for" the symbol noise if you actually do something custom.
For example:
void std::__1::vector<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator>, std::__1::allocator<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator> > >::__push_back_slow_path<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator> >(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator>&&)
would become
void std::__1::vector<nlohmann::json, std::__1::allocator<nlohmann::json> >::__push_back_slow_path<nlohmann::json>(nlohmann::json&&)

That wouldn't solve the forward declare problem, it would make it worse.  You need to have the full definition of the base class to derive from it.
You can make your own struct that derives that way if you find that it helps with debugging, but I don't think changing the type of nlohmann::json at this point is a good idea.

Yes, it would solve the forward declare problem because you have a distinct type:
namespace nlohmann {  struct json; }
You can't do that now because you're not allowed to forward-declare a type alias (ie, typedef).
I agree there could be complications with changing the type, but it might be worth it. (It doesnt actually change convertibility or is-a relationships because 'json' still is-a 'basic_json' thing. But it is a ABI change by definition, which can have consequences).

Ah, I misunderstood what you were suggesting.

@nlohmann It did improve my compilation times on my project by a small margin. For my project where I use quite a bit of nlohmann::json for serialization purposes the compilation time improved from 2:44 to 2:40 in a one time test. I would guess that more than 50% of the cpp-files saw a declaration of nlohmann::json and about 15% included json.hpp directly or indirectly.
I was very careful in my project to reduce compile times to a minimum. I'm not using precompiled headers at the moment. I mostly include the C++ standard library, Qt and only little boost. I was very surprised that using json_fwd.hpp did not help more than that. But I would like to know, if others have more success with it. I have to remark that I observed some strange (counter-intuitive) effects on my compile-times lately. So my test might not be that relevant.

@nlohmann I don't have time at the moment to fully profile what difference it makes, but for the project which raised this question using json_fwd.hpp seems to have a negligible impact: in the single test I ran, I got a time of 29.46 seconds, down from 31.03, for a full build. So it doesn't appear to be a game changer.

I'm not terribly surprised.  The size of this library is tiny compared to the rest of the c++ standard library that you're probably already including.

I think the reported numbers do not justify merging PR #314.

I found this thread on google. In my solution where I use lots of .o objects with json. Compiling lots of independent files with forward declaration is noticeably faster than with full header. For now, fwd proposed by @ralphtandetzky works well. +1 for merging this

@peku33 Thanks for checking by! Do you have concrete numbers?

@nlohmann
test_fwd.cpp:
#include <json_fwd.hpp>

int main()
{
        return 0;
}

test_hpp.cpp:
#include <json.hpp>

int main()
{
        return 0;
}

time g++ -std=c++11 -O2 test_hpp.cpp
real    0m2.762s
user    0m2.585s
sys     0m0.175s

time g++ -std=c++11 -O2 test_fwd.cpp
real    0m0.649s
user    0m0.592s
sys     0m0.047s

For lightweight objects 2.762s vs 0.649s is the difference :)
For 100 it will be 4.5min vs 1min.

Any update?

@peku33 That's only the difference of that file included in isolation.  The json.hpp file includes a lot of the STL.  If you normally include a lot of the STL in your source files, then the difference should be a lot smaller.
#include <algorithm>
#include <array>
#include <cassert>
#include <cctype>
#include <ciso646>
#include <cmath>
#include <cstddef>
#include <cstdint>
#include <cstdlib>
#include <cstring>
#include <functional>
#include <initializer_list>
#include <iomanip>
#include <iostream>
#include <iterator>
#include <limits>
#include <locale>
#include <map>
#include <memory>
#include <numeric>
#include <sstream>
#include <stdexcept>
#include <string>
#include <type_traits>
#include <utility>
#include <vector>

It might be an interesting test to review if all of these includes are really necessary, especially all the c* headers.

I recently used iwyu to check the headers and did not find headers that could be skipped. But maybe I overlooked something. I may add comments for at least one function of each header to document the actual need.

@gregmarr the problem is that I actually don't use heavy stl containers (string, vector, map, streams). That's why I'm posting here. As I mentioned before, the difference in compilation time is visible for me.
Because adding the fwd file wont change anything for users currently using the full hpp file - where is the problem with merging it?

@nlohmann sounds good.  I hadn't heard of that site.  I checked a couple of those manually, and they were all necessary.  std::isdigit is used in one place, so we need cctype, for example.
@peku33 Just want to make sure that there would be some significant benefit for the extra complexity.

@peku33 I'm still hesitant to add the header to the library. I shall have a look into it for the 3.0.0 release.

FYI: I could actually remove <cstring>, see e385417047fcb4a5e3724ecc3291718014917fd4.

Update: <cstring> was actually used for strlen... :-S Sorry.

+1 for @gregmarr 's idea:
struct json : basic_json<>
{
    using basic_json<>::basic_json;
};

That was @jaredgrubb

I did try it locally, but it ended up not working well.

For me it works well, all I needed to do is add a constructor from basic_json<> to resolve an ambiguity:
namespace marton78 {
    struct json : nlohmann::basic_json<>
    {
        using basic_json<>::basic_json;
        json(basic_json<> const& j) : basic_json<>(j) {}
    };
}

This would be a breaking change

@gregmarr Why would this be a breaking change? I assume this introduces a compatibility issue, but what exactly?

I didn't observe any drawbacks of using unmodified version of json (json.hpp) together with https://github.com/ralphtandetzky/json/blob/ralphtandetzky-json_fwd/src/json_fwd.hpp

As an experiment, I disabled ccache, and built a 146 compilation-unit project I maintain with:

#include<json.hpp> in (almost) every compilation unit - 3m17s
#include<json_fwd.hpp> in (almost) every compilation unit - 2m4s
include json_fwd.hpp/json.hpp only where needed - 1m40s

This was using the development branch of the nlohmann/json repo, where I extracted the forward declarations myself, along the lines of @ralphtandetzky's fork.
Even if the performance gain of using forward declarations was not quite as pronounced, the forward declarations should arguably be split out anyway: modular design promotes (re-) use, and should be encouraged as a software engineering best practice.  Everything Lakos wrote about organizing code in large projects in https://www.amazon.com/Large-Scale-Software-Design-John-Lakos/dp/0201633620 still holds to this day.

@JoroRoss what other headers did you include in your files?
I'm not 100% sure, but streams are pretty heavy to compile. If you use them anyway, there is probably going to be no significant gain.
In projects with lots of small compilation units, which does not include streams, this matters.

@peku33 I'm not @JoroRoss, but I'm using for over a year this forward declaration file:
#ifndef NLOHMANN_JSON_FWD_HPP
#define NLOHMANN_JSON_FWD_HPP

#include <cstdint>
#include <map>
#include <vector>
#include <string>

namespace nlohmann
{

template <
    template<typename U, typename V, typename... Args> class ObjectType,
    template<typename U, typename... Args> class ArrayType,
    class StringType,
    class BooleanType,
    class NumberIntegerType,
    class NumberUnsignedType,
    class NumberFloatType,
    template<typename U> class AllocatorType
    >
class basic_json;

using json = basic_json<std::map,std::vector,std::string,bool,std::int64_t,
                        std::uint64_t,double,std::allocator>;

} // namespace nlohmann

#endif // NLOHMANN_JSON_FWD_HPP

I guess this is a very minimal forward declaration, with very few nested includes.

Hey @peku33, thanks for getting back to me on this.
The code-base in question is reasonably well organized, limiting the use of large/complicated headers to compilation units that actually need them.  I suppose  and  end up in a lot of places, but we try to keep them in implementation files where possible.  We (of course) use  where appropriate - most compilation units will pick that up some way or other, as we declare iostream operators with most of our data model classes.  Only the data model compilation units (and a hand-full of collaborators) need the actual definitions.
For our own templates, we typically create two headers - one with declarations for clients to pick up, another with definitions.  We often introduce dedicated compilation units with template instantiations for the production classes, and include the definition files in a unit tests compilation unit for mocking/stubbing.  (We used to not do this; when we put in the effort, we got build times down from 25 minutes to around 3.)
So yeah, sort of a typical codebase, that encapsulates the use of expensive resources.
Regarding your speculations about the compilation time of , I'm afraid to say you are wrong. That is, unless every single compilation unit in your project needs every last template definition from , ,  and ... (if you're in that situation, you may want to consider restructuring your project!).
The overhead in build times typically comes from (redundant) parsing of headers, not from compilation (unused template declarations don't get compiled, just parsed).  The nlohmann/json project luckily uses , and including those forward declarations in all compilation units has an almost negligible impact on compilation times.  Hardly any compilation units actually need to include the actual definitions, and even if they do, they can often limit the impact by specifically including ,  or .
For what it's worth: I just repeated my experiment from before, this time including  in (almost) every compilation unit.  The build time went to 2m20s.  My test is producing a CMake Debug build, so the build time is dominated by IO and parsing.  The pre-compiler emits about twice as much code for the json header as it does for the iostream one (as can be readily verified by running g++ with the -E flag.)

@JoroRoss Have you seen this PR?  https://github.com/nlohmann/json/pull/700/files


Add serialization support

This adds the serialization support.
Refactored the SCNAssimpScene so that it generates a model SCNScene and an array of animation SCNScene's. By transforming to the SCNScene instance, serialization does not need any custom changes.
The tests are now updated and the test criteria is both the validity of the generated SCNScene as well as it's full serialization. mdl support dropped as a result.
The API has changed and is simpler, with just addition of an SCNScene animation to a given SCNScene model.
The example projects and documentation updated as per the API changes.

Current coverage is 76.33% (diff: 78.43%)

Merging #43 into master will decrease coverage by 5.15%

@@             master        #43   diff @@
==========================================
  Files             9         11     +2
  Lines          1232       1293    +61
  Methods          81        101    +20
  Messages          0          0
  Branches        103        110     +7
==========================================
- Hits           1004        987    -17
- Misses          171        249    +78
  Partials         57         57

Powered by Codecov. Last update 66505d4...9add299


Add command line flag for doing a full "regular" sync

What is wrong?
It should be possible to perform a "regular" sync. Meaning sequential download of blocks and sequential application of those blocks to build the chain state.
How can it be fixed
First, we need to fix an internal naming issue where we refer to our fast sync using SYNC_FULL.  This should be renamed to SYNC_FAST, and use the fast terminology in place of full everywhere that is currently used.
Then, add a new option for SYNC_FULL which just does a regular block-by-block sync.  The cli should support the following:

--sync-mode=light/full/fast with fast being the default.
--light as a shortcut for --sync-mode=light (already supported)
--fast as a shortcut for --sync-mode=fast
--full as a shortcut for --sync-mode=full

The trinity.sync module will need to be re-organized as well, moving trinity.sync.full to trinity.sync.fast and adding the new syncer under trinity.sync.full.

Note that the "first" and "then" steps above should probably be separate PRs to make sure all tests pass after the first step.
Hm, geth used to have those shortcuts (eg~ --fast) and seems to have dropped them since. We should consider not adding them in the first place.

Am I right to assume that we still need to ensure we have the entire state for the current head before we start applying blocks with their transactions (how should it work without that :thinking:?)

how should it work without that

Yes, regular sync should likely do a validate step upon startup (and honestly maybe before applying any block to check if the parent block's state root is present in the database so that we can throw a user friendly error in the event that it is not.

Cool, I feel like I'm starting to how this called Ethiriuum works.

It surely already raises an error if the state is missing (he speculated with out finding it in the code), but catching it and raising a friendlier message could be reasonable.

Would like to take a stab on this

but catching it and raising a friendlier message could be reasonable.

I guess we should even go one step further and hint the user that they could recreate the state using something like --rebuild-state.
When started with this command, trinity would take the genesis state and then apply all the blocks that it already has until it has build the state up to where its HEAD is. Then, from there, the user could run trinity --full and get into regular syncing. Does that make sense?

According to my simple math, it would take roughly 40 hours to rebuild the entire state for current mainnet (rather than fetch it from other peers)
6751752 / 46 bps* / 60 / 60 = 40.77

according to our benchmark

But I'm not suggesting this as an alternative to state sync, just as another tool to rebuild the state up to block x (e.g. for testing etc.)

I guess we should even go one step further and hint the user that they could recreate the state using something like --rebuild-state.

Sounds interesting. If we have a tool to rebuild the state for them, might as well just run it for them directly instead of telling them to run it.
But I'd say this is all something that could be captured in a separate issue so that this issue can stay a reasonable size.

Hm, geth used to have those shortcuts (eg~ --fast) and seems to have dropped them since. We should consider not adding them in the first place.

Yes, the shorthands were deprecated in geth v1.6.0.
Their --syncmode was added in commit https://github.com/ethereum/go-ethereum/pull/13875/commits/48339d4ef6dcbcf80f4c74ae64bc7426aa9053c3, part of PR https://github.com/ethereum/go-ethereum/pull/13875.
There is not much reasoning on why; but since this commit is in that PR, I'd guess that it's because of the config file looking much more self-explanatory with
SyncMode = "fast"

instead of
Fast = true

This helps to avoid accidental overrides, the likes of
SyncMode = "fast"
...
Light = true

which happens a lot with users cargo-culting when they "just want to blockchain real quick". (OT: Believe me, that happens more than you'd want to know.)
This also applies to CLI args, not just config files.

TL;DR: I'd recommend not giving several ways to configure the same parameter. Having just one way reduces confusion.

I'm still in favor of the pattern we have in place for CLI options.

Assuming we have a flag that takes a concise number of well named string arguments (--sync-mode fast/light/full).
It's very convenient to have quick shortcuts --fast/--light/--full

This doesn't need to apply to everything, but I think it's good UX (please tell me I'm wrong if you think so).

Our CLI parser already handles the mutual exclusivity of these commands so that shouldn't be a problem.
When we get around to doing config file stuff I don't plan on including those shortcuts so that shouldn't be a problem.

The main problem I see is when something like --fast can be easily conflated with meaning something else, but that is something we should be able to deal with on a case-by-case basis.

The main problem I see is when something like --fast can be easily conflated with meaning something else, but that is something we should be able to deal with on a case-by-case basis.

Another alternative would be to name the shorthands --sync-fast, --sync-light and --sync-full. Sits somewhere in between.
But this is just something that occurred to me. My brain has a very low resolution which basically eliminates all the subtle nuances of these UX issues. So, any solution is fine for :sweat_smile:

I'd recommend not giving several ways to configure the same parameter. Having just one way reduces the possibility confusion (and subsequent "bug" reports).

I tend to agree with this, but not so much that I would give a :-1: to a PR. So I guess I leave it up to the implementer.

It's very convenient to have quick shortcuts

Other than as a trinity developer (when I'm repeatedly testing a wide range of options), I have always ended up with a long line of config parameters, written in a single-line shell file. For that scenario it seems nicer to have exactly one way to do things, and have them be explicit. So I'm not convinced that the quickness is a big enough benefit for end users, compared to the potential maintenance costs.
Also, it's much easier to add --fast later than it is to remove it later.
Anyhow, I don't think I would give a :-1: to a PR for including the shortcut. It's only a moderate opinion, and clearly smart people differ on it. So I guess it's up to the implementer, unless some broader consensus forms.

I'm now convinced that the benefit isn't justified by the various downsides.  Lets remove the aliases.


Fonts and the Repo

Following the site style, i got the free Lato font, http://www.latofonts.com/lato-free-fonts/ for use in the app. But, i am not really sure how to put it in the project.
Should we just put a link in the readme for people who would like to work on the app to get? Put it in the repo along with the license file(open font license)?
I am not really familiar with this situation.
cc @leafo

maybe we can have a script that downloads any external dependencies?

Will look into it tomorrow. But a preliminary search says it's quite simple.

Here is how it would go on Windows: https://gist.github.com/Zetsaika/3b696e5530a64bcf4aaf

looks fine to me

Ok. Will commit it, if someone could do that for Linux/Mac it would be appreciated.


Add myself as maintainer

Checklist

[ ] Used a fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

noarch: python recipes are required to have a lower bound on the python version. Typically this means putting python >=3.6 in both host and run but you should check upstream for the package's Python compatibility.

@maresb Thanks for being a maintainer. Could you please follow the instructions of the linter and change the meta.yaml accordingly?

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@kmuehlbauer done. The linter looks happy now.

@conda-forge-admin, please rerender

@maresb Thanks,
let's try to rerender. If this doesn't work, we need to do this manually...

Well the other PR looks good. Couldn't we just merge that first?

@maresb Usually it's good practise to rerender, especially if the PR is that old. I'll try to fix this later today.

I see. Unfortunately I don't have write permissions on the regro-cf-autotick-bot fork, presumably because I'm not a maintainer yet. Circular problem...
I'm about to go out, will probably be back later this evening. Thanks!

By the way, what's the complication? Is there anything beyond just adding >=3.4 to the autotick PR and rerendering?

I'll try exactly that later, but there might be more involved. I've asked in the gitter channel for guidance, so hopefully this get sorted out soon.

@conda-forge-admin, please rerender

Weird, it looks like it's using Python 2.7.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

There are 1 too many lines.  There should be one empty line at the end of the file.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Yes, the py27 stuff would normally be removed by rerender, but somehow it doesn't work for the bot. I'll try to rerender manually, then.


Not able to access delete_series api from thanos query component

We are trying to clean metrics from s3 object using delete_Series thanos query
404 page not found error is coming. can you please tell us how to check

There is no such API at the moment.
See: https://github.com/thanos-io/thanos/issues/415
Closing as duplicate. (:


fix(createEdit): Prevent routing issues caused by original fix for #266

@wski Please test locally. I believe I have solved the issue found by @mparizernc and the issue you found where getDOMNode() is undefined.

I'm worried about how this will work in prod. with the reported 20s+ response times


Typescript: Add setting property EnumStyle with options (Enum, StringLiteral).

This PR adds support for union string literal types instead of enums. It resolves issue https://github.com/RicoSuter/NSwag/issues/1188

PR looks good to me. Thanks. One minor comment... Is the PR ready after that?

Is the PR ready after that?

I think so

https://github.com/RicoSuter/NJsonSchema/pull/1140


feat(typescript): Align isolatedDeclaration implementation with tsc

Description:

[x] Basic implementation. This also includes diagnostics and type inference.

[x] Function
[x] Class
[x] Variable declaration
[x] Enum

[x] Remove unused imports and declarations. Based on the result ast of fast dts transformation, we collect the usage of ids which also contain syntax context. Then we prune the ast to cut off the unused imports and declarations.
[x] Port tests and bug fixes. To ensure correctness, I'd like to port the fixtures as oxc does https://github.com/oxc-project/oxc/tree/main/crates/oxc_isolated_declarations/tests/fixtures.
[x] Benmarks
[x] Strip internal

Implementation:
Most code is direct translation of https://github.com/oxc-project/oxc/tree/main/crates/oxc_isolated_declarations. The differences come from:

swc and oxc use different ast.
The transformation of swc is mutation-based, which directly mutates the whole cloned ast, while the transformation of oxc is immutation-based, which constructs and copies the ast nodes on demand. Maybe mutation-based transformation is not better, but I think it's also annoyed to construct ast nodes and it could be easy to refactor if there is some demands in the future.
oxc transforms while collecting references information to prune unused idents. However, the ast nodes of swc contain syntax context, we don't need to collect the information again. So I build a reference graph and calculate the reachability of identifiers.
I think oxc could have bugs. I manually compare the result with tsc --declaration --isolatedDeclarations --emitDeclarationOnly --isolatedModules --noResolve --noCheck --strictNullChecks test.ts. I also reguard those cases where tsc can't compile as undefined behaviors.

😅 Actually I find my implementation of point 2 is not too good either. I meet many bad cases after my basic implementation and write many ugly patches. So issues are welcome.
Benchmarks:
Roughly test with https://raw.githubusercontent.com/oxc-project/benchmark-files/main/vue-id.ts under M3Pro.

parse without syntax context resolution
isolated declarations

swc
282.68ms
335.44ms

oxc
53.286ms
63.917ms

tsc
-
3.875s

Known problem:

Due to the comment emitting algorithm in swc, isolated declaration may also generate comments with wrong positions.

Related issue (if exists):
closes https://github.com/swc-project/swc/issues/9705
closes https://github.com/swc-project/swc/issues/9718

thanks for your help❤
😂 got an another blocking point of isolatedDeclaration
should generate the same error as tsc
src/App.tsx(8,10): error TS9007: Function must have an explicit return type annotation with --isolatedDeclarations.

I'll open another issue

FYI oxc uses immutable AST for a single transform pipeline.

FYI oxc uses immutable AST for a single transform pipeline.

I see. And I believe immutable AST gets better performance and other engineering benefits in isolated declaration transformation. After finishing, I have a little bit of regret... But maybe in the next pr 😁.

Sorry for the delay. I'm still reviewing this.


pruned model size no change and inference time is even longer

nni Environment:pytorch

nni version:1.4.1
nni mode(local|pai|remote):local
OS:ubuntu 16.04
python version:3.6
is conda or virtualenv used?: conda
is running in docker?:no
I run the example code: model_prune_torch.py
and the pretrain_naive model is 1.7M, the pruned_model is also 1.7M,the same with the mask.
The inference time using pretrain model is 0.4ms,but for the pruned model, time increase to 1.5ms.
I am so confused that what the function of the example? isn't is downscale the model and speedup?
and I also try the speedup method follow the example for my model base on YOLOv3 , still the same .
Please help me  what is going wrong ?
Thx!

@misslibra thanks for reporting this issue. It is expected that the pruned model is also 1.7M, because the pruners are responsible for finding weight masks that make the model still performs reasonably good. ModelSpeedup is responsible for making the model smaller based on the generated masks.
For your case, could you tell us how you measured the number 1.5ms? with pruner applied? or loading the saved model weight checkpoint to the original model (i.e., before pruning)? if the former, inference latency should be higher because weights should be multiplied by the masks in forward. if the latter, the inference latency should not be different.
For ModelSpeedup, it would be great if you can share the code with us, so that we can check whether your model is really compressed.

Thanks for your support!
I add measure time code in test,
 s_time = time.time() output = model(data) print('inference time is : ', (time.time() - s_time)*1000 )
before model pruning, time is 0.7ms, and after prune it is  1.5ms
now I understand that mask multiply take time.

`device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# config_list = [{'sparsity': 1, 'op_types': ['Conv2d']}]
model = Darknet(opt.model_def, img_size=opt.img_size).to(device)
model.load_state_dict(torch.load(opt.weights_path))
if compression == 'prune':
    print('do prune')
    config_list = [{ 'sparsity': 0.2, 'op_types': ['default'] }]##Conv2d
    pruner = L1FilterPruner(model, config_list)
    # pruner = ActivationMeanRankFilterPruner(model, config_list)
    pruner.compress()
    pruner.export_model('model.pth', 'mask.pth')

"""model inference time"""
if do_speedup_detection:
    # Get dataloader
    dataloader = DataLoader(
        ImageFolder(opt.image_folder, img_size=opt.img_size),
        batch_size=opt.batch_size,
        shuffle=False,
        num_workers=opt.n_cpu,
    )

    Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
    model.eval()
    masks_file = './mask.pth'
    apply_compression_results(model, masks_file)
    for batch_i, (img_paths, input_imgs) in enumerate(dataloader):
        input_imgs = Variable(input_imgs.type(Tensor))
        input_imgs = input_imgs.to(device)
        with torch.no_grad():
            start = time.time()
            detections = model(input_imgs)
            durable = time.time()
            print('inference time : ', 1000*(durable - start))
        # break`

`if name == "main":
parser = argparse.ArgumentParser()
parser.add_argument("--image_folder", type=str, default="data/demo_data/image_ori/", help="path to dataset")
parser.add_argument("--model_def", type=str, default="config/geely_yolo3d.cfg", help="path to model definition file")
# parser.add_argument("--weights_path", type=str, default="weights/yolov3.weights", help="path to weights file")
parser.add_argument("--class_path", type=str, default="data/geely.names", help="path to class label file")
parser.add_argument("--conf_thres", type=float, default=0.5, help="object confidence threshold")
parser.add_argument("--nms_thres", type=float, default=0.5, help="iou thresshold for non-maximum suppression")
parser.add_argument("--batch_size", type=int, default=1, help="size of the batches")
parser.add_argument("--n_cpu", type=int, default=0, help="number of cpu threads to use during batch generation")
parser.add_argument("--img_size", type=int, default=(192, 640), help="size of each image dimension")
parser.add_argument("--weights_path", type=str, default='./weights/best_model_Epoch_1060_step_619624_mAP_0.7210_lr_0.0001', help="path to checkpoint model")
opt = parser.parse_args()
# print(opt)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# config_list = [{'sparsity': 1, 'op_types': ['Conv2d']}]
model = Darknet(opt.model_def, img_size=opt.img_size).to(device)
model.load_state_dict(torch.load(opt.weights_path))
if compression == 'prune':
    print('do prune')
    config_list = [{ 'sparsity': 0.2, 'op_types': ['default'] }]##Conv2d
    pruner = L1FilterPruner(model, config_list)
    # pruner = ActivationMeanRankFilterPruner(model, config_list)
    pruner.compress()
    pruner.export_model('model.pth', 'mask.pth')

"""model inference time"""
if do_speedup_detection:
    # Get dataloader
    dataloader = DataLoader(
        ImageFolder(opt.image_folder, img_size=opt.img_size),
        batch_size=opt.batch_size,
        shuffle=False,
        num_workers=opt.n_cpu,
    )

    Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
    model.eval()
    masks_file = './mask.pth'
    apply_compression_results(model, masks_file)
    for batch_i, (img_paths, input_imgs) in enumerate(dataloader):
        input_imgs = Variable(input_imgs.type(Tensor))
        input_imgs = input_imgs.to(device)
        with torch.no_grad():
            start = time.time()
            detections = model(input_imgs)
            durable = time.time()
            print('inference time : ', 1000*(durable - start))
        # break`

`
if name == "main":
parser = argparse.ArgumentParser()

parser.add_argument("--image_folder", type=str, default="data/demo_data/image_ori/", help="path to dataset")

parser.add_argument("--model_def", type=str, default="config/geely_yolo3d.cfg", help="path to model definition file")

# parser.add_argument("--weights_path", type=str, default="weights/yolov3.weights", help="path to weights file")

parser.add_argument("--class_path", type=str, default="data/geely.names", help="path to class label file")

parser.add_argument("--conf_thres", type=float, default=0.5, help="object confidence threshold")

parser.add_argument("--nms_thres", type=float, default=0.5, help="iou thresshold for non-maximum suppression")

parser.add_argument("--batch_size", type=int, default=1, help="size of the batches")

parser.add_argument("--n_cpu", type=int, default=0, help="number of cpu threads to use during batch generation")

parser.add_argument("--img_size", type=int, default=(192, 640), help="size of each image dimension")

parser.add_argument("--weights_path", type=str, default='./weights/best_model_Epoch_1060_step_619624_mAP_0.7210_lr_0.0001', help="path to checkpoint model")

opt = parser.parse_args()

# print(opt)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# config_list = [{'sparsity': 1, 'op_types': ['Conv2d']}]

model = Darknet(opt.model_def, img_size=opt.img_size).to(device)

model.load_state_dict(torch.load(opt.weights_path))

if compression == 'prune':
    print('do prune')
    config_list = [{ 'sparsity': 0.2, 'op_types': ['default'] }]##Conv2d
    pruner = L1FilterPruner(model, config_list)
    # pruner = ActivationMeanRankFilterPruner(model, config_list)
    pruner.compress()
    pruner.export_model('model.pth', 'mask.pth')

"""model inference time"""
if do_speedup_detection:
    # Get dataloader
    dataloader = DataLoader(
        ImageFolder(opt.image_folder, img_size=opt.img_size),
        batch_size=opt.batch_size,
        shuffle=False,
        num_workers=opt.n_cpu,
    )

    Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
    model.eval()
    masks_file = './mask.pth'
    apply_compression_results(model, masks_file)
    for batch_i, (img_paths, input_imgs) in enumerate(dataloader):
        input_imgs = Variable(input_imgs.type(Tensor))
        input_imgs = input_imgs.to(device)
        with torch.no_grad():
            start = time.time()
            detections = model(input_imgs)
            durable = time.time()
            print('inference time : ', 1000*(durable - start))
        # break`

this  is my code to use speedup

I try to load new model exported by pruner.export_model, and inference time is still not cut down.
and how to use  ModelSpeedup to get a smaller model ?
`
"""model inference time"""
if do_speedup_detection:
print('------')
model_1 = Darknet(opt.model_def, img_size=opt.img_size).to(device)
model_1.load_state_dict(torch.load('model.pth'))
# Get dataloader
dataloader = DataLoader(
ImageFolder(opt.image_folder, img_size=opt.img_size),
batch_size=opt.batch_size,
shuffle=False,
num_workers=opt.n_cpu,
)
    Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
    model_1.eval()
    # masks_file = './mask.pth'
    # if use_mask:
    #     apply_compression_results(model, masks_file)
    # else:
    #     m_speedup = ModelSpeedup(model, input_imgs, masks_file)
    #     m_speedup.speedup_model()

    for batch_i, (img_paths, input_imgs) in enumerate(dataloader):
        input_imgs = Variable(input_imgs.type(Tensor))
        input_imgs = input_imgs.to(device)
        with torch.no_grad():
            start = time.time()
            detections = model_1(input_imgs)
            durable = time.time()
            print('inference time : ', 1000*(durable - start))
        # break`

and how to use  ModelSpeedup to get a smaller model ?
when I use ModelSpeedup, I get error:

/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:291: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if grid_size[0] != self.grid_size[0]:
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:257: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert self.stride == self.img_dim[1] / self.grid_size[1]
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:262: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
self.scaled_anchors = FloatTensor([(a_w / self.stride, a_h / self.stride) for a_w, a_h in self.anchors])
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:299: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
pred_boxes = FloatTensor(prediction[..., :4].shape)
Traceback (most recent call last):
File "/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/model_compression.py", line 111, in
m_speedup = ModelSpeedup(model_1, input_imgs, masks_file)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/nni/compression/speedup/torch/compressor.py", line 91, in init
self.trace_graph = torch.jit.trace(model, dummy_input)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 858, in trace
check_tolerance, _force_outplace, _module_class)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 1007, in trace_module
check_tolerance, _force_outplace, True, _module_class)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
return func(*args, **kwargs)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 676, in _check_trace
raise TracingCheckError(*diag_info)
torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
Graph diff:
graph(%self : ClassType,
%x.1 : Tensor):
%2 : ClassType = prim::GetAttrname="module_list"
%3 : ClassType = prim::GetAttrname="0"
%4 : ClassType = prim::GetAttrname="conv_0"

`
if name == "main":
parser = argparse.ArgumentParser()

parser.add_argument("--image_folder", type=str, default="data/demo_data/image_ori/", help="path to dataset")

parser.add_argument("--model_def", type=str, default="config/geely_yolo3d.cfg", help="path to model definition file")

# parser.add_argument("--weights_path", type=str, default="weights/yolov3.weights", help="path to weights file")

parser.add_argument("--class_path", type=str, default="data/geely.names", help="path to class label file")

parser.add_argument("--conf_thres", type=float, default=0.5, help="object confidence threshold")

parser.add_argument("--nms_thres", type=float, default=0.5, help="iou thresshold for non-maximum suppression")

parser.add_argument("--batch_size", type=int, default=1, help="size of the batches")

parser.add_argument("--n_cpu", type=int, default=0, help="number of cpu threads to use during batch generation")

parser.add_argument("--img_size", type=int, default=(192, 640), help="size of each image dimension")

parser.add_argument("--weights_path", type=str, default='./weights/best_model_Epoch_1060_step_619624_mAP_0.7210_lr_0.0001', help="path to checkpoint model")

opt = parser.parse_args()

# print(opt)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# config_list = [{'sparsity': 1, 'op_types': ['Conv2d']}]

model = Darknet(opt.model_def, img_size=opt.img_size).to(device)

model.load_state_dict(torch.load(opt.weights_path))

if compression == 'prune':
    print('do prune')
    config_list = [{ 'sparsity': 0.2, 'op_types': ['default'] }]##Conv2d
    pruner = L1FilterPruner(model, config_list)
    # pruner = ActivationMeanRankFilterPruner(model, config_list)
    pruner.compress()
    pruner.export_model('model.pth', 'mask.pth')

"""model inference time"""
if do_speedup_detection:
    # Get dataloader
    dataloader = DataLoader(
        ImageFolder(opt.image_folder, img_size=opt.img_size),
        batch_size=opt.batch_size,
        shuffle=False,
        num_workers=opt.n_cpu,
    )

    Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
    model.eval()
    masks_file = './mask.pth'
    apply_compression_results(model, masks_file)
    for batch_i, (img_paths, input_imgs) in enumerate(dataloader):
        input_imgs = Variable(input_imgs.type(Tensor))
        input_imgs = input_imgs.to(device)
        with torch.no_grad():
            start = time.time()
            detections = model(input_imgs)
            durable = time.time()
            print('inference time : ', 1000*(durable - start))
        # break`

@misslibra there are two issues in your code. First, after calling pruner.compress() you should fine tune your model. pruner.compress() generates masks based on for example model weights, but it does not fine tune model for you, you still need to write fine tune logic after calling pruner.compress(). Second, apply_compression_results is expected to make inference slower if you use v1.4.1, please try master branch which would not make inference slower.

@QuanluZhang Thank you so much ! I will try now and update the result

apply_compression_results simply multiplies generated masks to weights, it does not speedup model inference. ModelSpeedup does, but ModelSpeedup is still in Alpha release, it only supports torch 1.3.1, please refer to https://nni.readthedocs.io/en/latest/Compressor/ModelSpeedup.html for details.

apply_compression_results simply multiplies generated masks to weights, it does not speedup model inference. ModelSpeedup does, but ModelSpeedup is still in Alpha release, it only supports torch 1.3.1, please refer to https://nni.readthedocs.io/en/latest/Compressor/ModelSpeedup.html for details.

My torch version is 1.3.1

and how to use ModelSpeedup to get a smaller model ?
when I use ModelSpeedup, I get error:
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:291: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if grid_size[0] != self.grid_size[0]:
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:257: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert self.stride == self.img_dim[1] / self.grid_size[1]
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:262: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
self.scaled_anchors = FloatTensor([(a_w / self.stride, a_h / self.stride) for a_w, a_h in self.anchors])
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:299: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
pred_boxes = FloatTensor(prediction[..., :4].shape)
Traceback (most recent call last):
File "/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/model_compression.py", line 111, in
m_speedup = ModelSpeedup(model_1, input_imgs, masks_file)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/nni/compression/speedup/torch/compressor.py", line 91, in init
self.trace_graph = torch.jit.trace(model, dummy_input)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 858, in trace
check_tolerance, _force_outplace, _module_class)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 1007, in trace_module
check_tolerance, _force_outplace, True, _module_class)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
return func(*args, **kwargs)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 676, in _check_trace
raise TracingCheckError(*diag_info)
torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
Graph diff:
graph(%self : ClassType,
%x.1 : Tensor):
%2 : ClassType = prim::GetAttrname="module_list"
%3 : ClassType = prim::GetAttrname="0"
%4 : ClassType = prim::GetAttrname="conv_0"

looks like a bug in torch.jit, some related issues in pytorch:
https://github.com/pytorch/pytorch/issues/23993
https://github.com/pytorch/pytorch/issues/33491

and how to use ModelSpeedup to get a smaller model ?
when I use ModelSpeedup, I get error:
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:291: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if grid_size[0] != self.grid_size[0]:
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:257: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
assert self.stride == self.img_dim[1] / self.grid_size[1]
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:262: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
self.scaled_anchors = FloatTensor([(a_w / self.stride, a_h / self.stride) for a_w, a_h in self.anchors])
/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/models.py:299: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
pred_boxes = FloatTensor(prediction[..., :4].shape)
Traceback (most recent call last):
File "/home/cindy/Documents/3D/training/camera/geely_yolo3D_02/model_compression.py", line 111, in
m_speedup = ModelSpeedup(model_1, input_imgs, masks_file)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/nni/compression/speedup/torch/compressor.py", line 91, in init
self.trace_graph = torch.jit.trace(model, dummy_input)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 858, in trace
check_tolerance, _force_outplace, _module_class)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 1007, in trace_module
check_tolerance, _force_outplace, True, _module_class)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
return func(*args, **kwargs)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/torch/jit/init.py", line 676, in _check_trace
raise TracingCheckError(*diag_info)
torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
Graph diff:
graph(%self : ClassType,
%x.1 : Tensor):
%2 : ClassType = prim::GetAttrname="module_list"
%3 : ClassType = prim::GetAttrname="0"
%4 : ClassType = prim::GetAttrname="conv_0"

looks like a bug in torch.jit, some related issues in pytorch:
pytorch/pytorch#23993
pytorch/pytorch#33491

this error can be solved by this notice(from source code):
ModelSpeedup( model, dummy_input, masks_file)
The dummy input for jit.trace, users should put it on right device before pass in

@misslibra thanks for sharing the cause.

@QuanluZhang  hi ，when I try to apply ModelSpeedup with Pytorch model nesnet50
`
import torchvision.models as models
model = models.resnet50(pretrained=False)
`
I hit error :
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/nni/compression/speedup/torch/compressor.py", line 496, in infer_module_mask
self.infer_module_mask(_module_name, in_shape=output_cmask)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/nni/compression/speedup/torch/compressor.py", line 496, in infer_module_mask
self.infer_module_mask(_module_name, in_shape=output_cmask)
File "/home/cindy/anaconda3/envs/python36_env/lib/python3.6/site-packages/nni/compression/speedup/torch/compressor.py", line 474, in infer_module_mask
.format(m_type, module_name))
RuntimeError: Has not supported infering output shape from input shape for module/function: aten::_convolution, ResNet/Sequential[layer1]/Bottleneck[1]/Conv2d[conv1].aten::_convolution.1
and I try to add "aten::_convolution"  in map :  infer_from_inshape in infer_shape.py .
BUT, error still happen for another item "aten::_add"......
Is it because this function not suitable for resnet ? Or I still need to do other modify?

@misslibra ModelSpeedup relies on shape inference of operations to figure out what modules should be replaced and how. In the current alpha release, we only support limited operations/modules for shape inference. We are working on simplifying the process and interface for adding new operation/module support, will be included in future release.
Specifically for the error you encountered, seems like induced by a bug that has been already fixed. Could you pull the latest master branch, source install and try ModelSpeedup again?

@misslibra ModelSpeedup relies on shape inference of operations to figure out what modules should be replaced and how. In the current alpha release, we only support limited operations/modules for shape inference. We are working on simplifying the process and interface for adding new operation/module support, will be included in future release.
Specifically for the error you encountered, seems like induced by a bug that has been already fixed. Could you pull the latest master branch, source install and try ModelSpeedup again?

Hello, I also encounter problems when I tried to speed up ResNet. It seems like some conflicts occur between 2 shortcuts. For example, A -> conv_bn_relu_1 -> B, out1 = A+B -> conv_bn_relu_2 -> C, out2 = out1 + C, the mask of B should be apply on out1 and out2 because of successor relationships, but it conflicts with the mask of C...

@TangChangcheng could you provide an executable python script along with the mask file you use, so that we can reproduce the problem?

@TangChangcheng @misslibra your issue may be resolved by pr @2579

hi @misslibra which pytorch version were you using for L!filterpruner ?  I am having import error with pytorch 1.8.1 .


make rundemo not working

...
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/x/p/venv/lib/python3.9/site-packages/pinax/teams/apps.py", line 2, in <module>
    from django.utils.translation import ugettext_lazy as _
ImportError: cannot import name 'ugettext_lazy' from 'django.utils.translation' (/home/x/p/venv/lib/python3.9/site-packages/django/utils/translation/__init__.py)
make: *** [Makefile:84: demo] Error 1
(venv) x@lily:~/p/django-helpdesk$

which version of django and helpdesk are you using?

The trace looks like it's actually a problem with pinax, so will have to look into that as a dependency.

I'm using the helpdesk version from github.
I looked to see what version of django was installed, and there wasn't one. So I ran.
pip3 install django==3.2.13
After installing django, I get a new error.
  File "/Users/x/p/django-helpdesk/django-helpdesk/demo/demodesk/config/urls.py", line 30, in <module>
    path('', include('helpdesk.urls', namespace='helpdesk')),
  File "/Users/x/p/django-helpdesk/venv/lib/python3.8/site-packages/django/urls/conf.py", line 34, in include
    urlconf_module = import_module(urlconf_module)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/x/p/django-helpdesk/django-helpdesk/helpdesk/urls.py", line 175, in <module>
    url(r'^tickets/(?P<ticket_id>[0-9]+)/dependency/add/$',
NameError: name 'url' is not defined
make: *** [demo] Error 1

I think the problem is I assumed all I had to do was check out the code and run make rundemo. It seems there are other steps involved.

this is a pretty old script that I don't use very much anymore, so we'll have to either try to fix it or look for an alternative way to quickly run a demo and get users started. as for the url error here, I think this was fixed in #1022 so should be in the next bugfix release (0.4.1).

With the current version I can't get it to run as well.
Python 3.10.4 - Manjaro with 5.10 kernel
It crashed with the following dump:
× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [29 lines of output]
    running develop
    /usr/lib/python3.10/site-packages/setuptools/command/easy_install.py:157: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "<pip-setuptools-caller>", line 34, in <module>
      File "/home/rtfm/Documents/Docker/Flicket/django-helpdesk/setup.py", line 124, in <module>
        setup(
      File "/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 155, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
        return run_commands(dist)
      File "/usr/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
        dist.run_commands()
      File "/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 985, in run_command
        cmd_obj.ensure_finalized()
      File "/usr/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/usr/lib/python3.10/site-packages/setuptools/command/develop.py", line 52, in finalize_options
        easy_install.finalize_options(self)
      File "/usr/lib/python3.10/site-packages/setuptools/command/easy_install.py", line 276, in finalize_options
        self._fix_install_dir_for_user_site()
      File "/usr/lib/python3.10/site-packages/setuptools/command/easy_install.py", line 382, in _fix_install_dir_for_user_site
        self.create_home_path()
      File "/usr/lib/python3.10/site-packages/setuptools/command/easy_install.py", line 1338, in create_home_path
        if path.startswith(home) and not os.path.isdir(path):
    AttributeError: 'int' object has no attribute 'startswith'
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: There was an error checking the latest version of pip.
make: *** [Makefile:82: demo] Error 1

Thanks @js-on yeah it looks like it's about time to retire the script. We'll need to figure something else out. In the meantime if you're looking to get a demo, check out our demo website.


Python 3.5   SyntaxError: invalid syntax

import jose

  File "/py/env35/local/lib/python3.5/site-packages/jose.py", line 546
    print decrypt(deserialize_compact(jwt), {'k':key},
                ^
SyntaxError: invalid syntax

Is this package compatible with python 3.5?

Hi, I suppose you should use the python 3 branch

Nice, Is this in the pip library? or I have to do git+https://github.com/Demonware/jose.git@python3?

looking at https://pypi.python.org/pypi/jose/1.0.0 it seems the pip delivery is python 2 only. (mind you I'm no maintainer of this package)

python 3 branch worked well. Thanks


Latest flask release (2.0.0) cannot import functions from flask_restful package

Description
The new flask version that got released today has problems with the flask_restful package and fails to import _endpoint_from_view_func from flask.helpers. I then changed my environments to older versions of flask and it was working again.
How to replicate the bug:
Requirements:

install newest version of flask
install newest version of flask_restful

Code example:
from flask import Flask
from flask_restful import Api
app = Flask(name)
api = Api(app)
@app.route('/')
def hello_world():
return 'Hello, World!'
Exception upon execution:
app_1     | Traceback (most recent call last):
app_1     |   File "/app/app.py", line 3, in
app_1     |     from flask_restful import Api
app_1     |   File "/usr/local/lib/python3.9/site-packages/flask_restful/init.py", line 14, in
app_1     |     from flask.helpers import _endpoint_from_view_func
app_1     | ImportError: cannot import name '_endpoint_from_view_func' from 'flask.helpers' (/usr/local/lib/python3.9/site-packages/flask/helpers.py)
Environment:

Python version: tested with 3.8 to latest
Flask version: 2.0.0

This is an issue with Flask-Restful, see #4009.


Fix compile error when using timer in C++

In file included from E:/timer_test/main/main.cpp:11:0:
E:/esp-idf/components/soc/esp32/include/soc/timer_group_struct.h: At global scope:
E:/esp-idf/components/soc/esp32/include/soc/timer_group_struct.h:193:19: error: 'timg_dev_t TIMERG0', declared using anonymous type, is used but never defined [-fpermissive]
extern timg_dev_t TIMERG0;
^
E:/esp-idf/components/soc/esp32/include/soc/timer_group_struct.h:192:3: note: 'typedef volatile struct timg_dev_t' does not refer to the unqualified type, so it is not used for linkage
} timg_dev_t;

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

All the SoC header files miss the #ifdef __cplusplus extern "C" { #endif guards. Is this a bug or is there a reason for that? If it is a bug I think it should be corrected everywhere, not only for the timer...

I don't think all header files should have such guards. As you can see nearly all .h files in soc directory have only macros and structures. Those things don't need 'extern'. What 'extern "C"' does is to use C linker instead of C++ linker. The timer_group_struct.h has two external constants which are defined in a .c file. So it should be linked with C linker otherwise C++ linker cannot find them.

I guess what @yaqwsx means is that all the soc/xxx_struct.h files define extern global structures, not just the timer_group one.
Your change looks good though, thank you for sending this in. I can either add C++ guards to the rest of the _struct header files myself or wait for you to do this.

@igrr Yes, that's exactly what I meant.

@igrr  I think I have fixed all the remaining extern global structures in soc include directory.

Hi @imxieyi ,
Thanks for sending this. Sorry we didn't get back to you earlier.
As it turns out, we accidentally duplicated effort and there's a change coming which adds these guards. The commit is c7173e0d08. It's not on github right now, but it should appear in the next day or so.
Sorry for the mixup,
Angus


fix: find go.mod path correctly

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.81%. Comparing base (f2db3ac) to head (128c83f).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #246   +/-   ##
=======================================
  Coverage   83.81%   83.81%
=======================================
  Files          19       19
  Lines         791      791
=======================================
  Hits          663      663
  Misses        102      102
  Partials       26       26

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


How do I log in with my account and password

How do I log in with my account and password

https://core.telegram.org/tdlib/getting-started

I found the verification code login and url login in the document, but did not find the password login

machine translation *

There are no such things as "code login" and "url login". https://core.telegram.org/tdlib/getting-started contains all relevant information.


Connecting Front and Back End Apps

Hi,
Are the front and back end repos automatically integrated assuming that they're hosted on the same server?
Thanks!

Hello, yes. By default they are. In dev mode, they communicate through (localhost).
But in production, you should explicitly set PORT and FRONTEND_URL in backend env. (See application-prod.yml)
And for frontend, you should set BACKEND_URL. (See env.sample)

@scooter7 No problem. 😃If you face any problems, feel free to post an Issue here or in the Backend Repo

Will do! Thanks again!


Sanitize req.params

There does not seem to be an easy way to sanitize or escape request params.  What we ended up doing you can see here https://github.com/fingerfoodstudios/express-sanitize-escape/blob/master/lib/express-sanitize-escape.js#L40
module.exports.sanitizeParams = function(router, paramNames) {
    paramNames.forEach(function(paramName) {
        router.param(paramName, function(req, res, next) {
            req.params[paramName] = sanitize(req.params[paramName]);
            next();
        });
    });
}

And then using it in every router that we have where params are required.
'use strict';

var express = require('express');
var expressSanitized = require('express-sanitize-escape');
var controller = require('./user.controller');

var router = express.Router();

expressSanitized.sanitizeParams(router, ['id']);

router.get('/:id/', controller.show);

module.exports = router;

Is there any way to do this better?  So that all params can be escaped and sanitized?  If not can the router be changed to allow a wildcard for params functions?

There is no better way currently, no, but looking at what you are doing, I would argue that html escaping data should be the job of your view layer, not the router anyway.

Ok thanks

This was 2 years ago. Any update (or planned update) in Express that will handle this automatically or as an option?


DOIs from Elsevier v Sci-Hub

I'm curious about which articles Elsevier selected in their case against Sci-Hub. For each of these articles, Sci-Hub was fined $150,000.
I extracted the list from this document: elsevier-dois.txt. Here are the links to the 100 articles:

Expand to see articles

https://doi.org/10.1016/j.actpsy.2015.05.014
https://doi.org/10.1016/j.agwat.2015.08.001
https://doi.org/10.1016/j.apgeog.2015.03.016
https://doi.org/10.1016/j.apsusc.2015.05.178
https://doi.org/10.1016/j.bej.2015.04.006
https://doi.org/10.1016/j.cattod.2014.03.074
https://doi.org/10.1016/j.enbuild.2015.09.037
https://doi.org/10.1016/j.engappai.2013.04.008
https://doi.org/10.1016/j.eswa.2015.07.061
https://doi.org/10.1016/j.ibiod.2015.02.019
https://doi.org/10.1016/j.ijsolstr.2015.08.002
https://doi.org/10.1016/j.jallcom.2015.05.083
https://doi.org/10.1016/j.jbankfin.2015.01.013
https://doi.org/10.1016/j.jcis.2007.03.070
https://doi.org/10.1016/j.jcis.2008.03.041
https://doi.org/10.1016/j.jfluidstructs.2013.09.018
https://doi.org/10.1016/j.jfluidstructs.2013.09.020
https://doi.org/10.1016/j.jfluidstructs.2015.06.017
https://doi.org/10.1016/j.jmbbm.2015.08.043
https://doi.org/10.1016/j.jmmm.2015.02.045
https://doi.org/10.1016/j.jphotobiol.2015.05.018
https://doi.org/10.1016/j.jpowsour.2013.09.100
https://doi.org/10.1016/j.jvlc.2014.08.002
https://doi.org/10.1016/j.matdes.2014.08.042
https://doi.org/10.1016/j.measurement.2014.06.001
https://doi.org/10.1016/j.memsci.2013.04.034
https://doi.org/10.1016/j.nut.2013.09.007
https://doi.org/10.1016/j.oceaneng.2013.11.021
https://doi.org/10.1016/j.optmat.2015.05.024
https://doi.org/10.1016/j.pep.2007.12.005
https://doi.org/10.1016/j.poly.2013.11.038
https://doi.org/10.1016/j.powtec.2013.04.029
https://doi.org/10.1016/j.powtec.2013.11.035
https://doi.org/10.1016/j.synthmet.2015.04.015
https://doi.org/10.1016/j.tetlet.2013.04.043
https://doi.org/10.1016/j.tsf.2015.07.023
https://doi.org/10.1016/j.watres.2015.05.010
https://doi.org/10.1016/j.ympev.2015.07.026
https://doi.org/10.1016/j.urolonc.2015.02.009
https://doi.org/10.1016/j.ijpharm.2014.12.057
https://doi.org/10.1016/j.cellsig.2015.10.015
https://doi.org/10.1016/j.clgc.2013.07.004
https://doi.org/10.1016/j.finel.2015.07.001
https://doi.org/10.1016/j.ibusrev.2014.11.006
https://doi.org/10.1016/j.cca.2013.09.017
https://doi.org/10.1016/j.adiac.2013.03.007
https://doi.org/10.1016/j.ab.2015.09.020
https://doi.org/10.1016/j.acha.2014.06.006
https://doi.org/10.1016/j.ins.2015.03.027
https://doi.org/10.1016/j.ins.2015.02.033
https://doi.org/10.1016/j.ijpe.2014.12.034
https://doi.org/10.1016/j.scriptamat.2015.10.011
https://doi.org/10.1016/j.epsl.2015.04.002
https://doi.org/10.1016/j.jvb.2015.03.003
https://doi.org/10.1016/j.wear.2015.11.004
https://doi.org/10.1016/j.bej.2015.10.027
https://doi.org/10.1016/j.engfailanal.2015.10.002
https://doi.org/10.1016/j.biortech.2015.07.090
https://doi.org/10.1016/j.watres.2015.08.019
https://doi.org/10.1016/j.renene.2014.11.004
https://doi.org/10.1016/j.ijheatfluidflow.2015.09.001
https://doi.org/10.1016/j.jeurceramsoc.2014.05.016
https://doi.org/10.1016/j.triboint.2014.03.010
https://doi.org/10.1016/j.triboint.2014.09.016
https://doi.org/10.1016/j.apenergy.2014.12.055
https://doi.org/10.1016/j.applthermaleng.2015.07.021
https://doi.org/10.1016/j.surg.2015.09.016
https://doi.org/10.1016/j.respol.2014.12.008
https://doi.org/10.1016/j.eurpolymj.2014.07.016
https://doi.org/10.1016/j.eurpolymj.2014.08.004
https://doi.org/10.1016/j.eurpolymj.2014.04.004
https://doi.org/10.1016/j.jprot.2015.05.035
https://doi.org/10.1016/j.jaridenv.2014.04.010
https://doi.org/10.1016/j.toxicon.2014.08.063
https://doi.org/10.1016/j.jembe.2015.07.011
https://doi.org/10.1016/j.nbd.2014.07.014
https://doi.org/10.1016/j.scitotenv.2014.07.071
https://doi.org/10.1016/j.yexcr.2015.07.003
https://doi.org/10.1016/j.biortech.2014.12.072
https://doi.org/10.1016/j.nanoen.2015.02.024
https://doi.org/10.1016/j.euroecorev.2015.08.004
https://doi.org/10.1016/j.iref.2015.06.006
https://doi.org/10.1016/j.frl.2015.02.007
https://doi.org/10.1016/j.bioelechem.2015.11.001
https://doi.org/10.1016/j.cej.2015.08.100
https://doi.org/10.1016/j.econlet.2015.05.006
https://doi.org/10.1016/j.jpowsour.2015.07.103
https://doi.org/10.1016/j.aca.2015.07.019
https://doi.org/10.1016/j.advengsoft.2015.02.005
https://doi.org/10.1016/j.agwat.2015.06.018
https://doi.org/10.1016/j.amc.2015.04.116
https://doi.org/10.1016/j.aml.2014.10.013
https://doi.org/10.1016/j.anucene.2014.04.033
https://doi.org/10.1016/j.apcatb.2014.03.025
https://doi.org/10.1016/j.apm.2014.06.022
https://doi.org/10.1016/j.apsusc.2014.09.133
https://doi.org/10.1016/j.apsusc.2015.07.052
https://doi.org/10.1016/j.autcon.2014.02.016
https://doi.org/10.1016/j.autcon.2014.06.005
https://doi.org/10.1016/B978-0-12-411549-1.00016-8

Work number 28 in Elsevier's list is a 2014 article in Ocean Engineering titled Numerical simulation of water impact of solid bodies with vertical and oblique entries:

Interestingly, the article's copyright statement is:

Crown copyright © 2013 Published by Elsevier Ltd. All rights reserved.

I'm not familiar with UK Copyright Law. From an OUP document:

This means that the first publication of Crown copyright material does not benefit from the OGL and therefore still requires a licence. However, once the material has been published (e.g., by OUP), the specific material that has been published may fall under the OGL so that other publishers can re-publish it.

@rossmounce what do you make of this? Does Elsevier's copyright apply only to the typesetting?

In https://github.com/greenelab/scihub/commit/ab8ffda9bfa2e198fb6bbad7f9133f0d5d053142, I created a table of Crossref metadata for the 100 Elsevier works (see elsevier-suit-articles.tsv). This table contains columns for authorship and funding information as well as copyright statements. @stevemclaugh and @tamunro you may find this interesting.

@dhimmel at first I thought it was a mistaken assertion of UK Crown Copyright.
But then I dug deeper. If one checks the ESPRC grant codes given there may be clues:
http://gow.epsrc.ac.uk/NGBOViewGrant.aspx?GrantRef=EP/K037889/1
STFC Laboratories (http://www.stfc.ac.uk/) is listed as a project partner.
STFC is sponsored by the Department for Business, Energy & Industrial Strategy (BEIS), which is a UK Crown body and thus subject to Crown copyright protection under Section 163 of the Copyright Designs and Patents Act 1988. http://www.nationalarchives.gov.uk/information-management/re-using-public-sector-information/uk-government-licensing-framework/crown-copyright/uk-crown-bodies/
So... the paper has attracted Crown Copyright via project partnership, even though none of the authors themselves are directly affiliated to any UK Crown bodies. Hugely interesting example, well found!
"Does Elsevier's copyright apply only to the typesetting?" Probably. As well as Elsevier logo-stamp and journal front-cover stamp in the top left and top right hand corners.

Wow @rossmounce, amazing research. I do think this was probably an oversight in Elsevier's selection of infringed works. Certainly, there may be some copyright protection on petty aspects of the PDF, but if the entire scientific content should be available under an Open Goverment License, it seems crazy to fine someone $150,000! Contacting the authors for the related legal documents would be a good next step.
This is a great example of potential copyfraud. Also note that Elseiver is charging for reuse through RightsLink, even when the reuse is only of the content (and not typesetting or logos). There's big punishment for copyright infringement, but it seems like false copyright claims on open or public domain content have little recourse.


HiBitSoftware.HiBitUninstaller version <IP_ADDRESS>

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?
[x] This PR only modifies one (1) manifest
[x] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[x] Does your manifest conform to the 1.5 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Submission containing materials of a third party:
Author: Mohammed Raee
Publisher: HiBitSoftware
PublisherUrl: https://www.hibitsoft.ir/
PublisherSupportUrl: https://www.hibitsoft.ir/About.html
License: Proprietary
Copyright: Copyright @ 2017-2023 HiBitSoft.IR All Rights Reserved

Microsoft Reviewers: Open in CodeFlow

I assume the problem might be caused by editing the yml file manually to fix the copyright date. I will try publishing one last time after closing this request, without intervening with the auto generated manifest this time.


Booking module

Description
BDE Booking module
Checklist

[ ] Created tests which fail without the change (if possible)
[ ] All tests passing
[ ] Extended the documentation, if necessary

Unit tests are missing ?


feat: export createAxiosLanguageModel

Exporting createAxiosLanguageModel would solve the following problems

support custom axios config (e.g. users can pass a proxy)  related: #73 #55 #34
ability to pass custom parameters to the endpoint /v1/chat/completions

Personally I don't think the library should expose Axios at all, since it's an internal implementation detail.
For example, in another issue someone asked about how to take advantage of streaming responses from OpenAI. Best I can tell, Axios doesn't make that easy to do (or it might not provide the APIs to do so at all). I use this as a specific example, but my point is really that hooking up Axios or whatever language model provider API is easy enough because TypeChatLanguageModel just needs a single function at the moment.
Still, I'll let others weigh in here.

@DanielRosenwasser thete are likely legit n/w level overrides desired, but middleware is likely the better way forward imo rather than anything axios/fetch/ky specific.
I agree with exposing the model the way it is with the single function 👍

I don't think TypeChat needs to provide proxy settings, as it can support proxy access through higher-level configurations, such as proxychains4.

I think the package shall use undici as this will be the default client in the future Node versions
If anything needs to be exposed, then I think FetchEsque types from trpc can be borrowed, which can adapt to any fetch implementation.


Port To Android

Hi @Nabeelshar ,
Impressive job! I spent some time playing the Tetris game and I must admit that it's a testament to your skill in creating games using pure JavaScript. Have you considered adapting it for React Native to develop an Android/iOS version?

https://play.google.com/store/apps/details?id=com.nab.brickgame9999in1


Close WebSocket connections after server exists

Resolve #7.
Additionaly, new formatting is applied with the new version of SwiftFormat.

Fails

:no_entry_sign:

Sources/SwiftToolchain/FileDownloadDelegate.swift#L54 - Opening braces should be preceded by a single space and on the same line as the declaration. (opening_brace)

  Generated by :no_entry_sign: Danger Swift against c3f6a64c3fbc5d42d7220db08f9d7d74a57155a1


exceptions trigger crash in bugsnag

I'm running into an issue where a call to notify(), an exception, crashes my app. In the debugger it's shown as 'mapState is not a function' error.

 This is pretty odd. It seems to coming from deep in react-redux. Can you share more about your application, such as the React Native version, library version, and device/OS used?

Yeah it's indeed weird :)
This was on react-native 0.38 and 0.39, with the latest bugsnag release and on iOS simulator (iOS 10.1). It's not easily reproducible to be honest so this is as detailed as i can report it.

Was it just something like notify(new Error("…")) or potentially something more?

almost.. i did use the metadata feature:
this.client.notify(error, function(report) {
  report.metadata = { ...user };
});

The v1.1.4 release went out shortly after this, would you be able to retest this, @tijs? I haven’t been able to reproduce it, but maybe it was related to other issues which were fixed.

@kattrali will do! i have a release to finish up first but will have a look again next week.

Closing this out, but feel free to reopen if this issue persists. Thanks again, @tijs.


tutorial INBO citation style

Implements the tutorial mentioned in issue #168.
@florisvdh can you render this to html so the contents of the tutorial can be shared with and reviewed by Herwig? I don't know how to preview this tutorial.

@hansvancalster I recommend to extend the output key in the Rmd YAML header as:
output:
    md_document:
        preserve_yaml: true
        variant: gfm
    html_document: default

Knit as html (don't commit the html) and then drop the (zipped) generated html in a message here, so that you get a hyperlink.
Rendering the site with hugo separates css, javascript and other functionality from the actual html file, as the former are common to all pages - so it's less convenient for sharing as files. (but I do use it sometimes to check layouted results locally)

Thanks @florisvdh I will send Herwig the html file by e-mail for review.

@florisvdh Herwig did not have further comments. So I wait for your comments or approval to finalize this PR.

@florisvdh thanks for the great review (as always :+1: ). Can you have one final look? I think all concerns are addressed.

Olé https://inbo.github.io/tutorials/tutorials/style_inbo_bibliography/


can't tokenize last macro in file

foo.h
#define FOO 123
#define BAR 456

require "ffi/gen"

FFI::Gen.generate(
  module_name: "foo",
  ffi_lib:     "foo",
  headers:     ["foo.h"],
  cflags:      `/usr/local/Cellar/llvm/3.4.1/bin/llvm-config --cflags`.split(" "),
  prefixes:    [],
  output:      "foo.rb"

Result:
# Generated by ffi-gen. Please do not change this file by hand.

require 'ffi'

module foo
  extend FFI::Library
  ffi_lib 'foo'

  def self.attach_function(name, *_)
    begin; super; rescue FFI::NotFoundError => e
      (class << self; self; end).class_eval { define_method(name) { |*_| raise e } }
    end
  end

  FOO = 123

end

BAR is nowhere to be found.
Maybe related:
http://lists.cs.uiuc.edu/pipermail/cfe-dev/2011-January/012983.html

Closing as upstream issue.


fix: Remove the unused permission to record the audio on Android

Hi everyone,
The camera plugin asks for the RECORD_AUDIO permission.
As we don't need it in the app, we force it to be removed.
Will fix #3971

Codecov Report

Merging #4085 (dd960b8) into develop (b33118a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #4085   +/-   ##
========================================
  Coverage    10.92%   10.92%
========================================
  Files          271      271
  Lines        13470    13470
========================================
  Hits          1471     1471
  Misses       11999    11999

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Invalid hook call. Hooks can only be called inside of the body of a function component.

Describe the bug
Invalid hook call. Hooks can only be called inside of the body of a function component. This could happen for one of the following reasons:

You might have mismatching versions of React and the renderer (such as React DOM)
You might be breaking the Rules of Hooks
You might have more than one copy of React in the same app
See https://reactjs.org/link/invalid-hook-call for tips about how to debug and fix this problem.

To Reproduce
Steps to reproduce the behavior:

ReactJs 18.2.0, React Dom 18.2.0
Produce the minimal setup of the treeview

Expected behavior
Hooks should be called inside FUNCTIONAL COMPONENTS
Screenshots

Issue is on line number 35, where useEffect is called inside useMemo.

How did you solve this?


Default for no matching objects

It would be nice if one can specify a value/callable to use when no related objects match predicate.
In our system there is part where we classify traffic by some key. If there is no match for the key we consider it an Organic Traffic.
Unfortunately it's not possible to implement this business rule with just predicate itself.

Hi @scorpp, thanks for your feedback.
Can you give me a more concrete idea of the models and Relationships between them in your system, and the behaviour of your proposed change to the django-relativity API?
At least initially, this sounds beyond the scope of this project - the idea of the Relationship field is to be as consistent with the Django ORM's relation fields as possible from the perspective of application code. But I'm open to being convinced.


update vpc peering definition to include peer owner and peer role arn

We are using this package to write all of our production templates and need the definition for vpc peering to be updated and released asap please.  I made the same changes on local and can confirm they are successful

The aws types are updated dynamically using an AWS provided resource specification. Run cfndsl -u to update the file locally. The latest spec should have the missing properties defined.

Well, we are on a very, very old version.  Will upgrade for now and if anyone on the team has an issue we will re-open.  Thanks!


Integrate code generation into main package

Merge code generation into evalplus directory
One command to run both generation and evaluation
Package isolation

Implemented by:

a04f3251bcc44b151f42428f21636ea1d65e287b
ca0acd3e6c19c323e7ff5dd5037ce04daca6f7df


Can I export/import the training data?

Is there a way that I can export/import the training data?
After I trained the class with many images, I want to export(download) the training data and I want to reuse it.

I'm not sure if this is the correct way to do this, but how I've been doing it is by storing tensor data in storageDB and then retraining the model onLoad. To store tensor data, simply use the .data() method on the tensor:
// Clone the existing tensor
// .data() call is asynchronously, so we clone the old tensor
// in case it's disposed of by the time we generate our next tensor
let tensor = oldTensor.clone()
tensor.data().then((data) => {
  // Store the tensor
  saveTensorToStorageDB(data)
  tensor.dispose()
})

Then, to load it back in try something like:
let tensorShape = [227, 227, 3]
loadTensorsFromStorageDB.then((trainedData) => {
  trainedData.tensors.forEach((tensors, modelClass) => {
    tensors.forEach((data) => {
      let tensor = deeplearn.tensor(data, tensorShape)
      this.classifier.addImage(tensor, modelClass)
      tensor.dispose()
    })
  })
})

You can either download the trained data as a JSON file, save it to localStorage (5MB limit), or save it to StorageDB (50MB limit). You can also save it to Firebase or some other remote store. I wrote about it in more detail (with examples) here: https://learnwithoz.github.io/day-3-model-persistence

Thank you very much for your advice.
I wanted to see your examples, but was unable to see pages on https://learnwithoz.github.io/day-3-model-persistence.

@champierre hey looking for the same thing, did you ever figure out how to do it?

@atzin-em I could implement export/import feature on different project. Hope that they could help you.
download(export)
https://github.com/champierre/ml2scratch/blob/master/main.js#L341
upload(import)
https://github.com/champierre/ml2scratch/blob/master/main.js#L354

@champierre Thank you very much, help was greatly appreciated

@champierre would you happen to know how to also upload photos to the model manually instead of through the webcam?

@atzin-em It is interesting to try, but I have not yet.

Hi - I am wondering about this same issue.  A lot of the example links above are broken now.  Please advise.

Hi! A simple strategy is saving the model out as a JSON file with data from the classifier, then importing it back in to access the weights.
You can read more about this strategy here: https://js.tensorflow.org/tutorials/model-save-load.html

hi @mayaman - thanks for the tip, but I am very much a noob.  In trying to adapt this boilerplate code specifically, I can't quite pinpoint what they are calling the "model" if I was to apply the code below:
const saveResult = await model.save('localstorage://my-model-1');
Any ideas?

There is a new release coming later this year that will have this feature! You can read more about it here: https://teachablemachine.withgoogle.com/io19.

There is a new release coming later this year that will have this feature! You can read more about it here: https://teachablemachine.withgoogle.com/io19.

I've seen that, but is there any way to export this now? or how to do that?

There are a number of ways to save your trained model and also upload that trained model and continue from where you left off.
Here is an example for saving (Downloading a JSON file)
const downloadJSON = async (knn) => {     let dataset = this.knn.getClassifierDataset()    var datasetObj = {}    Object.keys(dataset).forEach((key) => {      let data = dataset[key].dataSync();      datasetObj[key] = Array.from(data);     });    let jsonStr = JSON.stringify(datasetObj)     let jsonModel = JSON.stringify(datasetObj);     let downloader = document.createElement('a');     downloader.download = "model.json";     downloader.href = 'data:text/text;charset=utf-8,' + encodeURIComponent(jsonModel);     document.body.appendChild(downloader);     downloader.click();     downloader.remove();   }
And here is an example to upload the JSON file in to the web application
const loadJSON = async (classifierModel, event) => {
let inputModel = event.target.files;
console.log("Uploading");
let fr = new FileReader();
if (inputModel.length>0) {
fr.onload = async () => {
var dataset = fr.result;
var tensorObj = JSON.parse(dataset);
Object.keys(tensorObj).forEach((key) => {
tensorObj[key] = tf.tensor(tensorObj[key], [tensorObj[key].length / 1000, 1000]);
});
classifierModel.setClassifierDataset(tensorObj);
console.log("Classifier has been set up! Congrats! ");
};
}
await fr.readAsText(inputModel[0]);
console.log("Uploaded");
}


Project deletion fails if it's using hardcoded namespace

Steps to reproduce:

create namespace and project (You can even let gardener create the namespace)

$ cat /tmp/project-sample
---
apiVersion: v1
kind: Namespace
metadata:
  name: nstest3
  labels:
    garden.sapcloud.io/role: project
    project.garden.sapcloud.io/name: nstest3
---
apiVersion: garden.sapcloud.io/v1beta1
kind: Project
metadata:
  name: nstest3
spec:
  owner:
    apiGroup: rbac.authorization.k8s.io
    kind: User
    name<EMAIL_ADDRESS>  namespace: nstest3

$ kubectl apply -f /tmp/project-sample
namespace/nstest3 created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
project.garden.sapcloud.io/nstest3 configured

Try to delete the project

$ KUBECONFIG=~/.kube/msa-sandbox-garden.kubecfg hack/delete project nstest3
project.garden.sapcloud.io/nstest3 annotated
project.garden.sapcloud.io "nstest3" deleted
^C ****(this will stuck waiting so break the command)***

Validate status

$ kubectl get project
NAME      OWNER                                                NAMESPACE              STATUS        READY                              AGE
nstest3   unknown                                              nstest3                Terminating   False (NamespaceDeletionFailed)    38s

Try to delete namespace directly

$  gardener/hack/delete ns nstest3
namespace/nstest3 annotated
Error from server (InternalError): Internal error occurred: failed calling admission webhook "validate-namespace-deletion.gardener.cloud": Post https://gardener-controller-manager.garden.svc:443/webhooks/validate-namespace-deletion?timeout=30s: dial tcp <IP_ADDRESS>:443: connect: connection refused

We actually have that svc defined and the ip is correct:
$ kubectl get svc -n garden
NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
gardener-controller-manager      ClusterIP   <IP_ADDRESS>     <none>        80/TCP,443/TCP   36d

I downgraded gardener to 0.10.2 from 0.12.0. All the namespaces in Termination state are gone.

After upgrading to 0.12.0 with a cleaned state,
I created a project without a given namespace. Gardener created a semi-random named namespace.
I tried to delete the project. It still failed to delete.
  $ > cat /tmp/project
apiVersion: garden.sapcloud.io/v1beta1
kind: Project
metadata:
  name: nstest1
spec:
  owner:
    apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: Dublin-MSA

  $ kubectl apply -f /tmp/project
project.garden.sapcloud.io/nstest1 configured

  $ kubectl get project
NAME      OWNER                                                NAMESPACE              STATUS    READY     AGE
nstest1   Dublin-MSA                                           garden-nstest1-9jcqq   Active    True      1m

  $ kubectl get ns
NAME                   STATUS    AGE
default                Active    37d
garden                 Active    37d
garden-nstest1-9jcqq   Active    1m
kube-public            Active    37d
kube-system            Active    37d

  $ hack/delete project nstest1
project.garden.sapcloud.io/nstest1 annotated
project.garden.sapcloud.io "nstest1" deleted

  $ kubectl get project
NAME      OWNER                                                NAMESPACE              STATUS        READY                              AGE
nstest1   Dublin-MSA                                           garden-nstest1-9jcqq   Terminating   False (NamespaceDeletionAllowed)   1m

aws: devopsdublin
oyalazi at OYALAZI in ~/Documents/PTC/k8s/k8s-clusters on upgrade*
  $ k8s-clusters > KUBECONFIG=~/.kube/msa-sandbox-garden.kubecfg kubectl get project
NAME      OWNER                                                NAMESPACE              STATUS        READY                             AGE
nstest1   Dublin-MSA                                           garden-nstest1-9jcqq   Terminating   False (NamespaceDeletionFailed)   4m

Error from server (InternalError): Internal error occurred: failed calling admission webhook "validate-namespace-deletion.gardener.cloud": Post https://gardener-controller-manager.garden.svc:443/webhooks/validate-namespace-deletion?timeout=30s: dial tcp <IP_ADDRESS>:443: connect: connection refused

This is a problem in your Gardener cluster. We now register a validating webhook to the Garden cluster and it seems that the kube-apiserver of your Garden cluster cannot reach the gardener-controller-manager pod.

Ok i checked the service description for the gcm. It should be listening for 2718 and 2719. But it's only listening on 2718.
Also apiserver can reach  gcm:80. gcm:443 is refusing connections as expected.
  $ kubectl get svc -n garden gardener-controller-manager -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-08-22T13:23:48Z
  labels:
    app: gardener
    chart: gardener-0.1.0
    heritage: Tiller
    release: garden
    role: controller-manager
  name: gardener-controller-manager
  namespace: garden
  resourceVersion: "5843792"
  selfLink: /api/v1/namespaces/garden/services/gardener-controller-manager
  uid: 9ab0177a-a60e-11e8-b857-0a58ac1f22de
spec:
  clusterIP: <IP_ADDRESS>
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 2718
  - name: https
    port: 443
    protocol: TCP
    targetPort: 2719
  selector:
    app: gardener
    release: garden
    role: controller-manager
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

  $ kubectl exec -n garden -ti gardener-controller-manager-598fc5799c-5rvg8  bash
bash-4.4# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 :::2718                 :::*                    LISTEN
bash-4.4# exit

  $ kubectl exec -n garden -ti gardener-apiserver-6679f4d697-xrmmv bash
bash-4.4# curl http://gardener-controller-manager
404 page not found
bash-4.4#

Ok turns out, it's about a certificate issue:
time="2018-09-28T14:08:58Z" level=error msg="Could not start HTTPS server: tls: failed to find any PEM data in certificate input"

I'll check this issue and report back. Thank you.

After adding the tls cert, project deletion started working.


Update package.json

fix openupm build failure, check this.

sorry i forgot it.. thanks!

Maybe new release is required so i will create it.


[Utils] fix downstream bundlers remove React 17 useId compatibility

[x] I have followed (at least) the PR section of the contributing guide.

Summary
Fixes #37182. Based on this commit from floating-ui. I've confirmed that this will resolve the issue by manually tweaking the useId code in a local install of @mui/utils and running it through the same bundle process that's stripping the string concatenation.
Bundle output before tweak to confirm fix, using string concatenation in utils source
Note that +"" has been stripped from React3["useId"+""] on line 15. This is what's causing the downstream build issue.
// ../../common/temp/node_modules/.pnpm/@mui+utils@5.11.13_react<EMAIL_ADDRESS>import * as React3 from "react";
var globalId = 0;
function useGlobalId(idOverride) {
  const [defaultId, setDefaultId] = React3.useState(idOverride);
  const id = idOverride || defaultId;
  React3.useEffect(() => {
    if (defaultId == null) {
      globalId += 1;
      setDefaultId(`mui-${globalId}`);
    }
  }, [defaultId]);
  return id;
}
var maybeReactUseId = React3["useId"];
function useId2(idOverride) {
  if (maybeReactUseId !== void 0) {
    const reactId = maybeReactUseId();
    return idOverride != null ? idOverride : reactId;
  }
  return useGlobalId(idOverride);
}

Bundle output after tweak to confirm fix, using .toString() in utils source
Note that the call to toString has been preserved on line 15.
// ../../common/temp/node_modules/.pnpm/@mui+utils@5.11.13_react<EMAIL_ADDRESS>import * as React3 from "react";
var globalId = 0;
function useGlobalId(idOverride) {
  const [defaultId, setDefaultId] = React3.useState(idOverride);
  const id = idOverride || defaultId;
  React3.useEffect(() => {
    if (defaultId == null) {
      globalId += 1;
      setDefaultId(`mui-${globalId}`);
    }
  }, [defaultId]);
  return id;
}
var maybeReactUseId = React3["useId".toString()];
function useId(idOverride) {
  if (maybeReactUseId !== void 0) {
    const reactId = maybeReactUseId();
    return idOverride != null ? idOverride : reactId;
  }
  return useGlobalId(idOverride);
}

Hey, thanks for looking into this. Can you please share simple reproduction app so that we can validate the fix?

Hi @mnajdova, do you have any suggestions on how to achieve that, since this only appears when an app is consuming another package that bundles MUI? Would I make a simple reproduction library and a simple reproduction app that attempts to use it?

Thanks for the review @mnajdova! What's the next step in the process here?

Wanted to check back in on this since it's a blocker for us. What needs to happen to get this merged?

It should be released today!

This PR is strange, we update useId but not useSyncExternalStore?
https://github.com/mui/material-ui/blob/046aac6fe9509111944633dc91f02b88969519ef/packages/mui-system/src/useMediaQuery/useMediaQuery.ts#L103