LUCENE-10335: Deprecate broken helper methods not compatible with module system; add ModuleResourceLoader and add integration test

See: https://issues.apache.org/jira/browse/LUCENE-10335

I have not yet checked Luke's image loading, as I touched this while reviewing all calls to getResource()/getResourceAsStream().

I fixed Luke. I will now check for other places that do getClassLoader().getResource...

We have some more in Benchmark module and tests of Spatial module. We may need to fix those and move resources to exported packages (not a root folder called "data/").

I found another problematic one: loadStopwordSet in StopAnalyzerBase. This does not work from inside nor/kuromoji and others. I will deprecated in same way.
I would like to make this method only visible from the analyzers-common module, but I am not sure how.

How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. gradlew check is totally happy for me.

How can we have failing tests for these problems? We need a test-driven approach to fixing this stuff. gradlew check is totally happy for me.

This is preparatory. We have a branch where it does not work yet.

To explain: We need to go away with a public static helper class that calls Class#getResourceAsStream() on behalf of somebody else.

I understand why the method doesnt work for modules: I just don't care.
I am only asking about failing tests.

@rmuir we will have a test-driven approach soon after xmas when Dawid finishes the conversion to module system. But all work that can be done before should be done before, as it will be a messy huge patch.

I understand why the method doesnt work for modules: I just don't care.
I am only asking about failing tests.

For now I can only make sure I don't break the status as is.

Also, are we sure this stuff isn't bugs in the JDK? Seems like a design flaw in modules.
I have the feeling this is going to break the analysis module left and right. I am concerned about things like analyzer factories, stopwordananalyzerbase, the list goes on and on. I cannot "see" the scope of the problem without failing tests LOL.

The problem is that getResourceAsStream() and getResource() are caller-sensitive and check who calls it.

Sorry, I have to say I am -1 at the moment to "gutting" all the analysis code, without tests, before first even trying to report these bugs to openjdk (it is THEIR FAULT). We should get it fixed upstream.

Also, are we sure this stuff isn't bugs in the JDK? Seems like a design flaw in modules.

I was in the discussion before Java 9 came out. It was a mess and I argued several times, because you were not even able to oad class files anymore as resources (now they have an exception). getResourceAsStream() was changed several times to work around some problem with legacy apps.
At the end we came out with this and it is clearly documented in the JLS and the API:

a Class instance located in unnamed modules (classpath) can be used to load any resource (it's the backwards compatibility layer)
a Class instance for a class located in a named module can only load resoures that the caller of getResourceAsStream can see (exported packages). This won't be problematic for analyzers module unless we use static utility methods.

In the analyzers module it is not a problem anymore, only a few issues are there.

Where is it clearly documented? I don't see any such documentation: BUG.

Robert, please let me finish this, otherwise I will trash down everything and stop working anymore on module system. If you want modules (and looks like you want) let us do our work and be calm. Thanks.
I will work more and commit this a bit later. These changes are not risky at all and IMHO I fixed a lot of other stuff around.

My problem is that the justification for fixing this stuff is that, "this code is buggy, it calls a CallerSensitive method".
CallerSensitive isn't even publicly documented. This is OpenJDK internal stuff. OpenJDK BUGS!
Uwe, I don't meant to discourage you from working on modules. It isn't a personal attack. I am angry about the OpenJDK behavior here and honestly believe this stuff is a bug.

(https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Class.html#getResourceAsStream(java.lang.String)):

Finds a resource with a given name.
If this class is in a named Module then this method will attempt to find the resource in the module. This is done by delegating to the module's class loader findResource(String,String) method, invoking it with the module name and the absolute name of the resource. Resources in named modules are subject to the rules for encapsulation specified in the Module getResourceAsStream method and so this method returns null when the resource is a non-".class" resource in a package that is not open to the caller's module.
Otherwise, if this class is not in a named module then the rules for searching resources associated with a given class are implemented by the defining class loader of the class. This method delegates to this object's class loader. If this object was loaded by the bootstrap class loader, the method delegates to ClassLoader.getSystemResourceAsStream(java.lang.String).

Thanks Uwe. I suppose "caller" there is enough to justify their shitty CallerSensitive shenanigans.
Please proceed, sorry for noise.

Thanks Uwe. I suppose "caller" there is enough to justify their shitty CallerSensitive shenanigans.

I was angry about that 4 or 5 years ago already. By the way the exception for "class" files was my work at that time.

I am thinking about a small solution to still use StopAnalyzerBase#loadStopwordSet(...,Class,...) and not duplicate code everywhere like in current patch.

I am thinking about a small solution to still use StopAnalyzerBase#loadStopwordSet(...,Class,...) and not duplicate code everywhere like in current patch.

But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue.

Maybe, we should just go back to hacky code to unzip jar file, open file on filesystem. after all, "resources" are practically always there on the filesystem anyway. We can just open them in a different way and stop using Class.getResourceAsStream!

I am thinking about a small solution to still use StopAnalyzerBase#loadStopwordSet(...,Class,...) and not duplicate code everywhere like in current patch.

But how will we fix analyzer factories (ClasspathResourceLoader) ? This seems like the biggest issue.

The class will work well in ClassPath mode, of course. But we should deprecate the constructor that takes a class.
The one taking a ClassLoader can be initialized by an app like a modularized Elasticsearch or Solr using a ClassLoader retrieved from the module. We did this inside Luke already.

 These tests are on a branch where I'm trying to fix classpath vs. module-path issues - the compilation and tests currently use classpath only; once you make them run with  true modules, things start to break. It is very messy conceptually - the conventions gradle has predate the module system and they're not always compatible. Every time I think I've solved something, two other things emerge. It is difficult. I am making progress though.

we will have a test-driven approach soon after xmas when Dawid finishes the conversion to module system

Thank you for your faith in me, Uwe. I hold on to your word. ;)

The code Uwe provided in the patch is fixing a true bug - it's just masked by the tests running in classpath only. In other words: if the tests pass after applying this patch, nothing changes for the classpath mode... and things will improve on the module-system layer side.

I am giving up for today, have to think about it a bit more.
@dweiss you can take my current PR for further testing.
From checking the code it looks like we need a ModuleResourceLoader class as complement to ClasspathResourceLoader. But I will open a separate PR about this when Dawid is done.

IMHO, it's a pain that so much reflection specific stuff is now caller sensitive (since Java 9), this makes utility methods impossible to expose.

Yes, I just did a quick git grep and immediately start finding bugs. first hit is RSLPStemmerBase.parse(). The class is public/abstract, so the code should be fixed. We can't reduce visibility of the class, it is needed by both Portuguese and Galician (separate packages). So we have to fix signatures etc. No tests will fail, but its a lurking "API bug", if someone tries to use it from the outside.
I am trying to think on some ideas to statically identify this stuff. Maybe we just need to use forbidden apis.

Thanks Uwe. I will integrate this patch into my working branch with gradle changes. I don't think I can fix everything related to module vs. classpath but I think it's close to being reasonably useful. Merry Christmas!

@rmuir - I do have tests that fail because of this. These tests are on a branch where I'm trying to fix classpath vs. module-path issues - the compilation and tests currently use classpath only; once you make them run with true modules, things start to break. It is very messy conceptually - the conventions gradle has predate the module system and they're not always compatible. Every time I think I've solved something, two other things emerge. It is difficult. I am making progress though.

Thanks Dawid. I didn't mean to complain, I was just confused at the sudden issues and wondering about the larger strategy to fix stuff. At this point I'm not even sure tests will be enough, due to the fact lucene is an API (see my example above about RSLP, first one i looked into). So I'm trying to think of other ways to make the build fail when our APIs do this.

The code Uwe provided in the patch is fixing a true bug - it's just masked by the tests running in classpath only. In other words: if the tests pass after applying this patch, nothing changes for the classpath mode... and things will improve on the module-system layer side.

Sure, I get that. But If i revert his commit or otherwise introduce regression, tests keep passing :) That's my point.

bq. But If i revert his commit or otherwise introduce regression, tests keep passing :) That's my point.
Yeah, I know. It's a vicious cycle though - I can't really progress refactoring the build to use modules if I am blocked by failures; I'm never sure if it's it's a regression I introduced or something else. I just filed an issue for this one because I knew it was a real problem. We don't have to merge it yet to main - I'll merge this from Uwe's branch.

Well, it does seem this particular method signature "can't work", unless any code that uses this helper explicitly opens up their module to lucene-core... and that's too confusing.
As far as what other stuff is broken by modules, that's my bigger concern: how can we test this? what else is broke? In this example it is literally just a couple words of javadocs that they slid into this method with jdk9 and now we have to deprecate APIs because of it.

how can we test this?

This is a problem because the code and tests can effectively run in many combinations. I don't have a clear picture of how this can be all reasonably arranged.

I merged the main branch to keep this up-to-date and fetch the change in luke module https://issues.apache.org/jira/browse/LUCENE-10340.

Browsing through this issue, I am wondering if it'd be helpful that we have an entry to forbid Class#getClassLoader() by default. When it is really needed (for example, on tests), we can suppress the linter by @SuppressForbidden.
lucene $ git diff
diff --git a/gradle/validation/forbidden-apis/defaults.lucene.txt b/gradle/validation/forbidden-apis/defaults.lucene.txt
index 0cc4edda6bd..a145000feb7 100644
--- a/gradle/validation/forbidden-apis/defaults.lucene.txt
+++ b/gradle/validation/forbidden-apis/defaults.lucene.txt
@@ -47,3 +47,6 @@ java.util.Arrays#copyOf(double[],int)
 java.util.Arrays#copyOf(boolean[],int)
 java.util.Arrays#copyOf(java.lang.Object[],int)
 java.util.Arrays#copyOf(java.lang.Object[],int,java.lang.Class)
+
+@defaultMessage Accessing class loader can be problematic with Java module system in combination with caller sensitive method calls such as getResource()
+java.lang.Class#getClassLoader()

# on main branch
lucene $ ./gradlew -p lucene/analysis/kuromoji/ forbiddenApi
> Task :lucene:analysis:kuromoji:forbiddenApisMain FAILED
Forbidden method invocation: java.lang.Class#getClassLoader() [Accessing class loader can be problematic with Java module system in combination with caller sensitive method calls such as getResource()]
  in org.apache.lucene.analysis.ja.dict.BinaryDictionary (BinaryDictionary.java:208)

I fixed the remaining deprecated method in StopwordAnalyzerBase.

Browsing through this issue, I am wondering if it'd be helpful that we have entries to forbid ~Class#getClassLoader()~ ClassLoadr#getResourceAsStream() (and so on) by default. When it is really needed (for example, on tests), we can suppress the linter by @SuppressForbidden.
Correction: not all methods in ClassLoader are caller sensitive (returns null when the package is not opened). getClassLoader() call itself may not be a problem.
lucene $ git diff
diff --git a/gradle/validation/forbidden-apis/defaults.all.txt b/gradle/validation/forbidden-apis/defaults.all.txt
index 6c6e0296cc5..62b1e8fa6e8 100644
--- a/gradle/validation/forbidden-apis/defaults.all.txt
+++ b/gradle/validation/forbidden-apis/defaults.all.txt
@@ -66,3 +66,6 @@ java.io.ObjectOutputStream
 @defaultMessage Don't set a dictionary on a Deflater using a method that takes an offset or ByteBuffer (JDK-8252739)
 java.util.zip.Deflater#setDictionary(byte[],int,int)
 java.util.zip.Deflater#setDictionary(java.nio.ByteBuffer)
+
+@defaultMessage caller sensitive method callq
+java.lang.ClassLoader#getResourceAsStream(java.lang.String)

# on main branch
lucene $ ./gradlew -p lucene/analysis/kuromoji/ forbiddenApi
> Task :lucene:analysis:kuromoji:forbiddenApisMain FAILED
Forbidden method invocation: java.lang.ClassLoader#getResourceAsStream(java.lang.String) [caller sensitive method callq]
  in org.apache.lucene.analysis.ja.dict.BinaryDictionary (BinaryDictionary.java:207)

The Classloader one should be forbidden, but Class#getResourceAsStream() should always be preferred, but we should make sure that we do  not use it in public utility methods that can be called from foreign code.
The code in dictionary loading should be fixed and we should completely get rid of those setting/enum classpath/file loading. Just add a method/ctor that takes an InputStream and done.

Correction: not all methods in ClassLoader are caller sensitive (returns null when the package is not opened). getClassLoader() call itself may not be a problem.

We should still not use getClassLoader as this requires higher security permissions. So avoid if possible! Then it might be possible to remove the getClassLoader security policy.

FYI, I did some tests / investigation today to work around the caller sensitiveness:

Use getResource() on the class file (use simple class name with .concat(".class")) to get the URL of the class passed into the IOUtils method
Then use a relative URL resolve to get the real file.

After some discussion with @dweiss and own tests, I found this is a "bad" workaround:

It won't work in non-module mode if classes es separate from resources (gradle layout)
We should not "work around" the module system (although it works)

This code is more repetition, but much more clear and is written like it should be done with module system. The static ctor should load the resource on its own and not pass it to a method in Lucene Core.
I only have some suggestions for future: Add more convenience methods to WordListLoader, so you can easily load wordlists from UTF-8 enoded input streams with default comment character ("#"). This would remove a few extra parameters repeated in every Analyzer class.

Hi Uwe. I took a closer look at the cross-module resource accessibility. So in fact resources could be accessed between modules if their packages are open (or if the entire module is open). Here is a repro (Windows) that shows this by force-opening the package with an internal resource:
https://github.com/dweiss/exp-module-resource-access
if you run it, you'll see something like this:
{code}
Classpath mode:
A.class.getResource("a.resource"): file:/C:/_tmp/repro/module_a/a/a.resource
Module mode:
A.class.getResource("a.resource"): null
Module mode with forced opens:
A.class.getResource("a.resource"): file:/C:/_tmp/repro/./module_a/a/a.resource
{code}
The same could be achieved by adding "opens a" or just declaring the module as open. While this does somewhat-work, I think changing the Lucene API is a better solution because it doesn't require people to open up entire packages (to reflection and resource-access) just so that IOUtil can open a single file. Yes, opens can be qualified at module-descriptor level but it introduces a level of sophistication I find hardly justifiable.

I added a ModuleResourceLoader and added documentation to ClassPathResourceLoader for usage with modules:

you need to add opens yourpackage_with_resources to org.apache.lucene.core; to your module descriptor
with modules it is recommended to use the new class, not ClasspathResourceLoader

@dweiss The integration test perfectly shows what needs to be done

The same could be achieved by adding "opens a" or just declaring the module as open. While this does somewhat-work, I think changing the Lucene API is a better solution because it doesn't require people to open up entire packages (to reflection and resource-access) just so that IOUtil can open a single file. Yes, opens can be qualified at module-descriptor level but it introduces a level of sophistication I find hardly justifiable.

Yes this PR fixes it. There is one place where applications may need resources: when they use analyzer builders, and there they can now use ModuleResourceLoader instead of ClasspathResourceLoader. But they have to declare in their module descriptor that the packages containing resources are open TO org.apache.lucene.core.
There are some remaining issues with code doing: Class.getClassLoader().getResource.... This code does not even work from inside the same module and always returns null. In most cases this is just a bug: From what I see it only affects BinaryDictionary in Kuromoji and Nori. As said before this should be refactored to not have this crazy File vs. Resource options from former times not applicable to Lucene anymore.

There are some remaining issues with code doing: Class.getClassLoader().getResource.... This code does not even work from inside the same module and always returns null. In most cases this is just a bug: From what I see it only affects BinaryDictionary in Kuromoji and Nori. As said before this should be refactored to not have this crazy File vs. Resource options from former times not applicable to Lucene anymore.

There are two problems and I think we can handle them separately.
As for the first one, "Class.getClassLoader().getResource....", all resources (dictionary data) contained in kuromoji and nori are placed in "org/apache/lucene/analysis/[ja|ko]/dict/", so they can be loaded by org.apache.lucene.analysis.[ja|ko].dict.BinaryDictionary#getResourceAsStream() by tweaking the paths a bit.
Quick fix is here and it passes tests.
https://github.com/mocobeta/lucene/commit/5432b09c980f029d03c6271b3ab7f50d0a9842e7
As for the second one, "File vs. Resource options", resolving this is not much trivial to me. I think another issue is needed for it.

Yeah - we came up with the same explanation independently, Uwe. In my opinion a simpler API that does not require special module-info clauses is a preference (if it can be done in a different way, avoiding that opens clause) - it took us a moment to realize why it doesn't work and it'll be a constant source of frustration to developers (and nobody reads the docs...).

Yeah - we came up with the same explanation independently, Uwe. In my opinion a simpler API that does not require special module-info clauses is a preference (if it can be done in a different way, avoiding that opens clause) - it took us a moment to realize why it doesn't work and it'll be a constant source of frustration to developers (and nobody reads the docs...).

That was funny, indeed. I did not see your message, so sorry for pinging you in chat at midnight :-)
Actually, except the dictionary stuff, all is solved here by deprecating the helper APIs and adding more convenience methods to WordListLoader taking InputStreams (with charset or default to UTF-8).
I also added a test that shows that resource loading won't work with non-open packages (exporting is not enough, it must be OPEN). Stupidly when reading the JDK code the naiming there is horrible: all the terms "exported", "open", "readable" are so close together and you simply get irritated, especially as the code is no deep. The main problem with JDK's naiming is that they never use "exported" in the API, only "module can read" and also "open" at other places. This is all a desaster as it makes it too hard to understand what's going on.
This PR also add for the CustomAnalyzers used in Solr/ES/Luke the ResourceLoadervinterface specialized for the Module system (see above). If you could have a look at that, please review!

Hi @mocobeta,

There are some remaining issues with code doing: Class.getClassLoader().getResource.... This code does not even work from inside the same module and always returns null. In most cases this is just a bug: From what I see it only affects BinaryDictionary in Kuromoji and Nori. As said before this should be refactored to not have this crazy File vs. Resource options from former times not applicable to Lucene anymore.

There are two problems and I think we can handle them separately.
As for the first one, "Class.getClassLoader().getResource....", all resources (dictionary data) contained in kuromoji and nori are placed in "org/apache/lucene/analysis/[ja|ko]/dict/", so they can be loaded by org.apache.lucene.analysis.[ja|ko].dict.BinaryDictionary.class#getResourceAsStream() by tweaking the paths a bit. A quick fix is here and it passes tests. I can make a PR or directly commit it into this branch, if you are okay with that. mocobeta@5432b09
As for the second one, "File vs. Resource options", resolving this is not much trivial to me. I think another issue is needed for it. I am sure we can deprecate protected BinaryDictionary(ResourceScheme resourceScheme, String resourcePath) and break it down into two constructors - one for resources from the class-path (no-arg constructor), another for external resources from the user's file system.

The problem with your patch is that you change public API to suddenly use relative resource names in public ctors. Instead we should deprecate those and add InputStream taking ctors. I will take care of this. I wanted to do this already 10 years ago when Robert and I imported the Kuromoji code, but the number of tests affected by this was too large. By deprecting I can keep the tests unmodified at beginning and let's cleanup later.
@dweiss / @mocobeta: Please make sure that you use the latest version of this PR, as I changed ALL static initializers since the one you merged. To test it outwith your branch I would solve all conflicts when merging by clicking on "theirs (whole file)" in Tortoise or similar.

I changed back to "draft", once I am done, I switch back!
We can then merge to main and or your branch and solve conflicts.

The problem with your patch is that you change public API to suddenly use relative resource names in public ctors. Instead we should deprecate those and add InputStream taking actors.

I am 99% (even though not 100%) sure anyone who uses the 2-args protected constructor uses it for loading external resources on their file system (since I know this was introduced to do so for really advanced users), and most users use no-arg constructor (this implicitly load resources from the default path); but in theory, I agree it can change the behavior of the constructor.

The problem with your patch is that you change public API to suddenly use relative resource names in public ctors. Instead we should deprecate those and add InputStream taking actors.

I am 99% (even though not 100%) sure anyone who uses the 2-args protected constructor uses it for loading external resources on their file system (since I know this was introduced to do so for really advanced users), and most users use no-arg constructor (this implicitly loads resources from the default path); but in theory, I agree it can change the behavior of the constructor.

We should clean that up in a later  PR. The quick fix I will use locally is to add another (possibly hidden) enum constant "CLASS".

The problem with your patch is that you change public API to suddenly use relative resource names in public ctors. Instead we should deprecate those and add InputStream taking actors.

I am 99% (even though not 100%) sure anyone who uses the 2-args protected constructor uses it for loading external resources on their file system (since I know this was introduced to do so for really advanced users), and most users use no-arg constructor (this implicitly loads resources from the default path); but in theory, I agree it can change the behavior of the constructor.

We should clean that up in a later PR. The quick fix I will use locally is to add another (possibly hidden) enum constant "CLASS".

We can fix the problem easier: If an external resource path is passed, we make it absolute by prefixing with a "/". Then I can use your patch easily.
Let me do this - thanks anyways.
Nevertheless, we should get rid of this and just supply a ctor that takes 3 InputStreams for advanced users.

Hi @mocobeta,
I cherrypicked your commit and modified it a bit: To still support full names, it just prefixes all paths with "/". After doing this I was able to revert the test changes which shows that its 100% backwards compatible.
In general a simple hotfix rule for Class#getClassLoader().getResource(xxx) is: Replace by Class#getResource("/".concat(xxx))
The main difference between ClassLoader and Class ist: For Class you can load non-open module-local resources, while ClassLoader ALWAYS needs open packages, also for callers from own module. The reason for that is simple: A ClassLoader can be shared between different modules, so the getResource() methods there have no idea from which module you want to get resources, so they block everything which is not open.
So we should really forbid usage of ClassLoader#getResource...() (separate issue).

In general a simple hotfix rule for Class#getClassLoader().getResource(xxx) is: Replace by Class#getResource("/".concat(xxx))

An alternative is Class#getModule().getResourceAsStream(xxx), which has same behaviour.

@uschindler Thank you. I looked through the changes.
I can't look closely at or review the whole change soon since I have to leave for a while (my father just passed away due to old age and lots of work is waiting for me), so please go ahead. I hope I'll be able to catch up on the progress of the jms support later.

Hi,

@uschindler Thank you. I looked through the changes. I can't look closely at or review the whole change soon since I have to leave for a while (my father just passed away due to old age and lots of work is waiting for me), so please go ahead. I hope I'll be able to catch up on the progress of the jms support later.

That's really bad news! I send you my deepest condolences! Take your time!
Most of this PR is mechanical work, so the only things to "review" are the API changes. I am sure, Dawid will also have a look.
Happy New Year,
Uwe

Thank you, Uwe. I am grateful for your kind concern.
Wish you all a Happy New Year!

@uschindler Thank you. I looked through the changes.
I can't look closely at or review the whole change soon since I have to leave for a while (my father just passed away due to old age and lots of work is waiting for me), so please go ahead. I hope I'll be able to catch up on the progress of the jms support later.

I'm sorry to hear this @mocobeta

@rmuir Thank you, for your concern.

@rmuir Thank you, for your concern.

I am very sorry for your loss, @mocobeta, please accept my condolences.

I am very sorry for your loss, @mocobeta, please accept my condolences.

@dweiss Thank you, I appreciate your concern.

@dweiss Thank you, I appreciate your concern.

Hi @dweiss: I will merge in the changes to your branch, as this is a bit more complicated (I also need to revert some changes on your branch)

No, please don't - I'll do it later.

Oh sorry, I fixed it already. You can revert if you like, but looks well now.

No worries, thanks Uwe.

Hm, now on main this causes an ECJ warning.... But it does not fail build. That's ok for now.

> Task :lucene:core.tests:ecjLintTest
----------
1. WARNING in C:\Users\Uwe Schindler\Projects\lucene\lucene\lucene\core.tests\src\test\module-info.java (at line 27)
        opens org.apache.lucene.core.testresources to
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The package org.apache.lucene.core.testresources does not exist or is empty
----------
1 problem (1 warning)

I will check how to fix this. Its a valid warning because the package has no class files. But that's wanted here for testing purposes.

I fixed the warning in main by a fake "package-info.java"


Published package doesn't work

The current package you get when installing @atixlabs/hardhat-time-n-mine doesn't work, because the built files are not included. I guess you forgot to run yarn build before publishing?

Fixed in version 0.0.5


Support better typehinting of the python armada code from other projects

I think this can be done my packaging the python stubs in when we release on mypy. More investigation needed.

Managed to fix this with no changes to armada. Apoligies!

This now is more import since gRPC version 1.48, since now the python GRPC generated files have no infomation on variables, types etc, so all imports will fail to most linters. By adding the mypy .pyi stubs, this should be prevented.


Number of Contributions reduces to zero on re-login

Summary:
Contibution fragment shows 0 contibutions(both failed and successful on re-login
Steps to reproduce:

First have some contributions either failed or successful
Logout of your account from you app
Log-in to the same account
Your contributions are all vanished

Expected

Previous succcessful as well as faile images must have been retained in the application
Also this empty fragment displays no such message like "You haven't made any contributions yet"

Device and Android version:
Vivo 1713 Android version 7.0
Screen-shots:

Would you like to work on the issue?
Yes

I can verify this happens after re-login. A very weird issue. @madhurgupta10 did you made any progress with this? Did you managed to find the root cause?

This happened to me again. I found it very annoying and obvious, will prioritize it

@neslihanturan I will prioritize it and deep dive to find the root cause to it.

Thanks for your interest @madhurgupta10 but I think it is assigned to @vanshikaarora . @vanshikaarora  can you please verify you are still working on this issue?

I am sorry, for no response since long. I'll start working on this soon.

@neslihanturan I was going through the issue what I noticed was that in the application contributions were added to the ContentProvider using
public void save(Contribution contribution) {
        ContentProviderClient db = clientProvider.get();
        try {
            if (contribution.getContentUri() == null) {
                contribution.setContentUri(db.insert(BASE_URI, toContentValues(contribution)));
            } else {
                db.update(contribution.getContentUri(), toContentValues(contribution), null, null);
            }
        } catch (RemoteException e) {
            throw new RuntimeException(e);
        } finally {
            db.release();
        }
    }

And while  logging out all files except the libraries are removed (including the ContentProviders) using the clearApplicationData()  function of CommonsApplication.java
 File cacheDirectory = context.getCacheDir();
        File applicationDirectory = new File(cacheDirectory.getParent());
        if (applicationDirectory.exists()) {
            String[] fileNames = applicationDirectory.list();
            for (String fileName : fileNames) {
                if (!fileName.equals("lib")) {
                    FileUtils.deleteFile(new File(applicationDirectory, fileName));
                }
            }
        }

Is the above conclusion correct? Please correct me if I am wrong :)

@neslihanturan Can you give please give your suggestion here? :)

@neslihanturan I have gone through the Issue closely.
Reason's for failure

Contribution and other data(like personal preferences in settings) are not fetched through API. They are stored locally in cache, files, databases and other storage's
While logging out we call the delete file method, that delete's the following 8 Files:

cache
code_cache
shared_prefs
files
app_ACRA-unapproved
app_ACRA-approved
lib-main
databases

These files contain data of the contributions and other preferences of the User
Solution
We can avoid deleting the data by removing these methods but then we need to take care of the case that this same information isn't used if the user is logged in using a different account
@neslihanturan Can you please share your opinion? :)

Does this happen just in master, or in an actual release? I experienced the same problem in master, but I have not received any reports from v2.10.2 users about it (and usually users are quite vocal about problems like this ;)).
If we can concur that it just happens in master, it may be a good idea to find out which commit is causing the problem rather than trying to solve from scratch.
BTW, I wonder why number of contributions are not fetched through the API anymore? They used to be IIRC.

@neslihanturan @misaochan Is this issue still there?

I have not noticed it recently, but I'm not sure. Could you test this and let us know what you find @madhurgupta10 ? :)

I would judge this "feature, not a bug", if I've understood it correctly. At least for failed contributions.
We delete our databases so that another user can login ,an unlikely scenario but possible.
To support this we would have to key the data in our databases by user and never delete them.
Previously successful contributions should be accessible from the endpoint though.

Previously successful contributions should be accessible from the endpoint though.

This is what I was thinking. AFAIK we DID have an API for this, but not sure what happened with that. @maskaravivek any idea?

I have not noticed it recently, but I'm not sure. Could you test this and let us know what you find @madhurgupta10 ? :)

I don't face this issue anymore, rather sometimes I get "Error occurred while loading images" error.

Happening no more on 2.13. Also as this is very old, closing this one, feel free to re-open if it happens again :)


Can't stop mongo docker container

I am having trouble to stop running mongo docker container. I got following message when I run "docker stop [container name]", Anybody having same issue?  the mongodb itself seems to be working fine. just can't stop the container.. so I have to restart VM in order to stop the mongo container.
I am using version 2.6 Dockerfile and build and run with docker-compose.
ERROR: for [container name]  HTTPSConnectionPool(host='X.X.X.X', port=2376): Read timed out. (read timeout=70)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

It seems it is related to http connection error with port 2376, but I couldn't figure out why it is happening, other container built and run by docker-compose works just fine and stopped precisely.

:+1:

Any ideas on reproducing this so that we can find the cause?  Are you all on OSX? Is there a common docker version (docker info)? Maybe a common docker-compose configuration?  Has it happened on other containers besides mongo?

@yosifkit yes, I am on OSX. I have updated to the mongo 2.6 image and this still happens. Hopefully this can provide you with better insights:
Using docker-compose 1.5.2, and docker info:
Containers: 8
Images: 73
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 89
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: YAUF:7G7V:ZLQJ:IXXG:2CRC:KPVH:O26R:YYLK:FOUW:BDBS:PX42:C4LW
Debug mode (server): true
 File Descriptors: 27
 Goroutines: 51
 System Time: 2016-01-07T13:49:42.657417574Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

The docker-compose.yml file has a simple python flask application (which exits correctly) and a  simple link to a direct mongo:2.6 image
Is it perhaps that the container subjected to the PID 1 problem ?
https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
Because the logs correctly say that mongo "Is really exiting now" but stays in that state. Here are the docker-container logs:
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] now exiting
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] dbexit:
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: going to close listening sockets...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] closing listening socket: 7
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] closing listening socket: 9
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: going to flush diaglog...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: going to close sockets...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: waiting for fs preallocator...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: lock for final commit...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: final commit...
dockerized_mongodb_1 | 2016-01-07T13:53:11.073+0000 [signalProcessingThread] shutdown: closing all files...
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [signalProcessingThread] closeAllFiles() finished
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [signalProcessingThread] journalCleanup...
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [signalProcessingThread] removeJournalFiles
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [conn4] end connection <IP_ADDRESS>:57284 (1 connection now open)
dockerized_mongodb_1 | 2016-01-07T13:53:11.074+0000 [conn5] end connection <IP_ADDRESS>:57285 (1 connection now open)
dockerized_mongodb_1 | 2016-01-07T13:53:11.076+0000 [signalProcessingThread] shutdown: removing fs lock...
dockerized_mongodb_1 | 2016-01-07T13:53:11.076+0000 [signalProcessingThread] dbexit: really exiting now

The output of docker-compose stop:
ERROR: for brancheserver_dockerized_mongodb_1  HTTPSConnectionPool(host='<IP_ADDRESS>', port=2376): Read timed out. (read timeout=70)

Output of docker-compose ps after stop:
david$ docker-compose ps
               Name                             Command               State      Ports
----------------------------------------------------------------------------------------
[my flask server name... irrelevant]       /branche/init_and_start.sh ...   Exit 0
brancheserver_dockerized_mongodb_1   /entrypoint.sh mongod            Up       27017/tcp

I had this problem in both Ubuntu 14.04.3 VMWare virtual machine (not sharing directories with the host system) and Mac OS X El Capitan (via docker-machine).
Now I'm not having this problem, using version 2.6.1.
I can't replicate this scenario anymore, but I think that could be related with any Linux kernel update on the Docker host (we had this problem with other containers like PHP-FPM).

Same problem on Ubuntu with mongo:2.6:
$ docker --version
Docker version 1.9.1, build a34a1d5
$ docker-compose --version
docker-compose version 1.5.2, build 7240ff3

My mongo definition on docker-compose.yml:
mongo:
  image: mongo:2.6
  volumes:
    - ./docker/conf/mongodb.yml:/etc/mongodb.conf:ro
  command: mongod --config /etc/mongodb.conf

The mongodb.yml file:
storage:
  journal:
    enabled: false

Sometime, mongo process got 100% CPU usage and can be stopped. I tried (in order):
docker-compose stop (^C)
sudo service docker stop # Docker service is stopped, mongo container still running.
kill -9 <mongo_process>

This container is like Chuck Norris.
Let me know if you need additional information.

Ok I think I found where is the problem.
On my mongo db configuration:
storage:
  journal:
    enabled: false

I disabled journal storage.
If I set it to true or just not specifying configuration, the issue is gone.
So we know where is the problem, but not why.

I have consistently had this problem for a few weeks now. It showed up when I upgraded from Docker 1.8.4 to Docker 1.9.x. I'm on a Mac using vmwarefusion or virtualbox. With or without shared file system support. I'm also using docker-compose 1.5.2. Happens with mongo versions 2.6 and 2.6.1. Happens with --journal and --no-journal.  The command I run is mongod --smallfiles.
This bug occurs during CI builds of my somewhat complicated product. I can't reproduce it by simply starting/stopping the container. I am working on making a smaller config to demonstrate the problem.
The "PID 1 problem" hypothesis sounds plausible, from what little I know. I'm running out of things to try so I might merge the mongo 2.6 image with phusion/baseimage and see what happens.

@damonomad If you want to test if it is the "PID 1" problem, then you shouldn't need all of phusion/baseimage, just use tini.

After much debugging this is an aufs bug. Which means it will be fixed in the newest release (I just tested boot2docker 1.10-rc1).  I was also able to change to overlay and was not able to hit this stopping problem.

See https://github.com/docker/docker/issues/18180#issuecomment-173436661 for where the current fixes are.

Upgraded mongo from 2.x to 3.x on ubuntu 14.04 has fixed this for me (so far)
https://docs.mongodb.org/v3.0/tutorial/install-mongodb-on-ubuntu

docker 1.10-rc1 was the definitive fix for me. I can use any version of the mongo image with this Docker. Thanks!

@damonomad you must mean boot2docker 1.10-rc1, which is not the samething as docker 1.10-rc1
Guys, the issue is a AUFS bug,  see https://github.com/docker/docker/issues/18180  at the end for workarounds.

@thefallentree  thanks, yes, I meant boot2docker 1.10-rc1. Which, because I'm on a Mac, I got via the DockerToolbox 1.10-rc1 installer. Upgrading the boot2docker image in my virtual machine was the key thing.

:+1:
I am using running docker on Ubuntu 14.04.3 and have had this problem on 1.7.1 and 1.9.1 as well.

This issue is fixed in boot2docker 1.10-rc1 and above. Thanks

I'm receiving the following error when trying to stop a mongo:2.4.10 image -
An error occurred trying to connect: Post https://<IP_ADDRESS>:2376/v1.21/containers/15e9bb28144b/stop?t=10: EOF
Error: failed to stop containers: [15e9bb28144b]

I'm running the following docker versions:
Docker version 1.9.1, build a34a1d5
docker-machine version 0.5.2 ( 0456b9f )
I've worked for a while with the 3.0.7 mongo image, and had none of this problems. Since I've downgraded the version to suit our production environment better I've started having trouble with stopping the mongo container.
Any idea how to solve this?

@yaron-idan, I believe the only solutions are to downgrade the kernel, update aufs, or use a different graph driver; none of which would be very easy in the boot2docker vm since you want to keep docker 1.9.1.
Are there specific reasons to be using 1.9 vs 1.10?  If your need to do docker cli commands to your production systems, you could downgrade your docker client on your host to 1.9.1 while running the 1.10 daemon in the VM (older clients can talk to newer daemons, while new clients cannot talk to old daemons).

I was actually not aware of the new docker version, I've updated it and I think it solved the problem. Thanks for the help.

I am also having the same issue. Mongo DB container is not stopping.
Output of sudo docker-compose down command :-
Stopping mongoServer ...
ERROR: for mongoServer  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=70)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
=====================================================================
Output of uname -a command
Linux ubuntu 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
=====================================================================
Output of docker info command
Containers: 3
Running: 1
Paused: 0
Stopped: 2
Images: 15
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 34
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.797GiB
Name: ubuntu
ID: QKF4:ZEQB:UUZQ:QK5G:6HKD:UXVB:AYIK:X5DV:OE2R:7XLZ:HFW6:E3HG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
<IP_ADDRESS>/8
Live Restore Enabled: false
WARNING: No swap limit support
=====================================================================
Output of docker-compose -version command
docker-compose version 1.11.2, build dfed245
Please help me to solve this issue.

Closing old issue, this should be fixed by using a newer docker daemon (1.10+). Feel free to comment on further troubleshooting if you feel this was closed in error or on the solution to help users running into the same problem.

Still reproduces on:
Linux IBelieveICanFly 4.2.0-23-generic #28~14.04.1-Ubuntu SMP Thu Dec 31 13:40:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
And last docker version.

@yosifkit I have the same problem:
$ uname -a
Linux ws212.open.ru 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u3 (2016-01-17) x86_64 GNU/Linux

$ docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:46 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:39:27 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 8 (jessie)"
NAME="Debian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=debian

$ ps fax | grep mongo -n3
117- 1053 ?        Ssl    1:31 /usr/bin/dockerd -H fd://
118- 1177 ?        Ssl    0:49  \_ docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
119- 5892 ?        Sl     0:00      \_ docker-containerd-shim ecd4b0b52c52847383dc784b692037bfb43a0e86d0c6aa94e81f5584862c2771 /var/run/docker/libcontainerd/ecd4b0b52c52847383dc784b692037bfb43a0e86d0c6aa94e81f5584862c2771 docker-runc
120: 5939 ?        Zsl   55:59          \_ [mongod] <defunct>

...

143- 1892 pts/0    Ss     0:00          |   \_ /bin/bash
144: 5822 pts/0    Sl+    0:00          |   |   \_ docker run mongo
145- 6020 pts/1    Ss     0:00          |   \_ /bin/bash
146- 6098 pts/1    Sl+    0:00          |   |   \_ docker stop blissful_raman

Note: the process is a zomby.
Mongo logs:
$ docker run mongo
2017-11-24T12:45:21.893+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=ecd4b0b52c52
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] db version v3.4.10
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1t  3 May 2016
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] modules: none
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] build environment:
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten]     distmod: debian81
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten]     distarch: x86_64
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2017-11-24T12:45:21.894+0000 I CONTROL  [initandlisten] options: {}
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten]
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2017-11-24T12:45:21.897+0000 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7473M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten]
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten]
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten]
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2017-11-24T12:45:21.918+0000 I CONTROL  [initandlisten]
2017-11-24T12:45:21.926+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2017-11-24T12:45:21.930+0000 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { version: 1 }, name: "incompatible_with_version_32", ns: "admin.system.version" }
2017-11-24T12:45:21.930+0000 I INDEX    [initandlisten] 	 building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2017-11-24T12:45:21.931+0000 I INDEX    [initandlisten] build index done.  scanned 0 total records. 0 secs
2017-11-24T12:45:21.931+0000 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 3.4
2017-11-24T12:45:21.931+0000 I NETWORK  [thread1] waiting for connections on port 27017
2017-11-24T12:45:35.611+0000 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] closing listening socket: 6
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] closing listening socket: 7
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2017-11-24T12:45:35.611+0000 I NETWORK  [signalProcessingThread] shutdown: going to flush diaglog...
2017-11-24T12:45:35.611+0000 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2017-11-24T12:45:35.613+0000 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2017-11-24T12:45:35.636+0000 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2017-11-24T12:45:35.636+0000 I CONTROL  [signalProcessingThread] now exiting
2017-11-24T12:45:35.636+0000 I CONTROL  [signalProcessingThread] shutting down with code:0

But the process is not finished and a command docker stop too.
And the most interesting thing is I can stop and remove this container only after a hard reset (by button) my machine.

I had the same problem with Mongo on Ubuntu 18.04.1 using this Dockerfile: https://github.com/docker-library/mongo/blob/6932ac255d29759af9a74c6931faeb02de0fe53e/4.0/Dockerfile
The solution turned out to be a Snap instance of docker installed during the Ubuntu OS setup, running alongside the one I installed via apt-get after the initial installation.
sudo snap remove docker


Remove cluster-autoscale from travis build.

Limit travis build to master and quota-management

Thank you!


ListView column widths

Category
[ ] Enhancement
[X] Bug
[ ] Question
Version
Please specify what version of the library you are using: [1.10.0]
Expected / Desired Behavior / Question
When I use the ListView control the spacing of the columns isn't great. It seems that the first column always gets about 2/3 of the overall width, and each of the other columns is squished to the right.
Here's an example screenshot.

My ViewFields looks like this:
  private ViewFields: IViewField[] = [
    {
      name: "ManufacturingMaterial.Title",
      displayName: "Manufacturing Material"
    }, {
      name: "LotNumber",
      displayName: "Lot #"
    }, {
      name: "QuantityType.Title",
      displayName: "Quantity Type"
    }, {
      name: "Quantity",
      displayName: "Quantity"
    }
  ];

and here's the render function:
  public render(): React.ReactElement<IViewMaterialProps> {
    return (
      <div>
        <ListView
          items={this.state.requestMaterial}
          viewFields={this.ViewFields}
          compact={true}
        />
      </div>
    );
  }

I've tried to remove as many settings as possible to reduce complexity, but always get a similar result. I'm also seeing similar results with different items in different components.
To maintain a responsive UI, I'd expect the columns to autospace like they do in the OOB list vies.

This is a default behavior of underlying DetailsList even in the latest version of Fluent UI.


Add offline scenario

DO NOT MERGE THIS.
This is just an example PR to show how to add a new task / benchmark (related to #304)

CI test on iOS failed because for iOS we have a config screen to choose batch size, so the batch_size from benchmark_setting is not used. The test model mobiledet.tflite used here seems to not support batching.


URP support??

hi dear,
does any plan to URP support?

This is my half baked attempt to port it, in URP 10 it could be improved by using the built-in DN texture rather than one generated manually
NNAO + DN.zip

This is my half baked attempt to port it, in URP 10 it could be improved by using the built-in DN texture rather than one generated manually
NNAO + DN.zip


Made some rundeck framework properties configurable.

This will allow us to set the rundeck server URL and the rundeck URL with the FQDN of the rundeck server. Otherwise once we login to the web console it will redirect to http://localhost:4440 which will obviously not work.

are you using RD via apache?  It should proxy that for your correctly.  Unless RD changed something, recently.

Well I am using RD without apache...With apache its throwing error like I don't have permission to access ...I shall check again(maybe because I changed the localhost to the hostname of the server in frameworks.properties configuration it is throwing error).

I changed the framework.properties file to point to localhost.. But something apache is blocking...I could not fix that problem. I am seeing below error
================
Forbidden
You don't have permission to access / on this server.

How about changing localhost in rundeck apache config to the FQDN of the rundeck server. In that case the system sill work both with and without apache reverse proxy. What do you say?
In my case somehow the apache configuration is not working. So I can work without the reverse proxy

@pcross616 Any thought on this?

testing this.. :)

cool :)


Run a seed.ts file when resetting the database locally

Is your feature request related to a problem? Please describe.
Currently, the Supabase CLI auto-seeds data when a seed.sql file is present in the Supabase directory. This is easy enough when starting up, but it can quickly grow out of control when lots of data must be seeded.
Describe the solution you'd like
We have a typescript script that can seed the data for us, but we have to manually call it every time after resetting the database. It would be great if the CLI also auto-calls the seed.ts file (we're using Deno) so that the seed process can run smoothly. This also helps do more complex stuff like randomly generated data or complex seeding logic impossible in simple SQL files.
Describe alternatives you've considered
Set up local process scripts that run both separate processes back to back.

We've manages this by using our own dev tools to run the deno script.

We've manages this by using our own dev tools to run the deno script.


Impossible to login Expo xde

Hello all,
I downloaded and installed Expo xde but when I try to connect with github and with my login, he puts me this: EACCES: permission denied, open '/Users/Baloo/.expo/state.json'
I have already uninstalled and reinstalled several times and even deleted the .json but nothing makes it I always have the same error.
I am running on OS X 10.9.5.

Hi! As the issue template requests, please file issues against https://github.com/expo/expo.
I'd recommend making sure that your user has write permission to your home directory, perhaps by running something like sudo chown -R Baloo:Baloo ~/.expo (WARNING: this might not work, it might cause problems on your machine, and you should understand what chown does before running it as root).

It's work !
it was sudo chmod -R 777 .expo/
Thank you very much for your help :D
2017-12-06 20:39 GMT+01:00 Adam Perry<EMAIL_ADDRESS>
Hi! As the issue template requests, please file issues against
https://github.com/expo/expo.
I'd recommend making sure that your user has write permission to your home
directory, perhaps by running something like sudo chown -R Baloo:Baloo
~/.expo (WARNING: this might not work, it might cause problems on your
machine, and you should understand what chown does before running it as
root).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/expo/exp/issues/90#issuecomment-349751608, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AgoOhNkO4nd7xamPXLbghO8MgokcKEswks5s9u2EgaJpZM4Q4WjA
.


merge alpha to master

:tada: This PR is included in version 4.0.2 :tada:
The release is available on:

4.0.2
GitHub release

Your semantic-release bot :package::rocket:


Yarn Versions Sometimes Fail to Load

Sometimes this route will load the data I expect, but sometimes it will not. Refreshing the browser seems to randomly provide valid or invalid responses. This happens for Yarn and NPM, but not the others, as far as I can tell.
https://semver.io/yarn.json
https://semver.io/npm.json
{"all":[],"stableVersions":[]}

{"stable":"1.16.0","unstable":"1.16.0","all":["0.1.0","0.1.1","0.1.2","0.1.3","0.15.1","0.16.0","0.16.1","0.17.0","0.17.2","0.17.3","0.17.4","0.17.5","0.17.6","0.17.7","0.17.8","0.17.9","0.17.10","0.18.0","0.18.1","0.18.2","0.19.0","0.19.1","0.20.0","0.20.3","0.20.4","0.21.0","0.21.1","0.21.2","0.21.3","0.22.0","0.23.0","0.23.1","0.23.2","0.23.3","0.23.4","0.24.0","0.24.1","0.24.2","0.24.3","0.24.4","0.24.5","0.24.6","0.25.1","0.25.2","0.25.3","0.25.4","0.26.0","0.26.1","0.27.0","0.27.1","0.27.2","0.27.3","0.27.4","0.27.5","0.28.1","0.28.4","1.0.0","1.0.1","1.0.2","1.1.0","1.2.0","1.2.1","1.3.1","1.3.2","1.4.0","1.5.0","1.5.1","1.6.0","1.7.0","1.8.0","1.9.1","1.9.2","1.9.4","1.10.0","1.10.1","1.11.0","1.11.1","1.12.0","1.12.1","1.12.3","1.13.0","1.14.0","1.15.0","1.15.1","1.15.2","1.16.0"],"stableVersions":["0.1.0","0.1.1","0.1.2","0.1.3","0.15.1","0.16.0","0.16.1","0.17.0","0.17.2","0.17.3","0.17.4","0.17.5","0.17.6","0.17.7","0.17.8","0.17.9","0.17.10","0.18.0","0.18.1","0.18.2","0.19.0","0.19.1","0.20.0","0.20.3","0.20.4","0.21.0","0.21.1","0.21.2","0.21.3","0.22.0","0.23.0","0.23.1","0.23.2","0.23.3","0.23.4","0.24.0","0.24.1","0.24.2","0.24.3","0.24.4","0.24.5","0.24.6","0.25.1","0.25.2","0.25.3","0.25.4","0.26.0","0.26.1","0.27.0","0.27.1","0.27.2","0.27.3","0.27.4","0.27.5","0.28.1","0.28.4","1.0.0","1.0.1","1.0.2","1.1.0","1.2.0","1.2.1","1.3.1","1.3.2","1.4.0","1.5.0","1.5.1","1.6.0","1.7.0","1.8.0","1.9.1","1.9.2","1.9.4","1.10.0","1.10.1","1.11.0","1.11.1","1.12.0","1.12.1","1.12.3","1.13.0","1.14.0","1.15.0","1.15.1","1.15.2","1.16.0"],"updated":"2019-06-06T16:07:40.066Z"}

This was causing our CI builds to fail.

Looks like the source of information is not accessible anymore:
curl https://skimdb.npmjs.com/
curl: (7) Failed to connect to skimdb.npmjs.com port 443: Connection refused

To bypass the issue, you need to specify the yarn version in your package.json:

I don't think this service is actively maintained any more. The Heroku team now uses another service for resolving versions, but I forget what it's called... perhaps @jmorrell can point you in the right direction.

NPM suggests to use replicate.npmjs.com  instead of skimdb.npmjs.com so we should switch to this version

The Heroku team now uses another service for resolving versions, but I forget what it's called...

We've now moved this logic into a binary in the buildpack itself to avoid relying on an external service

a binary in the buildpack itself

How does this binary know about all the Node.js versions? Do you have some automation in place to keep the binary up to date?

How does this binary know about all the Node.js versions? Do you have some automation in place to keep the binary up to date?

Currently it lists the S3 bucket, which has automations for new versions. Eventually there should be a manifest in the buildpack that a bot will open PRs against, but it might be a while before I get around to building those automations

https://github.com/heroku/semver.io/pull/69

I resolved this on my end by updating the buildpack to one that no longer uses semver.io: https://github.com/heroku/heroku-buildpack-ember-cli

This project + the semver.io site which runs it has been deprecated for some time, and due to be switched off soon - so it should no longer be used.


Crash with pivot_table and bcolz compressed hdf

The following code execution is fine (size is 1.7go HDF text+float), No Crash
############## 1.7 Go NO COMPRESS ####################################
df = dd.read_hdf( 'file_hdf_table_NOcompress.h5', key='df')
df.head()
df['company']= df['company'].astype('category')  ###Put in category some columns
df_pivot= df.pivot_table(index='brand', columns='company', values='purchaseamount', aggfunc='sum')
df_pivotr= df_pivot.compute()

But, I use the compressed file (bcolz compressed), pivot_table is crashing (Python.exe is crashing, so no error message). Total Memory usage is 2.2Go RAM on a 12GO RAM machine.
############## 1.7 Go NO COMPRESS ####################################
df = dd.read_hdf( 'file_hdf_table_bcolz_compress.h5', key='df')
df.head()
df['company']= df['company'].astype('category')  ###Put in category some columns
df_pivot= df.pivot_table(index='brand', columns='company', values='purchaseamount', aggfunc='sum')
df_pivotr= df_pivot.compute()

dask 0.13
pandas 0.19.2

Not sure about the relevance, but:
(1) I remember when I had a C library that caused a segfault, there was no error message in IPython, but if I simply started python in the terminal and executed my code there, I got an error message. Did you check if there might be an error message in the terminal?
(2) Also did you try to run the test suite of the pytables package to figure out, if everything is ok with your pytables setup?
import tables
tables.test()

Forget to add that:
    df = pandas.read_hdf( 'file_hdf_table_bcolz_compress.h5', key='df')
works well and I generate pivot table on it (after load it in memory).
Running this code takes (over 15mins...) .
import tables
tables.test()

Additionnal point (beyond this issue reporting).
To make it more efficient bug/issue report --> Check --> debugging/improvement as well as
to be better 'Bullet proof' tested on real user cases, propose the following:
Create a large "open directory" where  user put  :
1) Create sub-folder
2) In that folder create :
1) test_script.py
2) data folder when all the data are put (txt, hdf, ....)
3) .txt file when the problem is described.
To validate the folder, 'user 'or 'batch' has to run it and print output.
So,  you will get :
Community working for you to create real user case.
Reduce the number of regression testing you have to setup.
Bug/Issue reporting can be automatized....
You can even analyze the issues by Machine Learning models....
   You can focus on 'solving the engineering problem' rather 're-testing'.....

Your tought are welcome !

I believe this is an issue with thread safety in hdf5. This should be handled by setting lock=True in dd.read_hdf. Without a reproducible test case I'm not sure how we can help debug here.
I'm closing this for now as stale (sorry for just getting to it now). If you're still having this issue and have a reproducible test case you can share, please feel free to reopen.


Starting with node app.js throws error as of 1.2.3

Node version: 10
Sails version (sails): 1.2.3
ORM hook version (sails-hook-orm): 2.1.1
Sockets hook version (sails-hook-sockets): 2.0.0
Organics hook version (sails-hook-organics): N/A
Grunt hook version (sails-hook-grunt): 4.0.1
Uploads hook version (sails-hook-uploads): N/A
DB adapter & version (e.g. sails-mysql@5.55.5<EMAIL_ADDRESS>Skipper adapter & version (e.g. skipper-s3@5.55.5): N/A

When starting the app with node app.js instead of sails lift the app throws the following error:
To run an app using `node app.js`, you usually need to have a version of `sails` installed in the same directory as your app.
To do that, run `npm install sails`

Alternatively, if you have sails installed globally (i.e. you did `npm install -g sails`), you can use `sails lift`.
When you run `sails lift`, your app will still use a local `./node_modules/sails` dependency if it exists,
but if it doesn't, the app will run with the global sails instead!

I confirmed that sails is installed in the same directory of the app with nem install sails. This was working on version 1.0.2, and when updating to the latest version (1.2.3 at the time of this writing) this started occurring. Any ideas?
Thank you!
EDIT: Forgot to mention that when starting with sails lift the app starts correctly. On version 1.1.0 it's also working. I would assume some change on 1.2.x introduced this issue?

@joaomvfsantos - attempted to reproduce this with Sails v1.2.3 / node v11.13.0 - fresh sails app - and sails is lifting fine with node app.js:

I see you mentioned nem install sails did you mean npm install sails?  Have you tried a global update of sails with npm install sails -g?  Possibly reinstalling/updating node.js may be worth a try too.

@johnabrams7 thank you for your quick response. the nem install sails was actually just a typo when writing here on GitHub. Sorry for that.
So I continued investigating and realised that on my app.js I had the following require require('sails/node_modules/rc') which I realised does not exist anymore and was moved to sails/accessible/rc. This was the reason of failing to load.


ExtractInputType recommendation not appearing in nested graphql folder

Hi,
The ExtractInputType: include only mutations fix only works if the graphql folder is in the top directory.
I'd imagine many place their graphql folder in the rails app directory, leading to this recommendation not occurring.
I'd opt for placing a wild card before the below include so the graphql folder is picked up whether its nested or not.
config/default.yml:76
Include:
    - 'graphql/mutations/**/*.rb'

Include:
    - '**/graphql/mutations/**/*.rb'

Hey @arenclissold, may I ask you to make a PR with this change? 🙂

Yeah, can do
I tried pushing my branch to the remote but got permission denied. Do you need to give me access to push PRs?

@arenclissold  you need to  fork the repo, make branch there, push to it and GitHub will let you create a PR to this repo


"ProjectTo<TSource, TResult>(context)" from package HotChocolate.Data.AutoMapper does not work with EF when applying filters

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug
Problem
The where filter is not working properly when using projections. It only allows for filtering on fields that are being requested on the Query.
This works:
query {   persons(where: {lastNameDto:{contains:"M"}}) {     lastNameDto   } }
This does not work:
query {   persons(where: {lastNameDto:{contains:"M"}}) {     nameDto   } }
Context
We are returning a DTO from a Query using Filters and Projections.
To project the object from the Entity to the DTO we are using the suggested ProjectTo method from HotChocolate.Data.AutoMapper and we are also configuring the Explicit Expansion on AutoMapper.
Steps to reproduce
Simple App to Repro

Download or clone this
Edit connection string accordingly.
Run the project
Run the following Query on BCP:
query { persons(where: {lastNameDto:{contains:"M"}}) {  nameDto } }

Repro step if you don't want to use the demo project
### prerequisites

Setup the basic bootstrapping of HC with Entity Framework and AutoMapper.
Enable Filtering and Projection.
Make sure to use Explicit Expansion when mapping the DTO.

### Repro

Create a Query with a method that returns an IQueriable<DTOType>.
Add the UseProjection and UseFiltering attributes to the method.
On the Query fetch the entities using EF.
Project the of IQueriable into an IQueriable using the ProjectTo method from  HotChocolate.Data.AutoMapper

For more details please check the project.
Relevant log output
{
  "errors": [
    {
      "message": "Unexpected Execution Error",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "persons"
      ],
      "extensions": {
        "message": "The LINQ expression 'DbSet<Person>()\r\n    .Where(p => new PersonDto{ NameDto = p.Name }\r\n    .LastNameDto != null && new PersonDto{ NameDto = p.Name }\r\n    .LastNameDto.Contains(__p_0))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.",
        "stackTrace": "   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.<VisitMethodCall>g__CheckTranslated|15_0(ShapedQueryExpression translated, <>c__DisplayClass15_0& )\r\n   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)\r\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\r\n   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)\r\n   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)\r\n   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)\r\n   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.Execute[TResult](Expression expression)\r\n   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.System.Collections.IEnumerable.GetEnumerator()\r\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.<>c__DisplayClass58_0.<ExecuteResolverPipelineAsync>b__0()\r\n   at System.Threading.Tasks.Task`1.InnerInvoke()\r\n   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)\r\n   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\r\n--- End of stack trace from previous location ---\r\n   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)\r\n--- End of stack trace from previous location ---\r\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.ExecuteResolverPipelineAsync(CancellationToken cancellationToken)\r\n   at HotChocolate.Execution.Processing.Tasks.ResolverTask.TryExecuteAsync(CancellationToken cancellationToken)"
      }
    }
  ]
}

Additional Context?
Interestingly enough, if the call to .ForAllMembers(x => x.ExplicitExpansion()); is removed from the AutoMapper mapping, the filter starts to work fine, no matter what field is asked for, but projections are not enforced on the database, that is, the SQL SELECT projects all the fields, even when you are only asking for one.
Product
Hot Chocolate
Version
12.6

There is not much we can do here. The expression seems to be too complex for EF to translate.

Thank you for looking into it. I understand it can be too complex.
I would like to ask you, related to this limitation, what approach would you take? or what is the "HotChocolate" way for exposing information in a public API? What I mean is that we are using DTOs so we can encapsulate the underlaying models and have fine control on what we expose to the outside world. Should we just expose the full DB Entity? Is that "normal" or expected in GraphQL? I come from a more traditional RESTapi background so exposing the whole Entity "as is" feels a bit off to me, but I am open to suggestions.

In most cases your GraphQL Types are your DTO:
You can extend them with resolver, remove fields on them (black or whitelist, depends on you prefrence) etc.
You can also change the structure of types and everything.
// annotation based
[ExtendObjectType(typeof(User), IgnoreProperties = {nameofUser.SomeHiddenField} ]
public class UserExtensions

{

   public string DisplayName([Parent] User user) => $”{user.FirstName} {user.LastName}”

}

   //or code first
public class UserType : ObjectType<User>

{
    protected override void Configure(IObjectTypeDescriptor<User> descriptor)

    {

           descriptor.Ignore(x => x.SomeHiddenField);

           descriptor.Field(“displayName”).ResolveWith<Resolvers>(x => x.DisplayName(default));

     }

    public class Resolvers

    {

          public string DisplayName([Parent] User user) => $”{user.FirstName} {user.LastName}”
 ;
    }

}

A custom resolver cannot be sorted or filtered by. But this limitation is with a clean schema design normally no problem. As when you desgin you graphql schema, you should prefere
query GetUsers($search: String!) {

    searchUsers(search: $search) {
        name
        lastName
   }
}

over
query GetUsers($search: String!) {

   users(where: { OR [{ name: {contains: $search}}, {lastName: {contains:$search}}]}) {
        name
        lastName
   }
}


Validate CPF: casting String <-> Number back and forth.

In both calcFirstChecker and calcSecondChecker you expect a Number and then immediately cast it to String, which is no big deal, except that you were the one who casted it to a Number in the first place.
Maybe they should both receive string.
https://github.com/tiagoporto/gerador-validador-cpf/blob/7e65baaca305d0beb505d8cbab75acf98f879fb6/src/js/CPF.ts#L7-L11
https://github.com/tiagoporto/gerador-validador-cpf/blob/7e65baaca305d0beb505d8cbab75acf98f879fb6/src/js/CPF.ts#L110-L113

Hello @tloriato, thanks for your contribution.
Sure, I will take a lock on this.

Hello @tloriato, thanks for your colaboration.
This behavior created a bug #12
I fixed in 4.0.0-beta.1 version

https://4devs.net.br/gerador-cpf


[install]

Have you read and followed the installation tutorial at http://www.nokogiri.org/tutorials/installing_nokogiri.html?

[ x] Yes!

What is the complete output of gem install or bundle install?

$ bundle install
Fetching gem metadata from https://rubygems.org/.......
Resolving dependencies...
Using bundler 2.4.10
Fetching mini_portile2 2.8.2
Fetching racc 1.7.1
Installing mini_portile2 2.8.2
Installing racc 1.7.1 with native extensions
Fetching nokogiri 1.15.3
Installing nokogiri 1.15.3 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ext/nokogiri
/opt/pkg/bin/ruby32 extconf.rb --with-ldflags\=-L.\ -L/opt/pkg/lib\ -L/usr/lib
--with-xml2-include\=/opt/pkg/include/libxml2
checking for whether -std=c99 is accepted as CFLAGS... no
checking for whether -Wno-declaration-after-statement is accepted as CFLAGS... no
checking for whether -O2 is accepted as CFLAGS... no
checking for whether -g is accepted as CFLAGS... no
checking for whether -Winline is accepted as CFLAGS... no
checking for whether -Wmissing-noreturn is accepted as CFLAGS... no
checking for whether -Wshorten-64-to-32 is accepted as CFLAGS... no
checking for whether -Wno-error=unused-command-line-argument-hard-error-in-future is accepted as
CFLAGS... no
checking for whether -Wno-unknown-warning-option is accepted as CFLAGS... no
Building nokogiri using packaged libraries.
Static linking is enabled.
Cross build is disabled.
checking for iconv.h... yes
Using mini_portile version 2.8.2
checking for iconv... yes
---------- IMPORTANT NOTICE ----------
Building Nokogiri with a packaged version of libxml2-2.11.4.
Configuration options: --host\=x86_64-apple-darwin22 --enable-static --disable-shared
--libdir\=/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxml2/2.11.4/lib
--with-iconv\=yes --disable-dependency-tracking --without-python --without-readline --with-c14n
--with-debug --with-threads --disable-shared --enable-static RANLIB\=/usr/bin/ranlib AR\=/usr/bin/ar
CPPFLAGS\= CFLAGS\=-O2\ -U_FORTIFY_SOURCE\ -g\ -fPIC
The following patches are being applied:
  - 0001-Remove-script-macro-support.patch
  - 0002-Update-entities-to-remove-handling-of-ssi.patch
  - 0003-libxml2.la-is-in-top_builddir.patch
  - 0009-allow-wildcard-namespaces.patch
  - 0010-update-config.guess-and-config.sub-for-libxml2.patch
  - 0011-rip-out-libxml2-s-libc_single_threaded-support.patch

The Nokogiri maintainers intend to provide timely security updates, but if
this is a concern for you and want to use your OS/distro system library
instead, then abort this installation process and install nokogiri as
instructed at:

  https://nokogiri.org/tutorials/installing_nokogiri.html#installing-using-standard-system-libraries

Note, however, that nokogiri cannot guarantee compatibility with every
version of libxml2 that may be provided by OS/package vendors.

Extracting libxml2-2.11.4.tar.xz into tmp/x86_64-apple-darwin22/ports/libxml2/2.11.4... OK
Running git apply with
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/patches/libxml2/0001-Remove-script-macro-support.patch...
OK
Running git apply with
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/patches/libxml2/0002-Update-entities-to-remove-handling-of-ssi.patch...
OK
Running git apply with
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/patches/libxml2/0003-libxml2.la-is-in-top_builddir.patch...
OK
Running git apply with
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/patches/libxml2/0009-allow-wildcard-namespaces.patch...
OK
Running git apply with
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/patches/libxml2/0010-update-config.guess-and-config.sub-for-libxml2.patch...
OK
Running git apply with
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/patches/libxml2/0011-rip-out-libxml2-s-libc_single_threaded-support.patch...
OK
Running 'configure' for libxml2 2.11.4... OK
Running 'compile' for libxml2 2.11.4... OK
Running 'install' for libxml2 2.11.4... OK
Activating libxml2 2.11.4 (from
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxml2/2.11.4)...
Using mini_portile version 2.8.2
---------- IMPORTANT NOTICE ----------
Building Nokogiri with a packaged version of libxslt-1.1.38.
Configuration options: --host\=x86_64-apple-darwin22 --enable-static --disable-shared
--libdir\=/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxslt/1.1.38/lib
--disable-dependency-tracking --without-python --without-crypto --with-debug
--with-libxml-prefix\=/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxml2/2.11.4
--disable-shared --enable-static RANLIB\=/usr/bin/ranlib AR\=/usr/bin/ar CFLAGS\=-O2\
-U_FORTIFY_SOURCE\ -g\ -fPIC
The following patches are being applied:
  - 0001-update-config.guess-and-config.sub-for-libxslt.patch

The Nokogiri maintainers intend to provide timely security updates, but if
this is a concern for you and want to use your OS/distro system library
instead, then abort this installation process and install nokogiri as
instructed at:

  https://nokogiri.org/tutorials/installing_nokogiri.html#installing-using-standard-system-libraries

Extracting libxslt-1.1.38.tar.xz into tmp/x86_64-apple-darwin22/ports/libxslt/1.1.38... OK
Running git apply with
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/patches/libxslt/0001-update-config.guess-and-config.sub-for-libxslt.patch...
OK
Running 'configure' for libxslt 1.1.38... OK
Running 'compile' for libxslt 1.1.38... OK
Running 'install' for libxslt 1.1.38... OK
Activating libxslt 1.1.38 (from
/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxslt/1.1.38)...
checking for whether -DNOKOGIRI_PACKAGED_LIBRARIES is accepted as CPPFLAGS... no
xml2-config cflags:
-I/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxml2/2.11.4/include/libxml2
checking for whether -DNOKOGIRI_LIBXML2_PATCHES="\"0001-Remove-script-macro-support.patch
0002-Update-entities-to-remove-handling-of-ssi.patch 0003-libxml2.la-is-in-top_builddir.patch
0009-allow-wildcard-namespaces.patch 0010-update-config.guess-and-config.sub-for-libxml2.patch
0011-rip-out-libxml2-s-libc_single_threaded-support.patch\"" is accepted as CPPFLAGS... no
checking for -llzma... yes
xslt-config cflags:
-I/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxml2/2.11.4/include/libxml2
-I/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ports/x86_64-darwin22/libxslt/1.1.38/include
checking for whether
-DNOKOGIRI_LIBXSLT_PATCHES="\"0001-update-config.guess-and-config.sub-for-libxslt.patch\"" is
accepted as CPPFLAGS... no
checking for xmlParseDoc() in libxml/parser.h... yes
checking for xsltParseStylesheetDoc() in libxslt/xslt.h... yes
checking for exsltFuncRegister() in libexslt/exslt.h... yes
Using mini_portile version 2.8.2
---------- IMPORTANT NOTICE ----------
Building Nokogiri with a packaged version of libgumbo-1.0.0-nokogiri.
Configuration options: --disable-shared --enable-static CFLAGS\=-fPIC
Copying gumbo-parser files into
tmp/x86_64-apple-darwin22/ports/libgumbo/1.0.0-nokogiri/gumbo-parser...
Running 'compile' for libgumbo 1.0.0-nokogiri... OK
Activating libgumbo 1.0.0-nokogiri (from ports/x86_64-darwin22/libgumbo/1.0.0-nokogiri)...
checking for whether
-I/opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ext/nokogiri/ports/x86_64-darwin22/libgumbo/1.0.0-nokogiri/include
is accepted as CPPFLAGS... no
checking for gumbo_parse_with_options() in nokogiri_gumbo.h... yes
checking for xmlHasFeature()... yes
checking for xmlFirstElementChild()... yes
checking for xmlRelaxNGSetParserStructuredErrors()... yes
checking for xmlRelaxNGSetValidStructuredErrors()... yes
checking for xmlSchemaSetValidStructuredErrors()... yes
checking for xmlSchemaSetParserStructuredErrors()... yes
checking for rb_gc_location()... yes
checking for rb_category_warning()... yes
checking for whether -DNOKOGIRI_OTHER_LIBRARY_VERSIONS="\"libgumbo:1.0.0-nokogiri\"" is accepted as
CPPFLAGS... no
creating Makefile

current directory: /opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ext/nokogiri
make DESTDIR\= sitearchdir\=./.gem.20230716-69168-3r8sg6 sitelibdir\=./.gem.20230716-69168-3r8sg6
clean

current directory: /opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3/ext/nokogiri
make DESTDIR\= sitearchdir\=./.gem.20230716-69168-3r8sg6 sitelibdir\=./.gem.20230716-69168-3r8sg6
compiling gumbo.c
clang: warning: argument unused during compilation: '-diag-disable=175,188,1684,2259,2312'
[-Wunused-command-line-argument]
gumbo.c:32:10: fatal error: 'nokogiri_gumbo.h' file not found
#include "nokogiri_gumbo.h"
         ^~~~~~~~~~~~~~~~~~
1 error generated.
make: *** [gumbo.o] Error 1

make failed, exit code 2

Gem files will remain installed in /opt/sandbox/ngtest/bundle/ruby/3.2.0/gems/nokogiri-1.15.3 for
inspection.
Results logged to
/opt/sandbox/ngtest/bundle/ruby/3.2.0/extensions/x86_64-darwin-22/3.2.0/nokogiri-1.15.3/gem_make.out

  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:119:in `run'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:51:in `block in make'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:43:in `each'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:43:in `make'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/ext_conf_builder.rb:41:in `build'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:187:in `build_extension'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:221:in `block in build_extensions'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in `each'
  /opt/pkg/lib/ruby/3.2.0/rubygems/ext/builder.rb:218:in `build_extensions'
  /opt/pkg/lib/ruby/3.2.0/rubygems/installer.rb:852:in `build_extensions'
  /opt/pkg/lib/ruby/3.2.0/bundler/rubygems_gem_installer.rb:72:in `build_extensions'
  /opt/pkg/lib/ruby/3.2.0/bundler/rubygems_gem_installer.rb:28:in `install'
  /opt/pkg/lib/ruby/3.2.0/bundler/source/rubygems.rb:200:in `install'
  /opt/pkg/lib/ruby/3.2.0/bundler/installer/gem_installer.rb:54:in `install'
  /opt/pkg/lib/ruby/3.2.0/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /opt/pkg/lib/ruby/3.2.0/bundler/installer/parallel_installer.rb:156:in `do_install'
  /opt/pkg/lib/ruby/3.2.0/bundler/installer/parallel_installer.rb:147:in `block in worker_pool'
  /opt/pkg/lib/ruby/3.2.0/bundler/worker.rb:62:in `apply_func'
  /opt/pkg/lib/ruby/3.2.0/bundler/worker.rb:57:in `block in process_queue'
  /opt/pkg/lib/ruby/3.2.0/bundler/worker.rb:54:in `loop'
  /opt/pkg/lib/ruby/3.2.0/bundler/worker.rb:54:in `process_queue'
  /opt/pkg/lib/ruby/3.2.0/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing nokogiri (1.15.3), and Bundler cannot continue.

In Gemfile:
  nokogiri

If installation completed but is broken, what is the complete output from nokogiri -v?

the output of "nokogiri -v" goes here

If installation failed during compilation, what are the complete contents of the mkmf.log file generated during the failed installation?

mkmf.log is too big for this form. But we have many of these errors:
clang: error: argument unused during compilation: '-diag-disable=175,188,1684,2259,2312' [-Werror,-Wunused-command-line-argument]

And one iconv linker error (this may be the whole problem)...
try_link_iconv: checking for iconv... -------------------- yes

DYLD_FALLBACK_LIBRARY_PATH=.:/opt/pkg/lib "clang -o conftest -I/opt/pkg/include/ruby-3.2.0/x86_64-darwin22 -I/opt/pkg/include/ruby-3.2.0/ruby/backward -I/opt/pkg/include/ruby-3.2.0 -I. -I/opt/pkg/include -D_REENTRANT -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT   -O2 -I/opt/pkg/include -D_REENTRANT -fdeclspec -O3 -fno-fast-math -ggdb3 -Wall -Wextra -Wextra-tokens -Wdeprecated-declarations -Wdivision-by-zero -Wdiv-by-zero -Wimplicit-function-declaration -Wimplicit-int -Wmisleading-indentation -Wpointer-arith -Wshorten-64-to-32 -Wwrite-strings -Wold-style-definition -Wmissing-noreturn -Wno-cast-function-type -Wno-constant-logical-operand -Wno-long-long -Wno-missing-field-initializers -Wno-overlength-strings -Wno-parentheses-equality -Wno-self-assign -Wno-tautological-compare -Wno-unused-parameter -Wno-unused-value -Wunused-variable -diag-disable=175,188,1684,2259,2312 -Wundef -fno-common -pipe conftest.c  -L. -L/opt/pkg/lib -L. -L/opt/pkg/lib -L/usr/lib     -lruby<IP_ADDRESS>  -lpthread -lpthread "
Undefined symbols for architecture x86_64:
  "_libiconv", referenced from:
      _main in conftest-8b1441.o
  "_libiconv_open", referenced from:
      _main in conftest-8b1441.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
checked program was:
/* begin */
 1: #include "ruby.h"
 2:
 3: #include <stdlib.h>
 4: #include <iconv.h>
 5: int main(void)
 6: {
 7:     iconv_t cd = iconv_open("", "");
 8:     iconv(cd, NULL, NULL, NULL, NULL);
 9:     return EXIT_SUCCESS;
10: }
/* end */

If you're using Bundler:

what is the output from bundle version?

$ bundle version
Bundler version 2.4.10 (2023-07-17 commit unknown)

what is the output from bundle config? (Take care to redact any credentials)

$ bundle config
Settings are listed in order of priority. The top value will be used.
bin
Set for your local app (/opt/sandbox/ngtest/.bundle/config): "./bin"

build.nokogiri
Set for your local app (/opt/sandbox/ngtest/.bundle/config): "--with-ldflags='-L. -L/opt/pkg/lib -L/usr/lib' --with-xml2-include=/opt/pkg/include/libxml2"

build.racc
Set for your local app (/opt/sandbox/ngtest/.bundle/config): "--with-ldflags=-L. -L/usr/lib -L/opt/pkg/lib"

force_ruby_platform
Set for your local app (/opt/sandbox/ngtest/.bundle/config): true

path
Set for your local app (/opt/sandbox/ngtest/.bundle/config): "./bundle"

set
Set for the current user (/Users/louis/.bundle/config): "--local path .bundle"

If you're on MacOS, please note:

the version of XCode you have installed (if you know)
the output of gcc -v or clang -v

$ clang -v
Apple clang version 14.0.3 (clang-14<IP_ADDRESS>.1)
Target: x86_64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

If Linux or a BSD variant, please note:

the distro you're using
the output of uname -a
the contents of /etc/lsb-release.

$ uname -a
Darwin dhcp-223.xxxxxxxx 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64

If Windows, please note:

whether you're installing the precompiled gems, or compiling yourself with DevKit
the version of RubyInstaller you've installed
or if you're not using RubyInstaller, how did you install Ruby?

@louis-guillaume Sorry you're having trouble. I'll try to help.
I see that you've made quite a few tweaks in this environment to try to get Nokogiri to build, including setting some linker flags. Have you also set any environment variables like CFLAGS or LDFLAGS? (The mkmf.log might have told me this but you didn't include it.)
Can I ask why you're not using the precompiled gem, which would avoid all of these compilation and linking issues?

@flavorjones  - Honestly not sure where the --diag-disable is coming from. I did add the --with-ldflags and --with-xml-2-include. The latter was because the build was finding libxml headers elsewhere (probably in the base system). The ldflags - not sure I remember why - but I also needed it for the racc build. I'm not setting CFLAGS  or LDFLAGS in the environment.
mkmf.log was too big to include in the problem report. I did include a couple excerpts, though, where it seemed to be reporting significant errors. Looks like I can add it as an attachment though....
mkmf.log
The main reason I'm not using the precompiled gem is that I'm building for mixed platform deployment and don't want to depend on system gems. So my dev machine is MacOS, but will deploy to NetBSD and/or Linux hosts for production. As long as I have the correct ruby version and bundler, I'm expecting to have the target system re-bundle in place as part of the deployment.
LMK I can provide any other details. Thanks for looking!

don't want to depend on system gems

I'm not sure I understand this point. The precompiled gems use the same version of libxml2 and libxslt on all platforms; and these will be the same versions compiled on the platforms that don't support it (e.g., NetBSD). I strongly recommend you take a hard look at using the precompiled gems, because this is exactly the problem they were created to solve.
You should look at your system environment to determine where the --diag-disable flag is coming from, because something is injecting properties into your compile and link flags, and I'm not sure I can help until you can reproduce this issue with a clean environment (no bundler flags, no unknown or unwanted compiler or linker flags).

You may want to take a look at RbConfig::CONFIG["CFLAGS"] and ["LDFLAGS"] to see if some of these flags are coming from your ruby build.

Now I have no overrides in .bundle/config for nokogiri or racc and they build just fine

That's good news! Thanks for closing the loop with us. :heart:


rolltime in different timezone causes endofday error

existing code assume columns be GMT
/-change timeout to zero before eod flush .timer.repeat[.eodtime.nextroll-00:01;0W;1D;   (.rdb.timeoutreset;);"Set rdb timeout to 0 for EOD writedown"];
I had issue when the .eodtime.rolltimezone is set to be differently
e.g when i roll data after US market closed (18:00 NYTime, which .eodtime.nextroll=D23:00:00 in UTC)
when tp trigger to load at 18:00, .rdb.timeoutreset has not yet triggered, so .rdb.timeout is not defined and error out during endofday process when trying to restore timeout.
I think the fix should be:
.timer.repeat[.eodtime.nextroll+.eodtime.adjtime[]-00:01;0W;1D;   (.rdb.timeoutreset;);"Set rdb timeout to 0 for EOD writedown"];
Or without changing the core processes. I load custom appconfig/code/rdb.q after loading the original rdb.q
// Torq rdb assume in GMT, adjust to be in .proc.cp timezone update periodstart:periodstart+.eodtime.adjtime[], nextrun:nextrun+.eodtime.adjtime[] from .timer.timer where (first each funcparam)=.rdb.timeoutreset;
To be safe, I also define .rdb.timeout:system"T" in settings/rdb.q

@etennoQ can you pick this up please?
@luilui4320 thank you for the detailed report and the proposed fix!

Hello @luilui4320,
We are currently working on the issue you have raised.
When you got the error, were you using the -localtime flag on the rdb?
Kindest regards
Elliot

Yes I put in setting/process.csv
Essentially should be same
Sent from my iPhone

On 2 Dec 2020, at 01:06, etennoQ<EMAIL_ADDRESS>wrote:

Hello @luilui4320,
We are currently working on the issue you have raised.
When you got the error, were you using the -localtime flag on the rdb?
Kindest regards
Elliot
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.


Observability task：Monitoring buried points for abnormal conditions（异常情况的监控埋点）

[ ] I have searched the issues of this repository and believe that this is not a duplicate.

Describe the proposal

En:There may be some exceptions in the Dubbo call link. These exceptions are important. You need to bury some indicator items, collect these abnormal data, and then count them on the monitoring dashboard. Later, you can refine and monitor alarms
中文：Dubbo调用链路中可能会出现一些异常，这些异常是比较重要的，需要进行一些指标项埋点，将这些异常数据收集到然后先统计到监控的dashboard中，后续可以进行细化和监控告警

OK