Allow /static (only) to be written to disk in server mode

Currently, Hugo has 2 kinds of render destination memory or disk.
During local development, I prefer renderToMemory to renderToDisk in order to enjoy Hugo's advantage. I mean fast rendering.
However, staticfiles-intensive site development with rendreToMemory consumes machine memory linearly.
For example. 3GB static directory consumes 6GB memory on this setting.
(yes, this is exceptionally huge example but, unfortunately, it is true of mine)
To resolve this situation, I would like to introduce another render target, composite, using afero's composite backend.
This filesystem serves only dynamic contents (content and etc) from writable memory.
On the other hand, static files are served directly from os filesystem without syncing to memory.
Contrary to its smallness of diff, it significantly reduces memory consumption.
In addition, as far as I consider, it does not have any major disadvantages.
I attached simple benchmark results.
In this benchmark, ~60MB file was copied to static direcotry per 10 sec up to ~1.2GB (total ~200 sec).
memory.log
composite.log
I believe that this we could even replace renderToMemory mode with composite mode as long as we confirm its compatibility.
But this is prototyping. I am not so confident with Hugo's module system that I added composite as another option rather than rewriting renderToMemory for now.

I understand the problem: Limit the memory allocation while stile getting the speed benefits of files written often. But as /content seems to be the new /static, the above may work for you, but not for me (and others). I tend to have very few files in static.
But I agree that we need to fix this. And since we cannot determine "is it dynamic" by its location, maybe we can:

Write all files to this composite
Somehow detect the file size before write and have a threshold per file
Maybe also have a global threshold (if written more than n bytes and file does not exist already, write to disk)

Or something ....?

Thank you for your time @bep

as /content seems to be the new /static
Sorry, I cannot understand this. What is your concern?
It suppose to me that many of Hugo users have a little effect by this.
But I think it is positive effect at least.

we cannot determine "is it dynamic" by its location

I assume that "files in static directory can be serve as it is. without processing". Is this wrong?
At the implementation level, all files are written to this composite fs in this mode.
So I feel it works correctly.

Sorry, I cannot understand this phrase. What is your concern?

You say you have 3GB in your static directory. I assume these are mostly images. If I put these images inside /content I would have the same problem as you, but this PR does not solve it.

I assume these are mostly images

Yes. Most of them are media or data files in terms of file size.

If I put these images inside /content I would have the same problem as you, but this PR does not solve it.

I see.
So you mean that target directories are not only static but content and etc.
It was my intention that this targets only static dir because it is easy to determine their files are "static".
At least, we do not have to keep them inside memory.
As a result, logic is simple and it is easy to start trying without redundant disk space. we need just a single flag.
(In addition, it is bottle neck directory in my case.)
I bet you have got a my point. I mean, my original goal is reducing memory usage by static directory.
So if your method is more comprehensive, it will be ok.
At the same time, I think it will be beneficial (at least not harmful) for everyone even if we focus on static directory at first.
Anyway, thanks for your consideration!

Second thought.
We may be able to add content directory to this composite, too
Dynamic content results inside MemoryFs are always prior to read-only layer. they are always served correctly in this situation.
We do not need to care about dynamic content results.
In that case, unexpected items are only precursors of dynamic contents, such as content/post/foo.md.
Is there any good way to filter them? .. but it might be more simple than something like dependency tree.
Personally, I do not mind that Hugo serves markdown files during local development.

I am a bystander here, but would it not be possible to specify the big files in the config file à-la .gitignore that would allow the author to choose and select the files that are loaded from disk? This would permit the flexibility of storing these files wherever the user chooses.

@davidovich Thank you for your comment opinion.

would it not be possible to specify the big files in the config file à-la .gitignore that would allow the author to choose and select the files that are loaded from disk?

Possible.
If my understanding is correct, this discussion has 2 points.

how to determine target files
whether these targets are written to disk or directly served from original directories

Now, I agree with that more hugo users will have benefits when the setting is configurable or we can detect static or not.
In my opinion, this is hugo's config so it will be good to be written in Hugo's config file.
On the other hand, your comment hit the spot for me.
Because it implies glob pattern may be suitable in this case. like static/**/* content/**/!(*.md) assets/**/!(*.scss)
How do you think about this? @bep
canditates

file size based threshold
global threshold
directory-based (not be flexible enough)
glob pattern
media types?

Second, I'm wondering if we need to render these to another disk space.
If it is ok, Why not serve static files from original direcotries? I think it will be cheaper than disk writing operation.

Retried.

Write all files to this composite

At this time, I intended to wite all files into composite fs as @bep said.
Exception is ".md", ".html" and etc. It is based on file extension and follows currrent IsContentFile logic.
If we apply not whitelist (which files we should include) but the blacklist (which files we should exclude),
allmost all of the files are managed under comositeFs efficiently without further configurations.
To filter exceptions, FilenameFilterFs is added.
For now, I have specified only contentDir and staticDir since I assume that the other dirs do not have any "static" files.
(it will be useless to use composite Fs)
But even if necessary, it is as easy as contenteDir to add extra target dir (maybe assetDir?) with FilenameFilterFs
as far as I tested, it works well.
Any suggestions are welcome. @bep , @davidovich
Especially, I am wondring the possiblity of negative effects by this change

Retried.

Write all files to this composite

At this time, I intended to wite all files into composite fs as @bep said.
Exception is ".md", ".html" and etc. It is based on file extension and follows currrent IsContentFile logic.
If we apply not whitelist (which files we should include) but the blacklist (which files we should exclude),
allmost all of the files are managed under comositeFs efficiently without further configurations.
To filter exceptions, FilenameFilterFs is added.
For now, I have specified only contentDir and staticDir since I assume that the other dirs do not have any "static" files.
(it will be useless to use composite Fs)
But even if necessary, it is as easy as contenteDir to add extra target dir (maybe assetDir?) with FilenameFilterFs
as far as I tested, it works well.
Any suggestions are welcome. @bep , @davidovich
Especially, I am wondring the possiblity of negative effects by this change

benchmark of hugoDocs
memory usage was reduced by 10-20% even though hugoDocs' static directory is not so large
composite.memstats.log
memory.memstats.log

benchmark of hugoDocs
memory usage was reduced by 10-20% even though hugoDocs' static directory is not so large
composite.memstats.log
memory.memstats.log

I have added 2 configurations and related tests

file size threshold
filename-based exclusion

Here is example yaml configuration. you can add and configure any directories
staticDir:
    - static
    - static_en

staticFileRules:
   static:
   static_en:
   contentDir:
    ignore:
      - "\\.md$"
      - "^content/non-static/*$"
    size: 500 # KB

@bep
How do you think?
Do we have to care about any other points?

Hey,
And sorry for the late response on this; I agree that we really should add this, but the added complexity has been pulling me away from it ...
I have had a look and to me it looks a little too complicated.
Note that the below is probably a little too simplified, but:
In my head, we could add a new flag/config option for the server:
renderStaticFilesToDisk = true

If that is set to true we:

Use the OS filesystem when syncing in copyStaticTo
Use a composite for the server.

@bep
I am willing to simplify this.
In my case, I will simply apply this for the whole static direcoty when it is released, do not need further configuration.
ok, I will revise user-interfaced option/flag/configuration before merge.
By the way,

Use the OS filesystem when syncing in copyStaticTo

If my understanding is correct, you mean static files are written into public dir, right?
I have tried here
But this does not reduce memory eating.
Probably, this may be because afero.CopyOnWrite fs copy all public files into memory on full builds happen.

Any attempt to modify a file found only in the base will copy the file to the overlay layer before modification (including opening a file with a writable handle).

Previous implementation does not exec any ops of static files. thus, memory consumption is relieved.
Or, do I misunderstand something here?

@bep

Or, do I misunderstand something here?

It was my misunderstanding.  I have to consider files in contentDir.
Here, I implemented as --renderTo hybrid as you said.
I would like you to

check this implementation
compare hybrid with composite

My recommendation is composite because it is faster.
But It is ok for you to select hybrid considering complexity.
Finally, I will clean up and simplify configurations and so on using renderStaticFilesToDisk

If my understanding is correct, you mean static files are written into public dir, right?
I have tried here
But this does not reduce memory eating.

The "memory eating" comes from the static files being written to memory. So if we make sure that we don't do that, this should work.
Again, sorry if I'm oversimplifying it, but if the flag mentioned above is enabled:

Make sure that the FS in the static file syncer is the OS FS.
Make sure the FS used by the HTTP server is a composite of /public (OS) and the memory destination.

Do you have a test project online that demonstrates this "memory eating" problem?

this should work.
Yes. After fixation, both modes improve memory usage.
OK, I will simplify my implementation soon. Please wait.

Do you have a test project online that demonstrates this "memory eating" problem?

No but it is easy to reproduce this.

create minimal hugo site
copy some files into static or content directory

Hugo consumes about ~15MB memory in my environment. So you can also confirm this by copying 50-100MB file(s)

@bep Simplification done. Could you review again?
For testing, use this
hugo new site memory-eating-test
cd memory-eating-test
# generate 100MB static files
head -c 100000000 /dev/urandom > static/large-file
cp static/large-file content/large-file

@satotake OK, thanks for your pushing this, much appreciated. The latest patch looks very clean. I'm planning a Hugo release on Monday and will get this merged before then. I will test this a little myself ... The only thing I think about is that we may consider making write static files to disk the default option ... I suspect many have this issue without knowing it.

@bep
Revised or commented. My response may be delayed? If so, sorry

I added some test cases and caught one bug, which is related to file deletion event of static sync.
Do you have any considerations?, @bep

@satotake I'm sorry that this got lost in the backlog, I was reminded of this by a discussion on the forum. I will have a look at this today.

I have rebased and will complete this in here: #9626


metric: launched and killed walredo processes

In OOM situations, knowing exactly how many walredo processes there were at a time would help afterwards to understand why was pageserver OOM killed.
We need two counters:

walredo processes started
walredo processes shutdown

Good first issue:

pageserver/src/walredo.rs is the walredo implementation
We could increment the started counter near launching the process
We could increment the killed counter in the process waiting newtype wrapper
metric could be pageserver_walredo_processes, an IntCounterVec with "operation" key

the two concrete counters could be operation="started" and operation="shutdown"
the "shutdown" separate from "killed" if we'd ever want to distinguish on the ones we need to kill in addition to finding them having exited

@koivunej i would like to work on this

nov 6th notes:
discussed whether this aligns with the team priorities for Q4, considering to postpone and get back to this once other priorities are resolved
did i capture this right, @koivunej @jcsp @shanyp ?

@rmodpur feel free to do so and ping me once you have a draft! This should be more straightforward than the #5310 which I will eventually get to :)


[SUPPORT] - Range based partitioning to mitigate performance issues with large number of partitions

This is in reference to the github issue. https://github.com/apache/hudi/issues/2269
In order to mitigate the issue with a very high number of value based partitions, we want to shift to range based partitions. The value based partitions were coming out to be ~900K and growing which were leading to signficant increases in hudi writing time to our COW table. This was the case even with the auto cleaner turned off. So our understanding is that if the partitions are less and are more or less constant, that issue with write performance can be mitigated. With that in mind, we have the following questions:

We have already seeded a hudi table with ~900 K partitions based on value of a column. Since, the subsequent writes are becoming progressively slower(even with auto cleaner turned off), we wanted to see if there is any hudi utility to copy one hudi table to another hudi table? If there is one, then can that utility be used to change the existing partitioning strategy to something else (say range based in our case)

What should be the optimum number of partitions for hudi write to work efficiently in your opinion? We are thinking 1000 should be a good number, but wanted to see if this can be arrived at by some logic and/or other reasoning.

The reason we are doing range based partitions is that we want the ability to remove lesser value (older) range partitions from the hudi table as we continue to add higher value (newer) range partitions. The reason being we want to limit the number of partitions that the hudi table will have at a time, to a constant value (1000 if that can be a good constant number of partitions based on number 2 above). With this in mind, our questions are as follows:

a) Can we easily remove partitions from the hudi table by simply removing/deleting the underlying directory for the partition? Or is there any additional metadata in the hudi table structure as well which would be required to be updated?
b) Can we just copy the older partition folders to a different hudi table for the purposes of cold storage? If so, would it be possible to add them back to the hudi table manually, by restoring the partition directory of the hudi table. I guess this is related to question 3.a above.

@asharma4-lucid : The reason why any processing mechanism (hudi or others) takes longer time is the second order effects that the partitioning style causes. Having too many partitions implies the number of records per partition (per file) is small. and the processing engine needs to create a ton of small files (number of tasks increases).
There is no optimal number of partitions that could be suggested. It depends on the workload. To copy from one table to another, you can use spark.read.format("hudi")... and then write to another location in hudi format. I have personally seen datasets with partitions in the range of 10K.
For deleting partitions, there is an open PR to support it efficiently without individual records here : https://github.com/apache/hudi/pull/2254

Also, the slowdown caused due to file listings should be eliminated in next release when https://github.com/apache/hudi/pull/2189 and related diffs will be available. (https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+and+Query+Planning+Improvements)

Thanks @bvaradar for the details. Much appreciated.


can u please explain how to remove outputs

didn't understand :Note: Add definition DISABLE_OUTPUT to remove all console outputs

Well as you say you only have to add DISABLE_OUTPUT definition to your project, which will be processed here:
https://github.com/TheCruZ/kdmapper/blob/5a49a4eec2abe0a4bd1f1b761a2733fed56a36aa/kdmapper/utils.hpp#L3


.Net Core v2.0 - Nuget Packages are not getting restored

.Net Core v2.0 - Nuget Packages are not getting restored
Any application that is targeting .Net Core v2.0 framework, will not build as NugetPackages are not restored successfully. I have tried with both the Console Apps and Web Apps.
Error Messages I am getting:
Restoring NuGet packages...
To prevent NuGet from restoring packages during build, open the Visual Studio Options dialog, click on the Package Manager node and uncheck 'Allow NuGet to download missing packages during build.'
1>------ Build started: Project: ConsoleApp2, Configuration: Debug Any CPU ------
1>C:\Users\Super.Coder\AppData\Local\Temp.NETCoreApp,Version=v2.0.AssemblyAttributes.cs(4,20,4,26): error CS0400: The type or namespace name 'System' could not be found in the global namespace (are you missing an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(14,12,14,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(15,12,15,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(16,12,16,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(17,12,17,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(18,12,18,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(19,12,19,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(20,12,20,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(21,12,21,18): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>C:\Users\Super.Coder\AppData\Local\Temp.NETCoreApp,Version=v2.0.AssemblyAttributes.cs(4,71,4,97): error CS0518: Predefined type 'System.String' is not defined or imported
1>C:\Users\Super.Coder\AppData\Local\Temp.NETCoreApp,Version=v2.0.AssemblyAttributes.cs(4,99,4,119): error CS0246: The type or namespace name 'FrameworkDisplayName' could not be found (are you missing a using directive or an assembly reference?)
1>C:\Users\Super.Coder\AppData\Local\Temp.NETCoreApp,Version=v2.0.AssemblyAttributes.cs(4,122,4,124): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(14,55,14,68): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(15,61,15,68): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(16,59,16,80): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(17,59,17,68): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(18,68,18,75): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(19,55,19,68): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(20,53,20,66): error CS0518: Predefined type 'System.String' is not defined or imported
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(21,55,21,64): error CS0518: Predefined type 'System.String' is not defined or imported
1>Program.cs(1,7,1,13): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>C:\Users\Super.Coder\AppData\Local\Temp.NETCoreApp,Version=v2.0.AssemblyAttributes.cs(2,7,2,13): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>C:\Users\Super.Coder\AppData\Local\Temp.NETCoreApp,Version=v2.0.AssemblyAttributes.cs(3,7,3,13): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(11,7,11,13): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>obj\Debug\netcoreapp2.0\ConsoleApp2.AssemblyInfo.cs(12,7,12,13): error CS0246: The type or namespace name 'System' could not be found (are you missing a using directive or an assembly reference?)
1>Program.cs(5,11,5,18): error CS0518: Predefined type 'System.Object' is not defined or imported
1>Program.cs(7,26,7,32): error CS0518: Predefined type 'System.String' is not defined or imported
1>Program.cs(7,16,7,20): error CS0518: Predefined type 'System.Void' is not defined or imported
1>Done building project "ConsoleApp2.csproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
What tools I have tried with:

Visual Studio 2017 v15.3
Using New Project, default templates, they don't get built because nuget packages are not restored
CLI
Tried to make new apps using commands, and the nuget package restore failed.
After that tried to use DotNet Restore, that returned the same results.

What I have tried:

Checked on CLI that DotNet --Version returns 2.0.0
Repaired Visual Studio.

One of the web apps, I got a lot of errors similar to the following:
C:\Users\Super.Coder\source\repos\WebApplication8\WebApplication8\WebApplication8.csproj : error NU1202: Package Microsoft.AspNetCore 2.0.0 is not compatible with netcoreapp2.0 (.NETCoreApp,Version=v2.0). Package Microsoft.AspNetCore 2.0.0 supports: netstandard2.0 (.NETStandard,Version=v2.0)

Tried to make new apps using commands, and the nuget package restore failed.

Which dotnet CLI commands have you tried?
I'm running Windows 10 and have .NET Core SDK 2.0 installed and for me the following works fine:
> dotnet --version
2.0.0
> mkdir hwapp; cd hwapp
... snip
> dotnet new console
The template "Console Application" was created successfully.

Processing post-creation actions...
... snip
> dotnet run
Hello World!

Where the dotnet new console already contains a dotnet restore (since .NET Core 2.0).

@the-super-coder can you share what your project looks like?
/cc @rrelyea

I'm having the same issue.  I just updated to VS 15.3.5 two days ago on the machine that is having the issue.  I can't build from command line or inside VS with just project defaults.  I've repaired both the 2.0.0 sdk and VS 2017 with no luck.  This is from just the hello world new console app(dotnet new console -o hwapp).

What happens when you explicitly run dotnet restore?
/cc @rrelyea @emgarten could this be related to the fallback folder or other corruption I've seen in different issues?

Says the Restore completed, then the dotnet run fails in the same way though

Just a guess but maybe you are in this state? https://github.com/dotnet/core/issues/920#issuecomment-327222033. See the workarounds in the next few comments.

Thanks, deleting the C:\Program Files\dotnet\sdk\NuGetFallbackFolder (and deleting obj/bin folders of existing projects) fixed it for me.

Awesome! Glad it worked.

I closed this because I think its another instance of this problem. https://github.com/NuGet/Home/issues/5995. If people continue to see this, go ahead and add more detaild there.

@philippgille Sorry for late response, I was busy with some family work. Well, I am using .Net Core SDK v2.0
@Petermarcu Unfortunately, I cannot expose the personal project. However, I face the same issue with all the default templates for DotNetCore in VS2017 v15.3

Thanks all!!!

My solution was to do a dotnet restore from an elevated command prompt.
(after deleting the NuGetFallbackFolder, but I am not sure that is necessary)
Here is what I did:
Failed compiling with the error of this issue.
Did a dotnet restore and failed with a Permission denied to modify the 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder' folder.
Renamed the NuGetFallbackFolder
Asked VS2019 to do a Package Restore and failed.
Opened an elevated command prompt and dotnet restore. The NuGetFallbackFolder was created.
Asked VS2019 to do a Package Restore and succeeded.
I am not running VS2019 with elevated privileges.

I'm receiving the same set of errors and deleting NuGetFallbackFolder and running dotnet restore did not work for me.
https://github.com/dotnet/core/issues/4040

Thanks, deleting the C:\Program Files\dotnet\sdk\NuGetFallbackFolder (and deleting obj/bin folders of existing projects) fixed it for me.

I only deleted obj/bin folders and that fixed it


CUDA out of memory

Hi, author.
I use one RTX 2080 Ti for searching, but when the code do the validation step in training phrase, the 11G GPU memories can't meet the code, could you tell me how to deal with this? Thanks.
When I executed the command CUDA_VISIBLE_DEVICES=1 python train_search.py, the error happens:
14:37:50 params = 251.803792MB, FLOPs = 71.419396GB
architect initialized!
using downsampling: 2
Found 1487 images
using downsampling: 2
Found 1488 images
using downsampling: 2
Found 500 images
  0%|                                                    | 0/20 [00:00<?, ?it/s]25 14:37:53 True
25 14:37:53 search-pretrain-256x512_F12.L16_batch3-20210425-143740
25 14:37:53 lr: 0.02
25 14:37:53 update arch: False
[Step 495/495]: [26:39<00:00, 3.23s/it]                  | 0/20 [00:00<?, ?it/s]
[Epoch 1/20][validation...]:   0%|                       | 0/20 [26:39<?, ?it/s]25 15:04:32 Thread 0 handle 167 data.
25 15:04:32 Thread 1 handle 167 data.
25 15:04:32 Thread 2 handle 166 data.

...

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cjj/miniconda3/envs/real-time/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/home/cjj/miniconda3/envs/real-time/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/home/cjj/miniconda3/envs/real-time/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 109, in rebuild_cuda_tensor
    storage = storage_cls._new_shared_cuda(
RuntimeError: CUDA error: out of memory

Hi @Jun-jieChen ,
Thank you for your interest in our work!
If you always get OOM during [validation...] process, you may consider adding an argument threds=2 or threds=1 in the definition of SegEvaluator: https://github.com/VITA-Group/FasterSeg/blob/master/search/train_search.py#L115

@chenwydj
Thanks!


editing series 'subjects' via api fails

Describe the bug
When updating the metadata of a series via External API:
If I try to update the field 'subjects', the api responds with a weird error msg:
Not Found: Cannot find a metadata field with id 'subjects' from event with id '8e981748-69b1-4f8e-9839-8ebd2f2a9364' and the metadata type 'dublincore/series'.
Without the 'subjects' field, everything works fine.
Here's what the request looks like:
URL: https://cast.studer-raimann.ch/api/series/8e981748-69b1-4f8e-9839-8ebd2f2a9364/metadata?type=dublincore%2Fseries
Method: PUT
Body: [metadata] => [{"id":"contributor","value":["root user"]},{"id":"creator","value":["Opencast"]},{"id":"description","value":"asdasd123"},{"id":"subjects","value":["asdf"]},{"id":"title","value":"6: series 1.0.1"}]
Or via Curl:
curl --request PUT
--url 'https://cast.studer-raimann.ch/api/series/8e981748-69b1-4f8e-9839-8ebd2f2a9364/metadata?type=dublincore%2Fseries'
--header 'Authorization: Basic AUTH_CODE'
--header 'Content-Type: multipart/form-data; boundary=---011000010111000001101001'
--form 'metadata=[{"id":"contributor","value":["root user"]},{"id":"creator","value":["Opencast"]},{"id":"description","value":"asdasd123"},{"id":"subjects","value":["asdf"]},{"id":"title","value":"6: series 1.0.1"}]'
Environment Information
Opencast 10.3.0 - a750fa8 - built on 2021-11-05

The issue here seems to be that the corresponding GET endpoint returns the field 'subjects', but the PUT endpoint expects the field 'subject'. I would consider the latter the correct naming as per dublincore definition, so the issue probably lies somewhere within the GET endpoint

Have you tried to upgrade to OC11? I wonder if the issue has gone with the consolidation with the admin-UI's index.
I'm watching this ticket since our site also uses subject field for both episode and series metadata. We currently use the Series Service endpoint API, directly, to update series metadata, and a custom OC module for event metadata, so we are not running into this problem.


ci: Remove parallel limit from single tenant runners

The instances running these have either 32 or 48 CPUs. By explicitly setting the parallelism to 16, we are artificially making these runs slower. In the past, this was necessary since we had race conditions, and slower disks.
Having performed a bunch of local runs on an i9 13900KS and TMPFS (a.k.a. RAMDISK) this artificial limitation is making runs about 20% slower on average.
Let's see how well removing this limit works in our single tenant CI runners.

The non-split test suites are showing decent speed gains:

Engine & CLI
Before
After
Difference

lint
6m 39s
5m 21s
20% ⬆

test
18m 3s
15m 39s
13% ⬆

testdev
6m 38s
5m 49s
12% ⬆

The split test suites are showing marginal speed gains:

Engine & CLI split
Before
After
Difference

test-modules
5m 37s
5m 26s
3% ⬆

test-module-runtimes
7m 42s
7m 14s
6% ⬆

test-cli-engine
3m 34s
3m 39s
2% 🔻

test-everything-else
7m 56s
7m 42s
3% ⬆

I still think that this is worth merging. WDYT?

While this seemed like a good idea, test failed 3 times in a row when --parallel=16 is removed.
I will re-purpose this PR and just keep the testdev fix.


Make metric keys configurable

TL;DR
This PR allows manually configuring prometheus metric labels using metric keys. Currently FlytePropeller default metric keys include the workflow and task IDs. This means that as the number of workflows / tasks increases the cardinality of metrics will increase as well. For some deployments this is undesirable. By enabling manual configuration we can tweak metric cardinality to satisfy deployment requirements.
Type

[x] Bug Fix
[ ] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[x] Smoke tested
[ ] Unit tests added
[x] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
^^^
Tracking Issue
fixes https://github.com/flyteorg/flyte/issues/2565
Follow-up issue
NA

@katrogan did you end up setting the metric keys for single binary after the flyte admin PR? It is necessary right?

@hamersaw so I updated admin to set the metric keys outside of the Serve() entrypoint single binary uses so it wasn't necessary to change anything there (https://github.com/flyteorg/flyte/pull/2526)


[RKE2] etcd cert components are available on both etcd and control plane nodes

Rancher Server Setup

Rancher version: 2.6-head commit id: a31f5b861
Installation option (Docker install/Helm Chart): Helm install

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE1

Information about the Cluster

Kubernetes version: 1.22-RKE2
Cluster Type (Local/Downstream): AWS node driver RKE2 - 3 etcd, 2 cp and 3 worker nodes

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom) std user - cluster owner

Describe the bug
[RKE2] etcd cert components are available on both etcd and control plane nodes
To Reproduce

Deploy an rke2 node driver down stream cluster
Login to the etcd and control plane nodes /var/lib/rancher/rke2/server . etcd folder exists and the certs exist on these nodes.
Rotate certs on the cluster, select etcd component from the dropdown.
etcd certs on both the nodes get rotated.

Expected Result

etcd component should be available only on the etcd node and not on the control plane nodes
when cert rotate is triggered on etcd components, the etcd components on both the nodes rotate. only the etcd ones should rotate if ^^^ is not fixed.

This is by design that rke2 only has server and agent roles (v.s. separate etcd and control plane and worker), in rancher we add extra labels to mark nodes as "etcd" or "control plane",  so it is expected to see the etcd certs in both "etcd" and "control plane" nodes.
Moving this back to QA since it is the expected behavior.

Closing as per - https://github.com/rancher/rancher/issues/37590#issuecomment-1126297148


creating new MockObject inside of a when thunk breaks mach.ignoreMockedCallsWhen

creating the new MockObject resets the handler to the default handler

potentially store current handler in a global state and set that as the default handler?

👍


Update to Go 1.17.1

Signed-off-by: Phil Estes<EMAIL_ADDRESS>

@AkihiroSuda We've been using the full version form (e.g. #5842) for better build reproducibility.


[BUG] - Input label slot has a color bug

NextUI Version
2.2.9
Describe the bug
Input label slot has a color bug
Your Example Website or App
https://codesandbox.io/p/devbox/crimson-wood-zrgpsk?file=%2FApp.jsx%3A13%2C31
Steps to Reproduce the Bug or Issue
add custom color for label slot

Expected behavior
Input label slot can display correctly with custom color
Screenshots or Videos

Operating System Version
macOS
Browser
Chrome

Hi @thought7878 ,
This issue arises due to higher specificity in the default styles.
As a temporary workaround you can override this using the Important modifier in TailwindCSS.
classNames={{
   label: "!text-blue-500",
}}


Getting stuck while installing on stable version

It says something like
"Starting installation for Discord
Killing Discord Processes
Downloading BandagedBD package
Download seems to have failed, will try once more.
Downloading BandagedBD package"
Screenshots

Discord Version
<!-- Stable

Do you h ave access to GitHub in your region?


Line Intersections

Intersections function of the Line trait does not calculate the correct intersection point

Fixed it. I just forgot how to math apparently


Angular 4 : tsconfig file not detected

When I updated my project to Angular 4, I had to rename tsconfig.json to tsconfig.app.json. Since atom-typescript no longer detects it.
Thanks.

I confirm the issue, it happens also to me. Renaming the file resolve the problem.

Do you have an example project where this happens? Is tsc able find the config with the tsconfig.json name?

I confirm the issue, atom-typescript detects only ./tsconfig.ts in root directory and not loads src/tsconfig.app.ts. It would extend the first.


testshib.org problem

I got the following error from RetrieveAssertionInfo when testing with testshib.org:
error validating response: Missing signature referencing the top-level element

@ptman could you provide a little more info, please? The AssertionInfo that it's trying to validate would be especially helpful for understanding the problem

Hello, I think I'm experiencing a similar problem. When trying to retrieve the assertion info:
assertionInfo, err := sp.RetrieveAssertionInfo(r.FormValue("SAMLResponse"))
the program throws the error:
2017/02/10 17:22:33 http: panic serving [::1]:56367: missing AttributeStatement element
goroutine 7 [running]:
net/http.(*conn).serve.func1(0xc4201c6a00)
	/usr/local/Cellar/go/1.7.3/libexec/src/net/http/server.go:1491 +0x12a
panic(0x3acf40, 0xc4202384e0)
	/usr/local/Cellar/go/1.7.3/libexec/src/runtime/panic.go:458 +0x243
main.main.func1(0x586f40, 0xc42010e750, 0xc4200d2960)
	/Users/brian.auron/go/src/main.go:103 +0x5b7
net/http.HandlerFunc.ServeHTTP(0x41f8b8, 0x586f40, 0xc42010e750, 0xc4200d2960)
	/usr/local/Cellar/go/1.7.3/libexec/src/net/http/server.go:1726 +0x44
net/http.(*ServeMux).ServeHTTP(0x5a54e0, 0x586f40, 0xc42010e750, 0xc4200d2960)
	/usr/local/Cellar/go/1.7.3/libexec/src/net/http/server.go:2022 +0x7f
net/http.serverHandler.ServeHTTP(0xc420076a80, 0x586f40, 0xc42010e750, 0xc4200d2960)
	/usr/local/Cellar/go/1.7.3/libexec/src/net/http/server.go:2202 +0x7d
net/http.(*conn).serve(0xc4201c6a00, 0x587600, 0xc4201c4640)
	/usr/local/Cellar/go/1.7.3/libexec/src/net/http/server.go:1579 +0x4b7
created by net/http.(*Server).Serve
	/usr/local/Cellar/go/1.7.3/libexec/src/net/http/server.go:2293 +0x44d

I'm not sure what is going wrong, dumping the request out shows what appears to be an encoded response for SAMLResponse as in RetrieveAssertionInfo

@ptman I believe this should be resolved now, can you re-test?

Still getting the same result. I'll see if I can prepare a testcase. Have you tried testshib.org?

Can you get the same error using: https://gist.github.com/ptman/102629ca9aded1c2db0d45968dc998ed

https://stackoverflow.com/questions/45193627/no-providerid-parameter-given-in-shibboleth-sso-authentication-request-from-te helped me

So I actually got testshib.org working with the current master.

Now I'm having trouble with testshib.org again. It doesn't find metadata even though I successfully uploaded it. But at the same time I've had success with Azure AD SAML (ADFS?)


[DuckDB] Expose vector search capability via duckdb extension

Problem Statement
Lance rust dataset has the capability of querying over vector space, via Scannner::nearest(..), we should expose this to DuckDB via its extension mechanism.
For example, a potential implementation could be a duckdb table function lance_nearest
SELECT id, col
FROM lance_nearest("table", "vector_column", "query", top_k)
WHERE city = 'SF' and day = "02-03-2023"

Ideally duckdb extension would let us modify the query plan so if we see ORDER BY nearest(...) LIMIT 10 then this is pushed down as ANN search


Allow running individual tests

Usage:
run_lit script now accepts an argument. If this argument is given, which
can either be a file or directory, only tests in that file or directory
are executed.
Eg:
./run_lit test/Solver/
will execute all tests in Solver/ directory
./run_lit test/Sover/saturating1.ll
will execute only one test contained in saturating1.ll

LGTM, thanks!


Revert sigstore/cosign-installer update and ignore it

btw tests will fail because the image build is broken (this patch is fixing it) and it is using pull_request_target as trigger.


Drag and Drop "keep both files and rename the incoming one" prompt

Is your feature request related to a problem? Please describe.
When user drags and drops a file, user doesn't have option to rename dragged file in case there is a duplicate file of the same name
Describe the solution you'd like
Prompt the user for input on what they would like to rename the incoming file to on dropping the file in its new location in OutlineView
Describe alternatives you've considered
n/a
Additional context
Prerequisites:

https://github.com/CodeEditApp/CodeEdit/issues/1018
https://github.com/CodeEditApp/CodeEdit/pull/1031

What if there are multiple files we are “keeping both” for? We might look to Finder and follow it’s behavior.

Looks like when you click 'keep both' it appends copy to the end of the file name before the extension

I don’t know of a feature like that, no.

Yeah, I’d just append “copy” as Finder does. The user can rename manually afterwards.

yeah especially considering if we drag and drop multiple files, it would be cumbersome to prompt the user to input a name for each moved file


Plugin fails to start in kubernetes cluster.

Describe the bug
When the plugin is included in grafana 7.5.9 running in kubernetes cluster the plugin fails with this below trace.
t=2021-11-11T06:08:59+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl=/grafana socket=
t=2021-11-11T06:08:59+0000 lvl=eror msg="plugin init error" logger=plugins.backend pluginId=yesoreyeram-infinity-datasource error="open /tmp/plugin412270376: read-only file system"
t=2021-11-11T06:08:59+0000 lvl=eror msg="Failed to start plugin" logger=plugins.backend pluginId=yesoreyeram-infinity-datasource error="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply \n\n needs to be recompiled to support the latest protocol."
Version Details:

Grafana version : 7.5.9
Plugin version : latest

Can you provide more details on how to reproduce the issue? with plugin version, os version, how you are provisioning grafana, etc. Otherwise, I don't think I can reproduce this.


download internal tools from azure container.

Other than aka.ms, looks good.  Getting aka.ms url now


Add support for rename-command in redis.conf

Pull Request (PR) description
This pull request adds support for renaming or disabling commands in redis.conf.
This is best practice when running production instances.
Hiera example:
redis::rename_commands:
  FLUSHDB: "\"\""
  FLUSHALL: "\"\""
  CONFIG: "\"\""
  SHUTDOWN: "\"\""
  DEBUG: "\"\""

This Pull Request (PR) fixes the following issues
Fixes #242

Thanks! I will get started on the tests during the holidays.

Rebased on latest master and added tests

Rebased on latest master and added tests

+1 to merge when tests pass restarted the tests as the one that failed seemed to be a network timeout.

+1 to merge when tests pass restarted the tests as the one that failed seemed to be a network timeout.


Too many columns

On this website there is some csv files that I downloaded to test textql (the first one)
https://support.spatialkey.com/spatialkey-sample-csv-data/
when I run textql -sql "select count(*) from sample_data" FL_insurance_sample.csv
I get
2018/04/09 13:57:58 too many columns on FL_insurance_sample
The file has huge amount of rows, but only contains 18 columns.
Is the error correct?

The error is correct indeed.
Above file contains CR line terminators(\r). You can verify it with file <filename>.
Currently Golang csv parser used by textql do not support CR line terminators. You should replace all occurrences of \r with \n. See this Golang Issue.

Thanks for the feedback.
Can't the tool just detect the CR and fix the file accordingly?
In my opinion, people that could use the tool don't care about CR... for them is just a csv file, they just want for it to work.
If someone just sees that error... is going to immediately say... this tool is crap, which is not the case (in my opinion), just incomplete. Either the error needs to be much more explanatory or just make the tool fix the CR issue.
Also if someone is going to use this tool is because they have really huge csv, and nobody is going to open them to replace \r by \n.
Just an opinion.


Add an example of an Empty View with more customised content

Whilst this page does document the CollectionView.EmptyView feature, it does not give adequate examples of more customised views. The presented examples are:
A simple empty view
<CollectionView ItemsSource="{Binding EmptyMonkeys}"
                EmptyView="No items to display" />

A more complicated empty view
<CollectionView.EmptyView>
    <StackLayout>
        <Label Text="No results matched your filter."
                Margin="10,25,10,10"
                FontAttributes="Bold"
                FontSize="18"
                HorizontalOptions="Fill"
                HorizontalTextAlignment="Center" />
        <Label Text="Try a broader filter?"
                FontAttributes="Italic"
                FontSize="12"
                HorizontalOptions="Fill"
                HorizontalTextAlignment="Center" />
    </StackLayout>
</CollectionView.EmptyView>

These examples serve well for a chat application, however what happens if you want to display an image and copy the appearance of the native iOS empty table view? A commonly used library on iOS is DZNEmptyDataSet, I have personally used this library before and was looking to recreate the styling of the platform library with CollectionView.EmptyView. Below are the examples of empty views which I think should be provided by the xamarin documentation.

Lets try recreating one of these screens using the documentation. Experience suggests we would just need to define our content:
<CollectionView.EmptyView>
    <Grid HorizontalOptions="CenterAndExpand" VerticalOptions="CenterAndExpand">
        <Grid.RowDefinitions>
            <RowDefinition Height="100"/>
            <RowDefinition Height="auto"/>
            <RowDefinition Height="auto"/>
        </Grid.RowDefinitions>

        <Image Source="chat-icon.png"
               Grid.Row="0"/>
        <Label Text="No Chats Available"
               FontAttributes="Bold"
               HorizontalTextAlignment="Center"
               Grid.Row="1"/>
        <Label Text="Once you are sent a message, it will appear in this list"
               HorizontalTextAlignment="Center"
               Grid.Row="2"/>
    </Grid>
</CollectionView.EmptyView>

However if we look at the results, the empty view appears at the top of the CollectionView despite the fact that we set our HorizontalOptions and VerticalOptions to CenterAndExpand. This exact issue has been raised on the main Xamarin repository.

5041- cannot arrange content in center.

In the issue linked the reporter could not center their content. They are told that they will need to embed their view within a ContentView in order to center it. This happens because the child of the EmptyView does not respect its layout options and will always resize the element to fill all of the available space. As a result to center content a developer would need to wrap their EmptyView in a ContentView (or something that inherits from, such as Grid).
Our revised view then becomes:
<CollectionView.EmptyView>
    <ContentView>
        <Grid HorizontalOptions="CenterAndExpand" VerticalOptions="CenterAndExpand">
            <Grid.RowDefinitions>
                <RowDefinition Height="100"/>
                <RowDefinition Height="auto"/>
                <RowDefinition Height="auto"/>
            </Grid.RowDefinitions>

            <Image Source="chat-icon.png"
                Grid.Row="0"/>

            <Label Text="No Chats Available"
                FontAttributes="Bold"
                HorizontalTextAlignment="Center"
                Grid.Row="1"/>

            <Label Text="Once you are sent a message, it will appear in this list"
                HorizontalTextAlignment="Center"
                Grid.Row="2"/>
        </Grid>
    </ContentView>
</CollectionView.EmptyView>

Now the view is correctly centered and the developer can create a view that is similar to look and feel as the existing native empty views. I believe the best way to update this document would be to wrap the existing complex view to be wrapped in a ContentView with a sentence explaining how to layout the view.
I believe this change would make the documentation more clear for CollectionView and help future developers not get caught out by this simple yet annoying quirk!
Auto Generated
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 12322437-d338-f1b3-a352-ae86d0027992
Version Independent ID: cbfa2f8f-1b60-4bbd-a945-ee1181cf3613
Content: Xamarin.Forms CollectionView EmptyView - Xamarin
Content Source: docs/xamarin-forms/user-interface/collectionview/emptyview.md
Product: xamarin
Technology: xamarin-forms
GitHub Login: @davidbritch
Microsoft Alias: dabritch

Hi @Axemasta
Thanks for pointing this out. I've added it to the doc, in a PR that will go live tomorrow. Two things though:

I think it's still a bug as there's a redundant layout (a ContentView in this case). I'm checking up on this with the engineers.
CenterAndExpand only works on a StackLayout. It has no effect on a Grid. That applies to any of the ...AndExpand layout options. However, the redundant layout from (1) is still required to get centering working.

It's not a bug that the ContentView is required. It's because internally the EmptyView is added to a native container that doesn't provide any context for Xamarin.Forms layout. Therefore, to position the views that comprise your EmptyView, you must a root layout, whose child is a layout that can position itself within the root layout.
It's a requirement that will most likely be eliminated in future. There are just more important issues to be dealt with first.

Thanks for the insight 😁


Change Position Control With Css

How to change position my location button on map with css.
Please someone help to solve this issue. Thanks in advance for your help.
@wf9a5m75

The location button comes from the native api. Not this plugin. Only possible way is specifying the padding.
https://github.com/mapsplugin/cordova-plugin-googlemaps-doc/blob/master/v2.0.0/class/Map/setPadding/README.md

ok. Thanks for this reply. I will try


export WorkerId constructor

This PR exports the WorkerId constructor. The reasoning for this is it allows us to extract the string value so we can actually use and report the worker id.

Well... the right way to do this would probably to export it in an Internal module. But 🤷🏼

Well... the right way to do this would probably to export it in an Internal module. But 🤷🏼

do you mean something like this? https://stackoverflow.com/questions/9190638/how-why-and-when-to-use-the-internal-modules-pattern

Just took a closer look at this. So:

Yes, that SO link correctly describes the Internal pattern. Basically, sometimes you want to export things for testing purposes or for extensibility purposes that normal users shouldn't be using but will sometimes need to in order to hack stuff together. To signal that, you export the symbol in Foo.Internal instead of Foo. This is a well-understood Haskell convention which means "if you use stuff from this module, you can break the invariants that my package expects, so be very careful".
In this particular case, I actually don't think we should be exporting the WorkerId's underlying constructor. Instead, it seems like what we're trying to do semantically is to actually add a Show instance that returns the underlying String. These two approaches allow us to accomplish the same thing, but they differ in the semantic meaning that they convey and differ in the additional capabilities that they allow. For example, if you export the WorkerId constructor, you give consumers the ability to construct arbitrary WorkerId values, and that's not actually what we care about. We just want a descriptive string for reporting purposes, so we should use Show because it's the weakest power (i.e. least allows-invariant-breaking) tool that allows us to accomplish our goal.

would something like this be better then?
instance Show WorkerId where
  show (WorkerId wid) = wid

I tried adding Show to the deriving strategy but it's adding quotes to the value

Yes, that's exactly what I mean. You might be able to get this behavior automatically by deriving newtype.


HttpRequestData.Cookies does not contain request cookies

I have migrated a .NET Core 3.1 Function to .NET 5 and updated the code accordingly.
I used to be able to access cookies from HttpRequest.Cookies and I updated it to HttpRequestData.Cookies, but the collection seems to be empty on every request.
I was able to find the expected cookies using HttpRequestData.Headers and extracting them from there.
Req: Cookie: jwtToken=VALUE
Code:
[<Function("myfunc")>]
let run ([<HttpTrigger(AuthorizationLevel.Function, "get")>]req: HttpRequestData) =
    req.Cookies // This is an empty collection

.NET 5.0.300
Core Tools Version:       3.0.3568 Commit hash: e30a0ede85fd498199c28ad699ab2548593f759b  (64-bit)
Function Runtime Version: 3.0.15828.0

Thank you

Thanks for reporting. We'll take a look at this.

I've been able to reproduce this in the following environment:
.NET 5.0.301
Core Tools Version:       3.0.3477 Commit hash: 5fbb9a76fc00e4168f2cc90d6ff0afe5373afc6d  (64-bit)
Function Runtime Version: 3.0.15584.0
Investigating root cause.

The RpcHttp object used to construct GrpcHttpRequestData (implementation of HttpRequestData) has an empty collection for its Cookies property, which GrpcHttpRequestData.Cookies gets its value from:
https://github.com/Azure/azure-functions-dotnet-worker/blob/72b9d17a485eda1e6e3626a9472948be1152ab7d/src/DotNetWorker.Grpc/Http/GrpcHttpRequestData.cs#L69
Headers (including Cookie) come in on RpcHttp.NullableHeaders.

Moving this to sprint 105 as we may need to understand if there's a problem in the host not populating the cookies collection int he HTTP representation.

Moving this to 107 as it is a more realistic target given the current capacity

The Cookies field is never populated in the host:
https://github.com/Azure/azure-functions-host/blob/4764c25d885cdfb1d84af3b80861b181ce658faa/src/WebJobs.Script.Grpc/MessageExtensions/GrpcMessageConversionExtensions.cs#L115`
Cookies are only passed through the headers, presumably to avoid duplicate data in the payload. We will be handling this bug on the worker side so it will only be a worker update not a host update.

The root cause is now clear. Moving this to 110 for implementation.


Grant ownership to the package JSON file for installation step

Motivation and Context
The change is required for local development when building docker images for testing purposes.
It fixes the  #1083 issue.
Description

The npm ci is not able to run if it has no ownership over the package.json files. Similar to other COPY phase, the --chown=node:node option was added.
Contribution Checklist

[x] I have built and tested the code locally and in a deployed app
[x] For frontend changes, I have pulled the latest code from main, built the frontend, and committed all static files.
[x] This is a change for all users of this app. No code or asset is specific to my use case or my organization.
[x] I didn't break any existing functionality :smile:

The redundant chown commands were removed in the 207bc6d778d89a55958062ba31fd21ce24fcded6 commit.


Revert "Replace ideviceinfo and idevice_id with xcdevice"

Reverts flutter/flutter#49854
Expected device was marked unavailable:
stdout:              {
stdout:                "simulator" : false,
stdout:                "operatingSystemVersion" : "12.4.4 (16G140)",
stdout:                "interface" : "usb",
stdout:                "available" : false,
2020-02-05T17:54:53.284643: stdout:                "platform" : "com.apple.platform.iphoneos",
stdout:                "modelCode" : "iPhone7,2",
stdout:                "identifier" : "e7184ae5fe05c2ea2ad4654ac70b4ba19c92f2c6",
stdout:                "architecture" : "arm64",
stdout:                "modelName" : "iPhone 6",
stdout:                "name" : "iPhone",
stdout:                "error" : {
stdout:                  "code" : -1,
stdout:                  "failureReason" : "",
stdout:                  "underlyingErrors" : [
stdout:                    {
stdout:                      "code" : -1,
stdout:                      "failureReason" : "",
stdout:                      "description" : "dyld_shared_cache_extract_dylibs failed",
stdout:                      "recoverySuggestion" : "",
2020-02-05T17:54:53.284695: stdout:                      "domain" : "com.apple.dtdevicekit"
stdout:                    },
stdout:                    {
stdout:                      "code" : -1,
stdout:                      "failureReason" : "",
stdout:                      "description" : "dyld_shared_cache_extract_dylibs failed",
stdout:                      "recoverySuggestion" : "",
stdout:                      "domain" : "com.apple.dtdevicekit"
stdout:                    }
stdout:                  ],
stdout:                  "description" : "dyld_shared_cache_extract_dylibs failed",
stdout:                  "recoverySuggestion" : "",
stdout:                  "domain" : "com.apple.dtdevicekit"
stdout:                }

Then errored:
No devices found with name or id matching 'e7184ae5fe05c2ea2ad4654ac70b4ba19c92f2c6'
smoke_catalina_start_up_ios_eb4b6dc_2.log

Merging to green build.

LGTM


Re-throw exception from LoggingRunnable (re #1808)

There is a robust fix in #1818 that is probably too much to backport to 1.10. This fix simply re-throws the original exception (unless there is an exception when logging and that is more serious (i.e., it's an Error and the original Throwable was not an Error)).

@brianloss When you merge these in the UI, GitHub usually appends the PR number as (#1865) (in this case) to the end of the subject line of the commit message. It's helpful to have these in the history, even if the commit additionally references other issues in the body of the git commit message (or elsewhere in the subject line). I noticed in this PR (and your previous one), these references to the PR in which the work was done were not included in the commit message when it was merged, so I wanted to bring this to your attention, because it's helpful to keep these, so it's easier to find the PR from the commit.

@brianloss When you merge these in the UI, GitHub usually appends the PR number as (#1865) (in this case) to the end of the subject line of the commit message. It's helpful to have these in the history, even if the commit additionally references other issues in the body of the git commit message (or elsewhere in the subject line). I noticed in this PR (and your previous one), these references to the PR in which the work was done were not included in the commit message when it was merged, so I wanted to bring this to your attention, because it's helpful to keep these, so it's easier to find the PR from the commit.


feat(tools): Add Crowdin PR creator action

Checklist:

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[x] My pull request targets the main branch of freeCodeCamp.
[x] All the files I changed are in the same world language, for example: only English changes, or only Chinese changes, etc.

This PR helps DRY out the existing Crowdin download work flows that @nhcarrigan put together.  The core logic was created in the code he created in the workflow.  I just adapted it to be in a stand-alone action we can use in the repo.  We could potentially move this action to a more global location to use it in other repos if we want to go that route.  I have made it generic enough to accomplish that.
TO DO before before ready to QA:

[x] Fix some linting issues
[x] Test it thoroughly

Thanks @nhcarrigan for reviewing and making the suggestions.  I think it is ready for others to review.


Bluesky: update homepage

Anything I can do to help with this?

Thanks for offering! This needs both broad and deep knowledge of BF's functionality, so it may not be the best place to start contributing.
More importantly though, one of the biggest blockers to launch right now isn't this, or other issues here, but that Bluesky's federation beta test is currently limited to 10 users per federated server. Not sure what their timeline is for lifting that.

That's good to know. I suppose I was thinking moreso in terms of styling and such lol.

Done!


Appstudio update mvp-test-component

Pipelines as Code configuration proposal

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
20 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
12 seconds
deprecated-base-image-check

✅ Succeeded
11 seconds
sanity-inspect-image

✅ Succeeded
1 minute
clamav-scan

✅ Succeeded
19 seconds
clair-scan

✅ Succeeded
13 seconds
sbom-json-check

✅ Succeeded
17 seconds
sanity-optional-label-check

✅ Succeeded
17 seconds
sanity-label-check

✅ Succeeded
5 seconds
show-summary

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
20 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
11 seconds
sanity-inspect-image

✅ Succeeded
12 seconds
deprecated-base-image-check

✅ Succeeded
19 seconds
clair-scan

✅ Succeeded
1 minute
clamav-scan

✅ Succeeded
13 seconds
sbom-json-check

✅ Succeeded
17 seconds
sanity-optional-label-check

✅ Succeeded
17 seconds
sanity-label-check

✅ Succeeded
5 seconds
show-summary


Phone number not valid after upgrading to 8.5.1

What

After upgrading our library to 8.5.1 , a certain phone number does not work and throws the error "Not a valid phone number"

Any reason why this may be happening ?

Hi,
Thank you for using libphonenumber!
We need more information to fix the issue. Please see our contribution guidelines.

We update the metadata frequently, and it is expected that different versions will give different results.
If there is a specific range that you think we should validate and don't do so in the latest release, please file an issue with all the details. See our contribution guidelines to determine whether something is a real issue, and the details we need to be able to fix it.

Closing as it's not an issue. Please re-open if you respond so we're notified.


pretrained model

Hi, thanks for your excellent work!
Could you provide related trained models on DDAD datasets for inference?

Sorry, readme already provided the pretrained models.


pods warning

What are your changes?
What type of changes does your doc introduce?

[X] Bug fix
[ ] New feature

What is the urgency level of this change?

[ ] Blocker
[ ] High
[X] Medium

I have completed these items:

[ ] I have tested all commands and code snippets.
[ ] I have built locally and tested for broken links and formatting.
[X] I have made this change for all affected versions (e.g. 1.7, 1.8, and 1.9).
[ ] My doc follows the style guide: https://github.com/dcos/dcos-docs#contributing.

Ping @emanic @sascala @joel-hamill for reviewing pull request changes.
If you included a comment with your commit, it appears here:

@joel-hamill  I'm not really sure what the right thing is to do here -- we don't have the "Experimental" label for the OSS site, do we?

@sascala there's an [open ticket|https://dcosjira.atlassian.net/browse/SITE-66?filter=-3] to get the badges added, but we do have a feature maturity topic etc. please use the same verbiage and format how we did pre-badges in enterprise (e.g. https://github.com/dcos/dcos-docs/blob/d117818b177702a15c2b807f66df889daca99a1c/1.8/usage/containerizers.md).


Update Composer dependencies (2024-03-15-00-13)

Visual regression test passed!
View the visual regression test report


Issuing credits to Michele

Issue 100000 credits to @michelezanini.

 Follow up
On Monday, 08-Apr-24 15:09:41 UTC by Gov4Git dev
Issued 100000 credits to member @michelezanini.


Add flag to have a nodes translation, scale and rotation blended with its animation

This fixes a serious problem with AnimationController  which prevents a developer from blending custom transformations to nodes while they are animating.
For example, in the game I am currently making, this is a character which has body sliders applied to it:

However, when a animation is playing, some limbs regrow back to their normal proportions: (see the front leg)

This change adds a backwards-compatible flag to the Node class. The flag defaults to false so all existing projects will be unaffected.

For 1001 reasons, we shouldn't merge this :

order of transforms is arbitrary
some may want to blend more than two transforms
the API (Node class) was not designed for that : multiple transforms shouldn't be a concern at Node level.
it kinda still the concept of node transform for a totally different use
it adds unnecessary transforms when blending is disabled
it breaks most of documentation about it

In other words, it's a quick and dirty approach (mostly dirty) that you could keep in your own fork and use in your game.
For a more general solution, you could either :

modify your animations programmatically, based on original animations and some transforms on top
create a custom animation controller (inspired by the libgdx one) supporting multiple animation layers

create a custom animation controller (inspired by the libgdx one) supporting multiple animation layers

I will give this a go, thank you!


Import by Table Issues

Describe the bug

Import by Table does not fill status dropdown
Jubilife City encounter does not autofill with encounter from table import

To Reproduce
Steps to reproduce the behavior:

Copy Box Data Report from the provided save
Import via table.
Go to Encounters page, and notice that none of the encounters have statuses
Also see that Jubilife City doesn't have an encounter despite Ivysaur being that encounter

Expected behavior

Encounters should have statuses automatically input.
Jubilife should have an encounter

Save File for testing

Should be resolved with 73fd003e2c185737d6ee8f31d3f07d6d70b4b260


MultiSegmentAudioPlayer: player capable of playing many file segments

The idea is that developers will have classes that represents their audio clips/segments adopt the protocol "StreamableAudioSegment"
StreamableAudioSegment says that their class will have these properties:

var audioFile: AVAudioFile { get }
var startingTime: TimeInterval { get }
var endingTime: TimeInterval { get }
var fileStartTime: TimeInterval { get }
var fileEndTime: TimeInterval { get }
var completionHandler: AVAudioNodeCompletionHandler? { get }

Then, any array of objects (named segments in below example) that have adopted the StreamableAudioSegment protocol will be able to be scheduled and played by this new audio playing using a simple method call:
audioPlayer.playSegments(audioSegments: segments, referenceTimeStamp: studioTime)
Example video using one player and many segments (sorry - no audio):
https://user-images.githubusercontent.com/15333030/140840514-94d5dad4-9ccc-475a-9815-b73b470328f3.mov
Open to suggestions and future development on this concept :)

Not sure what's going on with all of these Trailing Whitespace Violation callouts. I ran Format File using SwiftFormat

This is cool, are there some tests that you could write to ensure that this starts off on the right foot?

@aure sorry for the delay (busy couple of weeks). Tests have been added


Add sbt-dynamodb to the sbt plugins community page

https://github.com/localytics/sbt-dynamodb

Hi @bdarfler,
Thank you for your contribution! We really value the time you've taken to put this together.
Before we proceed with reviewing this pull request, please sign the Typesafe Contributors License Agreement:
http://www.typesafe.com/contribute/cla


Need clarification

This is in regards to Raw materials/Stock parts available in github repository.
In the parts sourcing excel sheet - In row 109 :
Aluinum Block 6061 MISC macheing 40x40x70 bar1
https://amzn.to/3wI5uVH
The size mentioned in the sheet is 40×40×70 (I assume in mm) but in the provided Amazon link- the size of the AL bar is 50mm×50mm×150mm.
Well, I need a clarification regarding this size issue. Does or doesn't the size matters? If I order from that Amazon link- the size will be bigger than required according to the sheet. So do I have to cut it later to match the size? Honestly, instead of doing that myself, I will better order a customized one if size really matters.

@KRALYN

I think it just needs to be 40x40x70mm or bigger.

The stock sizes are just recommendations, I'll make this clear in the BOM


This content is pretty lame.

It's for "getting" a big object full of information and it has no explanation of the content of that object nor a link to somewhere that does. The "Outputs" section is entirely boiler-plate.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: eb7a7897-2dc8-48d1-ef9d-515cc285e175
Version Independent ID: 70fcd483-a0e1-54c4-877d-944e7ad7b626
Content: Get-HostedContentFilterPolicy (exchange)
Content Source: exchange/exchange-ps/exchange/antispam-antimalware/Get-HostedContentFilterPolicy.md
Service: exchange-powershell
GitHub Login: @chrisda
Microsoft Alias: chrisda

@PreventRage, we typically don't describe the output of a cmdlet (the exception is reporting cmdlets) because there are often unmodifiable properties in the output. Typically, you can get a reckoning of the interesting (modifiable) properties by the parameters on the corresponding New or Set cmdlets.
Regarding the Output section, that is boilerplate, because it links off to programmatically-generated results of the output types (if available). In the case of this cmdlet, if you click on the link and scroll down to Get-HostedContentFilterPolicy and click the link in the Output Types column, it takes you to https://msdn.microsoft.com/library/Microsoft.Exchange.Data.Directory.SystemConfiguration.HostedContentFilterPolicy.aspx, and under that is https://msdn.microsoft.com/en-us/library/microsoft.exchange.data.directory.systemconfiguration.hostedcontentfilterpolicy_properties.aspx which is really just a retelling of the parameters that are available on New-HostedContentFilterPolicy and Set-HostedContentFilterPolicy.
We realize the click maze isn't ideal, but it would be monumental to keep that same information current and updated within each topic.


Reproducing of Lora Model  Result on MT-Bench

Recently, I attempted to fit the DPO on my own dataset.
Initially, I tried to reproduce the results of your LORA model( 7.43 on MT-Bench).
However, I encountered some issues.
Despite using all your parameters and data, here are my results on MT-Bench:

Model
MT-Bench

Zephyr-SFT-Lora-Own
6.37

Zephyr-DPO-Lora-Own
6.95

Then, I downloaded your models from here, and the results were nearly the same as mine.

Model
MT-Bench

Zephyr-SFT-Lora
6.4

Zephyr-DPO-Lora
6.93

DPO does help improve performance on MT-Bench, but I can't achieve a score of 7.43. Is there any difference between the model described in your paper and the model available on your homepage?
Or could it be the difference between the full  and LORA?
By the way, I truly love the "yaml style" argument parser; it's clear and elegant!
@edbeeching @lewtun

Perhaps related, I compared two full DPO-trained checkpoints  HuggingFaceH4/zephyr-7b-beta and alignment-handbook/zephyr-7b-dpo-full. The MT-bench results seem to be different as well. HuggingFaceH4/zephyr-7b-beta match the results in the paper, and alignment-handbook/zephyr-7b-dpo-full does not. See https://github.com/huggingface/alignment-handbook/issues/44
Could you please clarify where the 7.43 MT-bench score for LoRA-DPO was reported? As far as I know, the models reported from the technical report didn't use LoRA. They mentioned:

We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods.

The MT-bench score of full DPO is 7.34 (Table 1).

Perhaps related, I compared two full DPO-trained checkpoints HuggingFaceH4/zephyr-7b-beta and alignment-handbook/zephyr-7b-dpo-full. The MT-bench results seem to be different as well. HuggingFaceH4/zephyr-7b-beta match the results in the paper, and alignment-handbook/zephyr-7b-dpo-full does not. See #44
Could you please clarify where the 7.43 MT-bench score for LoRA-DPO was reported? As far as I know, the models reported from the technical report didn't use LoRA. They mentioned:

We did not experiment with parameter-efficient techniques such as LoRA (Hu et al., 2021), but expect similar results to hold with these methods.

The MT-bench score of full DPO is 7.34 (Table 1).

Yeah, it appears we obtained the same results, regardless of whether we used -full or -lora. I also found a model named Zephyr-7b-α in zephyr-7b-beta, which has a MT-Bench score of 6.88, nearly identical to ours.
Perhaps you could evaluate zephyr-7b-sft-full and check if there is any difference at the sft stage.

@liutianlin0121 @wlhgtc ,
@liutianlin0121
I ran the official model weights: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta
On MT-Bench, this is the result based on the current GPT-4 API call (I named the model zep-hf).
########## First turn ##########
               score
model  turn
zep-hf 1     7.10625

########## Second turn ##########
             score
model  turn
zep-hf 2      6.45

########## Average ##########
           score
model
zep-hf  6.778125

Seems like GPT4 API is pretty unstable now?

@liutianlin0121 @wlhgtc ,
I ran the official model weights: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta
On MT-Bench, this is the result based on the current GPT-4 API call (I named the model zep-hf).
########## First turn ##########
               score
model  turn
zep-hf 1     7.10625

########## Second turn ##########
             score
model  turn
zep-hf 2      6.45

########## Average ##########
           score
model
zep-hf  6.778125

Seems like GPT4 API is pretty unstable now?

Edit: To anyone that wants to replicate MT-Bench, just note that it will cost you $3 USD to run one whole evaluation for a single model.

Thanks for all your questions and detailed analysis, there are a number of different things to address here.
LoRA training:
The official zephyr-7b-beta model used full training. We provide LoRA training configs as an example for lower resource machines, but we do not expect the LoRA runs to achieve parity with the full finetune.
I would recommend targeting all linear layers with LoRA rather than the subset we have chosen in the config files. In addition, there is probably a better combination of hyperparameters as these were copied from the full training config. A rule of thumb is to use 100x learning rate when using LoRA, I do not know if this is valid for the DPO step.
MT Bench scores
There appears to be a regression somewhere when comparing the original zephyr-7b-beta model and the models trained in the handbook. There are a number of potential sources of this regression which we have not managed to pin down yet.
Here are our ideas so far:

A regression in TRL's SFT trainer
A regression in TRL's DPO trainer
The GPT-4 model used in MT Bench has changed since our last evals
A mismatch better our internal codebase (h4) and the alignment handbook

We still need to investigate this, but we will keep you up to date as we do so. This may take several weeks.

Thanks all! This is very helpful.
In light of @timothylimyl 's result (running the original zephyr-7b-beta on MT-bench), it seems that the difference of GPT4 API is a contributing factor. About 2 weeks back, I ran the same MT-bench eval as @timothylimyl did, and at that time the MT-bench score of the original zephyr-7b-beta is 7.37, which closely matched with the score reported in the paper. A few days ago, I ran the handbook version of zephyr-7b-beta, and the MT-bench score is 7.09. See https://github.com/huggingface/alignment-handbook/issues/44

We identified the regression using MT bench scores as well. We are rerunning the evals and other exps internally to try and get to the root cause.

Thanks for all your questions and detailed analysis, there are a number of different things to address here.
LoRA training:
The official zephyr-7b-beta model used full training. We provide LoRA training configs as an example for lower resource machines, but we do not expect the LoRA runs to achieve parity with the full finetune.
I would recommend targeting all linear layers with LoRA rather than the subset we have chosen in the config files. In addition, there is probably a better combination of hyperparameters as these were copied from the full training config. A rule of thumb is to use 100x learning rate when using LoRA, I do not know if this is valid for the DPO step.
MT Bench scores
There appears to be a regression somewhere when comparing the original zephyr-7b-beta model and the models trained in the handbook. There are a number of potential sources of this regression which we have not managed to pin down yet. Here are our ideas so far:

A regression in TRL's SFT trainer
A regression in TRL's DPO trainer
The GPT-4 model used in MT Bench has changed since our last evals
A mismatch better our internal codebase (h4) and the alignment handbook

We still need to investigate this, but we will keep you up to date as we do so. This may take several weeks.

Thanks for your quickly response.
According to my understanding： the model zephyr-7b-dpo-full is identical to zephyr-7b-beta. They are expected to yield approximately 7.34 in the MT-Bench. However, zephyr-7b-dpo-lora is only a sample and its results might be less than 7.43 due to the influence of hyperparameters or other factors.
Did I get it right? @edbeeching

Hi @wlhgtc
In fact zephyr-7b-beta was trained using an internal codebase. zephyr-7b-dpo-full was trained using code from this repo, with the same parameters as the internal codebase. This repo contains a subset of the code from our internal codebase.
zephyr-7b-dpo-lora has hyper-parameters that are the same as the full finetune, but includes LoRA with peft adapters.
There is no gaurantee that zephyr-7b-beta and zephyr-7b-dpo-full have the same performance on MT-Bench. We aim for them to be the same, but there may be a regression somewhere.

Hello everyone, thank you for raising this issue - we hope to get to the bottom of the discrepancy soon!
In the meantime, I double-checked if the source of the diff could be from GPT-4 in MT-Bench and can confirm that it's still pretty stable. To be safe, I reran evals for openchat-3.5 and vicuna-7b-v1.5 so we can compare to the official leaderboard (new evals have a date suffix of 2023-11-23):
########## Average ##########
                                      score
model
openchat-3.5_2023-11-23            7.825000
openchat-3.5                       7.81
zephyr-7b-beta_2023-11-23          7.358491
zephyr-7b-beta                     7.34
vicuna-7b-v1.5                     6.17
vicuna-7b-v1.5_2023-11-23          6.034375

With this eliminated, the next thing is to check for the other points in @edbeeching's post above (regressions or issues with porting the code). We'll report back here once we have some new information!

@lewtun Thanks so much! Just to make sure that I understand: I suppose both zephyr-7b-beta_2023-11-23 and zephyr-7b-beta in the table above are based on the original model (HuggingFaceH4/zephyr-7b-beta), and not the handbook model (alignment-handbook/zephyr-7b-dpo-full), right?

Hello everyone, thank you for raising this issue - we hope to get to the bottom of the discrepancy soon!
In the meantime, I double-checked if the source of the diff could be from GPT-4 in MT-Bench and can confirm that it's still pretty stable. To be safe, I reran evals for openchat-3.5 and vicuna-7b-v1.5 so we can compare to the official leaderboard (new evals have a date suffix of 2023-11-23):
########## Average ##########
                                      score
model
openchat-3.5_2023-11-23            7.825000
openchat-3.5                       7.81
zephyr-7b-beta_2023-11-23          7.358491
zephyr-7b-beta                     7.34
vicuna-7b-v1.5                     6.17
vicuna-7b-v1.5_2023-11-23          6.034375

With this eliminated, the next thing is to check for the other points in @edbeeching's post above (regressions or issues with porting the code). We'll report back here once we have some new information!

Please correct me if I am misunderstanding this. Do you mean that the regression in results is caused a different release by hugging face compared to the official paper? The signal for this to me is running the official release zephyr-beta on HF on MT-Bench and observing the result now.
Another issue signalling regression due to porting of code (your internal to alignment handbook) is that I could not seem to train the model to be anywhere close. I went ahead and try using the official sft-full model and ran DPO on it as per the recipe and gotten this result:
########## First turn ##########
               score
model   turn
zep-own 1     6.6875

########## Second turn ##########
               score
model   turn
zep-own 2     5.8625

########## Average ##########
         score
model
zep-own  6.275

You can find my replication model for sft-full to dpo-full: https://huggingface.co/timlim123/zephyr-7b-dpo-full
Lastly, @lewtun, I am confused by the evaluation loss report provided.  The loss reported does not match any of the loss value in the training table.

Hello @timothylimyl can you share which commit of the FastChat repo you are using to compute the MT bench scores?
I just noticed there is a bug with the chat templates introduced 6h ago (https://github.com/lm-sys/FastChat/pull/2725) which I've now proposed a fix for in https://github.com/lm-sys/FastChat/pull/2727
It's possible that your generations are using the wrong chat template (Hermes2 instead of Zephyr) and that's why your scores are so much lower. A quick way to test this is to see what happens if you try to chat with your model via their CLI:
python3 -m fastchat.serve.cli --model-path timlim123/zephyr-7b-dpo-full --debug

Please correct me if I am misunderstanding this. Do you mean that the regression in results is caused a different release by hugging face compared to the official paper? The signal for this to me is running the official release zephyr-beta on HF on MT-Bench and observing the result now.

Yes, it seems there are two remaining sources of discrepancy:

either a regression in the TRL trainers for SFT/DPO occurred in the time between our paper & the handbook release
a bug was introduced when porting our code from the internal codebase to the handbook

We're trying to pin down the root cause and will report back here when we have some results.

Lastly, @lewtun, I am confused by the evaluation loss report provided. The loss reported does not match any of the loss value in the training table.

Can you please clarify exactly what you're comparing against?

Hello @timothylimyl can you share which commit of the FastChat repo you are using to compute the MT bench scores?
I just noticed there is a bug with the chat templates introduced 6h ago (lm-sys/FastChat#2725) which I've now proposed a fix for in lm-sys/FastChat#2727
It's possible that your generations are using the wrong chat template (Hermes2 instead of Zephyr) and that's why your scores are so much lower. A quick way to test this is to see what happens if you try to chat with your model via their CLI:
python3 -m fastchat.serve.cli --model-path timlim123/zephyr-7b-dpo-full --debug

Please correct me if I am misunderstanding this. Do you mean that the regression in results is caused a different release by hugging face compared to the official paper? The signal for this to me is running the official release zephyr-beta on HF on MT-Bench and observing the result now.

Yes, it seems there are two remaining sources of discrepancy:

either a regression in the TRL trainers for SFT/DPO occurred in the time between our paper & the handbook release
a bug was introduced when porting our code from the internal codebase to the handbook

We're trying to pin down the root cause and will report back here when we have some results.

Lastly, @lewtun, I am confused by the evaluation loss report provided. The loss reported does not match any of the loss value in the training table.

Can you please clarify exactly what you're comparing against?

Hi @lewtun ,
The FastChat repo I used is the current master (as of 23/11/23).
name = "fschat"
version = "0.2.32"

When running under FastChat inference debug mode for the dpo-full model that I trained,  the responses are good:

In regards the the loss discrepancy I am mentioning, the loss value posted cannot be found anywhere in the table of results:

@lewtun An add-on query: It seems that the system prompts are left blank? Is this done so internally too?

@timothylimyl

In regards the the loss discrepancy I am mentioning, the loss value posted cannot be found anywhere in the table of results

You mean the loss value 0.7496 reported in the "Training and evaluation" section did not occur in the table in the "Training results" section, or?
If so, this is because the loss value 0.7496 is the final loss after the training, and the table in the "Training results" reports evaluation every 100 steps. There are 5811 steps in total, so you cannot see the last step's loss in the table. I checked your tensorboard record, and the 0.7496 shows up there. See the screenshot attached, where I circled the final loss and the time step where this loss occurs.
Hope this helps!

@timothylimyl

In regards the the loss discrepancy I am mentioning, the loss value posted cannot be found anywhere in the table of results

You mean the loss value 0.7496 reported in the "Training and evaluation" section did not occur in the table in the "Training results" section, or?
If so, this is because the loss value 0.7496 is the final loss after the training, and the table in the "Training results" reports evaluation every 100 steps. There are 5811 steps in total, so you cannot see the last step's loss in the table.
Take your experiment https://huggingface.co/timlim123/zephyr-7b-dpo-full as an example. The final loss reported just below "zephyr-7b-dpo-full" is 0.7337, and it is not in the table. However, I checked your tensorboard record, and the 0.7496 shows up there. See the screenshot attached, where I circled the final loss and the time step where this loss occurs.
Hope this helps!

@liutianlin0121
Thanks for clearing that up!  I wrongly assumed that the loss reported would be the lowest validation loss, should had check the logs.
@lewtun
Is the official model deployed on hugging face then:
a) The final checkpoint/weight
b) The lowest training loss checkpoint
c) The lowest validation loss checkpoint

@lewtun  seems like on the new pull of fastchat (27/11/2023), the results are further degraded for the official mode zephry-beta modell:
########## First turn ##########
                    score
model        turn
zep-hf-fixed 1     7.1375

########## Second turn ##########
                   score
model        turn
zep-hf-fixed 2     6.175

########## Average ##########
                score
model
zep-hf-fixed  6.65625

@lewtun
Is the official model deployed on hugging face then:
a) The final checkpoint/weight b) The lowest training loss checkpoint c) The lowest validation loss checkpoint

Lewis may want to correct me, but I believe that its a) the final checkpoint. The reason is that the training loss or validation loss are not reliable indicators of the DPO-trained model's downstream performance, as mentioned in the paper:

In the process of training ZEPHYR-7B we observed that after one epoch of DPO training, the model would strongly overfit, as indicated by perfect training set accuracies in Figure 3. Surprisingly, this did not harm downstream performance on MT-Bench and AlpacaEval; as shown in Figure 3, the strongest model was obtained with one epoch of SFT followed by three epochs of DPO.

Given this, selecting the checkpoints based on train or validation loss seems to be no-more reliable than just use the last checkpoint.

@lewtun seems like on the new pull of fastchat (27/11/2023), the results are further degraded for the official mode zephry-beta modell:
########## First turn ##########
                    score
model        turn
zep-hf-fixed 1     7.1375

########## Second turn ##########
                   score
model        turn
zep-hf-fixed 2     6.175

########## Average ##########
                score
model
zep-hf-fixed  6.65625

Please ignore this, I realised that FastChat MT-Bench is not smart enough to detect the correct model, and uses the name instead, so I just needed to name the model zephyr instead of zep:
########## First turn ##########
               score
model  turn
zephyr 1     7.68125

########## Second turn ##########
             score
model  turn
zephyr 2     6.975

########## Average ##########
           score
model
zephyr  7.328125

@timothylimyl Thanks for verifying this! Really curious about what causes the regression...

@edbeeching @lewtun May I ask if there are updates regarding the regression? Even a partial solution or some intuitions would be tremendously helpful. Thanks!

Hi @edbeeching ! Thanks a lot! I went ahead and did some debugging myself. I just finished the training of zephyr-7b using the handbook recipe + 2 changes. I can reproduce the MT-bench scores now.

My zephyr model on huggingface hub: link

The MTbench evaluation on colab: link. The MT-bench score is 7.390625, matching the original result reported in the paper.

The two small changes I made are the following:

Instead of using a global batch size 64, use a global batch 32. The global batch size 32 is consistent with the number reported in the paper. It is also used in the official model. However, batchsize 64 was used to train the handbook model. So I switch back to 32.

Instead of using the SFT checkpoint alignment-handbook/zephyr-7b-sft-full, I used the SFT checkpoint HuggingFaceH4/mistral-7b-sft-beta.

The full training recipe of the DPO training is thus the following:
# Model arguments
model_name_or_path: HuggingFaceH4/mistral-7b-sft-beta

# Data training arguments
# For definitions, see: src/h4/training/config.py
dataset_mixer:
  HuggingFaceH4/ultrafeedback_binarized: 1.0
dataset_splits:
- train_prefs
- test_prefs
preprocessing_num_workers: 12

# DPOTrainer arguments
bf16: true
beta: 0.1
do_eval: true
evaluation_strategy: steps
eval_steps: 500
gradient_accumulation_steps: 1
gradient_checkpointing: true
hub_model_id: zephyr-7b-dpo-full-debug-regression
learning_rate: 5.0e-7
log_level: info
logging_steps: 10
lr_scheduler_type: linear
max_length: 1024
max_prompt_length: 512
num_train_epochs: 3
optim: rmsprop
output_dir: data/zephyr-7b-dpo-full-debug-regression
per_device_train_batch_size: 4 # With 8 gpus, the global batch size is 32.
per_device_eval_batch_size: 4
push_to_hub: true
save_strategy: "no"
save_total_limit: null
seed: 42
warmup_ratio: 0.1

See the tensorboard record for more details.
For now, I am not sure if both changes are needed. But I suspect that the SFT checkpoint plays a greater role.

@liutianlin0121
Did you run MT-Bench on HuggingFaceH4/mistral-7b-sft-beta?
I am curious on whether does the SFT model itself already achieve a high result on MT-Bench.


Add missing docs in RabbitMQ and NATS modules

PR to ballerina 1.2.x - https://github.com/ballerina-platform/ballerina-lang/pull/22219
Purpose

$title

Fixes #21652
Fixes #21676
Check List

[x] Read the Contributing Guide
[ ] Updated Change Log
[ ] Checked Tooling Support (#)
[ ] Added necessary tests

[ ] Unit Tests
[ ] Spec Conformance Tests
[ ] Integration Tests
[ ] Ballerina By Example Tests

[ ] Increased Test Coverage
[ ] Added necessary documentation

[ ] API documentation
[ ] Module documentation in Module.md files
[ ] Ballerina By Examples

Codecov Report

Merging #22253 into master will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #22253   +/-   ##
=======================================
  Coverage   14.59%   14.59%
=======================================
  Files          51       51
  Lines        1411     1411
  Branches      219      219
=======================================
  Hits          206      206
  Misses       1189     1189
  Partials       16       16

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 99fc76d...fb77ca7. Read the comment docs.


proxified images in gallery

@talhasch Gallery is only shown if USE_PRIVATE = 1 right?!

yes, it is. but need to modify new mobile image dialog for that. will push another PR soon. you can merge this if it is ok.


MultiplayerSynchronizer and 4.1

Sorry to create an issue for a question but couldn't find any other place to reach out.
I've been looking for a solution for my multiplayer FPS game that has functionality for predicition & reconciliation. Looks like the project is dead on however what's not clear from the doc is

is it compatible with 4.1 or upcoming 4.2
is it a complete replacement for MultiplayerSyncronizer and MultiplayerSpawner nodes or is it supposed to be complimentary
apart from the example project are there any documents? is this an addon/extension? does it need to be compiled, how to start using etc?

thank you

Hello, no problem for opening an issue and I'm sorry that I'm replaying only now. I don't receive any notification when someone opens a PR or an issue on this repository, and I'm unable to figure out why.
Anyway, regarding your questions:

I've been looking for a solution for my multiplayer FPS game that has functionality for predicition & reconciliation.

This is exactly what you are looking for.

... Looks like the project is dead on however ...

This project is not dead and is currently in development and maintained. Said that, there is no doc right now, but I plan to add it soon.

is it compatible with 4.1 or upcoming 4.2

Yes it is.

is it a complete replacement for MultiplayerSyncronizer and MultiplayerSpawner nodes or is it supposed to be complimentary

It's supposed to be complementary to MultiplayerSpawner and a replacement for MultiplayerSyncronizer.

apart from the example project are there any documents?

The only documentation you can find is inside the C++ classes. I'm planning to write some docs soon.

is this an addon/extension?

This is a module and not a extension. Maybe in future this will became an extension, but not yet.

does it need to be compiled, how to start using etc?

You should be able to compile this module just by copying (or cloning using a submodule) this project into the godot modules directory.

Ask any other question here https://github.com/GameNetworking/network_synchronizer/discussions/categories/general I'll take care to answer there. In this way we can keep the issue section clear and each question on its own page.


Substract current automaton size to Wp-method's maxDepth

I probably got it wrong, but after checking the original paper it seems that the "middle part" length is the difference between implementation's bound on number of states (m) and the current spec size (n), so when using Wp-method as equivalence query the maxDepth could be shrunk dynamically (by substracting the current automaton.size()). Right?
Right now the length is fixed idependently of the candidate automaton: https://github.com/LearnLib/automatalib/blob/master/util/src/main/java/net/automatalib/util/automata/conformance/WpMethodTestsIterator.java#L81
I just started looking at the code base, so I'm might be missing something. Should/Can I update the equivalence oracle inside my learning loop to reduce the max depth?

You are correct, currently the parameter is independent of the hypothesis size and used to insert all tuples of length l \in {0, ..., maxDepth} between elements of the P and W sets. I'll have to look at the paper to see what the authors intentions for this case are (I haven't originally implemented it in AutomataLib and only know the Wp-method with maxDepth = 0).
What I see as a potential problem with shrinking the value is, that a bad estimate will degrade performance in later equivalence checks, because once n > m you basically have no more look-ahead. With a fixed exploration depth (as it is now), you continue to look ahead even in later iterations of refinement cycles. On the contrary, a too low value for the current value of maxDepth probably gives you the same problems in the earlier equivalence checks. Maybe, we can use two parameters (one for m and one for the minimal exploration depth depth), and always choose max(depth, m - n), so users have more fine-grained control over the behaviour. What do you think?
For now, to simulate shrinking exploration depth, you can just instantiate a new equivalence for every refinement loop, like (untested):
DFALearner<I> learner = ...;
learner.startLearning();

int m = ...;
boolean finished = false;

while (!finished) {
    DFA<?, I> hyp = learner.getHypothesisModel();

    DFAEquivalenceOracle<I> eqOracle = new DFAWpMethodEQOracle<>(mqOracle, m - hyp.size());
    DefaultQuery<I, Boolean> ce;

    if ((ce = eqOracle.findCounterExample(hyp, inputs)) != null) {
        learner.refineHypothesis(ce);
    } else {
    	finished = true;
    }
}

Thanks for the quick answer.
My main point is that if I know the upper bound size for the automaton I want to learn (m in the paper), having a fixed value will require a bunch of unnecessary queries when n (size of current hypothesis) gets bigger. But I agree that a bad estimate (or no information) can also be a problem.
Your solution makes sense to me: m would be the upper bound (which can be optional), and depth the minimal exploration depth (in the worst case, i.e. if no counterexamples are found before).


Calling present? on an iframe will return an error if iframe cannot be located

Calling present? should return a boolean. When called on an iframe it is able to return true but will throw an error instead of returning false.
logs:
https://gist.github.com/WildeRunner/18a9898e99a1edccba07bd4e104348bd

Do you have the code you used to get this log?
Our specs are passing for it: https://github.com/watir/watir/blob/master/spec/watirspec/elements/iframe_spec.rb#L44-L55
You might also try with Watir > 6.4 since we made a significant update to how things are located & errors are rescued.

If you can send us a way to duplicate this bug or verify it is still an issue on the latest version of Watir, I will reopen. Thanks.


.net standard compatibility

Ca your cardiview be compatible with .net standard in the future ?

@MaxenceMax Hi
It's compatible now.
Can't you install package?

You got right, sorry for this question, i didn't try to install it, i just read the readme. Nice work on this !

@MaxenceMax thanks for feedback
I will be grateful if you set star on this pepo) it helps me to develop this plugin