[BUG] Timestamp incorrect in event logs, dated in the future by 7 days/1 week

Describe the bug
While analyzing event and debug logs generated by Falco, we discovered that the events were dated in the future.
For example, we saw an debug entry on<PHONE_NUMBER> (around Mon, 27 Feb 2023 19:50:44 GMT), I identified the following entry with the following timestamp 2023-03-06T11:56:34.693685797Z:
{
  "hostname": "redacted",
  "output": "Falco internal: syscall event drop. 116192 system calls dropped in last second.",
  "output_fields": {
    "ebpf_enabled": "0",
    "n_drops": "116192",
    "n_drops_buffer_clone_fork_enter": "0",
    "n_drops_buffer_clone_fork_exit": "0",
    "n_drops_buffer_connect_enter": "0",
    "n_drops_buffer_connect_exit": "0",
    "n_drops_buffer_dir_file_enter": "0",
    "n_drops_buffer_dir_file_exit": "0",
    "n_drops_buffer_execve_enter": "0",
    "n_drops_buffer_execve_exit": "0",
    "n_drops_buffer_open_enter": "0",
    "n_drops_buffer_open_exit": "0",
    "n_drops_buffer_other_interest_enter": "0",
    "n_drops_buffer_other_interest_exit": "0",
    "n_drops_buffer_total": "116192",
    "n_drops_bug": "0",
    "n_drops_page_faults": "0",
    "n_drops_scratch_map": "0",
    "n_evts": "430758"
  },
  "priority": "Debug",
  "rule": "Falco internal: syscall event drop",
  "time": "2023-03-06T11:56:34.693685797Z"
}

Around an event generated on
date +%s ; date -u
1677527950
Mon 27 Feb 2023 07:59:10 PM UTC

The following event was observed with timestamp<PHONE_NUMBER>443642000, 2023-03-06T13:27:09.443642063Z:
{
  "hostname": "redacted",
  "priority": "Warning",
  "rule": "Read sensitive file untrusted",
  "source": "syscall",
  "tags": [
    "T1020",
    "T1083",
    "T1212",
    "T1552",
    "T1555",
    "container",
    "filesystem",
    "host",
    "mitre_credential_access",
    "mitre_discovery"
  ],
  "time": "2023-03-06T13:27:09.443642063Z",
  "output_fields": {
    "container.id": "host",
    "container.image.repository": null,
    "evt.time":<PHONE_NUMBER>443642000,
    "fd.name": "/etc/pam.d/common-session",
    "proc.aname[2]": "systemd",
    "proc.aname[3]": null,
    "proc.aname[4]": null,
    "proc.cmdline": "sshd -D -R",
    "proc.name": "sshd",
    "proc.pid": 3225785,
    "proc.pname": "sshd",
    "user.loginuid": -1,
    "user.name": "root"
  }
}

How to reproduce it
Unsure, installation was from HEAD @ 0de9af9 via systemd on-metal.
Generate events and look at log messages?
Expected behaviour
Timestamp should be accurate / not in the future.
Since the epoch times are pretty similar,<PHONE_NUMBER>443642000 and<PHONE_NUMBER>; perhaps the string generated in the time field is incorrect (~7 days in the future, or<PHONE_NUMBER>000 in nanoseconds epoch)?
Environment

Falco version:

{
  "default_driver_version": "e1d0fd9b043f1c7dfd91c9d030c11cfe2c062931",
  "driver_api_version": "4.0.0",
  "driver_schema_version": "2.0.0",
  "engine_version": "16",
  "falco_version": "0.34.1-1-da4e36e4",
  "libs_version": "e1d0fd9b043f1c7dfd91c9d030c11cfe2c062931",
  "plugin_api_version": "2.0.0"
}

System info:

{
  "machine": "x86_64",
  "nodename": "redacted",
  "release": "5.15.90-custom",
  "sysname": "Linux",
  "version": "#1 SMP Mon Sep 27 00:00:00 UTC 2010"
}

OS:

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel:

Linux redacted 5.15.90-custom #1 SMP Mon Sep 27 00:00:00 UTC 2010 x86_64 GNU/Linux

Installation method:

from source, HEAD @ 0de9af9 via DEB/systemd on-metal

Additional context
Possibly related to https://github.com/falcosecurity/libs/pull/595 ? (Did a search from prior PRs related to time.)
Discussion on Slack community: https://kubernetes.slack.com/archives/CMWH3EH32/p1677517061599169

Hi Stanley, thanks for reporting this. During development of https://github.com/falcosecurity/libs/pull/595 you linked to I noticed some time travel issues as well, that is why I force changed my entire dev Linux box to UTC not just the configs and then the times made sense again.
What was changed in above PR was the derivation of the host boot ts, new approach vs we kept the old one as fall-back. Rest of the changes were around the new fields introduced so they are unrelated. The new approach to derive boot time is the same approach we use to derive the process start time when doing the initial proc scan. Boot time is involved in deriving epochs when the kernel only offers monotonic times.
Wondering if someone has more context around this?

Hey team,
Did some preliminary investigation:

not a problem with the way my misbehaving systems were configured (timezone, etc.). Confirmed by copying the functions/code used in https://github.com/falcosecurity/falco/blob/93ae6bb6098e7563f94738839715c2d7ee8f5ab5/userspace/falco/logger.cpp into a test.cpp to generate a timestamp

not a problem with JSON output. Confirmed by turning json_output: false and time_format_iso_8601: false

Will do some more digging as time permits. My next "excavation" will be where/how the timestamp is recorded when an event is traced/processed.

yeah probably I would try to reproduce it with the scap-open example, if we cannot see something probably we can go with the sinsp-example... I've tried both on my machine for some hours but nothing happens :(
I'm pretty sure that the issue is related to what we print and not to what we collect from drivers but still not sure how to spot it :/

Hey team,
Thanks for the pointers in this ticket, the Slack discussion, and the community call. I was able to reproduce the erroneous timestamp on one of my Falco-evaluation systems by building the libs and running scap-open.
Here is a sample output (note: I'm not sure I'm using the scap-open arguments correctly so please correct me if I'm wrong!):
schan@host:~/libs/build$ date -u && date +%s && date +%s%N; sudo ./libscap/examples/01-open/scap-open --modern_bpf --evt_type execve --num_events 10
Wed 22 Mar 2023 08:20:11 PM UTC
1679516411
1679516411159811048

[SCAP-OPEN]: Hello!

---------------------- SCAP SOURCE ----------------------
* Modern BPF probe, 1 ring buffer every 1 CPUs
-----------------------------------------------------------

--------------------- CONFIGURATIONS ----------------------
* Print single event type: 0 (`-1` means no event to print).
* Run until '10' events are catched.
-----------------------------------------------------------

---------------------- INTERESTING SYSCALLS ----------------------
* All sc codes are enabled!
------------------------------------------------------------------

* OK! modern BPF probe correctly loaded: NO VERIFIER ISSUES :)
* Live capture in progress...
* Press CTRL+C to stop the capture

------------------ EVENT: 0 TID:612821
------ HEADER
timestamp:<PHONE_NUMBER>583734
tid: 612821
len: 34
type: 0
num params: 2
------
------ PARAMS
PARAM 0: 16
PARAM 1: 172
------
------------------

------------------ EVENT: 0 TID:612821
------ HEADER
timestamp:<PHONE_NUMBER>646014
tid: 612821
len: 34
type: 0
num params: 2
------
------ PARAMS
PARAM 0: 108
PARAM 1: 135
------
------------------

------------------ EVENT: 0 TID:769419
------ HEADER
timestamp:<PHONE_NUMBER>763854
tid: 769419
len: 34
type: 0
num params: 2
------
------ PARAMS
PARAM 0: 178
PARAM 1: 115
------
------------------

---------------------- STATS -----------------------
Events correctly captured (SCAP_SUCCESS): 10
Seen by driver: 259
Time elapsed: 0 s
Number of dropped events: 0
Number of timeouts: 1
Number of 'next' calls: 11
Number of dropped events caused by full buffer (total / all buffer drops - includes all categories below, likely higher than sum of syscall categories): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_exit syscall category): 0
Number of dropped events caused by full scratch map: 0
Number of dropped events caused by invalid memory access (page faults): 0
Number of dropped events caused by an invalid condition in the kernel instrumentation (bug): 0
Number of preemptions: 0
Number of events skipped due to the tid being in a set of suppressed tids: 0
Number of threads currently being suppressed: 0
-----------------------------------------------------

[SCAP-OPEN]: Bye!

Additionally, I can validate that all the hosts I am evaluating Falco on reports an incorrect timestamp; the timestamp "offset" is inconsistent and does not seem to stem from incrementing the correct timestamps by the same amount.

I think a next good step, is to trace through where the ts value gets set in the scap_evt object.   Could add a print statement there, to see if the timestamp changes from assignment to where it's being printed in scap-example.

Hey, @Happy-Dude this is not a bug is a feature! In this way you can read events from the future! :rofl:
Apart from jokes, some minor notes:
scpa-open usage
Scap-open is a really low-level tool so unfortunately, it cannot understand something like --evt_type execve after --evt_type it wants an event num, you can find them here :point_down:
https://github.com/falcosecurity/libs/blob/f5412a4ced0e0b6b46b735e48d4e12900e0b0c85/driver/ppm_events_public.h#L1098
So in this case if you want the execve exit event you have to use --evt_type 293. The full command will be
sudo ./libscap/examples/01-open/scap-open --modern_bpf --evt_type 293 --num_events 10

Probably  --num_events 10 is not enough to catch something interesting so feel free to bump it or to remove directly it stopping the capture with CTRL+C when you want
timestamp issue
Coming back to the issue, I've tried the same scap-open setup but again i didn't see almost anything strange :( BTW looking at your data it seems that the boot time in your case is completely wrong. The event timestamp is composed of 2 parts: the boot_time and the number of nanosecond from the boot_time. The boot_time is computed at initialization time by us, while the nanoseconds from the boot_time are provided by a bpf helper bpf_ktime_get_boot_ns
hdr->ts = maps__get_boot_time() + bpf_ktime_get_boot_ns();

Now supposing that there is no bug in the bpf helper (this doesn't seem the case because if I look at the increment of your events in the scap-open logs I can see that the increment in nanoseconds seems almost good) probably there is something wrong in the boot time computation...
This is a minimal branch https://github.com/Andreagit97/libs/tree/timestamp_issue that prints some info about the boot_time when scap-open starts, you could compile it from this branch and maybe try it again, WDYT? Of course, patches are really simple https://github.com/Andreagit97/libs/commit/da7f52cb17503022cbbc8a2646d7fc6b38e38c9d so you can apply them by hand to your libs version if you prefer
Another possible test would be to use scap-open with the kmod (--kmod) in this case the event timestamp is completely computed by the kernel so in this case i don't expect any issue

@Happy-Dude i'd also check the output of stat /proc/1; can you share it?
Thank you!

So in this case if you want the execve exit event you have to use --evt_type 293. The full command will be
sudo ./libscap/examples/01-open/scap-open --modern_bpf --evt_type 293

Awesome, appreciate the context + explanation and will use this command.

This is a minimal branch https://github.com/Andreagit97/libs/tree/timestamp_issue that prints some info about the boot_time when scap-open starts, you could compile it from this branch and maybe try it again, WDYT?

Just built a new scap-open and got the following output:
schan@host:~/libs/build$ sudo ./libscap/examples/01-open/scap-open --modern_bpf --evt_type 293 --num_events 75000

[SCAP-OPEN]: Hello!

---------------------- SCAP SOURCE ----------------------
* Modern BPF probe, 1 ring buffer every 1 CPUs
-----------------------------------------------------------

--------------------- CONFIGURATIONS ----------------------
* Print single event type: 293 (`-1` means no event to print).
* Run until '75000' events are catched.
-----------------------------------------------------------

---------------------- INTERESTING SYSCALLS ----------------------
* All sc codes are enabled!
------------------------------------------------------------------

************ proc_dir: /proc/1/
************ proc_dir: /proc/1/
************ boot_time:<PHONE_NUMBER>0

* OK! modern BPF probe correctly loaded: NO VERIFIER ISSUES :)
* Live capture in progress...
* Press CTRL+C to stop the capture

------------------ EVENT: 293 TID:4034839
------ HEADER
timestamp:<PHONE_NUMBER>411786
tid: 4034839
len: 858
type: 293
num params: 27
------
------ PARAMS
PARAM 0: 0
PARAM 1: /usr/bin/dig
PARAM 2: <omitted>
PARAM 3: 4034839
PARAM 4: 4034839
PARAM 5: 3437214
PARAM 6: is empty
PARAM 7: 65536
PARAM 8: 0
PARAM 9: 4
PARAM 10: 420
PARAM 11: 4
PARAM 12: 0
PARAM 13: dig
PARAM 14: <omitted>
PARAM 15: <omitted>
PARAM 16: 0
PARAM 17: 4034839
PARAM 18: -1
PARAM 19: 0
PARAM 20: 8192
PARAM 21: 8192
PARAM 22: 8192
PARAM 23: 1375
PARAM 24:<PHONE_NUMBER>0
PARAM 25:<PHONE_NUMBER>000000000
PARAM 26: 2138
------
------------------

<omitted some events>

---------------------- STATS -----------------------
Events correctly captured (SCAP_SUCCESS): 75000
Seen by driver: 75760
Time elapsed: 0 s
Number of dropped events: 0
Number of timeouts: 239
Number of 'next' calls: 75239
Number of dropped events caused by full buffer (total / all buffer drops - includes all categories below, likely higher than sum of syscall categories): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_exit syscall category): 0
Number of dropped events caused by full scratch map: 0
Number of dropped events caused by invalid memory access (page faults): 0
Number of dropped events caused by an invalid condition in the kernel instrumentation (bug): 0
Number of preemptions: 0
Number of events skipped due to the tid being in a set of suppressed tids: 0
Number of threads currently being suppressed: 0
-----------------------------------------------------

[SCAP-OPEN]: Bye!

i'd also check the output of stat /proc/1

schan@host:~/libs/build$ stat /proc/1
  File: /proc/1
  Size: 0               Blocks: 0          IO Block: 1024   directory
Device: 15h/21d Inode: 20203       Links: 9
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 1970-01-01 00:00:12.721000000 +0000
Modify: 1970-01-01 00:00:12.721000000 +0000
Change: 1970-01-01 00:00:12.721000000 +0000
 Birth: -

So, the issue here is that /proc/1 has a super old change time (basically epoch).
But i'd expect event timestamps to be relative to 1 jan 1970, not in the future.
We will investigate further.

Awesome awesome; any further actions on my side atm?
If this might be a specific config in our environment, I can discuss with internal teams on why this might be the case.
Also forgot to answer Andrea re: kmod option; unfortunately, won't be able to do this easily on my side as loading kernel modules would require communication, approvals, etc. with my org and stakeholders.

Can you share uptime output? :)

You know what ahhaah there is no issues here, just that sites like epochconverter automatically switch base depending on number of digits of the number.
Therefore, in epochconverter,

12721000000 is  GMT: Saturday 10 February 2373 19:06:40
1946299696411786 is GMT: Thursday 4 September 2031 14:48:16.411

Using another tool, ie: https://www.epoch101.com/, you'll discover that:

12721 (ie:<PHONE_NUMBER>0 parsed as milliseconds) means  Thu, 01 Jan 1970 00:00:12 GMT
1946299696 (ie:<PHONE_NUMBER>411786 parsed as milliseconds)  means Fri, 23 Jan 1970 12:38:19 GMT

Basically, the even is happening ~22days after you booted up your PC (and that is a validated information from your uptime output.)
The issue here is not about events timestamp, but is all about why your /proc/1 has a very low epoch change time...

Most probably you are mounting your proc filesystem with noatime mount option, right?
Can you share mount | grep proc?

mount | grep proc

schan@host:~/libs/build$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

Just built and ran:
schan@host:~/libs/build$ sudo ./libscap/examples/01-open/scap-open --modern_bpf --evt_type 293 --num_events 75000

[SCAP-OPEN]: Hello!

---------------------- SCAP SOURCE ----------------------
* Modern BPF probe, 1 ring buffer every 1 CPUs
-----------------------------------------------------------

--------------------- CONFIGURATIONS ----------------------
* Print single event type: 293 (`-1` means no event to print).
* Run until '75000' events are catched.
-----------------------------------------------------------

---------------------- INTERESTING SYSCALLS ----------------------
* All sc codes are enabled!
------------------------------------------------------------------

************ boot_time:<PHONE_NUMBER>260643054

* OK! modern BPF probe correctly loaded: NO VERIFIER ISSUES :)
* Live capture in progress...
* Press CTRL+C to stop the capture

------------------ EVENT: 293 TID:60964
------ HEADER
timestamp:<PHONE_NUMBER>031082745
tid: 60964
len: 878
type: 293
num params: 27
------
------ PARAMS
PARAM 0: 0
PARAM 1: /usr/bin/dig
PARAM 2: <omitted>
PARAM 3: 60964
PARAM 4: 60964
PARAM 5: 3437895
PARAM 6: is empty
PARAM 7: 65536
PARAM 8: 0
PARAM 9: 3
PARAM 10: 420
PARAM 11: 4
PARAM 12: 0
PARAM 13: dig
PARAM 14: <omitted>
PARAM 15: <omitted>
PARAM 16: 0
PARAM 17: 60964
PARAM 18: -1
PARAM 19: 0
PARAM 20: 8192
PARAM 21: 8192
PARAM 22: 8192
PARAM 23: 1375
PARAM 24:<PHONE_NUMBER>0
PARAM 25:<PHONE_NUMBER>000000000
PARAM 26: 2138
------
------------------

<omitted events>

---------------------- STATS -----------------------
Events correctly captured (SCAP_SUCCESS): 75000
Seen by driver: 75617
Time elapsed: 0 s
Number of dropped events: 0
Number of timeouts: 303
Number of 'next' calls: 75303
Number of dropped events caused by full buffer (total / all buffer drops - includes all categories below, likely higher than sum of syscall categories): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_exit syscall category): 0
Number of dropped events caused by full scratch map: 0
Number of dropped events caused by invalid memory access (page faults): 0
Number of dropped events caused by an invalid condition in the kernel instrumentation (bug): 0
Number of preemptions: 0
Number of events skipped due to the tid being in a set of suppressed tids: 0
Number of threads currently being suppressed: 0
-----------------------------------------------------

[SCAP-OPEN]: Bye!

ok with this method timestamps have again some sense:

boot_time1677634320260643054 is 2023-03-01 02:32:00.260643072 +0100 CET
event_time<PHONE_NUMBER>031082745 is 2023-03-23 16:19:34.031082752 +0100 CET

So just to have a summary:

The extraction method with stat /proc/1 doesn't work on your machine because for some reason your proc change time is epoch :/
The fallback method with clock_gettime correctly works on your machine as expected.

Not sure what we want to do here, this proc anomaly even if strange could happen to other users :/ do we want to always use the fallback method instead of proc one, or maybe we want to check that the value returned from proc is big enough otherwise we will use the fallback :thinking:
WDYT ? @incertum @terylt @FedeDP

on your side @Happy-Dude it would be amazing to understand why in all your machines /proc has this strange change time (~epoch)

The /proc/1 timestamp may be something specific to our ARM kernels.
BUT small plot twist: I'm building scap-open on a x64 system to see if I can reproduce the problem, as our kernel does have the right stat /proc/1 output but still exhibit this problem

Hey @Happy-Dude it was changed as it was imprecise between agent reboots as it is also used for some other new time related fields, see https://github.com/falcosecurity/libs/issues/932#issuecomment-1447102203
Now the bigger question is we use this type of approach for our entire initial proc scan in scap, so if this approach is not robust for some environments or settings we need to check if it only concerns the boot time derivation or if the entire proc scan timestamps are off as well, not just for the clone ts, but also the newer ctime and mtime or pidns_init_start_ts fields?
In summary, we can revert to the old fall back method or possibly consider a completely new robust and stable solution, but maybe it won't fix possible proc scan issues and we need a broader adjustment?
Would also be curious to understand what can cause the wrong stat info, please keep us posted.
Would you mind also poking around the entire proc folder and checking ctime of pid dirs?

ok with this method timestamps have again some sense:

boot_time1677634320260643054 is 2023-03-01 02:32:00.260643072 +0100 CET
event_time<PHONE_NUMBER>031082745 is 2023-03-23 16:19:34.031082752 +0100 CET

So just to have a summary:

The extraction method with stat /proc/1 doesn't work on your machine because for some reason your proc change time is epoch :/
The fallback method with clock_gettime correctly works on your machine as expected.

Not sure what we want to do here, this proc anomaly even if strange could happen to other users :/ do we want to always use the fallback method instead of proc one, or maybe we want to check that the value returned from proc is big enough otherwise we will use the fallback 🤔 WDYT ? @incertum @terylt @FedeDP

@Andreagit97 wrt to boot time there are many ways to get it, so we probably could contrast a few options. Preferably would be to retrieve a constant value vs the old fall back method wouldn't give that. If we don't find a good constant approach, I think we have to just revert ...
@terylt @FedeDP @Andreagit97 what are your thoughts around broader implications around the scap proc scan? see comment above https://github.com/falcosecurity/libs/issues/932#issuecomment-1481516184

@terylt @FedeDP @Andreagit97 what are your thoughts around broader implications around the scap proc scan? see comment above https://github.com/falcosecurity/libs/issues/932#issuecomment-1481516184

Uhm let's say this proc scan code is quite old and nobody ever noticed something, this doesn't mean that it is right but at least should be quite robust, btw your concerns are valid, let's ask to our proc-scan expert @gnosek

Hey y'all,
Our internal kernel folks are still investigating the /proc/1 epoch time problem, but they have a suggestion for Falco:
Instead of stat /proc/1, can stat /proc/1/cmdline be used instead? This apparently similar to what Docker is doing.
user@host:~$ strace -v -etrace=statx -y stat /proc/1 /proc/1/cmdline
statx(AT_FDCWD, "/proc/1", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_blksize=1024, stx_attributes=0, stx_nlink=9, stx_uid=0, stx_gid=0, stx_mode=S_IFDIR|0555, stx_ino=21510, stx_size=0, stx_blocks=0, stx_attributes_mask=STATX_ATTR_AUTOMOUNT|STATX_ATTR_MOUNT_ROOT|STATX_ATTR_DAX, stx_atime={tv_sec=12, tv_nsec=17000000} /* 1970-01-01T00:00:12.017000000+0000 */, stx_btime={tv_sec=0, tv_nsec=0}, stx_ctime={tv_sec=12, tv_nsec=17000000} /* 1970-01-01T00:00:12.017000000+0000 */, stx_mtime={tv_sec=12, tv_nsec=17000000} /* 1970-01-01T00:00:12.017000000+0000 */, stx_rdev_major=0, stx_rdev_minor=0, stx_dev_major=0, stx_dev_minor=21}) = 0
  File: /proc/1
  Size: 0             Blocks: 0          IO Block: 1024   directory
Device: 15h/21d    Inode: 21510       Links: 9
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 1970-01-01 00:00:12.017000000 +0000
Modify: 1970-01-01 00:00:12.017000000 +0000
Change: 1970-01-01 00:00:12.017000000 +0000
 Birth: -

statx(AT_FDCWD, "/proc/1/cmdline", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_blksize=1024, stx_attributes=0, stx_nlink=1, stx_uid=0, stx_gid=0, stx_mode=S_IFREG|0444, stx_ino=70683560, stx_size=0, stx_blocks=0, stx_attributes_mask=STATX_ATTR_AUTOMOUNT|STATX_ATTR_MOUNT_ROOT|STATX_ATTR_DAX, stx_atime={tv_sec=1679590867, tv_nsec=687421205} /* 2023-03-23T17:01:07.687421205+0000 */, stx_btime={tv_sec=0, tv_nsec=0}, stx_ctime={tv_sec=1679590867, tv_nsec=687421205} /* 2023-03-23T17:01:07.687421205+0000 */, stx_mtime={tv_sec=1679590867, tv_nsec=687421205} /* 2023-03-23T17:01:07.687421205+0000 */, stx_rdev_major=0, stx_rdev_minor=0, stx_dev_major=0, stx_dev_minor=21}) = 0
  File: /proc/1/cmdline
  Size: 0             Blocks: 0          IO Block: 1024   regular empty file
Device: 15h/21d    Inode: 70683560    Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-03-23 17:01:07.687421205 +0000
Modify: 2023-03-23 17:01:07.687421205 +0000
Change: 2023-03-23 17:01:07.687421205 +0000
\ Birth: -
+++ exited with 0 +++

ACK, this would certainly be a super simple refactor ...

Agree! It is a minor change that seems to fix weird cases like this one! +1 from me for this tiny patch

Just created a patch; happy to incorporate any feedback!
I tested it briefly on the ARM system and it looked alright, could use more testing and validation though 🙏
schan@host:~/libs/build$ sudo ./libscap/examples/01-open/scap-open --modern_bpf --evt_type 293 --num_events 75000

[SCAP-OPEN]: Hello!

---------------------- SCAP SOURCE ----------------------
* Modern BPF probe, 1 ring buffer every 1 CPUs
-----------------------------------------------------------

--------------------- CONFIGURATIONS ----------------------
* Print single event type: 293 (`-1` means no event to print).
* Run until '75000' events are catched.
-----------------------------------------------------------

---------------------- INTERESTING SYSCALLS ----------------------
* All sc codes are enabled!
------------------------------------------------------------------

************ proc_cmdline: /proc/1/cmdline
************ proc_cmdline: /proc/1/cmdline
************ boot_time:<PHONE_NUMBER>623759387

* OK! modern BPF probe correctly loaded: NO VERIFIER ISSUES :)
* Live capture in progress...
* Press CTRL+C to stop the capture

------------------ EVENT: 293 TID:727523
------ HEADER
timestamp:<PHONE_NUMBER>579184804
tid: 727523
len: 816
type: 293
num params: 27
------
------ PARAMS
PARAM 0: 0
PARAM 1: /usr/bin/dig
PARAM 2: <omitted>
PARAM 3: 727523
PARAM 4: 727523
PARAM 5: 3437315
PARAM 6: is empty
PARAM 7: 65536
PARAM 8: 0
PARAM 9: 4
PARAM 10: 420
PARAM 11: 4
PARAM 12: 0
PARAM 13: dig
PARAM 14: <omitted>
PARAM 15: <omitted>
PARAM 16: 0
PARAM 17: 727523
PARAM 18: -1
PARAM 19: 0
PARAM 20: 8192
PARAM 21: 8192
PARAM 22: 8192
PARAM 23: 1375
PARAM 24:<PHONE_NUMBER>0
PARAM 25:<PHONE_NUMBER>000000000
PARAM 26: 2138
------
------------------

---------------------- STATS -----------------------
Events correctly captured (SCAP_SUCCESS): 75000
Seen by driver: 76799
Time elapsed: 0 s
Number of dropped events: 0
Number of timeouts: 265
Number of 'next' calls: 75265
Number of dropped events caused by full buffer (total / all buffer drops - includes all categories below, likely higher than sum of syscall categories): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_exit syscall category): 0
Number of dropped events caused by full scratch map: 0
Number of dropped events caused by invalid memory access (page faults): 0
Number of dropped events caused by an invalid condition in the kernel instrumentation (bug): 0
Number of preemptions: 0
Number of events skipped due to the tid being in a set of suppressed tids: 0
Number of threads currently being suppressed: 0
-----------------------------------------------------

[SCAP-OPEN]: Bye!

schan@host:~/libs/build$ git diff
diff --git a/userspace/libscap/scap.c b/userspace/libscap/scap.c
index 324c0639..58073079 100644
--- a/userspace/libscap/scap.c
+++ b/userspace/libscap/scap.c
@@ -1230,13 +1230,13 @@ int32_t scap_get_boot_time(char* last_err, uint64_t *boot_time)
        struct timespec tv_now = {0};
        uint64_t now = 0;
        uint64_t uptime = 0;
-       char proc_dir[PPM_MAX_PATH_SIZE];
+       char proc_cmdline[PPM_MAX_PATH_SIZE];
        struct stat targetstat = {0};

        /* More reliable way to get boot time */
-       snprintf(proc_dir, sizeof(proc_dir), "%s/proc/1/", scap_get_host_root());
-       printf("************ proc_dir: %s\n", proc_dir);
-       if (stat(proc_dir, &targetstat) == 0)
+       snprintf(proc_cmdline, sizeof(proc_cmdline), "%s/proc/1/cmdline", scap_get_host_root());
+       printf("************ proc_cmdline: %s\n", proc_cmdline);
+       if (stat(proc_cmdline, &targetstat) == 0)
        {
                /* This approach is constant between agent re-boots */
                *boot_time = targetstat.st_ctim.tv_sec * (uint64_t) SECOND_TO_NS + targetstat.st_ctim.tv_nsec;

Unfortunate news:
The change didn't fix the timestamp on my nodes; it's possible that /proc/1/cmdline may have an incorrect timestamp -- confirmed on both a x86-64 host and aarch64 host.
Changed the PR to (WIP)
Here is some output from scap_open:
schan@host:~/libs/build$ date -u && date +%s && date +%s%N && uptime && stat /proc/1/cmdline; sudo ./libscap/examples/01-open/scap-open --modern_bpf --evt_type execve --num_events 10

Thu 23 Mar 2023 11:00:42 PM UTC

1679612442

1679612442422449367

 23:00:42 up 56 min,  1 user,  load average: 1.58, 1.63, 1.70

  File: /proc/1/cmdline
  Size: 0               Blocks: 0          IO Block: 1024   regular empty file
Device: 15h/21d Inode: 29705       Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-02-15 20:36:33.542000000 +0000
Modify: 2023-02-15 20:36:33.542000000 +0000
Change: 2023-02-15 20:36:33.542000000 +0000
 Birth: -

[SCAP-OPEN]: Hello!

---------------------- SCAP SOURCE ----------------------
* Modern BPF probe, 1 ring buffer every 1 CPUs
-----------------------------------------------------------

--------------------- CONFIGURATIONS ----------------------
* Print single event type: 0 (`-1` means no event to print).
* Run until '10' events are catched.
-----------------------------------------------------------

---------------------- INTERESTING SYSCALLS ----------------------
* All sc codes are enabled!
------------------------------------------------------------------

************ proc_cmdline: /proc/1/cmdline
************ proc_cmdline: /proc/1/cmdline
************ boot_time:<PHONE_NUMBER>542000000

* OK! modern BPF probe correctly loaded: NO VERIFIER ISSUES :)
* Live capture in progress...
* Press CTRL+C to stop the capture

------------------ EVENT: 0 TID:73151
------ HEADER
timestamp:<PHONE_NUMBER>579307218
tid: 73151
len: 34
type: 0
num params: 2
------
------ PARAMS
PARAM 0: 176
PARAM 1: 113
------
------------------

---------------------- STATS -----------------------
Events correctly captured (SCAP_SUCCESS): 10
Seen by driver: 247
Time elapsed: 0 s
Number of dropped events: 0
Number of timeouts: 1
Number of 'next' calls: 11
Number of dropped events caused by full buffer (total / all buffer drops - includes all categories below, likely higher than sum of syscall categories): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_clone_fork_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_execve_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_connect_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_open_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_dir_file_exit syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_enter syscall category): 0
Number of dropped events caused by full buffer (n_drops_buffer_other_interest_exit syscall category): 0
Number of dropped events caused by full scratch map: 0
Number of dropped events caused by invalid memory access (page faults): 0
Number of dropped events caused by an invalid condition in the kernel instrumentation (bug): 0
Number of preemptions: 0
Number of events skipped due to the tid being in a set of suppressed tids: 0
Number of threads currently being suppressed: 0
-----------------------------------------------------

[SCAP-OPEN]: Bye!

Yes I think that would be a great data point for us to see if the old approach works.
In addition, surfing the internet for any options available e.g. see this, maybe it would be helpful to check on all of them just to give all of us more information to work with and help.
Anyone else having more ideas?
Also would it perhaps be possible to reboot one server and check again on all options?

Also would it perhaps be possible to reboot one server and check again on all options?

Possible depending on which option; essentially have to make new packages iterating through each because config management will install the latest package in our repo on my evaluation hosts 😅
I'll look to creating a package that comments out the boot time code in scap.c like in https://github.com/falcosecurity/libs/compare/master...Andreagit97:libs:timestamp_issue
Does that mean I also need to undo the /proc/1/cmdline changes made elsewhere?

Maybe just run the different commands from a shell first to check?

The change didn't fix the timestamp on my nodes; it's possible that /proc/1/cmdline change may be insufficient in producing the right timestamp -- confirmed on both a x86-64 host and aarch64 host.

What do you mean by "The change didn't fix the timestamp on my nodes"? if I understand correctly now using /proc/1/cmdline you should see the right boot_time, is it true? if yes, what is the issue now :thinking: ?

Just updated the PR and added a comment, https://github.com/falcosecurity/libs/pull/1003#issuecomment-1483538662

Removed the (WIP) tag and submitted a new commit (and rebases with fixes to suggestions).
I tested a build with these two commits applied as a patch and first impressions are very positive; Falco is logging events with the right timestamps!
Summary:
After chatting in #932 and a Slack thread, we discovered that /proc/1/cmdline did not return the right boot time as we had thought. This was verified on a host in my org.
This lead us to do a round of debugging involving running the following commands on problematic systems:
* `uptime -s`, `who -b`, `stat /proc /proc/1 /proc/1/cmdline`, `cat /proc/stat | grep btime`: revealed that the output from who -b was incorrect, but the output from uptime was correct

* `ls -lahtr`: revealed that `/proc/uptime` had timestamps closer to when the system was started

Most importantly, the cat /proc/stat | grep btime had a btime value closest to the boot time. Speaking to a few engineers in my org also revealed that the same parts of the kernel inform/feed values into /proc/utime and /proc/stat's btime, so they are "essentially" equivalent.
The Slack thread also discussed ways of getting the system boot time:
* btime in /proc/stat

* calculation via clock_gettime(CLOCK_REALTIME - CLOCK_BOOTTIME)

* calculation via time(NULL) - sysinfo().uptime

Overall, the maintainers preferred btime from /proc/stat for the following reasons:
* value does not depend on calculation using current timestamp

* btime is "static" and doesn't change once set

* btime is available in kernels from 2008

* CLOCK_BOOTTIME is available in kernels from 2011 (2.6.38)

@gnosek mentioned:

By scraping btime from /proc/stat, it is both the heaviest and most likely to succeed

And this PR (as it is right now) is a result of that discussion:
* use `btime` from `/proc/stat` for boot time

* use `/proc/<pid>/root/proc/1/cmdline` for container start time


Fix deadlock in H.264 encoder

There was a case where two mutexes were acquired in different orders
causing a deadlock.
Signed-off-by: Naushir Patuck<EMAIL_ADDRESS>

@davidplowman


Add an ability to supply the custom subnet in AKS nodeclass

Tell us about your request
Currently with karpenter on AKS, the way it works is it provisions nodes in the subnet specified during the cluster creation.
For our usecase, the subnet specified during the cluster creation is not the one where we want to provision our worker nodes.
For eg: We create 3 subnets, master, worker and system. During cluster creation we use command az aks cluster create --vnet-subnet-id "mastersubnetID"  but we dont want our workers to get provisioned in mastersubnetID, we want to supply workerSubnetID to have workers created in that IP space.
We need a way to supply our custom subnet after the cluster has been created in AKS nodeclass where the workers will be provisioned. This could be through the tag selector or having subnet supplied just as is.
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Outcome we want to achieve is AKS nodeclass supports having a custom subnet field other than one used during cluster creation.
Are you currently working around this issue?
N/A
Additional Context
No response
Attachments
No response
Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Fixed by https://github.com/Azure/karpenter-provider-azure/pull/486


[SPARK-17144] [SQL] Removal of useless CreateHiveTableAsSelectLogicalPlan

What changes were proposed in this pull request?
CreateHiveTableAsSelectLogicalPlan is a dead code after refactoring.
How was this patch tested?
N/A

Test build #64028 has started for PR 14707 at commit 2b65086.

Test build #64028 has finished for PR 14707 at commit 2b65086.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64028/
Test PASSed.

cc @cloud-fan

LGTM, but let's wait for https://github.com/apache/spark/pull/14155 to avoid conflict, thanks!

retest this please

Test build #64239 has started for PR 14707 at commit 2b65086.

Test build #64239 has finished for PR 14707 at commit 2b65086.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64239/
Test PASSed.

thanks, merging to master!


Error: Cannot find where you keep your Bower packages

Hi,
I've create a yo angular app with yo angular Followed the steps but I get the following error:
remote: -----> Found Gruntfile, running grunt heroku: task remote: Running "clean:dist" (clean) task remote: >> 14 paths cleaned. remote: remote: Running "wiredep:app" (wiredep) task remote: Warning: Error: Cannot find where you keep your Bower packages. Use --force to continue.
Any ideas?
Thanks

Here is the default wiredep
     // Automatically inject Bower components into the app     wiredep: {       app: {         src: ['<%= yeoman.app %>/index.html'],         ignorePath:  /\.\.\//       },


Delete existing links that are broken

This PR attempt to fix an issue that occur when the storage link already exists but is somehow broken. By "broken", I mean link that point to a non-existing target.
When running the storage:link command, we get an error message and the link is not updated.
$ php artisan storage:link

In Filesystem.php line 263:

  symlink(): No such file or directory

To fix this issue, we add an additional check to detect and delete broken links when running the storage:link command.
Implementation details
When a link exists but is broken, file_exists($link) return false. And when symlink($link, $target) is called on a broken link, a PHP Warning is returned and the link is not updated. To fix this, we add an additional check using is_link($link) (which return true, even if the link is broken) to detect and delete broken links.

@pgrenaud can you send this to 6.x instead please? Thanks

Ok! I have created #36470.
Note that this new patch wont apply easily on 8.x since features were added in the StorageLinkCommand class.

Okay, I'm re-opening this one as well so we can have the proper fix for 8.x as well. We'll handle the merge conflicts.


Fix PHP7 error "Cannot use 'String' as a class name".

This enables the 2.4.x version to run on PHP7 with MongoDB

Actually


Running "docker-compose up" doesn't watch for file changes

Right now I can't see file changes instantly when developing.
To see changes I need to kill the server and rebuild using docker-compose up --build.
Is there a way to have docker watch for file changes without rebuilding each time?

@anniegiang Good point, I created a production ready container so it won't watch for changes unfortunately. But it's possible to create a Dockerfile.dev and watch for changes, I'm gonna work on that today

@neemiasjnr sounds good!


fix(db): export query db type

Description of change

Export QueryDatabaseParameters and QueryDatabaseResponse

Pull-Request Checklist

[ ] Code is up-to-date with the main branch
[ ] npm run lint passes with this change
[ ] npm run test passes with this change
[ ] This pull request links relevant issues as Fixes #0000
[ ] There are new or updated unit tests validating the change
[ ] Documentation has been updated to reflect this change
[ ] The new commits follow conventions outlined in the conventional commit spec

:tada: This PR is included in version 1.3.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Creating Universal Libraries for Vue 2 and 3

I came across this plugin vue-demi that claims to create universal libraries for Vue 2 and 3 without having to maintain different branches. I haven't tried it but planning to for my next project.
Will this be compatible with the vue-sfc-rollup setup? Not sure how all of this works, hence the question. I want to know if it's possible to include the plugin in this current setup without breaking anything

I have seen this, and while at the outset it looks great, there is one downside. Use of vue-demi requires use of the composition api. This will make vue-sfc-rollup unusable for anyone who wishes to continue using the options api. As such, I don't think it will become part of the default output of this utility. It might be worth investigating it as an option to select during init, however.
That being said, I don't think there is anything that would prevent the use of vue-demi with the output of vue-sfc-rollup, if you choose to. As long as you are aware that you are passing a new dependency to your users, and will need to write your component/library with the composition api in mind, it should work fine.

Didn't know you had to use the composition API. Thanks for the detailed explanation :+1:


If is_prealloc is set to 1, is it necessary to call rknn_outputs_release?

You'd better call it, although nothing is currently done in it when is_prealloc is set to 1.


Examples in text

the page states that this applies to 2022, 2019 and 2016, but all of the examples are from older versions. Can we just substitute new versions whenever appropriate, or is there more effort involved? Please clarify what, if anything should be done differently, in the newer systems
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a3cfb096-8fc8-7f33-f42f-8082371940c2
Version Independent ID: a41c5888-b33a-7090-64c2-ff209b41ee70
Content: Monitoring Active Directory for Signs of Compromise
Content Source: WindowsServerDocs/identity/ad-ds/plan/security-best-practices/Monitoring-Active-Directory-for-Signs-of-Compromise.md
Product: windows-server
Technology: ad-ds
GitHub Login: @iainfoulds
Microsoft Alias: daveba

#assign:wscontent

The following users don't have valid GitHub IDs, or are not collaborators on this repo: wscontent

#assign:windowsServerDocs

The following users don't have valid GitHub IDs, or are not collaborators on this repo: windowsserverdocs

#assign:IngridAtMicrosoft

Hello @DeanGross,
We'd like to express our gratitude for your commitment to our documentation. Unfortunately, we’ve recently fallen short of our usual standards in addressing documentation concerns in a timely manner and offer our apologies for any inconvenience we’ve caused.
Because this issue is now outdated, we’re closing this request. If we've addressed the documentation issue, this thread will also be closed. However, if you believe there’s still a documentation concern, you're encouraged to open a new GitHub issue. We welcome any additional suggestions for improving our documentation to provide a more satisfactory experience in the future. Thanks for your understanding and continued support.
#please-close


Windows Installation

How to install on a Windows machine ? I would like to install in a windows machine...to work in my desktop

Hi @prog20901,
Do you have any experience installing/managing web apps using something like apache?
Unfortunately this is a web hosted app so there's no easy installer, You'll need to manage separate web-server+php & database components to run BookStack.

I was able to run Bookstack on my windows machine by installing Docker and running the script that's from solidnerd.
Pre-requisite: Make sure that you have a version of Windows that supports VMs, e.g., Windows 10 Pro.

Got it..Thanks a lot..


linux_{5_0,5_1,testing}: restore _kernel_fpu{begin,restore} [19.03]

Motivation for this change
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Currently building all three kernels...


Lab 05 - Part 3, Question 4

Hello,
This question asks for Average Casualties (fatalities or injuries) per Harmful Accident. Does this mean the x axis should be something calculated and related to Harmful Accident or is it Hour of the day?
Thanks

What does the graph that you have to replicate suggest?

The graph says its the hour of the day, so I guess the x-axis should reflect the hours.
However, I am confused while trying to get the formula for the average casualties. I am not sure if I'm thinking about it the right way.
The graph shows it the average number of passengers hurt, so I am trying to divide the sum of passengers by the number of crashes but I am getting stuck while translating this into code, especially inside the function summarize.  I am posting what I'm trying to do:

Issue #21 addresses this exact question - check it out and let me know if you still need any clarification!

However, @gzbib, just for a succinct response - just about everything in your code looks on point.  In mutate(), your newly-created variable total should technically work.  It's the same as doing this:
%>% mutate(casualties = Totalinjuries + Totalfatalities) %>%

This is because, if only adding injuries or fatalities greater than "0", you're doing the same as summing up all injuries and fatalities (since "0" would not add to either total, whether you remove 0 or not via > 0).  Does that help?

Hello Sir,
Thank you for your reply. I checked issue 21 but I am not sure whether it addresses my concern in this question. I am still not getting it like why they keep telling me that they can't find function %>%  whereas I believe the logic is right and the syntax is right.
Sorry if this is taking much of your time.

Hi @gzbib, ah, I didn't understand the issue you were having.  You need to run library(dplyr) in the beginning of your .rmd script (as well as any time you use %>% or other dplyr in a new session).
Your error is telling you that R can't find %>%, and that typically occurs because the packages that use function %>% are not loaded first.  That should do the trick!

Ohhh okay ! it worked thank you Sir

Awesome, sure thing!


Use abstract logging interface

Based on Patrick Ohly comment here
Don't use "log" for logging. It's way too limited in practice.

A reusable package should use some abstract logging interface.
Whoever composes some binary then gets to decide which logging implementation to use.

For now, I recommend go-logr for this. In some future Go release, slog can replace it.

I see a lot of usage for https://github.com/sirupsen/logrus as well


const { post, errors } = useForm(initialFormValues) not working

Hi, noticed that the post/errors are not working.
When I use post, the "data" values sent to post are not being update by setData (the initial values are sent). I also tried to log out the errors, that is also empty. Works as expected with useForm.
Does it work fine for you?

Could you provide an example, maybe on codesandbox or something similar?

Not sure if it will be possible to do it in the sandbox, as we need the full inertia/laravel setup, but you should be able to easily replicate locally.
We should be able to test it by just using the following:
const initialFormValues = {
title: ""
}
const { data, setData, post } = useInertiaForm(initialFormValues); // This doesn't work
vs
const { data, setData, post } = useForm(initialFormValues); // this works

const submitForm = (e: FormEvent<HTMLFormElement>): void => {
    e.preventDefault();
    post("/");
}

return (
    <form onSubmit={(e) => submitForm(e)}>
        <input
            id="title"
            name="title"
            value={data.title}
            onChangeHandler={(e) => setData("title", e)}
         />
     </form>
)

I can see in the network request, that in the useForm post, the updated payload is sent as expected, but in useInertiaForm, the payload is not updated, and the initialFormValues is being sent on post instead.

The first thing I notice is that you have onChangeHandler in your input, which is not a prop. You're also passing the input change event directly to setData, which is expecting a primitive value type.
If I paste the code you provided into a project as is, I get console errors, but changing it to the following works for me:
const SomeFormComponent = () => {
  const initialFormValues = {
    title: '',
  }

  const { getData, setData, post } = useInertiaForm(initialFormValues)

  const handleSubmit = (e: React.FormEvent<HTMLFormElement>) => {
    e.preventDefault()
    post('/')
  }

  return (
    <form onSubmit={ handleSubmit }>
      <input
        id="title"
        name="title"
        value={ getData('title') }
        onChange={ e => setData('title', e.target.value) }
      />
    </form>
  )
}

Not sure if that was copied directly from your project, but if it was, the call to setData never would have triggered because onChangeHandler would have been ignored as a non-existent html attribute by React. If these mistakes didn't exist in your project, I'll need to get a reproducible error to track down the issue. You could fork the example I have on codesandbox for a project with the proper dependencies already installed.

My apologies, while I was trying to add a simplified version of the form and forgot to change the onChangeHandler to onChange. The example you provided is exactly what I meant.
I might be doing something wrong, but the codesandbox doesn't allow post or router.post, throws (0 , G.default) is not a function.
https://codesandbox.io/p/sandbox/brave-mendel-34sycv - this is the example in theory. The console log would log out the correct data value, but the post request would send the initialFormValues in the actual network request.

So, I was able to reproduce the error on a local project. You were indeed correct that the HTTP verb methods weren't working. I believe it was due to memoizing them in an attempt to make them stable for dependency arrays. I removed the memoization and they now appear to be functioning as expected for me. #14
Thank you for bringing this to my attention. I don't use the hook itself, I use the Form component from this package, so I never had the opportunity to see it fail. Please let me know if this fixes things for you.

Yes, looks like this fixes it. Thanks a bunch.


Websocket errors occur silently

Because of https://github.com/coderstephen/slack-client/blob/master/src/RealTimeClient.php#L116, errors in websocket connections don't actually print errors. I spent two hours trying to work out why my program was exiting without complaining about anything. There must be a better way of doing this, perhaps a spoofing object of some sort?

Fixed by #28


Fix DSL parser for expression index support in ActiveRecord 5.0

Changes Unknown when pulling ee0a6007c4157a026bc57cc2eac38cd7bdf94733 on eagletmt:ar5-expression-index into ** on winebarrel:v0.6.5**.

Thanks!


Add AudioStreamPolyphonic icon

Adds an icon for polyphonic audio streams. A plus at the negative space of the note, similar to MultiMesh, seems appropriate.

Thanks!


Container not found

I installed the package on DS918+, and when I try to start it I get a "container not found" error.

Can you provide the settings you used?

I found the issue -I was mapping a volume that had a space in it.  The diskstation allows shared folder names with spaces.
John Pacione

On Dec 9, 2017, at 6:31 AM, Jocelyn Le Sage<EMAIL_ADDRESS>wrote:
Can you provide the settings you used?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.


17982 - Unable to obtain or confirm information.

*Issue:17982 *https://github.com/bcgov/entity/issues/17982
Description of changes:

added message box component (alert, warning, info ...)
added section in add new individual that is dealing with "Unable to obtain or confirm information" data

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the namex license (Apache 2.0).

Temporary Url for review: https://bcregistry-btr-dev--pr-63-ew8sa3uw.web.app

/gcbrun

Temporary Url for review: https://bcregistry-btr-dev--pr-63-ew8sa3uw.web.app

/gcbrun

/gcbrun

Temporary Url for review: https://bcregistry-btr-dev--pr-63-ew8sa3uw.web.app

Temporary Url for review: https://bcregistry-btr-dev--pr-63-ew8sa3uw.web.app


Telegram

Dear Pylakey,
Thank you for the update. I acknowledge receipt of your email and confirm
that I am aware that the issue has been closed.
Best regards,
Dr. Ignacio Tapia V
Dr. Ignacio P. Tapia Vargas
www.drtapiavargas.net http://www.drtapiavargas.net
http://www.drtapiavargas.net
WhatsApp  +59176123237
El vie, 5 jul 2024 a la(s) 11:10 p.m., pylakey @.***)
escribió:

Closed #132 https://github.com/pylakey/aiotdlib/issues/132 as completed.
—
Reply to this email directly, view it on GitHub
https://github.com/pylakey/aiotdlib/issues/132#event-13409485088, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/A2LM53ARPRG6EQKF4FLTOXLZK4DVFAVCNFSM6AAAAABKLSXSKSVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGQYDSNBYGUYDQOA
.
You are receiving this because you authored the thread.Message ID:
@.***>


stat: replace custom reverse logic with slices.Reverse

This PR introduces a single modernization change from https://github.com/gonum/gonum/pull/1934. Benchmarks will follow.
Follow-on to https://github.com/gonum/gonum/pull/1934, which was my first attempt at this but was too large to properly benchmark.
Blocked by https://github.com/gonum/gonum/pull/1940.

Actually, slices.ROC has no benchmarks, so there's no need to run anything here.

As discussed in https://github.com/gonum/gonum/pull/1956#issuecomment-1988326786, I'll make some new benchmarks as soon as I can.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 72.34%.  Consider uploading reports for the commit 82397bc to get more accurate results

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1943   +/-   ##
=======================================
  Coverage   72.34%   72.34%
=======================================
  Files         555      555
  Lines       65005    65005
=======================================
  Hits        47026    47026
  Misses      15380    15380
  Partials     2599     2599

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@kortschak Running the benchmarks in 82397bc, it looks like a few cases are better and worse than master by +-1-2%, with a slight increase in geomean by 0.48%.
 Thank you for merging this in. 👍


feat(ad-api): add door control

Description
Add the door control API. Close autowarefoundation/autoware#3232.
https://autowarefoundation.github.io/autoware-documentation/pr-415/design/autoware-interfaces/ad-api/features/vehicle-doors/
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The Reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Wait for https://github.com/autowarefoundation/autoware-documentation/pull/407

It seems like DCO check is failing. Could you fix the unsigned commit?


Don't let Justin Palace beat you this time.

Thanks for your cooperation earlier!
Please check it.

thank you!


Layout does not scale correctly, makes it impossible to join room on certain resolutions.

Repro steps

Go to the join room https://ffc.staging.jimber.org/room/Patat/join

Expected Result
Possible to join room
Either by making it fit the screen or by adding a scrollbar
Actual Result
no scrollbar and no join button, impossible to join on current window size

System Info
v2.0.0

@ken-de-moor I don't think this will be an issue as this page is removed

I see this still has to be merged check #123


dev: Add Links to stopwatch and countdown timer

Add router links and routes to stopwatch and countdown timer.

Done.


Añadido reto 3

Hello @avarez! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file reto-03/avarez/main.py:

Line 28:80: E501 line too long (99 > 79 characters)


FLW-735 Add Confirmation payout screen

Staking payout confirmation screen

Codecov Report

:exclamation: No coverage uploaded for pull request base (develop@d83ed2f). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             develop     #201   +/-   ##
==========================================
  Coverage           ?   24.85%
==========================================
  Files              ?      397
  Lines              ?    14916
  Branches           ?        0
==========================================
  Hits               ?     3708
  Misses             ?    11208
  Partials           ?        0


fix: remove temporary link element after csv download

the _createAndDownloadFile method in extensions/measurement-tracking/src/_shared/downloadCSVReport.js creates a temporary link element to download the data
currently this temporary element is not removed after download, which leads to many <a> elements being appended to the body when performing multiple downloads
with this PR the temporary element is removed after download

Good catch! Thanks!


UnicodeDecodeError when loading pretrained GloVe embeddings

Hi all,
I'd like to use GloVe vectors as pretrained embeddings when training a text classifier. I downloaded the glove.840B.300d.zip vectors, unzipped them, and specified in my model definition file the following lines:
input_features:
    -
        name: text
        type: text
        encoder: rnn
        cell_type: lstm_cudnn
        pretrained_embeddings: /home/victor/data/glove.840B.300d.txt

This is the error I get as soon as I launch the training:
embeddings = load_glove(embeddings_path)
  File "/home/victor/ludwig/lib/python3.6/site-packages/ludwig/utils/data_utils.py", line 178, in load_glove
    for line in f:
  File "/usr/local/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1387: ordinal not in range(128)

All my files are encoded in UTF-8. I'm using Debian stable (Stretch), Python 3.6.8, ludwig 0.1.2 and TF 1.13.1.
Any ideas? Thanks in advance,

Just pushed a quick fix that should solve the problem. Please install Ludwig from the latest commit on master with pip install git+https://github.com/uber/ludwig.git and let me know if the problem is solved.

Hi,
Great, thanks for the fix! It works now!
However, I just found out that some GloVe files don't have a regular format. For instance, glove.840B.300d.zip is supposed to contain 301-item lines (the token and its 300 weights) but some lines do contain more than one str tokens. As a consequence, when trying to convert some of these unexpected strings into floats in this line, it raises an exception.
Maybe we can safely ignore those files containing lines not matching the expected embedding size.
Thanks again!

@vitojph  could you please post here some examples line that raise exceptions so that I can figure out how to account for them?

hi @w4nderlust,
When loading ie. Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download): glove.840B.300d.zip, some lines seems to be ill-formed.

I see. Thanks for inspecting this. I will do 2 things: 1) reproduce your result and try to figure out if those are actually malformed lines or if there are weird utf-8 characters that the split function doesn't recognize (or uses for splitting), 2) add more checks, discarding lines that are malformed. Stay tuned

So it turned out there are some weird utf-8 space characters in that file. The reason is likely that in the glove c code they split using only the ASCII space.
I initially implemented a solution that was dealing with it by joining all the elements in the split that were not part of the embeddings, but that was slow and useless because those characters would not be present in tokens in Ludwig because of how Ludwig's tokenizers work.
So in the end I ended up just skipping those "malformed" lines, printing a warning.
@vitojph please let me know if now everything works as expected.

@w4nderlust I am trying to load embeddings for my text input features but as soon as I start to train the model i get the following error:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2438: character maps to <undefined>
I already tried the it again by installing from pip install git+https://github.com/uber/ludwig.git but it still doesn't work.
The commoncrawl embeddings file is from https://nlp.stanford.edu/projects/glove/

@abc1110 use the code from master: pip uninstall ludwig & pip install git+http://github.com/uber/ludwig.git
The issue has been solved in master, i was able to load all the pretrained GloVe embeddings files.

@w4nderlust I have tried that solution, but I am using the programmatic API where this issue persists..

@abc1110 the command line and API use exactly the same code. Please make sure you are running the updated code. If you are, and still get an error, please make a reproducible example, because I tested wit all those retrained embeddings files and they all work.


Daemonize doesn't create pid file in 0.14.0

Using

fluentd 0.14.0
CentOS 6.7
Empty fluent.conf

Running
 fluentd -c fluent.conf  --daemon /tmp/test.pid
Expected result:
/tmp/test.pid with process id
Actual result:
No /tmp/test.pid file

+1

@tagomoris It seems urgent issue. Could you handle this?

OK.

The code to fix this issue was merged into master.

Yes, thank you!


update elm-version

Hey @jcollard!
Thanks for the cool module, I quite like it.
I started defining the constants I use manually, before I discovered your publshed module.
Before I can use it, I need the elm-version to be updated, I tested this change and it all seems  to work. Could you please merge this in @jcollard?
Thanks :)

LGTM! Thanks!

Thanks for the quick merge @jcollard, any future date when this will be published as a new module version?

Unless they've changed the way elm-publish works, anyone should be able to publish it as a new module. Of course, I could be wrong. Let me know if that is not the case and I can make it happen.

Oh it's because I can't see the new version tag in your GitHub repository. And on the Package website only v1.0.0 is available. So I think you haven't update your tag & published correctly. I might be wrong though. Let me know if you can find it :dancers:

Ah I see what's going on. I don't have the tool chain installed correctly. Could you run whatever command is needed to generate the correct elm-dependencies.json? It should generate the correct tag that is needed. If you do a PR, I will merge and you should be able to publish it.


Ability to specify a custom logger

A simple option that allows replacing the default grunt logger with something else would do. Would it be ok?

Hi Rafał!
What exactly are you trying to achieve, and what API for specifying another logger do you suggest?
Cheers
-Christian

Hi, I don't remember now :/ I'll close this and reopen when I come back to the issue!


Removal of TodoReturn alias to void

Geode Version
v2.0.0
Related Problem
No response
Your Solution
TodoReturn can still be aliased to void behind a flag, which only exists for people to migrate over its new behavior. For stable release, alias will be fully removed.
Additional Information
No response

done in 0c35a92ececb84fe9d4aa2d17b676fc7741319ec


NewsBlur sync should not include hidden

https://twitter.com/_rwatt/status/1384887096586870790
https://blog.newsblur.com/post/168431864825/intelligence-training-comes-to-newsblurs-android
Jeff Hellman on Slack: “The newsblur apps themselves set include_hidden=false but NNW seems to set include_hidden=true”

Fixed on Mac and iOS.


What's the difference between the accuracy in the training script and one in the infer_all scirpt?

First thanks for your reimplementation! great work!
I would like to apply the code to my own dataset, but I found a huge gap between the accuracy in the training script and one in the infer_all script.
During the inference stage in the training script, I got the following output:
test top-1 acc: 0.4992
test top-2 acc: 0.6064, test top-3 acc: 0.6559,
test top-5 acc: 0.7034, test top-10 acc: 0.7487,
test top-20 acc: 0.7768, test top-50 acc: 0.8026,
test top-100 acc: 0.8170

However, when I used the infer_all.sh, the accuracy dropped rapidly:
Test Top-1 accuracy:  6.358%
Test Top-3 accuracy:  7.160%
Test Top-5 accuracy:  7.253%
Test Top-10 accuracy:  7.275%
Test Top-20 accuracy:  7.278%
Test Top-50 accuracy:  7.280%
Test Top-100 accuracy:  7.280%
Test Top-200 accuracy:  7.280%

Which figure is more reliable?
Any reply would be greatly appreciated!

Hi, I'm happy to hear that my reimplementation has been of help.
In theory, the SMILES matching accuracy from infer_all.sh must be equal or higher than the template matching accuracy printed by the training script. This is because there is only one groundtruth template specified (which is not real since multiple templates can give the same, correct reactant SMILES). Put another way, neuralsym may have found an alternative template, which doesn't match the groundtruth template in the test set (thus lowering the template accuracy), but nevertheless gives the exact same reactant SMILES as the groundtruth template (thus increasing the SMILES accuracy.
In your case, I suspect something wrong is happening when the code is trying to string match the groundtruth reactant SMILES in your dataset against the reactant SMILES produced by applying the predicted template by neuralsym.
Are the reactant SMILES in your test set canonicalised with the same rdkit version you're running neuralsym with? Are there any excess salts or other species in your groundtruth reactant SMILES?

It would be helpful for debugging to print examples of reactions where the predicted template is correct but somehow the predicted reactant SMILES doesn't match what is notated in your groundtruth csv

Great thanks for your detailed explanation and advice!
I have ensured that all the code is running with the same rdkit version.
Taking your advice, I tried to print some examples of reactions where the predicted template is correct but somehow the predicted reactant SMILES doesn't match. And it seemed that desired reactants could not be generated correctly with correct template.
Here is a example:
The reaction with atom mapping is:
[F:1][c:2]1[c:3]([N+:10](=[O:11])[O-:12])[cH:4][c:5]([F:9])[c:6]([F:8])[cH:7]1.[NH2:13][c:14]1[s:15][cH:16][cH:17][c:18]1[C:19]#[N:20]>>[c:2]1([NH:13][c:14]2[s:15][cH:16][cH:17][c:18]2[C:19]#[N:20])[c:3]([N+:10](=[O:11])[O-:12])[cH:4][c:5]([F:9])[c:6]([F:8])[cH:7]1

And the top-1 predicted reaction template, which is also the desired template, is:
[#16&a:1]:[c:2]-[N&H1&D2&+0:3]-[c&H0&D3&+0:5](:[c:6]):[c:7]>>[#16&a:1]:[c:2]-[N&H2&D1&+0:3].[F&H0&D1&+0:4]-[c&H0&D3&+0:5](:[c:6]):[c:7]

These are their corresponding images:

I used the following function to get reactant:
def gen_precs(templates_filtered, preds, phase_topk, task):
    i, prod_smi_nomap = task
    # generate predictions from templates
    precursors, dup_count = [], 0
    pred_temp_idxs = preds[i]
    for idx in pred_temp_idxs:
        template = templates_filtered[idx]

        try:
            rxn = rdchiralReaction(template)
            prod = rdchiralReactants(prod_smi_nomap)
            precs = rdchiralRun(rxn, prod)
            precursors.extend(precs)
        except Exception as e:
            print(e)
            continue

    # remove duplicate predictions
    seen = []
    for prec in precursors: # canonicalize all predictions
        prec = Chem.MolToSmiles(Chem.MolFromSmiles(prec), True)
        if prec not in seen:
            seen.append(prec)
        else:
            dup_count += 1

    if len(seen) < phase_topk:
        seen.extend(['9999'] * (phase_topk - len(seen)))
    else:
        seen = seen[:phase_topk]

    return precursors, seen, dup_count

Then an error occured:
Traceback (most recent call last):
  File "/nfs3-p1/LH/Synthesis/neuralsym/a.py", line 76, in <module>
    precs = rdchiralRun(rxn, prod)
  File "/home/LH/anaconda3/envs/Synthesis/lib/python3.6/site-packages/rdchiral/main.py", line 171, in rdchiralRun
    for m in outcome for a in m.GetAtoms() if a.HasProp('old_mapno')}
  File "/home/LH/anaconda3/envs/Synthesis/lib/python3.6/site-packages/rdchiral/main.py", line 171, in <dictcomp>
    for m in outcome for a in m.GetAtoms() if a.HasProp('old_mapno')}
KeyError: 4

It seems to be a rdchiral error?  Is there any method to solve it?
Thanks again for your help!

hi @LaHauzel,
I do not remember having encountered myself before, unfortunately. does this happen for many of the reactions (and hence the main cause of the very low reactant SMILES-matching accuracy ?
I'm not terribly familiar with the inner workings of rdchiral so I would recommend you to create an Issue on rdchiral's github repository for this one.
https://github.com/connorcoley/rdchiral

hi @LaHauzel,
I do not remember having encountered myself before, unfortunately. does this happen for many of the reactions (and hence the main cause of the very low reactant SMILES-matching accuracy ?
I'm not terribly familiar with the inner workings of rdchiral so I would recommend you to create an Issue on rdchiral's github repository for this one. Sorry I couldn't be of much help here.
https://github.com/connorcoley/rdchiral
Just to confirm, did you use the function in this repo to extract the template from the reaction SMILES? or did you extract them elsewhere?

The reaction templates are all extracted by the provided script prepare_data.py.
I think I have solved the problem. With the reaction function provided by rdkit, I successfully generated the desired result. All of my modification is changing the following code
rxn = rdchiralReaction(template)
prod = rdchiralReactants(prod_smi_nomap)
precs = rdchiralRun(rxn, prod)
precursors.extend(precs)

to
rxn = AllChem.ReactionFromSmarts(template)
prod = [Chem.MolFromSmiles(prod_smi_nomap)]
precs = rxn.RunReactants(prod)
for prec in precs:
    prec = [Chem.MolToSmiles(p) for p in prec]
    precursors.append(".".join(prec))

This new function works as desired, resulting in a much better result which meets your view that the SMILES matching accuracy from infer_all.sh must be equal or higher than the template matching accuracy printed by the training script.
The final top-1 accuracy is ~53% and top-100 accuracy is ~85%.
Thanks again for your faithful reimplementation and help! You could close the issue now.

hi, @LaHauzel , very happy hear that you found a fix. seems that the way I was using rdchiral was outdated.
could I trouble you to create a PR to merge this fix into master? that would be very much appreciated :)

hi, @LaHauzel , very happy hear that you found a fix. seems that the way I was using rdchiral was outdated. 53% top-1 accuracy is also very nice! all the best with your tinkering.
could I trouble you to create a PR to merge this fix into master? that would be very much appreciated :)

I am glad you like it, but i am not sure if these two functions works exactly the same.
I tried the provided uspto-50k dataset with these two, and got different results, especially before removing duplicates:
(rdchiral version)
Avg # dups per product: 5.079888156580787

Calculating ranks before removing duplicates

Test Top-1 accuracy:  46.195%
Test Top-3 accuracy:  66.667%
Test Top-5 accuracy:  73.877%
Test Top-10 accuracy:  80.907%
Test Top-20 accuracy:  85.660%
Test Top-50 accuracy:  87.737%
Test Top-100 accuracy:  87.977%
Test Top-200 accuracy:  87.977%

Calculating ranks after removing duplicates

Test Top-1 accuracy:  46.195%
Test Top-3 accuracy:  67.525%
Test Top-5 accuracy:  74.536%
Test Top-10 accuracy:  82.085%
Test Top-20 accuracy:  86.279%
Test Top-50 accuracy:  87.897%
Test Top-100 accuracy:  87.977%
Test Top-200 accuracy:  87.977%

(rdkit version)
Calculating ranks before removing duplicates

Test Top-1 accuracy:  29.778%
Test Top-3 accuracy:  43.659%
Test Top-5 accuracy:  48.672%
Test Top-10 accuracy:  53.685%
Test Top-20 accuracy:  56.980%
Test Top-50 accuracy:  58.618%
Test Top-100 accuracy:  58.818%
Test Top-200 accuracy:  58.818%

Calculating ranks after removing duplicates

Test Top-1 accuracy:  46.175%
Test Top-3 accuracy:  67.446%
Test Top-5 accuracy:  74.556%
Test Top-10 accuracy:  82.025%
Test Top-20 accuracy:  86.239%
Test Top-50 accuracy:  87.957%
Test Top-100 accuracy:  88.037%
Test Top-200 accuracy:  88.037%

If you could live with this difference, I'd happy to create a PR.

that seems quite strange. did you have a chance to look at the duplicates with rdchiral vs rdkit?
it has been a while, but i assume the duplicates here refer to duplicated reactant-sets predicted by neuralsym. the strangest thing that stands out to me immediately is that the top-1 accuracy increased after removing duplicates. if the duplicates are indeed true duplicates, then removing the top-N duplicated predictions (starting from the top-1 prediction) does not change the top-1 prediction. Therefore, the top-1 accuracy must be the same in both cases. This might point to a bug in either the de-duplication code, or the accuracy-calculation code.

I see, it's the order between the reactants causing the problem. The output of the rdkit version may get multiple identical results that only differs in the order or the reactants, such as c1ccccc1.O and O.c1ccccc1. If I convert each output to canonical SMILES, the new reuslt is as follows, which is much normal:
(rdkit version)
Avg # dups per product: 5.310964649490713

Calculating ranks before removing duplicates

Test Top-1 accuracy:  46.175%
Test Top-3 accuracy:  66.667%
Test Top-5 accuracy:  73.797%
Test Top-10 accuracy:  80.807%
Test Top-20 accuracy:  85.360%
Test Top-50 accuracy:  87.617%
Test Top-100 accuracy:  87.857%
Test Top-200 accuracy:  87.857%

Calculating ranks after removing duplicates

Test Top-1 accuracy:  46.175%
Test Top-3 accuracy:  67.486%
Test Top-5 accuracy:  74.496%
Test Top-10 accuracy:  81.945%
Test Top-20 accuracy:  86.079%
Test Top-50 accuracy:  87.777%
Test Top-100 accuracy:  87.857%
Test Top-200 accuracy:  87.857%

Here is the new version of gen_precs:
def gen_precs(templates_filtered, preds, phase_topk, task):
    i, prod_smi_nomap = task
    # generate predictions from templates
    precursors, dup_count = [], 0
    pred_temp_idxs = preds[i]
    for idx in pred_temp_idxs:
        template = templates_filtered[idx]
        try:
            # rxn = rdchiralReaction(template)
            # prod = rdchiralReactants(prod_smi_nomap)
            # precs = rdchiralRun(rxn, prod)
            # precursors.extend(precs)
            rxn = AllChem.ReactionFromSmarts(template)
            prod = [Chem.MolFromSmiles(prod_smi_nomap)]
            precs = rxn.RunReactants(prod)
            precs = list(set([".".join([Chem.MolToSmiles(p) for p in prec] ) for prec in precs]))
            precs = list(Chem.CanonSmiles(prec) for prec in precs)
            precursors.extend(precs)
        except:
            continue

    # remove duplicate predictions
    seen = []
    for prec in precursors: # canonicalize all predictions
        try:
            prec = Chem.MolToSmiles(Chem.MolFromSmiles(prec), True)
        except:
            prec = None
        if prec and prec not in seen:
            seen.append(prec)
        else:
            dup_count += 1

    if len(seen) < phase_topk:
        seen.extend(['9999'] * (phase_topk - len(seen)))
    else:
        seen = seen[:phase_topk]

    return precursors, seen, dup_count

Cool! Glad that you found and fixed the bug. Would you be willing to make a PR for this?

Cool! Glad that you found and fixed the bug. Would you be willing to make a PR for this?

Of course, I have created a PR and you can check it now.

addressed by #4


[repo] docs: updated getting started docs for additional details

#minor
updated the getting started README docs with minor additional details for clarity and setting customer expectations
Attestation Checklist

[x] My code follows the style guidelines of this project

I have checked for/fixed spelling, linting, and other errors

I have commented my code for clarity

I have made corresponding changes to the documentation (we use TypeDoc to document our code)

My changes generate no new warnings

I have added tests that validates my changes, and provides sufficient test coverage. I have tested with:

Local testing
E2E testing in Teams

New and existing unit tests pass locally with my changes

Additional information

Feel free to add other relevant information below

Thanks @Carter425!


Update to Jetty 9.3.11.M0

This fixes a race when using async responses

looks good

This should be done in airbase.

I think it's better to have it in the airlift pom. Putting them in airbase is error prone, as it's less obvious what version is being used. imo, we should remove Jetty from airbase entirely

I'm fine with removing it, but don't just update it here, because any projects using airbase will pull the wrong version.


chore: add node10 in circleci

And update some dependencies.

Codecov Report

Merging #32 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #32   +/-   ##
=======================================
  Coverage   94.87%   94.87%
=======================================
  Files           1        1
  Lines          39       39
  Branches        8        8
=======================================
  Hits           37       37
  Misses          2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fbed169...d8ec9b8. Read the comment docs.


Add util test cases

Creating this PR to check if this problem persists.
Created test class:

TestUtil

Can one of the admins verify this patch?

test this please

ok to test


Add files via upload

This is the Govt. Institution that provides training to all the teachers of School Education Department of Government of Punjab, Pakistan. There are many computer programming courses that are offered.  Here is the url of School Education Department: https://schools.punjab.gov.pk/
Here is the URL where academics and trainings offered for teaching staff:
http://qaed.edu.pk/pages/subpages/6
Here are the computer courses offered in Punjab Schools
https://sabaq.pk/book-page.php?b=p&c=12&s=cs
https://sabaq.pk/book-page.php?b=p&c=11&s=cs

@seaatif First stage of verification passed, official domain is qaed.edu.pk as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

@seaatif I have to close this request without merging, because you requested to add a non-existent domain to this repository: ;; connection timed out; no servers could be reached Please ensure that you are familiar with this repository rules located at the bottom of this page and submit a new request with the correct information. Thank you for your understanding!

The hosting server might have been down due to some problem. But its up again and working now. Please recheck. http://qaed.edu.pk/


corporate: Clean up some sponsorship related billing code.

Notes

The first commit changes is_sponsored_or_pending in BillingSession to be implemented on the abstract class vs the child classes. This logic is used when getting the initial upgrade page context and should do the same checks for all classes.
The second commit removes the now unused is_realm_sponsored as it's been replaced by the is_sponsored method on the RealmBillingSession class.

Self-review checklist

[ ] Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

[ ] Explains differences from previous plans (e.g., issue description).
[ ] Highlights technical choices and bugs encountered.
[ ] Calls out remaining decisions and concerns.
[ ] Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

[ ] Each commit is a coherent idea.
[ ] Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

[ ] Visual appearance of the changes.
[ ] Responsiveness and internationalization.
[ ] Strings and tooltips.
[ ] End-to-end functionality of buttons, interactions and flows.
[ ] Corner cases, error conditions, and easily imagined bugs.

Merged, thanks @laurynmm!


Move "Language Service Extensibility"

PR Microsoft/TypeScript#12231 was merged at Feb 15, 2017. And v2.2.1 is tagged at Feb 18, 2017.
So this feature is available with typescript:^2.2.1, isn't this ?

ok. Thanks for your reply.


deterministic_matching_rules -> rules_to_estimate_prior

Is your proposal related to a problem?

The various rules that a user can define in the training of a Splink model can get confusing as they all do different things (despite the rules themselves looking the same). Blocking Rules are being addressed in #1389, but we should make some decisions on how we refer to the rules for estimating the prior.
Describe the solution you'd like

Rename deterministic_matching_rules in estimate_probability_two_random_records_match to rules_to_estimate_prior. (thanks to @zslade for the suggestion)
We should also change the references to deterministic_rules in the tutorials and example notebooks to ensure consistency across the docs to something more explicit. Possibly rules_to_estimate_prior, but I haven't thought too hard about the naming at this stage.
Describe alternatives you've considered

Additional context

I agree this is a bit confusing.  It should be clear that these are rules to estimate the prior, but I'm not sure about the suggestion for two reasons:

We don't use the terminology prior in the Splink public API itself.  Expert users are likely to be familiar with this terminology from reading record linkage theory, but I'm not sure other users would be familiar with it.  Instead, we use probability_two_random_records_match"
This only appears as an argument to the https://moj-analytical-services.github.io/splink/linker.html#splink.linker.Linker.estimate_probability_two_random_records_match.  Since it's an argument to the function, it's arguably tautological to repeat it in the function call.

The other place it appears is the docs, e.g. here.  The first time it appears is as a variable assignment:
deterministic_rules = [
    "l.first_name = r.first_name and levenshtein(r.dob, l.dob) <= 1",
...

We could certainly rename this to be more specific, which may help; it would also have the advantage of not breaking backwards compatibility

Yea that makes sense. Happy for it to leave the function argument and make it a terminology convention (possibly overlapping with Sam's issue #1401) in the docs and examples (and probably in our own pipelines too to make things consistent).


Allow empty string identifiers

This PR fixes a subtle bug that disallowed the UnitOfWork from containing entities with a '' identifier.
Note: this fix targets ORM 2.6, because empty identifiers are still subject to bug https://github.com/doctrine/doctrine2/issues/5923

hiding

@deeky666 thanks!


Better 404 page

Right now, going to a page that doesn't exist,  it just shows huge svgs or a blank page.

https://docs.axiom.co/getting-started/  -> should be a 404 page
https://docs.axiom.co/getting-started-2 -> should be a 404 page

similar to: https://github.com/axiomhq/axiom/issues/2168
Closing for now as we will be migrating docs fully to axiom.co


Scrape rate without account restrictions?

To avoid my personal LinkedIn account from being banned, I used new alternate accounts. Unfortunately, each of these (3) new accounts were locked down after less than 50 scrapes each. Using my own account I can scrape significantly more (~300) without being restricted. I need to scrape many more profiles (~3000), so I'm wondering if someone has successfully scraped that many without having their account restricted. If so, what are config values were you using maxConcurrentCrawlers and profileLoadWaitTime?
Thanks!!!
(PS I was using Default config (2 & 5000) for those runs)

Wrong repo


Added in related person structure definition based on CP ITI 1207.

Based on CP 1207 I added in a RelatedPerson Structure definition for PMIR for review.

@lduncan-intrah I accepted the pull request


Conduct integration tests

Test whether new functionalities hinder previously implemented ones. If so, these problems have to be solved as quickly as possible, as this would break the backwards compatibility.

Can't be conducted before #1


Use Sphinx for documentation

Following on from #205, I have integrated Sphinx documentation into the project.
This PR creates the necessary config and index files for the change. The recommonmark extension is included to allow for Markdown documentation, such as the current guides. sphinx-apidoc is used to automatically generate additional documentation from the current docstrings.
Upon initially running the documentation build, I found a number of docstring syntax errors that I have fixed in the second commit 😄.
The build process for the documentation is now:
m -f docs/sanic.rst
rm -f docs/modules.rst
sphinx-apidoc -o docs/ sanic
sphinx-build -b html docs docs/_build

HTML is generated in the docs/_build directory.
It should now be possible to integrate the repository with ReadTheDocs. This would hopefully make Sanic more accessible to new users and increase adoption 🎉.

@wilcoschoneveld Thanks for the detailed feedback! I'll outline some of my reasoning below:
Gitignore changes, apidoc build location
I agree with you on these ones - looks good.
Separate index.rst and README.md
This one really annoyed me when I was setting it up! I'd love to just include the current README.md in the index, but unfortunately the Sphinx include directive only supports RestructuredText. Are you aware of another way around this issue :smile:?
Makefile
I debated a Makefile but decided against it because I didn't want to add infrastructure like that for only one part of the build process - it seemed a bit overkill to me. However, I think it would be great to have one that covers docs, testing, cleaning etc. Maybe one of us could make a separate PR?
sphinx-apidoc
Another issue for me when I was setting up. Generally I'm a fan of manual documentation as you suggested. However,  for a couple of reasons I decided to include it:

For a pre-1.0 release I think it's helpful to have the docs adapt automatically to API changes to make sure they don't get out of sync, although this could be alleviated if we made it a requirement for all PRs to contain the corresponding doc changes.
Currently the module is missing some docstrings, and this could be a good way to encourage greater coverage.

Please let me know what you think and I'll make the changes later today!

Maybe then convert README.md to .rst? GitHub would also render it. Using sphinx-apidocs, at least until the api is stable, sounds like a good idea. Right now there are some conflicts that need to be resolved before a merge can happen. @seemethere, what do you think?

OK, I've converted the README and moved the autogenerated documentation to the docs/_api directory.

I can work on fixing the merge conflicts, I do like the direction though.

Looks good to me in the current state! @kdelwat awesome work! @seemethere can you merge this?

Going to cut a release to 0.2.0 will move this to the next milestone

@kdelwat I can no longer fix the merge conflicts on the web client. Can you please resolve them locally?

@seemethere Sure, I'll get onto it today.

@seemethere Do you have a preference between a merge commit or rebase?

@kdelwat I think a merge commit would be better in this case and I'll just a squash and merge when we do eventually merge this

@seemethere All done! I've also added the new guides to the documentation index.

More merge conflicts fixed (I'm fighting a losing battle!) and @wilcoschoneveld's note added - I didn't know about the -f flag, thanks!


Undefined document introduced when using import.meta.url in worker

Describe the bug
// main.js
new Worker(new URL("./worker.js", import.meta.url), { type: 'module' });

// worker.js
console.log(import.meta.url);

Build result
// main
new Worker(new URL("/assets/worker-f523fd11.js", self.location), {
  type: "module",
});

// worker
(function () {
  "use strict";
  console.log(
    (document.currentScript && document.currentScript.src) ||
      new URL("assets/worker-f523fd11.js", document.baseURI).href
  );
})();

I'm 100% sure document does not exist in web worker and my worker will panic.
Reproduction
https://stackblitz.com/edit/vitejs-vite-vyu6ig?file=worker.js,index.html&terminal=dev
Steps to reproduce
run vite build
System Info
-

Used Package Manager
npm
Logs
I think use typeof document != 'undefined' && before document.currentScript is better
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

Yep can confirm, been having similar issue with using Web Worker in Nuxt 3


Implement quadcopter state space (dynamic model/search space)

The quadcopter's state space has been implemented on badcaae04ce7619a7875fd276b0ce9f70ba6de65. It will be closed as soon as it is tested enough.