Integrate npm

✈  This is the flight deck for the Integrate npm project. ✈

Current open-source crowdfunding options (Kickstarter, Patreon, Gratipay, OpenCollective, etc.) are consumer-grade. Our hunch is that a business-grade product with better aggregation can better serve the companies that want to pay for open source, because companies use hundreds or thousands of open source packages, not just a few.
Picking up from https://github.com/gratipay/gratipay.com/pull/4135#issuecomment-255122149 and https://github.com/gratipay/inside.gratipay.com/issues/852#issuecomment-255098337 ...
For wider context see:

The fundamental idea is to get a viral loop going again like we had under Gittipay 1.0.
The burn I feel

JavaScript is the most popular language in open source and npm is the most popular package manager for JavaScript. A good first concrete step towards helping companies pay for open source (#4135), therefore, will be to add the ability to pay for any package on npm. Once we have npm deployed, we will have enough experience to inform a partnership with Libraries.io for the rest of the package managers.
Target
Our goal is to announce this feature in my lightning talk on Thursday, October 26 at Red Hat's All Things Open conference (https://github.com/gratipay/inside.gratipay.com/issues/757).
Our goal is to incrementally improve this feature throughout the first half of 2017, with an eye towards OSCON and $ustain in May.
Package names to test with
From https://github.com/gratipay/gratipay.com/pull/4135#issuecomment-262672635:
http://localhost:8537/on/npm/async/
http://localhost:8537/on/npm/iframe-resizer/
http://localhost:8537/on/npm/mongoose/
http://localhost:8537/on/npm/nodemon/
http://localhost:8537/on/npm/react/
http://localhost:8537/on/npm/react-helmet/
http://localhost:8537/on/npm/react-modal/
http://localhost:8537/on/npm/react-redux/
http://localhost:8537/on/npm/react-router/
http://localhost:8537/on/npm/react-router-redux/
http://localhost:8537/on/npm/redux/
http://localhost:8537/on/npm/redux-thunk/
http://localhost:8537/on/npm/webpack/
Todo
Prerequisites

[x] Land the Decouple milestone—https://github.com/gratipay/inside.gratipay.com/issues/432
[ ] Finish Project Basics—https://github.com/gratipay/gratipay.com/milestone/18

Checkpoint 1: Inert /on/npm/foo/ Pages

[x] wrap marky-markdown (#4154)
[x] load up npm (#4153, #4158)
[x] load up npm readmes (#4159, #4160, #4161, #4162, #4164, #4176, #4179, #4180, #4181, #4182)
[x] stop storing readmes after all (#4211)
[x] add inert (can't actually give to them) /on/npm/foo pages (#4212)

Checkpoint 2: Giving to Packages

[ ] link packages to teams (#4155)
[ ] opt in or out from /on/npm/foo/ pages via email verification
[ ] add payment widget on /on/npm/foo/ pages

Checkpoint 3: Easy Sign-up

[ ] bulk manage payments from /on/npm/ pages

Nice to Have

[x] move search into {% content %} (leaving social network jump in sidebar)
[ ] merge current result types together (~user via username, ~user via profile statement)
[ ] add Teams to local index for inclusion in results
[ ] mix npm packages into search (#4147)
[ ] pledging to as-yet-unclaimed packages
[ ] load up npm incrementally (reticket from #4153)
[ ] deal intelligently with unpublished packages (reticket from https://github.com/gratipay/gratipay.com/issues/4148#issuecomment-256525647)
[ ] we seem to get two H1s on https://www.npmjs.com/package/resize-img
[ ] syntax highlight

Promotion

[ ] update the "★ announcement ★" link (#4149)
[ ] Publish a blog post.
[ ] Promote the heck out of it!

✈  This is the flight deck for the Integrate npm project. ✈

@mattbk You wanna draft a blog post?

@aandis How about I start hacking on this today and tomorrow, and I will leave notes for you over the weekend based on how far I get. Ya?

I've started a task list in the ticket description.

@aandis How about I start hacking on this today and tomorrow, and I will leave notes for you over the weekend based on how far I get. Ya?

Yep. That works. A quickstart on https://github.com/gratipay/gratipay.com/pull/4147 will help in implementing the exact functionality you have in mind.

@aandis What's your availability? I was assuming Saturday and Sunday would be the time you could work on this but maybe I'm wrong?

@whit537 I am free over weekend, yes. I'll be keeping a close eye on this ticket for the next couple of days.

I've reached out to npm, Inc. in private email to let them know about our plans and invite them to give feedback and/or cross-promote with us next week.

Is there a rate limit on npm registry API requests? Can it be up'd with authentication? I find no docs on this.

We should think about possibly merging existing teams into these.

The code that npm uses to talk to the registry.

https://github.com/npm/npm-registry-client

http://nipstr.com

@whit537 I am not seeing any official well documented api for searching packages over npm. Best I could find is https://github.com/wires/npm-keywordsearch through this.

Hashed this out a bit with @kaguillera. The problem we're solving is companies funding open source software. That's a market. The two sides are companies (buyers) and projects (sellers). We need to make it as easy as possible for companies to give to open source software; that's #4135. This is the other side, making it as easy as possible for developers to receive payments for open source.
/on/npm/
One thing we realized is that a single developer will have multiple projects. "Easy as possible" means letting them configure multiple projects at once. Doug Wilson is maintaining 103 packages. We can't make him go through 103 separate verification workflows. He needs one worfklow.
We're thinking that that workflow should live at /on/npm/.
-anon—Sign in to configure payments for your npm packages.
-auth—Verify the email you use on npm to configure payments your npm packages.
-no packages—We didn't find any npm packages with<EMAIL_ADDRESS>as an author or maintainer.
-packages—We found 103 npm packages with<EMAIL_ADDRESS>as an author or maintainer.
We should account for the case where another ~user has already claimed the package.
Added to todo. ^^^ Bring this over to a PR when the time comes.
Going all in?
The other thing we realized is that the 400,000 packages on npm dwarf the 200 projects currently on Gratipay. We should consider redoing the homepage to contain a search box (for givers and the curious) and a link to /on/npm/ (for developers ready to start receiving payments). We don't need to list "our Teams" on the homepage. With this change, "our Teams" instantly explodes to the size of npm.

Oh! And we should bring back pledging: givers should be able to set up payment instructions to projects before the projects opt-in. This is where opt-out starts to matter, because it prevents pledging.

We're getting back to the same dynamic as Gittipay 1.0, but with a focus on packages on open source software repositories (→ Teams [Projects]) rather than users of social media networks (→ ~users).

Also, it seems unlikely that we'll get this all done before next Thursday. Let's do what we can ... and have fun doing it! :-)

Alright, @aandis, I've stubbed out a few PRs and updated the todo in the description here. Hopefully that gives you something to go on before I get online tomorrow!
!m @aandis

@aandis Can we use this thread to communicate and coordinate about what specifically we're each working on? I'm seeing work from you on #4154 and #4155 (!m @aandis). What are you working on now? Do you think the todo on this ticket is right and in the right order or do we need to modify it?
I'm going to start at the top of the todo with #4117 and then move down to join you on the actual coding bits. :-)
Let's do this! 💃  🌻

@whit537 I am gonna wrap up #4154 and then work on #4153 and #4155 to make #4151 possible. I think  we should move pledging out of #4155 into it's own ticket and focus on mvp to be able to give to a package. Pledging is a nice to have feature but only when the groundwork is laid.

@aandis Cool. Strictly speaking, #4155 isn't necessary for #4151. In other words, we could have /on/npm/foo/ pages without any possibility of giving to them. Is it worth batching #4154 #4153 and #4151 as our first checkpoint, and tackle #4155 after that first checkpoint is deployed?

{#4154, #4153, #4151} feels like something we could potentially get deployed before the end of the day today. Deployment itself will take some effort because of the need to spin up a worker dyno—something we haven't done before—to handle the continual npm syncing. Sure, inert /on/npm/foo/ pages won't be much to brag about, but it'll at least give us a feeling of win. :)
Thoughts?

Sounds good. :)

@aandis Cool. I've updated the todo.

Alright, sooooooo ... it's emerging that the Decouple milestone, which should really get done before this (so we are free to accept all of the new Teams → projects we're hoping to get from this), is going to have some code implications. In particular we really ought to reflect a couple name changes in our codebase to avoid drift and tech debt.

Teams → projects
members → collaborators

I guess that's just a heads-up that we're looking at some potentially serious rebases of the work under here once the dust settles under https://github.com/gratipay/inside.gratipay.com/issues/432.

once the dust settles under gratipay/inside.gratipay.com#432

This will be a happy day.

Maybe Tuesday? 😳

I guess that's just a heads-up that we're looking at some potentially serious rebases of the work under here once the dust settles under gratipay/inside.gratipay.com#432.

Scratch that, @aandis. It doesn't make sense to block this on the full codebase renames on gratipay/inside.gratipay.com#432. We can still get the important stuff done (Terms of Service update, in particular), without renaming tables.

@aandis Heads up that I'm going to be offline today. I've been going pretty hard and need a break before I fly out tomorrow for https://github.com/gratipay/inside.gratipay.com/issues/757. Whatever you get done under this will be awesome! Tomorrow afternoon/evening I should be able to pick up where you leave off.
Also, I think it may be better to focus on bulk sign-up on /on/npm/ instead of adding npm packages to search. The former is of more value to developers, which is who we're aiming at here. The latter is for companies/givers, which is less of the focus this week.

!m @aandis 💃

@whit537 I am just gonna finish what I can on https://github.com/gratipay/gratipay.com/pull/4153 since that's anyway going to be needed for email signups. I'll try to find time this week for more. Good luck for https://github.com/gratipay/inside.gratipay.com/issues/757 :)

Thank you! :-)

Also, I think it may be better to focus on bulk sign-up on /on/npm/ instead of adding npm packages to search.

Todo updated to reflect this.
I just compiled my schedule for this week. I should have a little time tomorrow and Tuesday, and then I should be free from lunchtime onwards on Wednesday until I fall asleep. I would really rather not be trying to hack on Thursday morning before the lightning talk, so my goal would be to get everything done by Wednesday night.
Please forgive me, @aandis @rohitpaulk et al.: I may well cut lots of corners and self-merge PRs and whatnot in order to get this done. I really really want to demonstrate something real and interesting on Thursday! :rage4: Let's go!

I'm gonna pause on decoupling since that's pretty far along. The remaining changes to the About pages are low-risk. I'm going to focus here over the next couple days and come back to decoupling on Wednesday.

The remaining changes to the About pages are low-risk.

If I get some time I'll dip into https://github.com/gratipay/gratipay.com/pull/4157 to move it along.

Okay! #4153 is deployed, and I've scheduled a daily job for 15:00 UTC.

Gonna get ready to head over to https://github.com/gratipay/inside.gratipay.com/issues/757, I'll check Papertrail before I leave. I won't be back for ~6 hours, at which point I plan to start hacking on the rest of this ticket.

Your account has reached 100% log data transfer for this period. Log processing has been paused.

End of the month! Would you believe we got cut off, like two hours ago? 😞

Nothing yet. 😕
gratipay::MAROON=> select count(*) from packages;
┌───────┐
│ count │
├───────┤
│     0 │
└───────┘
(1 row)

😕
[gratipay] $ heroku logs -a gratipay | grep scheduler
[gratipay] $

"Last run" is now Oct 26 15:00 UTC. Maybe I will try it from heroku run bash ...

[gratipay] $ heroku run -a gratipay bin/sync-npm.sh > first-run.log 2>&1

[gratipay] $ tail -f first-run.log
Running bin/sync-npm.sh on gratipay... starting, run.1126 (Hobby)
Running bin/sync-npm.sh on gratipay... connecting, run.1126 (Hobby)
Running bin/sync-npm.sh on gratipay... up, run.1126 (Hobby)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 24 27.2M   24 6784k    0     0   946k      0  0:00:29  0:00:07  0:00:22  825k

It's processing packages ...
econds
processed 2000 packages in   3 seconds
  0  201M    0 1167k    0     0   253k      0  0:13:32  0:00:04  0:13:28  253kprocessed 3000 packages in   4 seconds
  1  201M    1 2063k    0     0   367k      0  0:09:21  0:00:05  0:09:16  367kprocessed 4000 packages in   5 seconds
processed 5000 packages in   5 seconds
  1  201M    1 2895k    0     0   435k      0  0:07:52  0:00:06  0:07:46  503kprocessed 6000 packages in   6 seconds
processed 7000 packages in   6 seconds
  1  201M    1 3727k    0     0   490k      0  0:07:00  0:00:07  0:06:53  573kprocessed 8000 packages in   7 seconds
  2  201M    2 4431k    0     0   513k      0  0:06:41  0:00:08  0:06:33  809kprocessed 9000 packages in   8 seconds
  2  201M    2 5263k    0     0   549k      0  0:06:15  0:00:09  0:06:06  821kprocessed 10000 packages in   9 seconds
  2  201M    2 5967k    0     0   560k      0  0:06:07  0:00:10  0:05:57  776kprocessed 11000 packages in  10 seconds
  3  201M    3 6415k    0     0   549k      0  0:06:14  0:00:11  0:06:03  699kprocessed 12000 packages in  11 seconds

Oh heck yeah. Streaming! 💃
If I'm reading that right, it's sending data to postgres as it's discovered, not waiting to read the whole file.

Well, sorry. It's serializing as it downloads. I don't think it's uploading as it serializes.

Though some additional logging would help confirm that.
P.S. This play-by-play should maybe still be on #4153. :o)

Back from https://github.com/gratipay/gratipay.com/pull/4158. Rescheduling the sync ...

Scheduled job has kicked off!

2016-10-27T00:35:37.400471+00:00 heroku[scheduler.9462]: Process exited with status 0

Okay, we're live with the npm syncer.
gratipay::MAROON=> select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 372271 │
└────────┘
(1 row)

gratipay::MAROON=>

2016-10-27T00:35:26.231474+00:00 app[scheduler.9462]: processed 372271 packages in 247 seconds
2016-10-27T00:35:35.102567+00:00 heroku[scheduler.9462]: source=scheduler.9462 dyno=heroku.1942532.7efd96ec-cd0b-4718-b884-a779ee6f7de1 sample#load_avg_1m=0.77 sample#load_avg_5m=0.98
2016-10-27T00:35:35.102790+00:00 heroku[scheduler.9462]: source=scheduler.9462 dyno=heroku.1942532.7efd96ec-cd0b-4718-b884-a779ee6f7de1 sample#memory_total=140.20MB sample#memory_rss=34.25MB sample#memory_cache=105.94MB sample#memory_swap=0.00MB sample#memory_pgpgin=220943pages sample#memory_pgpgout=185564pages sample#memory_quota=512.00MB
2016-10-27T00:35:37.420115+00:00 heroku[scheduler.9462]: State changed from up to complete
2016-10-27T00:35:37.400471+00:00 heroku[scheduler.9462]: Process exited with status 0

Okay, so that looks like 247 seconds for the download/serialization, and another 11 seconds for the upsert. Call it ~260 seconds == 4.333 minutes.

We have a nightly backup from last night. I am taking another one just for safety.

Looks like packages adds 15 MB, or 21%.

Curveball: some packages are unpublished without having a security package in their place. They show up in all and also on the registry, but their JSON is actually 404 and they have an unpublished time. They don't show up on site search or HTML.

I'm adding the node buildpack back to our app so we can use npm install marky-markdown.
[gratipay] $ heroku run "npm install marky-markdown && sync-npm readmes"
Running npm install marky-markdown && sync-npm readmes on ⬢ gratipay... up, run.4576 (Hobby)
bash: npm: command not found
[gratipay] $ heroku buildpacks
=== gratipay Buildpack URL
https://github.com/gratipay/buildpack-python.git#gratipay-prod
[gratipay] $ heroku buildpacks:add heroku/nodejs
Buildpack added. Next release on gratipay will use:
  1. https://github.com/gratipay/buildpack-python.git#gratipay-prod
  2. heroku/nodejs
Run git push heroku master to create a new release using these buildpacks.
[gratipay] $ git push heroku master

gratipay::MAROON=> select count(*) from packages where readme_raw is not null;
┌───────┐
│ count │
├───────┤
│    22 │
└───────┘
(1 row)

gratipay::MAROON=>

Okay! Gonna set this up in the scheduler.

Blorg. Hitting a memory limit. :-/
2016-10-27T03:30:45.084125+00:00 app[scheduler.6359]: svg2css
2016-10-27T03:30:45.088103+00:00 app[scheduler.6359]: splat-points-2d
2016-10-27T03:30:45.092127+00:00 app[scheduler.6359]: signeds3
2016-10-27T03:30:45.125973+00:00 app[scheduler.6359]: saltyrtc-task-webrtc
2016-10-27T03:30:45.127890+00:00 app[scheduler.6359]: resst-request
2016-10-27T03:30:45.723554+00:00 app[scheduler.6359]: 404 for zzzzzzzzzzzzzzzzzzz
2016-10-27T03:30:45.724534+00:00 app[scheduler.6359]: zzzttt
2016-10-27T03:30:45.979220+00:00 app[scheduler.6359]: 404 for zzzttt
2016-10-27T03:30:45.979404+00:00 app[scheduler.6359]: zzzss
2016-10-27T03:31:00.393328+00:00 heroku[scheduler.6359]: source=scheduler.6359 dyno=heroku.1942532.ee6fcf22-109a-49cf-aa0a-0136911a86ac sample#memory_total=641.52MB sample#memory_rss=103.62MB sample#memory_cache=4.98MB sample#memory_swap=532.91MB sample#memory_pgpgin=251526pages sample#memory_pgpgout=223723pages sample#memory_quota=512.00MB
2016-10-27T03:31:00.394012+00:00 heroku[scheduler.6359]: Process running mem=641M(124.3%)
2016-10-27T03:31:00.394066+00:00 heroku[scheduler.6359]: Error R14 (Memory quota exceeded)

Started again under scheduler ...

[gratipay] $ heroku logs --tail | grep scheduler
2016-10-27T03:50:20.064405+00:00 heroku[api]: Starting process with command `sync-npm readmes` by scheduler@ad
dons.heroku.com
2016-10-27T03:50:27.921066+00:00 heroku[scheduler.4775]: Starting process with command `sync-npm readmes`
2016-10-27T03:50:28.518982+00:00 heroku[scheduler.4775]: State changed from starting to up
2016-10-27T03:50:35.273088+00:00 heroku[scheduler.4775]: source=scheduler.4775 dyno=heroku.1942532.ea097600-5c
50-446b-bc32-d9ebbacfd652 sample#memory_total=35.95MB sample#memory_rss=31.01MB sample#memory_cache=4.95MB sam
ple#memory_swap=0.00MB sample#memory_pgpgin=14112pages sample#memory_pgpgout=5419pages sample#memory_quota=512
.00MB
2016-10-27T03:50:42.900770+00:00 app[scheduler.4775]: zzzzzzzzzzzzzzzzzzz
2016-10-27T03:50:42.933254+00:00 app[scheduler.4775]: utilise.clone
2016-10-27T03:50:42.936615+00:00 app[scheduler.4775]: svg2base64
2016-10-27T03:50:42.948038+00:00 app[scheduler.4775]: semver-bumper-for-file-text
2016-10-27T03:50:43.226081+00:00 app[scheduler.4775]: 404 for zzzzzzzzzzzzzzzzzzz
2016-10-27T03:50:43.226131+00:00 app[scheduler.4775]: zzzttt
2016-10-27T03:50:43.458823+00:00 app[scheduler.4775]: 404 for zzzttt
2016-10-27T03:50:43.458926+00:00 app[scheduler.4775]: zzzazzz
2016-10-27T03:50:44.033136+00:00 app[scheduler.4775]: 404 for zzzazzz
2016-10-27T03:50:44.033150+00:00 app[scheduler.4775]: zzz_012_censorify
2016-10-27T03:50:54.621207+00:00 app[scheduler.4775]: utilise.client
2016-10-27T03:50:55.719408+00:00 heroku[scheduler.4775]: source=scheduler.4775 dyno=heroku.1942532.ea097600-5c50-446b-bc32-d9ebbacfd652 sample#memory_total=360.68MB sample#memory_rss=355.62MB sample#memory_cache=5.00MB sample#memory_swap=0.06MB sample#memory_pgpgin=158661pages sample#memory_pgpgout=67876pages sample#memory_quota=512.00MB
2016-10-27T03:50:55.774905+00:00 app[scheduler.4775]: svg2android
2016-10-27T03:50:56.292132+00:00 app[scheduler.4775]: semver-bumper

gratipay::MAROON=> select count(*) from packages where readme_raw is not null;
┌───────┐
│ count │
├───────┤
│    82 │
└───────┘
(1 row)

gratipay::MAROON=>

Call it 50 per minute. That's over five days. 😶

Alright, I'm about done here. We didn't make it.
Oh well. :-)

I've turned off both recurring tasks.

Sorry, I've removed them from the scheduler so that they don't recur. The existing process is still chugging along. It just crossed 1,000 readmes. We're coming up on 1100 at 20 minutes. Holding steady about 54 per minute, so I guess we let this run for five days?

I don't think so @mattbk. If I am not mistaken @whit537 the five days is for the initial loading and processing of the npm readme. The future updates would be less since it should only update the ones that have changed and add new ones that should not be that much.

!m @whit537

Thank goodness.

Yes, though I had forgotten that the Heroku scheduler kills processes when the time limit is up, so I am just restarting it now.

😞
On Oct 27, 2016 5:57 PM, "Chad Whitacre"<EMAIL_ADDRESS>wrote:

Yes, though I had forgotten that the Heroku scheduler kills processes when
the time limit is up, so I am just restarting it now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gratipay/gratipay.com/issues/4148#issuecomment-256781714,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABNTSUTOuEBz5np9LhcZPd_tTissHeW8ks5q4R45gaJpZM4KcQZM
.

from packages where readme_raw is not null) / (select count(*) from packages)::float;
┌───────────────────┐
│     ?column?      │
├───────────────────┤
│ 0.114843756295817 │
└───────────────────┘
(1 row)

gratipay::MAROON=>

  7 remote

26%

35%
I count 109,354 unique email addresses.
select unnest(emails) as email, count(name) from packages group by email;

npackages
nemails

1,000+
8

100-999
462

10-99
10,266

2-9
52,567

1
46,051

Looks like npm is stripping a leading H1.

Hmm ... https://www.npmjs.com/package/marky-markdown#npm-packages

Syncers are running again! https://github.com/gratipay/inside.gratipay.com/issues/884#issuecomment-259320916
I guess when we upsert we'll want to go off the time from back when we first started loading things up a couple weeks ago, back at https://github.com/gratipay/gratipay.com/issues/4148#issuecomment-256370122, to make sure we pick up any changes since then.

Fetcher crashed!
2016-11-09T02:45:47.778173+00:00 app[scheduler.7320]: Traceback (most recent call last):
2016-11-09T02:45:47.779195+00:00 app[scheduler.7320]:   File "/app/.heroku/python/bin/sync-npm", line 9, in <module>
2016-11-09T02:45:47.779275+00:00 app[scheduler.7320]:     load_entry_point('gratipay', 'console_scripts', 'sync-npm')()
2016-11-09T02:45:47.779327+00:00 app[scheduler.7320]:   File "/app/gratipay/package_managers/sync.py", line 160, in main
2016-11-09T02:45:47.779601+00:00 app[scheduler.7320]:     globals()[args.command.replace('-', '_')](env, args, db)
2016-11-09T02:45:47.779619+00:00 app[scheduler.7320]:   File "/app/gratipay/package_managers/sync.py", line 140, in fetch_readmes
2016-11-09T02:45:47.779835+00:00 app[scheduler.7320]:     k-component-page_readmes.fetch(db)
2016-11-09T02:45:47.779868+00:00 app[scheduler.7320]:
2016-11-09T02:45:47.779887+00:00 app[scheduler.7320]:   File "/app/gratipay/package_managers/readmes.py", line 93, in fetch
2016-11-09T02:45:47.781538+00:00 app[scheduler.7320]:     threaded_map(Fetcher(db), dirty, 4)
2016-11-09T02:45:47.781590+00:00 app[scheduler.7320]:   File "/app/gratipay/utils/threaded_map.py", line 23, in threaded_map
2016-11-09T02:45:47.781767+00:00 app[scheduler.7320]:     raise e.args[0]
2016-11-09T02:45:47.782241+00:00 app[scheduler.7320]: psycopg2.ProgrammingError: can't adapt type 'dict'

Meanwhile the Processor is starting.

Also crashed!
2016-11-09T02:50:45.011994+00:00 app[scheduler.7918]: Traceback (most recent call last):
2016-11-09T02:50:45.012249+00:00 app[scheduler.7918]:   File "/app/.heroku/python/bin/sync-npm", line 9, in <module>
2016-11-09T02:50:45.012320+00:00 app[scheduler.7918]:     load_entry_point('gratipay', 'console_scripts', 'sync-npm')()
2016-11-09T02:50:45.012355+00:00 app[scheduler.7918]:   File "/app/gratipay/package_managers/sync.py", line 160, in main
2016-11-09T02:50:45.012543+00:00 app[scheduler.7918]:     globals()[args.command.replace('-', '_')](env, args, db)
2016-11-09T02:50:45.012573+00:00 app[scheduler.7918]:   File "/app/gratipay/package_managers/sync.py", line 144, in process_readmes
2016-11-09T02:50:45.012635+00:00 app[scheduler.7918]:     _readmes.process(db)
2016-11-09T02:50:45.012663+00:00 app[scheduler.7918]:   File "/app/gratipay/package_managers/readmes.py", line 97, in process
2016-11-09T02:50:45.012887+00:00 app[scheduler.7918]:     dirty = db.all('SELECT id, package_manager, name, description, readme_raw '
2016-11-09T02:50:45.012917+00:00 app[scheduler.7918]:   File "/app/.heroku/python/lib/python2.7/site-packages/postgres/__init__.py", line 548, in all
2016-11-09T02:50:45.013917+00:00 app[scheduler.7918]:     return cursor.all(sql, parameters)
2016-11-09T02:50:45.013947+00:00 app[scheduler.7918]:   File "/app/.heroku/python/lib/python2.7/site-packages/postgres/cursors.py", line 145, in all
2016-11-09T02:50:45.014009+00:00 app[scheduler.7918]:     self.execute(sql, parameters)
2016-11-09T02:50:45.014036+00:00 app[scheduler.7918]:   File "/app/.heroku/python/lib/python2.7/site-packages/psycopg2/extras.py", line 288, in execute
2016-11-09T02:50:45.014475+00:00 app[scheduler.7918]:     return super(NamedTupleCursor, self).execute(query, vars)
2016-11-09T02:50:45.014555+00:00 app[scheduler.7918]: psycopg2.ProgrammingError: syntax error at or near "BY"
2016-11-09T02:50:45.014557+00:00 app[scheduler.7918]: LINE 1: ... packages WHERE readme_needs_to_be_processedORDER BY package...

I'm surprised the test suite didn't catch that one. Hmm ...

Ah! Now I see that we weren't testing that code. 😊

PR in https://github.com/gratipay/gratipay.com/pull/4176 ...

Hmm ... readme processor is running out of memory.
$ heroku logs --tail | grep scheduler
2016-11-09T03:50:28.311155+00:00 heroku[api]: Starting process with command `sync-npm process-readmes` by<EMAIL_ADDRESS>2016-11-09T03:50:35.164998+00:00 heroku[scheduler.8487]: Starting process with command `sync-npm process-readmes`
2016-11-09T03:50:35.846204+00:00 heroku[scheduler.8487]: State changed from starting to up
2016-11-09T03:50:47.749213+00:00 heroku[scheduler.8487]: source=scheduler.8487 dyno=heroku.1942532.1fbd95bf-81f1-4a26-aa9f-a30e88765924 sample#memory_total=142.36MB sample#memory_rss=137.41MB sample#memory_cache=4.95MB sample#memory_swap=0.00MB sample#memory_pgpgin=41624pages sample#memory_pgpgout=5691pages sample#memory_quota=512.00MB
2016-11-09T03:51:07.948771+00:00 heroku[scheduler.8487]: source=scheduler.8487 dyno=heroku.1942532.1fbd95bf-81f1-4a26-aa9f-a30e88765924 sample#memory_total=371.86MB sample#memory_rss=366.91MB sample#memory_cache=4.95MB sample#memory_swap=0.00MB sample#memory_pgpgin=100375pages sample#memory_pgpgout=5691pages sample#memory_quota=512.00MB
2016-11-09T03:51:28.261222+00:00 heroku[scheduler.8487]: source=scheduler.8487 dyno=heroku.1942532.1fbd95bf-81f1-4a26-aa9f-a30e88765924 sample#load_avg_1m=0.29
2016-11-09T03:51:28.261321+00:00 heroku[scheduler.8487]: source=scheduler.8487 dyno=heroku.1942532.1fbd95bf-81f1-4a26-aa9f-a30e88765924 sample#memory_total=1107.29MB sample#memory_rss=494.66MB sample#memory_cache=0.00MB sample#memory_swap=612.63MB sample#memory_pgpgin=406200pages sample#memory_pgpgout=280077pages sample#memory_quota=512.00MB
2016-11-09T03:51:28.261959+00:00 heroku[scheduler.8487]: Process running mem=1107M(216.3%)
2016-11-09T03:51:28.261959+00:00 heroku[scheduler.8487]: Error R15 (Memory quota vastly exceeded)
2016-11-09T03:51:28.262044+00:00 heroku[scheduler.8487]: Stopping process with SIGKILL
2016-11-09T03:51:28.473986+00:00 heroku[scheduler.8487]: Process exited with status 137
2016-11-09T03:51:28.519921+00:00 heroku[scheduler.8487]: State changed from up to complete

It's never even getting to the point where it emits a log line about processing even a single readme.

Ah, I bet it's because we're including readme_raw in the initial query over all unprocessed packages.

PR in https://github.com/gratipay/gratipay.com/pull/4177.

Blorg.
Traceback (most recent call last):
  File "/app/.heroku/python/bin/sync-npm", line 9, in <module>
    load_entry_point('gratipay', 'console_scripts', 'sync-npm')()
  File "/app/gratipay/package_managers/sync.py", line 160, in main
    globals()[args.command.replace('-', '_')](env, args, db)
  File "/app/gratipay/package_managers/sync.py", line 144, in process_readmes
    _readmes.process(db)
  File "/app/gratipay/package_managers/readmes.py", line 100, in process
    threaded_map(Processor(db), dirty, 4)
  File "/app/gratipay/utils/threaded_map.py", line 23, in threaded_map
    raise e.args[0]
OSError: net.js:639
    throw new TypeError('invalid data');
    ^

TypeError: invalid data
  at Socket.write (net.js:639:11)
  at /app/bin/our-marky-markdown.js:20:18
  at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:404:3)

Unable to reproduce locally. Too jetlagged, more tomorrow ...

Lessee here ...

I can repro the bug using a one-off dyno.

Still works locally. Hmm ...

https://github.com/gratipay/gratipay.com/issues/4148#issuecomment-259330483 points to net.js and fs.js. Let's see if we can get a better error log ...

~ $ sync-npm process-readmes
zzzzzzzzzzzzzzzzzzz
utilise.emitterify
svg2png-cli
semver-max
zzzzzzxl
react-router-native
plane-to-polygon
Traceback (most recent call last):
  File "/app/gratipay/utils/threaded_map.py", line 15, in g
    return func(*a, **kw)
  File "/app/gratipay/package_managers/readmes.py", line 72, in process
    processed = markdown.render_like_npm(raw)
  File "/app/gratipay/utils/markdown.py", line 44, in render_like_npm
    raise OSError(err)
OSError: net.js:639
    throw new TypeError('invalid data');
    ^

TypeError: invalid data
  at Socket.write (net.js:639:11)
  at /app/bin/our-marky-markdown.js:20:18
  at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:404:3)

Traceback (most recent call last):
  File "/app/.heroku/python/bin/sync-npm", line 9, in <module>
    load_entry_point('gratipay', 'console_scripts', 'sync-npm')()
  File "/app/gratipay/package_managers/sync.py", line 160, in main
    globals()[args.command.replace('-', '_')](env, args, db)
  File "/app/gratipay/package_managers/sync.py", line 144, in process_readmes
    _readmes.process(db)
  File "/app/gratipay/package_managers/readmes.py", line 100, in process
    threaded_map(Processor(db), dirty, 4)
  File "/app/gratipay/utils/threaded_map.py", line 23, in threaded_map
    raise e.args[0]
OSError: net.js:639
    throw new TypeError('invalid data');
    ^

TypeError: invalid data
  at Socket.write (net.js:639:11)
  at /app/bin/our-marky-markdown.js:20:18
  at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:404:3)

planet-news
planet-names
planet-maps-raster-2
planet-maps-raster
planet-maps
planet-icons
planet-hacker
planet-forty
planet-feeds
planetfall-archive
planet-facts
planet-express
planet-digest
planet-css
planet-client
planet-classic
planet-blank
planetary-system
planetary.js
planetary
planet
planer
planepacker
planemo
plane-mesh

Is it a problem with stdout?

The 1st argument given to .write() is expected to be a String or Buffer.

http://stackoverflow.com/a/26851940

What is marky returning that is not a String or Buffer?

I made a local modification to just process packages with names starting with pla, but wasn't able to trigger the error that way.

I overwrote the our-marky-markdown.js script on a Heroku dyno (using here strings) to give some debugging output.
html for nullisn't a string, it's a functionfunction (selector, context, r, opts)

Alright, so we need to call .html() on the output of marky(). But why doesn't it fail locally? I have node 4.3.1 where Heroku has 5.11.1. Maybe the write API changed?

Nope.

Maybe I don't have the right data loaded up locally?

Well, they do differ.
Local
gratipay=# select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 372147 │
└────────┘
(1 row)

gratipay=# select count(*) from packages where readme_raw is not null;
┌───────┐
│ count │
├───────┤
│   134 │
└───────┘
(1 row)

gratipay=#

Remote
gratipay::MAROON=> select count(*) from packages;                                                             ┌────────┐
│ count  │
├────────┤
│ 372271 │
└────────┘
(1 row)

gratipay::MAROON=> select count(*) from packages where readme_raw is not null;
┌────────┐
│ count  │
├────────┤
│ 138260 │
└────────┘
(1 row)

gratipay::MAROON=>

Oh! I'm seeing marky-markdown v9.0.0 locally, but 8.1.0 on Heroku.

Cached build?

For the fetcher bug (https://github.com/gratipay/gratipay.com/issues/4148#issuecomment-259322332), I think we'll need to unthread in order to get better error messaging.

Actually, threaded_map is supposed to output the original traceback to stdout. Are we losing that? I guess we'll have to reschedule to see ...

Well that is consternating. Even after https://github.com/gratipay/gratipay.com/pull/4178 I am still seeing 8.1.0 on a Heroku dyno. Hmm ...

Got it!

sync-npm process-readmes is working from a one-off dyno. 👍

I'm not seeing an error from sync-npm fetch-readmes in a one-off dyno. I've rescheduled both that and process-readmes.

Alright, the original traceback is in there, I had just confused it for a dupe.
2016-11-09T23:51:23.098776+00:00 app[scheduler.8465]: jstuningTraceback (most recent call last):
2016-11-09T23:51:23.098779+00:00 app[scheduler.8465]:   File "/app/gratipay/utils/threaded_map.py", line 15, i
n g
2016-11-09T23:51:23.098780+00:00 app[scheduler.8465]:     return func(*a, **kw)
2016-11-09T23:51:23.098781+00:00 app[scheduler.8465]:   File "/app/gratipay/package_managers/readmes.py", line
 55, in fetch
2016-11-09T23:51:23.098781+00:00 app[scheduler.8465]:     , dirty.name
2016-11-09T23:51:23.098782+00:00 app[scheduler.8465]:   File "/app/.heroku/python/lib/python2.7/site-packages/
postgres/__init__.py", line 374, in run
2016-11-09T23:51:23.098783+00:00 app[scheduler.8465]:     cursor.run(sql, parameters)
2016-11-09T23:51:23.098784+00:00 app[scheduler.8465]:   File "/app/.heroku/python/lib/python2.7/site-packages/
postgres/cursors.py", line 92, in run
2016-11-09T23:51:23.098784+00:00 app[scheduler.8465]:     self.execute(sql, parameters)
2016-11-09T23:51:23.098785+00:00 app[scheduler.8465]:   File "/app/.heroku/python/lib/python2.7/site-packages/
psycopg2/extras.py", line 288, in execute
2016-11-09T23:51:23.098786+00:00 app[scheduler.8465]:     return super(NamedTupleCursor, self).execute(query,
vars)

Interesting. So maybe name or something can be a dict coming from npm?

That doesn't sound right ...

I'm working on a PR to add Sentry support to these procs so we have better visibility into and resiliency in the face of errors (surely these won't be the last).

PR in #4179.

So it turns out that the registry includes packages that have been unpublished—a lot of them, from what I can tell. They appear as JSON with a 404. E.g. below.
Currently we log "404" for these and leave readme_raw untouched, which means it'll still be null next time around and we'll refetch the same 404. We should notice this case and probably drop the record from our database, though we'll want to be careful to bring it back again when someone else claims it.
I'm a little surprised at how many 404s I'm seeing. How does unpublishing relate to deleting or removing a package?
http://registry.npmjs.com/mysql-schema
https://www.npmjs.com/package/mysql-schema
{
    "_id":"mysql-schema",
    "_rev":"4-c527f6a64778f8d0afbaf6fd4754085e",
    "name":"mysql-schema",
    "time":{
        "modified":"2013-12-08T03:17:01.297Z",
        "created":"2013-12-08T03:16:59.506Z",
        "0.0.1":"2013-12-08T03:17:01.297Z",
        "unpublished":{
            "name":"carlosmarte",
            "time":"2014-07-26T06:27:28.217Z",
            "tags":{
                "latest":"0.0.1"
            },
            "maintainers":[
                {
                    "name":"carlosmarte",
<EMAIL_ADDRESS>                }
            ],
            "description":"mysql queries helper",
            "versions":[
                "0.0.1"
            ]
        }
    },
    "_attachments":{

    }
}

Okay! Fetcher and processor appear to be chugging along. We want to see these both hit zero, though they won't until we delete or skip 404s.
gratipay::MAROON=> select count(*) from packages where readme_raw is null;
┌────────┐
│ count  │
├────────┤
│ 223221 │
└────────┘
(1 row)

gratipay::MAROON=> select count(*) from packages where readme_needs_to_be_processed;
┌────────┐
│ count  │
├────────┤
│ 323828 │
└────────┘
(1 row)

gratipay::MAROON=>

Done in https://github.com/gratipay/gratipay.com/pull/4181 and deployed.

Deleting! 👍
2016-11-10T07:50:24.124188+00:00 app[scheduler.4899]: fetching practiceone
2016-11-10T07:50:23.863759+00:00 app[scheduler.9696]: fetching node-u2f
2016-11-10T07:50:23.939573+00:00 app[scheduler.9696]: fetching node-typograph
2016-11-10T07:50:23.974219+00:00 app[scheduler.9696]: no readme in killdrev
2016-11-10T07:50:23.974273+00:00 app[scheduler.9696]: fetching kill-desktop-osx
2016-11-10T07:50:24.020858+00:00 app[scheduler.9696]: fetching infinigon-tag
2016-11-10T07:50:24.027898+00:00 app[scheduler.9696]: fetching kill-dash-nine
2016-11-10T07:50:24.037882+00:00 app[scheduler.9696]: 404 for spurious-js-aws-sdk-helper
2016-11-10T07:50:24.038176+00:00 app[scheduler.9696]: fetching spur-di
2016-11-10T07:50:24.081055+00:00 app[scheduler.9696]: fetching node-typo
2016-11-10T07:50:24.152517+00:00 app[scheduler.4899]: yet-another-module is 404; deleting
2016-11-10T07:50:24.155182+00:00 app[scheduler.4899]: fetching yet-another-friendly-dependency
2016-11-10T07:50:24.185761+00:00 app[scheduler.4899]: fetching practice-npm-package
2016-11-10T07:50:24.203793+00:00 app[scheduler.4899]: fetching jspm-nodelibs-process
2016-11-10T07:50:24.228972+00:00 app[scheduler.4899]: fetching jspm-nodelibs-path
2016-11-10T07:50:24.273577+00:00 app[scheduler.4899]: fetching hwsl2
2016-11-10T07:50:24.279907+00:00 app[scheduler.4899]: fetching jspm-nodelibs-os
2016-11-10T07:50:24.304598+00:00 app[scheduler.4899]: yet-another-friendly-dependency is 404; deleting
2016-11-10T07:50:24.308818+00:00 app[scheduler.4899]: fetching yet-another-express-routing
2016-11-10T07:50:24.348231+00:00 app[scheduler.4899]: fetching practice_npm
2016-11-10T07:50:24.358175+00:00 app[scheduler.4899]: fetching jspm-nodelibs-net
2016-11-10T07:50:24.367534+00:00 app[scheduler.4899]: yet-another-express-routing is 404; deleting
2016-11-10T07:50:24.370328+00:00 app[scheduler.4899]: fetching yet-another-express-router
2016-11-10T07:50:24.132520+00:00 app[scheduler.9696]: fetching node-typhoon
2016-11-10T07:50:24.162294+00:00 app[scheduler.9696]: 404 for spur-di
2016-11-10T07:50:24.162538+00:00 app[scheduler.9696]: fetching sptitesmith-stylus-retina-template
2016-11-10T07:50:24.182196+00:00 app[scheduler.9696]: fetching kill-combo
2016-11-10T07:50:24.218516+00:00 app[scheduler.9696]: 404 for sptitesmith-stylus-retina-template
2016-11-10T07:50:24.218923+00:00 app[scheduler.9696]: fetching spruce
2016-11-10T07:50:24.253262+00:00 app[scheduler.9696]: no readme in spruce
2016-11-10T07:50:24.253270+00:00 app[scheduler.9696]: fetching sprout-object
2016-11-10T07:50:24.268288+00:00 app[scheduler.9696]: fetching node-typewriter
2016-11-10T07:50:24.291538+00:00 app[scheduler.9696]: 404 for sprout-object

I'm curious to see how many records we're left with after weeding out the 404s.
Okay! I'm gonna let this run overnight ...

Almost done fetching! Processing is slower to catch up ...
gratipay::MAROON=> select count(*) from packages where readme_raw is null;
┌───────┐
│ count │
├───────┤
│  3349 │
└───────┘
(1 row)

gratipay::MAROON=> select count(*) from packages where readme_needs_to_be_processed;
┌────────┐
│ count  │
├────────┤
│ 258387 │
└────────┘
(1 row)

gratipay::MAROON=> select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 345044 │
└────────┘
(1 row)

No movement on first and third. I think we have some packages that aren't 404 but also maybe don't have a readme? I think that's how readme_raw is ending up null for a percentage.
gratipay::MAROON=> select count(*) from packages where readme_raw is null;
┌───────┐
│ count │
├───────┤
│  3349 │
└───────┘
(1 row)

gratipay::MAROON=> select count(*) from packages where readme_needs_to_be_processed;
┌────────┐
│ count  │
├────────┤
│ 230868 │
└────────┘
(1 row)

gratipay::MAROON=> select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 345044 │
└────────┘
(1 row)

27,519 readmes processed in five hours, call it 5,000 an hour, so ... 45-50 hours remaining? Should be done over the weekend?

230,868 - 207,714 = 23,154 in about five hours. 👍

Fetcher crashed!

Captured with #4179: https://sentry.io/gratipay/gratipay-com/issues/179430708/. 👍

mysql-prettify has a readme of {"private": true}. It shows as a string on the HTML npm; looking at the json I do see an object there (though my browser is not prettifying it for some reason? ironic given the name 😛 ).

I don't see a repo for mysql-prettify. I was gonna check package.json to see if that's where the bad value is coming from. Do we want to stringify it or count is as "no readme"?

Alright, the {"private": true} issue should be fixed in #4182.

I'm going to fix the other Sentry bug that I introduced: https://sentry.io/gratipay/gratipay-com/issues/179441800/.

PR in #4183. Waiting for Travis.
gratipay::MAROON=> select count(*) from packages where readme_raw is null;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

gratipay::MAROON=> select count(*) from packages where readme_needs_to_be_processed;
┌───────┐
│ count │
├───────┤
│ 97562 │
└───────┘
(1 row)

gratipay::MAROON=> select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 345044 │
└────────┘
(1 row)

gratipay::MAROON=>

207,714 - 97,476 = 110,238 in about 19 hours = 5,802. On track! 👍

gratipay::MAROON=> select count(*) from packages where readme_raw is not null;
┌────────┐
│ count  │
├────────┤
│ 345042 │
└────────┘
(1 row)

gratipay::MAROON=> select count(*) from packages where readme_needs_to_be_processed;
┌───────┐
│ count │
├───────┤
│     3 │
└───────┘
(1 row)

gratipay::MAROON=> select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 345044 │
└────────┘
(1 row)

gratipay::MAROON=>

Here are the three remaining to be processed:
https://www.npmjs.com/package/testing233
https://www.npmjs.com/package/testing234
https://www.npmjs.com/package/kendo-ui-react-jquery-stockchart
The first two give "ERROR: No README data found!" The third results in a 504!

Is this part of a plan to build an index of all the popular libraries? Next is Maven rep for Java, if this works out?

Yeah, something like that. Ideally we can partner with Libraries.io and bring a bunch online at once.

¯\_(ツ)_/¯
gratipay::MAROON=> select count(*) from packages where readme_raw is not null;
┌────────┐
│ count  │
├────────┤
│ 345044 │
└────────┘
(1 row)

gratipay::MAROON=> select count(*) from packages where readme_needs_to_be_processed;
┌───────┐
│ count │
├───────┤
│     0 │
└───────┘
(1 row)

gratipay::MAROON=> select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 345044 │
└────────┘
(1 row)

gratipay::MAROON=>

Alright! READMEs initially loaded! 💃
Let's get some pages on the site! #4151

Slight change of plans. #4151 is too much of a rabbit hole. We don't to get ourselves into the business of processing and securing READMEs across 30+ package managers. Our existing /on/network/foo/ pages aren't very contentful, there's no reason /on/npm/foo/ need be, either. My current plan is to make a PR to remove the README processing that is already deployed (we still want package fetching and syncing, since emails are in there and that's our key for linking with users), and then move on to Checkpoint 2: Giving to Packages.

@kaguillera and I are talking IRL about how much tech debt we want to take on underneath Relax Open Work Requirement in order to fast-track the npm feature here. Over there, we are renaming Teams to projects and members to collaborators, and we are also now talking about removing tip migration. It's basically a question of whether we change names in the UI only, or also remove code and drop/rename database tables and such. The trade-off is that if we only make surface changes over there, then the next @JessaWitzel that comes along will have even more confusion to deal with ("Wait—projects are stored in the teams table? WTF!")—but I am going to be so incredibly mad if we're not first to market with npm pledging. :rage4:
@JessaWitzel Can we please please please go into debt here? I PROMISE I will fix it and make it all better in January or February. Or March. 🙏

Discussing in slack.

January or February. Or March.

😄

Ok. I will bless this technical debt with my fairy wand but request a deadline for fixing it.
12 weeks after feature launch

@JessaWitzel at slack

We have reached Checkpoint 1: Inert /on/npm/foo/ Pages! 💃
https://gratipay.com/on/npm/react-router/

Anything I can do to accelarate this? It seems top priority for now.
One question: is NPM cool with this, and are they willing to partner/coordinate in any way?

Thanks for bumping this, @nobodxbodon! As mentioned in slack, I hoping to spec this out this week while also bringing Relax Open Work Requirement in for landing (this was blocked on that).

is NPM cool with this, and are they willing to partner/coordinate in any way?

I emailed them and didn't hear back. I think if we get some traction with this, that will be the time to reapproach a conversation with them.

any workload estimate and roadmap with timeline?

Last month I guesstimated (one, two) that the two projects together would take six weeks of calendar time. It's now been four and a half weeks and we're not done with either yet. We're likely to finish the first by the end of this week (Week 5), with next week being Week 1 on Integrate npm. It seems unlikely to take a week. ;-) I was off by a factor of 2.5 on the Relax Open Work time estimate (I figured two out of the six). That suggests 10 weeks of calendar time for Integrate npm.
Roadmap and further estimation tbd when I can spec this out.

who are main developers and any dividing of tasks?

Me and maybe @aandis? Anybody else wanna volunteer? :-) Division of tasks tbd.

any blockers?

https://github.com/orgs/gratipay/projects/5?fullscreen=true

I've put both projects on the calendar, with an initial target date of March 17 for Integrate npm.

Honestly this is probably going to go into April or May, but we can take that as it comes. :-)

I'd really like to see this happening on gratipay so I am ready to help as much as I can. We need a development plan so that things are moving along.

I've put both projects on the calendar

This would be more useful in Gantt chart format, IMO. The calendar is cool for events, but in terms of lining up work, it's just a bunch of blue bars to me.

Conversation about product developement and wire-framing, might try https://gomockingbird.com/.

Closing per https://github.com/gratipay/inside.gratipay.com/issues/987#issuecomment-272985143.


Arm Cue Action

Using the Arm Cue Action causes the module to throw this error: "ERROR
Can't connect to QLab cue_id is not defined" however, buttons continue to function despite the module claiming it cannot connect and all of the buttons having a red square in the upper right corner.
It doesn't seem to matter whether its set to Arm, Disarm, or Toggle

Which companion and QLab versions?

companion 3.3.1
QLab 5.3.6

Beta release tonight should fix this


Philo no bonus badge

not my best work but here is the philo no bonus badge.

It looks great! Picasso would be jealous if he saw it. 😅

If you open a pull request, I can pull it 😊


Can we calculate the Token costs (aka figure out what OpenAI will charge)

First:  This is really great, and I can see tremendous value that me and others can get out.
And more of a question/than an issue...
Anyway to build the data model/sample questions to estimate costs?  As always with any tool we bring in its about ROI - so knowing some way to evaluate it (I assume questions are tokens, but also sending the data to openAI etc. is also)

Hey @lrauhockey, I like this idea but I don't have time to add it, feel free to add a PR if you can think of a nice kwarg or path to test this.
For when running locally, if you change this line
https://github.com/approximatelabs/sketch/blob/85f65360136d4bec042daeee2e3b0fadfb69c686/sketch/pandas_extension.py#L221
you can extract the exact prompt that will be sent to the endpoint with the command:
prompt_string = howto_prompt.prompt_template.render(dfname=dfname, data_description=description, how=how)

From this you could use the openai tokenizer to get the total number of requested tokens.


New package: WrapIt v1.1.1

Registering package: WrapIt
Repository: https://github.com/grasph/WrapIt.jl
Created by: @grasph
Version: v1.1.1
Commit:<PHONE_NUMBER>659a295e3f37d4a8df242830106ec3
Reviewed by: @grasph
Reference: https://github.com/grasph/WrapIt.jl/issues/1#issuecomment-1963886419

[noblock]
To me it looks like the Julia package is a little disconnected from the main functionality by being in its own repository. Are you aware that you can place the Julia package in a subdirectory of the main repository?
If you like the way it's set up now you can just ignore this message. I just wanted to point out the option if you didn't know about it.

[noblock]
Then all you need to do is to edit your comment to include [noblock], so you don't block yourself from an automatic merge when the waiting time is up.


Updated function signature.

OnDeleteItem function signature should have an int as it's first parameter.

@isaac868 : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Benchmark - Is this scalable for a large dataset ?

How long will this take to export a table with over  1million records to the excel ? Is this scalable?

I'd suggest benchmarking this with your own dataset to be sure. Feel free to share the results.


Add readiness/liveness probe values to falco-exporter-chart

What type of PR is this?
/kind feature
/kind chart-release
Any specific area of the project related to this PR?
/area falco-exporter-chart
What this PR does / why we need it:
Add variables for readiness and liveness probes to configure timings such as initialDelaySeconds
Which issue(s) this PR fixes:
Special notes for your reviewer:
My first PR, hopefully I have considered everything.
Checklist

[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] CHANGELOG.md updated

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: hardwarefresser
Once this PR has been reviewed and has the lgtm label, please assign leogr for approval. For more information see the Kubernetes Code Review Process.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Welcome @hardwarefresser! It looks like this is your first PR to falcosecurity/charts 🎉

LGTM label has been added.  Git tree hash: a4c0500d1d3862c601b5d22ce0f996ec8f4481da

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: alacuku, hardwarefresser, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


More query param handling

This adds better handling of query parameters, including default values & mapping key/values to different strings for url.
Default Values
Default values can now be specified with the withDefault wrapper function
export const useProjectFilters = makeQueryHandler({
  onlyMultipleEngagements: withDefault(BooleanParam(), false),
});

The underlying library has this function as well but we are using a custom one now to:

Remove default values from the url when using setter
Supplies default values to the hooks current value even when omitted from the url

The types also now correctly account for the value being undefined when withDefault is not used.
Different keys for url
Our query param keys are usually pretty specific due to whatever is best for code, and for syncing with API keys.
However, these are not always the best keys to use in urls.
The best example is onlyMultipleEngagements flag which just got renamed (formally "cluster").
This is really long for urls, so now we can map that to something shorter like multi.
https://github.com/SeedCompany/cord-field/blob/92dcead3797b4201a05d8163595c7338291ba85f/src/scenes/Projects/List/ProjectFilterOptions.tsx#L44
Transforms
Along the same lines as keys, it would also be nice to be able to map values in the url.
I've added a withTransform wrapper function.
Previously useSort transformed values by referencing the underlying functions directly:
https://github.com/SeedCompany/cord-field/blob/a57b6e10ea80b54ea0ee9da90ef8de85b249c041/src/components/Sort/useSort.ts#L16-L19
Now with the decorator we have those functions given to us:
https://github.com/SeedCompany/cord-field/blob/3c2d8f4548c39c97ed9f3940b91840eac7619e40/src/components/Sort/useSort.ts#L15-L18
I'm not exactly thrilled with the API - I would've liked to make it simpler. However, we don't know if we should be calling the inner decoder/encoder before or after the transform function, so I just opted to give it to the transform function. This way it can decided whether to call it before or after the transformation.
I also updated project filter status & sensitivity definitions to lowercase their values.

I also know there's that render error on status change due to an infinite loop. I opted to not solve that here.

I also know there's that render error on status change due to an infinite loop. I opted to not solve that here.

This also happens when I change sensitivity

Err I believe that's what I meant. I think it's a problem with the SwitchField. If I comment out one of the fields the other works.

Err I believe that's what I meant. I think it's a problem with the SwitchField. If I comment out one of the fields the other works.

Coolcool, just checking


Add example FASTA sequences

@sdash-github requested adding example FASTA sequences to the Home page, to avoid having to find a sequence somewhere and paste it into the text area.

First cut: clicking any of the "example" radio buttons pastes an appropriate sequence in the text area.
http://dev.lis.ncgr.org:50080/shiny/Funnotate/

Example sequences are no longer hard coded, now you store the example FASTA files in static/examples/.
http://dev.lis.ncgr.org:50080/shiny/Funnotate/

Committed (f68d2df) and pulled to production. Feel free to replace the files in static/examples/ with your own examples.
To do: Automatically determine whether they are DNA or protein sequences, to save another potential click.

To do: Automatically determine whether they are DNA or protein sequences, to save another potential click.

Done (2d536f0).


Youtube Ads

I'm scratching my head here. I have the default hosts file installed and working for most sites, but youtube ads are not being blocked. Is this just me, or have others experienced this?

YouTube ads can't be blocked by host files without blocking the actual video afaik.

Thanks Thomas @thomasthomas11235, Keegan @gkeegan is correct.  Those ads come from the same domain as the vid.  It's painful, I know...
Closing.

Do those ads not all come from pagead2.googlesyndication.com?
https://superuser.com/a/882747/601553

No. @gloschtla


Did it done in Wechat of China?

How about Wechat?Can it run?
The browser in  Chinese Wechat is different.It is always not the same in different versions.
Do you have to follow the version of WeChat updates?
Thanks!

Can you test this page on WeChat on iOS 9? Does it autoplay? http://bfred-it.github.io/iphone-inline-video/demo/
I cannot test it


Boot test AMIs

This PR rewrites the osbuild-upload-aws command to a new boot-aws that supports three subcommands: setup, teardown, and run.

setup uploads an image to AWS and boots it, setting up user credentials and security policies for SSH.  It prints the IP address and saves the IDs of all newly created objects to a file (resources.json by default).
teardown reads the file created by the setup command and cleans up all the resources/objects.
run runs setup, uploads an executable to the remote host, runs it, then runs teardown.

This is set up to run every time a new AMI is built in CI.  It's a basic boot test that simply makes sure that the image boots and can run some basic commands to print information about the image (rpm -qa, cat /etc/os-release, uname -a).

Let's rebuild all the test images while we're at it :)

Things got a bit messy with some outdated local copies and minor mistakes in the command line arguments to the boot test script.  Should be okay now.

Things got a bit messy with some outdated local copies and minor mistakes in the command line arguments to the boot test script. Should be okay now.

5 force-pushes later, I think it might actually be okay this time.

There's one thing that I don't like too much, specifically the two forks of the internal AWS cloud library in this repo and osbuild-composer repo. This PR modifies it, which will only make things worse going forward. Did you think about actually making the library public in one of these repos and using it in the other? 🤔

Yes, I think we should make this one public and import it in composer.  We've talked a few times (all of us I think) about whether it's weird to have cloud upload + boot code in images vs composer and I think we all agree (to varying degrees maybe) that it's okay to have it here.
None of the changes here are very big, so I think it wont be much work to adapt composer.

I think it might be that we don't set PYTHONUNBUFFERED: 1 when running the build, boot, and upload scripts like we do for the generators.  Since it was a bash script, I guess I didn't add it before but we should now.  That should fix the log output.  I'll add it now.

The PYTHONUNBUFFERED fixed the ordering of the Testing image at ./build/fedora_40-x86_64-ami-empty/image/image.raw message, but I'm still not seeing the script output.  Looking into it.

Found the issue.  It was actually a bad combination of two things:

The scp command was failing because it was running scp path/to/file host:path/to/file and failing because path/to doesn't exist on the host.
The error from the scp command was being clobbered by the cleanup functionality returning without errors and so the exitCheck() was returning success.

Fixed and updated.  Running a local test in the meantime as well.

I checked RHEL 9.0 x86 AMI and it's degraded because kdump fails to start with kdump: No memory reserved for crash kernel.

Did some testing on EC2 and locally and I think increasing the memory of the machines might solve the issue.  I switched them to small from micro.  It still doesn't explain why some were failing and others weren't, within the same major distro version.  Maybe 1 GiB is just borderline?
One thing I did when testing the RHEL 9.0 x86 AMI in EC2 was (sort of accidentally) upgrade the kernel to the one in 9.2 and on reboot, kdump was fine.  Not sure if this means that kdump will fail with specific kernel versions when the amount of memory isn't large enough.
Let's see if the change in instance type fixed it and take it from there.

Yes, that's a bit of an issue.  I've been thinking about this and I don't have the best solution yet.  The problem is when PRs like this get rebased a lot and different images get built at different times.  If you go into the generator for each distro version, you can see the commit SHA and a GitHub link to the commit when it was built last.  Also, in the top level coverage check, it now lists the images that were built in the CI run.
The last run here only built the RHEL 9.0 aarch64 AMI, because that was the last one that was failing to boot cleanly before I increased the instance memory for ARM.
I think it might be worth spending some time figuring out a good strategy to report image builds on a PR.  But it shouldn't just list every image built at every rebase.  I'm imagining an edge case where a PR changes an image accidentally and is then rebased to revert the change.  You wouldn't want it listed in the PR changes in the end.  So maybe what we need is a job that collects every image that differs from main and lists the images somewhere with a link to the commit.

This for example links to the last time the Fedora 38 x86 images were built for each one
https://gitlab.com/redhat/services/products/image-builder/ci/images/-/jobs/5329921859

This is good, but it is still not obvious whether the commit when an image was built was/is part of the PR. Anyway, this is probably not something to figure out in this PR, but a general issue of the testing approach for images and trying to make it efficient.

Agree 100%.  I'll figure something out :)


Option to set scan directory missing in admin area

Followed the recommended Docker-compose method.
I have modified only the path to volumes under the backend inside the docker-compose.yml file (as shown)
  backend:
    image: reallibrephotos/librephotos:dev
    # For development uncomment the "build" line below and comment the "image" line above
    # build: .
    container_name: librephotos-backend
    restart: always
    volumes:
      - $HOME/Pictures/librephotos/librephotos_images:/data # CHANGE ME
      - $HOME/Pictures/librephotos/librephotos_media:/code/protected_media # CHANGE ME
      - $HOME/Pictures/librephotos/librephotos_logs:/code/logs # CHANGE ME
      - $HOME/Pictures/librephotos/librephotos_cache:/root/.cache # CHANGE ME

Logged in as admin but the link to set the scan directory is missing.

Not sure if this helps but found this issue in the backend logs on running docker logs librephotos-backend
Traceback (most recent call last):
File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
self.handle_request(listener_name, req, client, addr)
File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
super().handle_request(listener_name, req, sock, addr)
File "/miniconda/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/miniconda/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 133, in call
response = self.get_response(request)
File "/miniconda/lib/python3.8/site-packages/django/core/handlers/base.py", line 75, in get_response
response = self._middleware_chain(request)
File "/miniconda/lib/python3.8/site-packages/django/core/handlers/exception.py", line 36, in inner
response = response_for_exception(request, exc)
File "/miniconda/lib/python3.8/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception
response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
File "/miniconda/lib/python3.8/site-packages/django/core/handlers/exception.py", line 125, in handle_uncaught_exception
return debug.technical_500_response(request, *exc_info)
File "/miniconda/lib/python3.8/site-packages/django/views/debug.py", line 91, in technical_500_response
text = reporter.get_traceback_text()
File "/miniconda/lib/python3.8/site-packages/django/views/debug.py", line 341, in get_traceback_text
return t.render(c)
File "/miniconda/lib/python3.8/site-packages/django/template/base.py", line 171, in render
return self._render(context)
File "/miniconda/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "/miniconda/lib/python3.8/site-packages/django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "/miniconda/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
return self.render(context)
File "/miniconda/lib/python3.8/site-packages/django/template/base.py", line 986, in render
output = self.filter_expression.resolve(context)
File "/miniconda/lib/python3.8/site-packages/django/template/base.py", line 693, in resolve
obj = template_localtime(obj, context.use_tz)
File "/miniconda/lib/python3.8/site-packages/django/utils/timezone.py", line 186, in template_localtime
return localtime(value) if should_convert else value
File "/miniconda/lib/python3.8/site-packages/django/utils/timezone.py", line 204, in localtime
timezone = get_current_timezone()
File "/miniconda/lib/python3.8/site-packages/django/utils/timezone.py", line 97, in get_current_timezone
return getattr(_active, "value", get_default_timezone())
File "/miniconda/lib/python3.8/site-packages/django/utils/timezone.py", line 83, in get_default_timezone
return pytz.timezone(settings.TIME_ZONE)
File "/miniconda/lib/python3.8/site-packages/pytz/init.py", line 181, in timezone
raise UnknownTimeZoneError(zone)
pytz.exceptions.UnknownTimeZoneError: ''
`

I have found the issue. Please add at a timezone to TIME_ZONE or use the default America/Chicago. Does this fix your issue?

Thanks for the quick response. Setting the TIME_ZONE fixed the issue. I am now able to see the link to the change the directory.
I see that there is another issue. Once i select the scan directory and hit scan photos, it shows "11 items processed" (I had 11 images) but there are no photos under the "Photos" or "Albums" section. Also, I noticed that the scan happened too quickly (as compared to hooram/ownphotos which I tried earlier, took 10 seconds approx.)

This is what I see in the backend logs after hitting "scan photos"
[2020-12-29 19:33:21 +0000] [116] [INFO] Starting gunicorn 20.0.4
[2020-12-29 19:33:21 +0000] [116] [INFO] Listening at: http://<IP_ADDRESS>:8001 (116)
[2020-12-29 19:33:21 +0000] [116] [INFO] Using worker: gevent
[2020-12-29 19:33:21 +0000] [120] [INFO] Booting worker with pid: 120
[2020-12-29 19:33:21 +0000] [121] [INFO] Booting worker with pid: 121
19:33:36 Worker rq:worker:157901144ee446eeb871591551bedbdc: started, version 1.6.1
19:33:36 Subscribing to channel rq:pubsub:157901144ee446eeb871591551bedbdc
19:33:36 *** Listening on default...
19:33:36 Cleaning registries for queue: default
Bad Request: /api/nextcloud/listdir/
Not Found: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
Not Found: /api/jobs/
<IP_ADDRESS> - - [2020-12-29 19:33:57] "POST /build/ HTTP/1.1" 200 139 0.001662
Not Found: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
<IP_ADDRESS> - - [2020-12-29 19:35:47] "POST /build/ HTTP/1.1" 200 139 0.001184
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
Bad Request: /api/nextcloud/listdir/
<IP_ADDRESS> - - [2020-12-29 19:37:27] "POST /build/ HTTP/1.1" 200 139 0.001161
Bad Request: /api/nextcloud/listdir/
Bad Request: /api/nextcloud/listdir/
Not Found: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
Not Found: /api/jobs/
Not Found: /api/jobs/
Not Found: /api/jobs/
Bad Request: /api/nextcloud/listdir/
Not Found: /api/jobs/
Not Found: /api/jobs/

Are the items photos that are .jpg, .png, .gif or .bmp? If that isn't the case could you maybe send me the ownphotos.log file? It is in the mapped logs folder.

I think the issue was with how I provided the path to the volume. Earlier when I ran docker-compose with $HOME/... in volume paths, which did not create any files(logs etc) inside the local volumes. I changed the path to absolute (/home/username/) and ran docker-compose. Everything seems to be working fine now. Docker container was pointing to root (with $HOME) whereas my images were in the default user directory.

Ah okay! I will close the issue then.


Toast as a flash message

Firstly, thanks for a great library. This does reduce the entry barrier for anyone who wants to implement Surface Components inside Phoenix LiveView.
Do you have any plans to implement bootstrap toast message as the flash message component?
I tried, multiple ways to implement toast inside LiveView - but - could not resolve the problem. In a regular Phoenix View, I could make it work. However, inside a LiveView, I could not implement the auto disappearing toast message synchronize with lv:clear-flash event?
I ventured to ask because I saw that you are using bootstrap native - and - handling the javascript in hooks. Just wanted to know if there is a solution.

I am going to release this within a week! And again sorry as I write on every issue, I try to check often but I am struggling a lot to get notifications via email for stuff in this repo.
Only direct mentions seems to work.


ascend support generator api

ascend support generator api

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Compatibility from django 1.4 to 1.11 with python 2.7

It would be great to have a release including this ! 👍

I've forked this project and released a new version compatible with Django 1.11 and 2.0 https://pypi.python.org/pypi?:action=display&name=django-async-messages-redux&version=0.4.0


Coloring difference between nox and overmind

Here's what Nox looks like when I boot my app

Here's what Overmind looks like

There's a few differences here:

Nox shows the yarn run v1.22.17 line under "web" as where Overmind just has it on its own line. I'm not sure which is better...
Same goes for the actual browser-sync command.
The [Browsersync] in "web" on Nox isn't colorized, but in Overmind it is. Notice in the assets it is though
Same goes for some links

I think it might be connected to the fact that Overmind uses PTY (pseudo-terminals) which causes the command to think it's outputting to the terminal, while Nox does not so the command thinks it's outputting to a file. I would like for it to look no different than Overmind's output, but Crystal doesn't have a library that implements PTY right now.
https://evilmartians.com/chronicles/introducing-overmind-and-hivemind
You could verify this by running it with foreman since it doesn't use PTY either and compare the output.
There might be some stuff that isn't attributed to this, though. Like the yarn run ... thing. I'll try to look into it.

Ah cool. Yup. Tried with foreman, and it looks the same. I'm cool with how it looks, I just wanted to make sure we weren't losing out on something. I did try doing some puts "***".colorize.yellow and it came out fine, so we're at least catching that.

Yep, pretty sure it only affects utilities that have different output styling based on what it's outputting to. The article's example is Sidekiq. The article also mentions that there could be lag with a lot of output, but I'm sure it will be fine 😟

It's weird coming back to this issue, checking to see if there are any PTY shards out there yet, finding one and thinking "ahh, but it's kinda old" and then realizing that it's newer than this conversation we had almost 3 years ago.
Anyways: https://github.com/crystal-posix/pty.cr might be helpful


Should also export a viewing key

The Zcash protocol also has support for view keys. Views keys can determine if a tx has been received but not spend them.
I don't believe there is any support for view keys in the full node wallet yet but we could generate and export a view key that could immediately be imported by a user's full node.

Ah wonderful, I thought they didn't exist and wished they did.
Yep, let's add them, and add a mode to input the secret key instead of generating it, to get address/view key from a saved key.


Minesweeper/Beesweeper

I have had the issue of when I implement your code into a webpage it tries to save the canvas as a canvas.png file every time I click on a square. Why would it do this?
Thank you for any attention you may give my question,
LankySmurph

https://github.com/CodingTrain/Rainbow-Code/blob/791870b687e3112a22ea12e9fda1a3fd8465e338/CodingChallenges/CC_71_minesweeper/sketch.js#L70-L84
This is probably happening because of line 83, which you can just remove.

Weird. I think this was added by accident. Fixed it now, thanks.


Draft: Add test to test single kwarg in subscriber func

Description
Fixes #848
Type of change
Please delete options that are not relevant.

[ ] Documentation (typos, code examples, or any documentation updates)
[ ] Bug fix (a non-breaking change that resolves an issue)
[ ] New feature (a non-breaking change that adds functionality)
[ ] Breaking change (a fix or feature that would disrupt existing functionality)
[ ] This change requires a documentation update

Checklist

[ ] My code adheres to the style guidelines of this project (scripts/lint.sh shows no errors)
[ ] I have conducted a self-review of my own code
[ ] I have made the necessary changes to the documentation
[ ] My changes do not generate any new warnings
[ ] I have added tests to validate the effectiveness of my fix or the functionality of my new feature
[ ] Both new and existing unit tests pass successfully on my local environment by running scripts/test-cov.sh
[ ] I have ensured that static analysis tests are passing by running scripts/static-anaylysis.sh
[ ] I have included code examples to illustrate the modifications

Not needed anymore as this is not a bug. Please refer https://github.com/airtai/faststream/discussions/857 for more info.


Suggestion: Warning upon sending spine commands before init

When the spine has not yet been initialized, a warning should appear in the command-line, for easy notice that something is not being done correctly.

Setting the motors and LEDs should already return errors if the spine isn't initiated. It is best practice to not have functions in a wrapper print directly to console, and for every stdout to be done by user-created code. vic-go is not at that point, but eventually I would like to get it to be. I did miss it in some spots though and that should now be fixed. Make sure to check for errors on your own like:
err := ThingFunction()
if err != nil {
      fmt.Println(err)
      return
}


Debug variables aren't accessible over an async/promise call

Issue Type: Bug
VS Code version: Code - Insiders 1.30.0-insider (241923763dbeaf3b64443187e935d55b29f6125d, 2018-12-07T22:31:41.479Z)
OS version: Windows_NT x64 10.0.18290
When using the nodejs/extension debugger, variables/watch/etc aren't accessible over an async/promise call.
Before the async/promise call -- variables are accessible

After the async/promise call -- nothing is accessible

It would be great to still be able to inspect the context (variables, etc) when walking back the call stack over an async/promise call.

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (8 x 4008)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: enabledvideo_decode: enabledvideo_encode: enabledwebgl: enabledwebgl2: enabled

Memory (System)
31.93GB (15.64GB free)

Process Argv
C:\Users\Eric\code\codestream\vscode-codestream\codestream.code-workspace

Screen Reader
no

VM
0%

This is a limitation from chrome/node. When you look into an async frame, you are looking into the past, so the data for those variables doesn't exist anymore. Nothing we can do on our end unfortunately.

Damn :(


How to transfer a repository?

I'm the author of base16-kermit and I'm curious about how to transfer a repository to be under this organization. Is there a process that I should follow? Thanks.

Hey @orhun I'm looking into creating a document regarding this now. It's been addressed here at least: https://github.com/tinted-theming/home/issues/84
If you're still interested in transferring and joining tinted theming I can help you with that before the document is completed.

Yup! I'm still interested.

Great! I'll invite you to be a member of the team in a minute, once you accept you can go to your repo -> settings -> Transfer ownership (at the bottom of the default settings page).

Awesome! I transferred the repository. Is there anything else that I need to do?

We have a document which describes this process now so I'm going to close this issue: https://github.com/tinted-theming/home/pull/90


enh: config with multiple promises

Fixes #2711

@EugeneHlushko updated, PTAL. ✨

@EugeneHlushko seems fixed now, PTAL 😄

Thanks!


Add changes in arch_test.h file, split macros into 2 files and enable multiple Privilege Modes in Trap Handler

This PR is co-authored-by: allenjbaum<EMAIL_ADDRESS>and modification of PR-267.
As requested by @pawks, the trap handler and macro definitions have been split into two different files, arch_test.h contains definitions of trap handler, macros to change privilege modes, and other necessary macros while macros to manage storing signatures, and ops dependent general macros are defined in test_macros.h file.
The major changes with respect to PR-267 are as follows:

Removed the bugs in RVTEST_GOTO_LOWER_MODE macro
Removed the bugs in RVTEST_GOTO_MMODE macro and defined strap_routine directive.
Updated RVTEST_TRAP_SAVEAREA and RVTEST_TRAP_EPILOG macros to refine the definitions of per mode save area macros.

Hello @pawks, @allenjbaum and I have updated the trap handler, and now it is working perfectly. Please review and merge it

I'm unclear what has been deprecated; do you mean that #warning has been
deprecated in the toolchain?
On Tue, Sep 20, 2022 at 7:12 AM InspireSemi @.***>
wrote:

@.**** requested changes on this pull request.
In test_macros.h
The #warnings have been commented out. Either we uncomment these or just
remove the this whole section from the file. Lines 768 - 818.
These macros have been deprecated and now is as good a time as any to
remove them.
—
Reply to this email directly, view it on GitHub
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#pullrequestreview-1113906240,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHPXVJUCZERQXXUX5DVEU3DV7HA4RANCNFSM56PYCGHQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

The macros in this section of the header are marked as being deprecated.
Someone commented out the #warning messages that say this macro is deprecated since 3.0….
So either uncomment the #warning messages and leave the macros in place or remove the whole block of macros.
  If the reader of this message is not the intended recipient or agent responsible for delivering the message to the intended recipient, then you are hereby notified that any dissemination or copying of this communication is strictly prohibited.  If you have received this electronic transmission in error, please delete it from your system without copying it and notify the sender by reply e-mail  so that our address record can be corrected.  Thank you.
From: Allen Baum @.>
Sent: Tuesday, September 20, 2022 2:23 PM
To: riscv-non-isa/riscv-arch-test @.>
Cc: Marc Karasek @.>; Comment @.>
Subject: Re: [riscv-non-isa/riscv-arch-test] Add changes in arch_test.h file, split macros into 2 files and enable multiple Privilege Modes in Trap Handler (PR #277)
I'm unclear what has been deprecated; do you mean that #warning has been
deprecated in the toolchain?
On Tue, Sep 20, 2022 at 7:12 AM InspireSemi<EMAIL_ADDRESS>wrote:

<EMAIL_ADDRESS>requested changes on this pull request.
In test_macros.h
The #warnings have been commented out. Either we uncomment these or just
remove the this whole section from the file. Lines 768 - 818.
These macros have been deprecated and now is as good a time as any to
remove them.
—
Reply to this email directly, view it on GitHub
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#pullrequestreview-1113906240,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHPXVJUCZERQXXUX5DVEU3DV7HA4RANCNFSM56PYCGHQ
.
You are receiving this because you were mentioned.Message ID:
<EMAIL_ADDRESS>
—
Reply to this email directly, view it on GitHubhttps://github.com/riscv-non-isa/riscv-arch-test/pull/277#issuecomment-1252745250, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AR3S6YZ5ZUGV7NHPMDXHBW3V7H6IVANCNFSM56PYCGHQ.
You are receiving this because you commented.Message ID<EMAIL_ADDRESS>

Ah, those macros. I thought you meant some other macros.
Yes, I think we can remove those.
I'll make sure these are removed when we merge the new trap handler, which
splits these off into test_macros.h
On Tue, Sep 20, 2022 at 11:25 AM InspireSemi @.***>
wrote:

The macros in this section of the header are marked as being deprecated.
Someone commented out the #warning messages that say this macro is
deprecated since 3.0….
So either uncomment the #warning messages and leave the macros in place or
remove the whole block of macros.
 If the
reader of this message is not the intended recipient or agent responsible
for delivering the message to the intended recipient, then you are hereby
notified that any dissemination or copying of this communication is
strictly prohibited. If you have received this electronic transmission in
error, please delete it from your system without copying it and notify the
sender by reply e-mail so that our address record can be corrected. Thank
you.
From: Allen Baum @.>
Sent: Tuesday, September 20, 2022 2:23 PM
To: riscv-non-isa/riscv-arch-test @.>
Cc: Marc Karasek @.>; Comment @.>
Subject: Re: [riscv-non-isa/riscv-arch-test] Add changes in arch_test.h
file, split macros into 2 files and enable multiple Privilege Modes in Trap
Handler (PR #277)
I'm unclear what has been deprecated; do you mean that #warning has been
deprecated in the toolchain?
On Tue, Sep 20, 2022 at 7:12 AM InspireSemi<EMAIL_ADDRESS>wrote:

<EMAIL_ADDRESS>requested changes on this pull
request.
In test_macros.h
The #warnings have been commented out. Either we uncomment these or just
remove the this whole section from the file. Lines 768 - 818.
These macros have been deprecated and now is as good a time as any to
remove them.
—
Reply to this email directly, view it on GitHub
<
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#pullrequestreview-1113906240>,

or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/AHPXVJUCZERQXXUX5DVEU3DV7HA4RANCNFSM56PYCGHQ>

.
You are receiving this because you were mentioned.Message ID:
<EMAIL_ADDRESS>
—
Reply to this email directly, view it on GitHub<
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#issuecomment-1252745250>,
or unsubscribe<
https://github.com/notifications/unsubscribe-auth/AR3S6YZ5ZUGV7NHPMDXHBW3V7H6IVANCNFSM56PYCGHQ>.
You are receiving this because you commented.Message ID:
<EMAIL_ADDRESS>—
Reply to this email directly, view it on GitHub
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#issuecomment-1252748194,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHPXVJQYL56O5UV4I3GVB5LV7H6STANCNFSM56PYCGHQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

Hello,
As sent by @allenjbaum , this final commit is ready to be merged. Allen found and fixed a bunch of bugs related to:

code paths I haven't exercised that have vectored interrupt modes
better indications of when the trap signature area is overrun
ensuring that the trap handler keeps the trampoline 64B aligned (need for CLIC spec)
fixing the handling of when xTVEC was not initialized and also could not be written
removing code that was compensating for Sail bugs
rewriting the save_GPR macro to it now compiles
some miscellaneous optimizations
comment fixes

A changelog entry is required. Please make this a "minor" update and in the Changelog be as detailed as possible in terms of the changes and feature additions that this PR introduces.

A changelog entry is required. Please make this a "minor" update and in the Changelog be as detailed as possible in terms of the changes and feature additions that this PR introduces.

Done.

There are some merge conflicts between the arch_test.h files on the main and the PR. Please pull the recent changes from main and resolve the conflicts correctly.

There are some merge conflicts between the arch_test.h files on the main and the PR. Please pull the recent changes from main and resolve the conflicts correctly.

Done!

The changes on the master are being overwritten. Especially the canary definitions here.

The CANARY error has been resolved. Now all (90) tests of RV32 are passing with the updated definition of CANARY.

Added changes proposed in #294 into this PR.

Were you able to run the tests under rv64i_m/C directory?  This is probably because of a .option norvc in the template macros which is not clubbed with a .option push ... .option pop sequence.  An instance of the error is shown below:
Error: unrecognized opcode `c.addi16sp x2,0x100'

An overflow?Not sure what that means.  The code above (or below, I guess)
is not the correct fix.
All Instances of SREG or FSREG, offset+SIGALIGN(_BR)
should be changed to     FSREG, offset(_BR)
because the CHK_OFFSET(_BR,SIGALIGN,1) pre-increment the offset.
On Thu, Dec 22, 2022 at 9:17 PM S Pawan Kumar @.***>
wrote:

@.**** commented on this pull request.
In riscv-test-suite/env/test_macros.h
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#discussion_r1056044807
:

+/* It stores both an Xreg and an Freg, first adjusting base & offset to	 /
+/ to keep offset < 2048. SIGALIGN is set to the max(FREGWIDTH, REGWIDTH)/
+/ _BR - Base Reg, _R - FReg, _F - Fstatus Xreg				 */
+#define RVTEST_SIGUPD_F(_BR,_R,_F,...)			;\

.if NARG(VA_ARGS) == 1				;\

.set offset,_ARG1(__VA_OPT__(__VA_ARGS__,0))	;\

.endif						;\
.if (offset&(SIGALIGN-1))!=0				;
+/* Throw warnings then modify offset to target */	;\

.warning "Incorrect signature Offset Alignment."	;\

.set offset, offset&(SIGALIGN-1)+SIGALIGN		;\

.endif						;\
CHK_OFFSET(_BR, SIGALIGN, 0)				;\
FSREG _R,offset(_BR)					;\
CHK_OFFSET(_BR, SIGALIGN, 1)				;\
SREG _F,offset+SIGALIGN(_BR)				;\

The latest macros on the main have been fixed(link
https://github.com/riscv-non-isa/riscv-arch-test/blob/e302d3bab41a46ec388691b1d961aa09fe2a4bc4/riscv-test-suite/env/arch_test.h#L661).
By introducing the offset+SIGALIGN the tests do not compile because an
overflow occurs after a few instances of the macro in the test,
specifically at line 106. In its current form, it also results in the flag
values of preceding test instance getting overwritten by the result of the
next instance.
—
Reply to this email directly, view it on GitHub
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#discussion_r1056044807,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHPXVJRNWILXUQCILOPA3FDWOUYV5ANCNFSM56PYCGHQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

More correctly, in case this is misinterpreted:
All Instances of            SREG/FSREG, offset+SIGALIGN(_BR)
should be changed to  SREG/FSREG, offset(_BR)
On Fri, Dec 23, 2022 at 12:14 AM Allen Baum @.***>
wrote:

An overflow?Not sure what that means.  The code above (or below, I guess)
is not the correct fix.
All Instances of SREG or FSREG, offset+SIGALIGN(_BR)
should be changed to     FSREG, offset(_BR)
because the CHK_OFFSET(_BR,SIGALIGN,1) pre-increment the offset.
On Thu, Dec 22, 2022 at 9:17 PM S Pawan Kumar @.***>
wrote:

@.**** commented on this pull request.
In riscv-test-suite/env/test_macros.h
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#discussion_r1056044807
:

+/* It stores both an Xreg and an Freg, first adjusting base & offset to	 /
+/ to keep offset < 2048. SIGALIGN is set to the max(FREGWIDTH, REGWIDTH)/
+/ _BR - Base Reg, _R - FReg, _F - Fstatus Xreg				 */
+#define RVTEST_SIGUPD_F(_BR,_R,_F,...)			;\

.if NARG(VA_ARGS) == 1				;\

.set offset,_ARG1(__VA_OPT__(__VA_ARGS__,0))	;\

.endif						;\
.if (offset&(SIGALIGN-1))!=0				;
+/* Throw warnings then modify offset to target */	;\

.warning "Incorrect signature Offset Alignment."	;\

.set offset, offset&(SIGALIGN-1)+SIGALIGN		;\

.endif						;\
CHK_OFFSET(_BR, SIGALIGN, 0)				;\
FSREG _R,offset(_BR)					;\
CHK_OFFSET(_BR, SIGALIGN, 1)				;\
SREG _F,offset+SIGALIGN(_BR)				;\

The latest macros on the main have been fixed(link
https://github.com/riscv-non-isa/riscv-arch-test/blob/e302d3bab41a46ec388691b1d961aa09fe2a4bc4/riscv-test-suite/env/arch_test.h#L661).
By introducing the offset+SIGALIGN the tests do not compile because an
overflow occurs after a few instances of the macro in the test,
specifically at line 106. In its current form, it also results in the flag
values of preceding test instance getting overwritten by the result of the
next instance.
—
Reply to this email directly, view it on GitHub
https://github.com/riscv-non-isa/riscv-arch-test/pull/277#discussion_r1056044807,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHPXVJRNWILXUQCILOPA3FDWOUYV5ANCNFSM56PYCGHQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


[Bug] Blank Page on Sagemaker Workspace Connect

Describe the bug
When connecting to Sagemaker workspaces, there is an intermittent issue where a blank browser launches instead of Sagemaker.  The issue presents for workspaces that are newly created as well as for workspaces that were already created, but were stopped and are being restarted.
To Reproduce
Steps to reproduce the behavior:
STEP 1: Login as an Admin
STEP 2: Create a workspace (SageMaker)
STEP 3: Start the Workspace
STEP 5: Click "Connect"
STEP 6: A new blank web browser tab opens
STEP 7: Click "Connect" again, another blank web browser tab opens
User receives a "Something Went Wrong" general error in SWB at Step 6
In the client logs for the browser, there is also this error noted:
"name": "x-cache",
"value": "Error from cloudfront"
Expected behavior
Sagemaker workspace launch in browser
Screenshots
If applicable, add screenshots to help explain your problem.
Versions (please complete the following information):

Release Version installed 3.0.0

Additional context
User has cleared cache and it solved the issue, but for one of her employees clearing the cache did not solve the issue.
The issue is experienced approximately once a week. Sometimes clearing cache solves the issue. Other times going to incognito, and it does not solve the issue.

The relevant code snippet for this issue is here.
Let's try to track down what the issue is. Are we unable to get the Sagemaker URL or is the browser failing to


keyForAttribute declaration appears to be incorrect

The Serializer.keyForAttribute property type is declared as (attribute: string) => string | caseOptions. Providing a function that returns one of these caseOptions prevents correct serialization of any attributes. If the property is assigned to a caseOptions literal string instead serialization works. Should the type be declared as ((attribute: string) => string) | caseOptions instead (note the parens)?

Please add the brackets, the are necessary for the ts compiler.

(attribute: string) => string | caseOption => ((attribute: string) => string) | caseOption

Done 👍
Thanks @snortblt and @christianengledercadt

Released v2.1.2 on npmjs.
https://www.npmjs.com/package/ts-jsonapi


Remove duplicate

Accidentally saw this in your video and couldn't ignore it :smile:

Now you have pointed it out, I can't unsee it!


How to get gfx running?

Hello there,
the instructions (Windows 10, Visual Studio Code, Go), for installing sdl, img, ttf, mix are all the same: download the devel package, unpack the package, copy x86_64-w64-mingw32, go get -v github.com/veandco/go-sdl2/XXX, hit run -> it works fine.
sadly this wont work with the gfx package because it doesnt contain any folder named x86_64-w64-mingw32. It is some visual studio (not code) project folder that must be build in order to get what you want.
It would be great to have some seperate install instructions for the gfx package or some nice guy that builds the project and links the copy & pastable folder so everyone that is not a compile master can use sdl_gfx.

Hi @MarcusTU, thanks for pointing that out. I'd guess that for the GCC version you can run ./configure; make and copy the library into the same directory as other libraries.
I'll have a look at this and update the guide when I have the time (probably this weekend).

So I had to edit the configure script at line 13142 to make sure they escape spaces in $SDL_CONFIG. Then I edit sdl2-config to use the correct $prefix. But looking at config.log, I found out that you need autoconf, autoheader, and automake in the end which aren't provided by mingw-w64.
I will try with MSYS2 if it can be used to successfully compile the project since it does provides those autotools.

I found a nice person who shares his SDL2_gfx.dll which I confirmed to be working with go-sdl2/gfx! Here's the link: https://github.com/Vladar4/sdl2_nim/blob/master/LINKS.md
The header files of SDL2_gfx (.h files) will need to be copied into include folder of your MinGW. They can be obtained from the source code here: http://www.ferzkopp.net/wordpress/2016/01/02/sdl_gfx-sdl2_gfx/

Thank you very much :)
This method is not complicated and works fine.


operators/bpf: Fix perf ring buffer fallback logic

Logic to fallback to a perf ring buffer when ring buffer is not supported
was introduced in ba2bfca2aa77 ("run: Add functions to use perf or ring
buffer."), however it wasn't ported in d28c76d0ef85
("pkg/operators/ebpf: ebpf layer operator").
Testing
option 0
Grab a machine with a kernel 5.4 and try to run some gadgets
option 1
Merge this on top of #3631
Apply the following diff (until we fix other gadgets on this PR)
$ git diff
diff --git a/gadgets/Makefile b/gadgets/Makefile
index 6699b7403..a73faf253 100644
--- a/gadgets/Makefile
+++ b/gadgets/Makefile
@@ -126,7 +126,7 @@ pull:
        GADGET_REPOSITORY=$(GADGET_REPOSITORY) \
        VIMTO=$(VIMTO) \
        sudo -E $(VIMTO) -kernel $(KERNEL_REPOSITORY):$* \
-       -memory 4096M -- go test -v ./.../$(UNIT_TEST_DIR)/...
+       -memory 4096M -- go test -v ./trace_open/$(UNIT_TEST_DIR)/...

 .PHONY:
 test-unit: build

run the test
$ make -C gadgets/ IG_VERIFY_IMAGE=false GADGET_TAG=latest 5.4/test-kernel -o pull

The patch "operators/bpf: Fix perf ring buffer fallback logic" looks good to me.
The second patch has a TODO.

The patch "operators/bpf: Fix perf ring buffer fallback logic" looks good to me.
The second patch has a TODO.

Right, that seems also to be fixed by #3542. So I'll wait for that one.