Picking and cleaning datasets

Background
For this project, we need to use the chronic disease dataset and augment it with another dataset. The two datasets must be somewhat related for them to be used together. This task will involve choosing the data set to augment the chronic disease data set and writing scripts to clean both data sets. This work will be completed in a branch called "eda"
Tasks

[x] EDA of chronic disease data to see its features
[x] selection of augmenting data so there is some feature overlap
[ ] data cleaning to make overlap more evident and data more usable

There are 34 separate fields in the chronic data set:

YearStart
YearEnd
LocationAbbr
LocationDesc
DataSource
Topic
Question
Response
DataValueUnit
DataValueType
DataValue
DataValueAlt
DataValueFootnoteSymbol
DataValueFootnote
LowConfidenceLimit
HighConfidenceLimit
StratificationCategory1
Stratification1
StratificationCategory2
Stratification2
StratificationCategory3
Stratification3
Geolocation
LocationID
TopicID
QuestionID
ResponseID
DataValueTypeID
StratificationCategoryID1
StratificationID1
StratificationCategoryID2
StratificationID2
StratificationCategoryID3
StratificationID3

Many of them have vague meanings, but it seems like each entry is a question asked to a specific individual like "Diabetes among adults". The features whos meaning we would need to identify in this case would be anything to do with stratification or datavalue

When displaying the entries that do not have NAN in the "response" column, an empty data frame is returned. There are 109 different questions asked. The most frequent question is "Binge drinking frequency among adults who binge drink" at 5720 instances. Many answers seem like they should be true or false, such as "Food insecure in the past 12 months among households" or "No broadband internet subscription among households" or it could be the number of yes answers from an aggregated survey. Because of the year start and year end features, it might be that the answer is the number of incidents in that given time period

YearStart
YearEnd
LocationAbbr
DataValueUnit
DataValueType
DataValue

25076
2019
2021
AK
%
Crude Prevalence
9.5

18425
2019
2021
AL
%
Crude Prevalence
13.1

30716
2019
2021
AR
%
Crude Prevalence
15

18702
2019
2021
AZ
%
Crude Prevalence
10.1

26942
2019
2021
CA
%
Crude Prevalence
9.6

30776
2019
2021
CO
%
Crude Prevalence
10.5

28062
2019
2021
CT
%
Crude Prevalence
9.6

29477
2019
2021
DC
%
Crude Prevalence
9

38811
2019
2021
DE
%
Crude Prevalence
11.2

31599
2019
2021
FL
%
Crude Prevalence
9.9

32639
2019
2021
GA
%
Crude Prevalence
9.9

31072
2019
2021
GU
%
Crude Prevalence
nan

51521
2019
2021
HI
%
Crude Prevalence
9.1

45921
2019
2021
IA
%
Crude Prevalence
7

44209
2019
2021
ID
%
Crude Prevalence
9.8

43543
2019
2021
IL
%
Crude Prevalence
9.4

50978
2019
2021
IN
%
Crude Prevalence
9.7

46560
2019
2021
KS
%
Crude Prevalence
10.2

40045
2019
2021
KY
%
Crude Prevalence
12.3

42348
2019
2021
LA
%
Crude Prevalence
14.5

65876
2019
2021
MA
%
Crude Prevalence
8.4

60423
2019
2021
MD
%
Crude Prevalence
8.7

59576
2019
2021
ME
%
Crude Prevalence
9.5

65847
2019
2021
MI
%
Crude Prevalence
11.4

58773
2019
2021
MN
%
Crude Prevalence
7.4

57794
2019
2021
MO
%
Crude Prevalence
12

65500
2019
2021
MS
%
Crude Prevalence
15.3

53444
2019
2021
MT
%
Crude Prevalence
10.4

66918
2019
2021
NC
%
Crude Prevalence
10.9

66152
2019
2021
ND
%
Crude Prevalence
7.7

77580
2019
2021
NE
%
Crude Prevalence
10.6

77195
2019
2021
NH
%
Crude Prevalence
5.4

67001
2019
2021
NJ
%
Crude Prevalence
8.3

70966
2019
2021
NM
%
Crude Prevalence
11.5

75368
2019
2021
NV
%
Crude Prevalence
10.2

70292
2019
2021
NY
%
Crude Prevalence
10.3

82520
2019
2021
OH
%
Crude Prevalence
10.8

85652
2019
2021
OK
%
Crude Prevalence
13.8

87941
2019
2021
OR
%
Crude Prevalence
10.3

83502
2019
2021
PA
%
Crude Prevalence
9.2

89665
2019
2021
PR
%
Crude Prevalence
nan

84235
2019
2021
RI
%
Crude Prevalence
8.4

91723
2019
2021
SC
%
Crude Prevalence
12.6

88092
2019
2021
SD
%
Crude Prevalence
8.7

102851
2019
2021
TN
%
Crude Prevalence
11.2

92585
2019
2021
TX
%
Crude Prevalence
13.7

99930
2019
2021
US
%
Crude Prevalence
10.4

104635
2019
2021
UT
%
Crude Prevalence
11.2

94758
2019
2021
VA
%
Crude Prevalence
7.8

104227
2019
2021
VI
%
Crude Prevalence
nan

93287
2019
2021
VT
%
Crude Prevalence
7.9

98754
2019
2021
WA
%
Crude Prevalence
7.9

105791
2019
2021
WI
%
Crude Prevalence
9.9

117168
2019
2021
WV
%
Crude Prevalence
14

109942
2019
2021
WY
%
Crude Prevalence
11.2

Here are some of the other features

LowConfidenceLimit
Stratification1
StratificationCategory1
Geolocation
LocationID
DataValue

18425
9.4
Overall
Overall
POINT (-86.63186076199969 32.84057112200048)
1
13.1

18702
7.1
Overall
Overall
POINT (-111.76381127699972 34.865970280000454)
4
10.1

25076
6.3
Overall
Overall
POINT (-147.72205903599973 64.84507995700051)
2
9.5

26942
8.6
Overall
Overall
POINT (-120.99999953799971 37.63864012300047)
6
9.6

28062
6.2
Overall
Overall
POINT (-72.64984095199964 41.56266102000046)
9
9.6

29477
6.8
Overall
Overall
POINT (-77.036871 38.907192)
11
9

30716
11.6
Overall
Overall
POINT (-92.27449074299966 34.74865012400045)
5
15

30776
7.2
Overall
Overall
POINT (-106.13361092099967 38.843840757000464)
8
10.5

31072
nan
Overall
Overall
POINT (144.793731 13.444304)
66
nan

31599
8.2
Overall
Overall
POINT (-81.92896053899966 28.932040377000476)
12
9.9

32639
7.5
Overall
Overall
POINT (-83.62758034599966 32.83968109300048)
13
9.9

38811
7.5
Overall
Overall
POINT (-75.57774116799965 39.008830667000495)
10
11.2

40045
9.2
Overall
Overall
POINT (-84.77497104799966 37.645970271000465)
21
12.3

42348
11.7
Overall
Overall
POINT (-92.44568007099969 31.31266064400046)
22
14.5

43543
7.4
Overall
Overall
POINT (-88.99771017799969 40.48501028300046)
17
9.4

44209
7.4
Overall
Overall
POINT (-114.3637300419997 43.682630005000476)
16
9.8

45921
4.2
Overall
Overall
POINT (-93.81649055599968 42.46940091300047)
19
7

46560
7.7
Overall
Overall
POINT (-98.20078122699965 38.34774030000045)
20
10.2

50978
7.4
Overall
Overall
POINT (-86.14996019399968 39.766910452000445)
18
9.7

51521
6.4
Overall
Overall
POINT (-157.85774940299973 21.304850435000446)
15
9.1

53444
7.8
Overall
Overall
POINT (-109.42442064499971 47.06652897200047)
30
10.4

57794
8.7
Overall
Overall
POINT (-92.56630005299968 38.635790776000476)
29
12

58773
4.8
Overall
Overall
POINT (-94.79420050299967 46.35564873600049)
27
7.4

59576
6.6
Overall
Overall
POINT (-68.98503133599962 45.254228894000505)
23
9.5

60423
6.1
Overall
Overall
POINT (-76.60926011099963 39.29058096400047)
24
8.7

65500
11.7
Overall
Overall
POINT (-89.53803082499968 32.745510099000455)
28
15.3

65847
8.7
Overall
Overall
POINT (-84.71439026999968 44.6613195430005)
26
11.4

65876
6.3
Overall
Overall
POINT (-72.08269067499964 42.27687047000046)
25
8.4

66152
5.5
Overall
Overall
POINT (-100.11842104899966 47.47531977900047)
38
7.7

66918
8.3
Overall
Overall
POINT (-79.15925046299964 35.466220975000454)
37
10.9

67001
6.1
Overall
Overall
POINT (-74.27369128799967 40.13057004800049)
34
8.3

70292
8.5
Overall
Overall
POINT (-75.54397042699964 42.82700103200045)
36
10.3

70966
7.1
Overall
Overall
POINT (-106.24058098499967 34.52088095200048)
35
11.5

75368
7.7
Overall
Overall
POINT (-117.07184056399967 39.493240390000494)
32
10.2

77195
3.5
Overall
Overall
POINT (-71.50036091999965 43.65595011300047)
33
5.4

77580
7.8
Overall
Overall
POINT (-99.36572062299967 41.6410409880005)
31
10.6

82520
8.6
Overall
Overall
POINT (-82.40426005599966 40.06021014100048)
39
10.8

83502
7.2
Overall
Overall
POINT (-77.86070029399963 40.79373015200048)
42
9.2

84235
5.4
Overall
Overall
POINT (-71.52247031399963 41.70828019300046)
44
8.4

85652
10.2
Overall
Overall
POINT (-97.52107021399968 35.47203135600046)
40
13.8

87941
7.7
Overall
Overall
POINT (-120.15503132599969 44.56744942400047)
41
10.3

88092
6.2
Overall
Overall
POINT (-100.3735306369997 44.353130053000484)
46
8.7

89665
nan
Overall
Overall
POINT (-66.590149 18.220833)
72
nan

91723
9.9
Overall
Overall
POINT (-81.04537120699968 33.998821303000454)
45
12.6

92585
11.9
Overall
Overall
POINT (-99.42677020599967 31.827240407000488)
48
13.7

93287
5.8
Overall
Overall
POINT (-72.51764079099962 43.62538123900049)
50
7.9

94758
5.7
Overall
Overall
POINT (-78.45789046299967 37.54268067400045)
51
7.8

98754
6.1
Overall
Overall
POINT (-120.47001078999972 47.52227862900048)
53
7.9

99930
10.1
Overall
Overall
nan
59
10.4

102851
9
Overall
Overall
POINT (-85.77449091399967 35.68094058000048)
47
11.2

104227
nan
Overall
Overall
POINT (-64.896335 18.335765)
78
nan

104635
8.5
Overall
Overall
POINT (-111.58713063499971 39.360700171000474)
49
11.2

105791
6.9
Overall
Overall
POINT (-89.81637074199966 44.39319117400049)
55
9.9

109942
8.6
Overall
Overall
POINT (-108.10983035299967 43.23554134300048)
56
11.2

117168
8.9
Overall
Overall
POINT (-80.71264013499967 38.66551020200046)
54
14

It looks like the stratifications are ways to group the data, like by age or sex

| Question
|    0 |
|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----:|
| Binge drinking frequency among adults who binge drink
| 5720 |
| Binge drinking intensity among adults who binge drink
| 5680 |
| Diabetic ketoacidosis mortality among all people, underlying or contributing cause
| 5616 |
| Diseases of the heart mortality among all people, underlying cause
| 5616 |
| Coronary heart disease mortality among all people, underlying cause
| 5616 |
| Cerebrovascular disease (stroke) mortality among all people, underlying cause
| 5616 |
| Diabetes mortality among all people, underlying or contributing cause
| 5616 |
| Chronic liver disease mortality among all people, underlying cause
| 5616 |
| Asthma mortality among all people, underlying cause
| 5616 |
| Chronic obstructive pulmonary disease mortality among adults aged 45 years and older, underlying or contributing cause
| 5304 |
| Chronic obstructive pulmonary disease mortality among adults aged 45 years and older, underlying cause
| 5304 |
| Diabetes among adults
| 5060 |
| Chronic obstructive pulmonary disease among adults
| 5060 |
| Routine checkup within the past year among adults
| 5060 |
| Depression among adults
| 5060 |
| Recent activity limitation among adults
| 5060 |
| Current smoking among adults with chronic obstructive pulmonary disease
| 5060 |
| Current cigarette smoking among adults
| 5060 |
| 2 or more chronic conditions among adults
| 5060 |
| Fair or poor self-rated health status among adults
| 5060 |
| Frequent mental distress among adults
| 5060 |
| Frequent physical distress among adults
| 5060 |
| Binge drinking prevalence among adults
| 5060 |
| Average recent physically unhealthy days among adults
| 5060 |
| Average mentally unhealthy days among adults
| 5060 |
| Obesity among adults
| 5060 |
| No leisure-time physical activity among adults
| 5060 |
| Influenza vaccination among adults
| 5060 |
| Adults with any disability
| 5060 |
| Quit attempts in the past year among adult current smokers
| 5060 |
| Current asthma among adults
| 4895 |
| Influenza vaccination among adults 18–64 who are at increased risk
| 4840 |
| Lack of health insurance among adults aged 18-64 years
| 4840 |
| Pneumococcal vaccination among adults aged 18–64 years who are at increased risk
| 4840 |
| Pneumococcal vaccination among adults aged 65 years and older
| 4400 |
| Arthritis among adults
| 3795 |
| Physical inactivity among adults with arthritis
| 3795 |
| Hospitalization for heart failure as principal diagnosis, Medicare-beneficiaries aged 65 years and older
| 3744 |
| Hospitalization for chronic obstructive pulmonary disease as principal diagnosis, Medicare-beneficiaries aged 65 years and older
| 3744 |
| Hospitalization for chronic obstructive pulmonary disease as any diagnosis, Medicare-beneficiaries aged 65 years and older
| 3744 |
| Invasive cancer (all sites combined), incidence
| 2544 |
| Consumed vegetables less than one time daily among adults
| 2530 |
| Visited dentist or dental clinic in the past year among adults
| 2530 |
| Taking medicine to control high blood pressure among adults with high blood pressure
| 2530 |
| High cholesterol among adults who have been screened
| 2530 |
| Have taken an educational class to learn how to manage arthritis symptoms among adults with arthritis
| 2530 |
| Taking medicine for high cholesterol among adults
| 2530 |
| Short sleep duration among adults
| 2530 |
| Consumed fruit less than one time daily among adults
| 2530 |
| Received health care provider counseling for physical activity among adults with arthritis
| 2530 |
| Provided care for a friend or family member in the past month among adults
| 2530 |
| Provided care for someone with dementia or other cognitive impairment in the past month among adults
| 2530 |
| High blood pressure among adults
| 2527 |
| Breast cancer mortality among all females, underlying cause
| 2496 |
| Cervical cancer mortality among all females, underlying cause
| 2496 |
| Colon and rectum (colorectal) cancer mortality among all people, underlying cause
| 2496 |
| Lung and bronchial cancer mortality among all people, underlying cause
| 2496 |
| Prostate cancer mortality among all males, underlying cause
| 2496 |
| Invasive cancer (all sites combined) mortality among all people, underlying cause
| 2496 |
| Subjective cognitive decline among adults aged 45 years and older
| 2422 |
| Discussed symptoms of subjective cognitive decline with a health care professional among adults aged 45 years and older with subjective cognitive decline
| 2422 |
| No teeth lost among adults aged 18-64 years
| 2420 |
| Severe joint pain among adults with arthritis
| 2420 |
| Work limitation due to arthritis among adults aged 18-64 years with arthritis
| 2420 |
| Activity limitation due to arthritis among adults with arthritis
| 2420 |
| All teeth lost among adults aged 65 years and older
| 2200 |
| Colorectal cancer screening among adults aged 45-75 years
| 2200 |
| Six or more teeth lost among adults aged 65 years and older
| 2200 |
| Mammography use among women aged 50-74 years
| 1758 |
| Binge drinking prevalence among high school students
| 1540 |
| Current electronic vapor product use among high school students
| 1540 |
| Consumed regular soda at least one time daily among high school students
| 1540 |
| Current smokeless tobacco use among high school students
| 1540 |
| Consumed fruit less than one time daily among high school students
| 1540 |
| Current tobacco use of any tobacco product among high school students
| 1540 |
| Short sleep duration among high school students
| 1540 |
| Obesity among high school students
| 1540 |
| Consumed vegetables less than one time daily among high school students
| 1540 |
| Met aerobic physical activity guideline among high school students
| 1540 |
| Alcohol use among high school students
| 1540 |
| Receipt of evidence-based preventive dental services in the past 12 months among children and adolescents aged 1-17 years
| 1430 |
| Visited dentist or other oral health care provider in the past 12 months among children and adolescents aged 1-17 years
| 1430 |
| Unable to pay mortgage, rent, or utility bills in the past 12 months among adults
| 1265 |
| Met aerobic physical activity guideline for substantial health benefits, adults
| 1265 |
| Lack of social and emotional support needed among adults
| 1265 |
| Lack of reliable transportation in the past 12 months among adults
| 1265 |
| Short sleep duration among children aged 4 months to 14 years
| 1248 |
| Children and adolescents aged 6-13 years meeting aerobic physical activity guideline
| 1248 |
| Unemployment rate among people 16 years and older in the labor force
| 1040 |
| Living below 150% of the poverty threshold among all people
| 1040 |
| High school completion among adults aged 18-24
| 1040 |
| Cigarette smoking during pregnancy among women with a recent live birth
| 1026 |
| Preventive dental care in the 12 months before pregnancy among women with a recent live birth
| 1026 |
| Postpartum depressive symptoms among women with a recent live birth
| 1026 |
| Postpartum checkup among women with a recent live birth
| 1026 |
| Gestational diabetes among women with a recent live birth
| 1026 |
| Health insurance coverage after pregnancy among women with a recent live birth
| 1026 |
| Health insurance coverage in the month before pregnancy among women with a recent live birth
| 1026 |
| Cervical cancer screening among women aged 21-65 years
|  880 |
| Current poor mental health among high school students
|  770 |
| Obesity among WIC children aged 2 to 4 years
|  432 |
| Life expectancy at birth
|  312 |
| Proportion of the population protected by a comprehensive smoke-free policy prohibiting smoking in all indoor areas of workplaces and public places, including restaurants and bars
|  165 |
| Per capita alcohol consumption among people aged 14 years and older
|  165 |
| Infants who were breastfed at 12 months
|  122 |
| Infants who were exclusively breastfed through 6 months
|  122 |
| No broadband internet subscription among households
|  104 |
| Incidence of treated end-stage kidney disease
|  104 |
| Food insecure in the past 12 months among households
|   55 |

I think there are a lot of candidates for augmenting datasets

food prices for nutrition: comparing nutrition access to disease **
climate and development report: looking at how air quality and infrastructure affects health outcomes
sustainable mobility for all: how transportation affects health
human capital index: looks at factors that improve health and education **
resilience and equity
adjusted net savings
health nutrition and population statistics **
millennium development goals

I think I will incorporate the data indicated with asterisks because it seems the most related if not overlappign

I also added some data from 538 because they have data on the state-level. These include:

district urbanization
state urbanization
redlining
governers problems emphasized per state

Here is a table of all the features from each incorporated dataset. To connect the datasets we will need to identify keys/features that can be related. Below is a list of how each dataset can be connected with the main one. Also, there is a list of how some features will be changed.

main
urbanization_dist
food_prices
nutrition
human_capital
metro_grade
sots_index
sots_words
urbanization_state

0
YearStart
stcd
Classification Name
Country Name
Country Name
metro_area
state
phrase
state

1
YearEnd
state
Classification Code
Country Code
Country Code
holc_grade
governor
category
urbanindex

2
LocationAbbr
cd
Country Name
Series Name
Series Name
white_pop
party
d_speeches

3
LocationDesc
pvi_22
Country Code
Series Code
Series Code
black_pop
filename
r_speeches

4
DataSource
urbanindex
Series Name
1960 [YR1960]
2010 [YR2010]
hisp_pop
url
total

5
Topic
rural
Series Code
1961 [YR1961]
2011 [YR2011]
asian_pop

percent_of_d_speeches

6
Question
exurban
2017 [YR2017]
1962 [YR1962]
2012 [YR2012]
other_pop

percent_of_r_speeches

7
Response
suburban
2018 [YR2018]
1963 [YR1963]
2013 [YR2013]
total_pop

chi2

8
DataValueUnit
urban
2019 [YR2019]
1964 [YR1964]
2014 [YR2014]
pct_white

pval

9
DataValueType
grouping
2020 [YR2020]
1965 [YR1965]
2015 [YR2015]
pct_black

10
DataValue

2021 [YR2021]
1966 [YR1966]
2016 [YR2016]
pct_hisp

11
DataValueAlt

1967 [YR1967]
2017 [YR2017]
pct_asian

12
DataValueFootnoteSymbol

1968 [YR1968]
2018 [YR2018]
pct_other

13
DataValueFootnote

1969 [YR1969]
2019 [YR2019]
lq_white

14
LowConfidenceLimit

1970 [YR1970]
2020 [YR2020]
lq_black

15
HighConfidenceLimit

1971 [YR1971]

lq_hisp

16
StratificationCategory1

1972 [YR1972]

lq_asian

17
Stratification1

1973 [YR1973]

lq_other

18
StratificationCategory2

1974 [YR1974]

surr_area_white_pop

19
Stratification2

1975 [YR1975]

surr_area_black_pop

20
StratificationCategory3

1976 [YR1976]

surr_area_hisp_pop

21
Stratification3

1977 [YR1977]

surr_area_asian_pop

22
Geolocation

1978 [YR1978]

surr_area_other_pop

23
LocationID

1979 [YR1979]

surr_area_pct_white

24
TopicID

1980 [YR1980]

surr_area_pct_black

25
QuestionID

1981 [YR1981]

surr_area_pct_hisp

26
ResponseID

1982 [YR1982]

surr_area_pct_asian

27
DataValueTypeID

1983 [YR1983]

surr_area_pct_other

28
StratificationCategoryID1

1984 [YR1984]

29
StratificationID1

1985 [YR1985]

30
StratificationCategoryID2

1986 [YR1986]

31
StratificationID2

1987 [YR1987]

32
StratificationCategoryID3

1988 [YR1988]

33
StratificationID3

1989 [YR1989]

34

1990 [YR1990]

35

1991 [YR1991]

36

1992 [YR1992]

37

1993 [YR1993]

38

1994 [YR1994]

39

1995 [YR1995]

40

1996 [YR1996]

41

1997 [YR1997]

42

1998 [YR1998]

43

1999 [YR1999]

44

2000 [YR2000]

45

2001 [YR2001]

46

2002 [YR2002]

47

2003 [YR2003]

48

2004 [YR2004]

49

2005 [YR2005]

50

2006 [YR2006]

51

2007 [YR2007]

52

2008 [YR2008]

53

2009 [YR2009]

54

2010 [YR2010]

55

2011 [YR2011]

56

2012 [YR2012]

57

2013 [YR2013]

58

2014 [YR2014]

59

2015 [YR2015]

60

2016 [YR2016]

61

2017 [YR2017]

62

2018 [YR2018]

63

2019 [YR2019]

64

2020 [YR2020]

65

2021 [YR2021]

66

2022 [YR2022]

I added a file from colab that is looking at the states scoring the "worst" in SDOH categories vs life expectancy (sorry I still don't know how to make files like yours).
Also, what do you mean by features? Is that just the column you are merging on?
Each SDOH is looking at five worst states for each category, then counting the number of times they are the worst, then comparing that with 10 lowest states for life expectancy (my thought worst SDOH=lower life exectancy) which is kind of true, but not the not appealing graph I've ever seen.
I think the SDOH avenue has a lot of other potential correlations we can make though, like avg income per household, maternal mortality rates, education performance. Theres also a few questions on insurance covered before/after birth and we could look at that vs maternal mortality rates. Maybe political leanings if we can find a good survey. Let me know what you think! I added a life expectancy to the data set.

@Aerlenbeck

Also, what do you mean by features? Is that just the column you are merging on?

yea i think for the most part a feature is the same as a column
Yea i think there are a lot of good analyses that can come from that. Where did you get the life expectancy data from? did you get a chance to look at any of the other data sets?

@jacob-umich
Life expectancy data is from CDC, not one of her sources for data but I think it should be fine since they are reliable.
I see the datasets and how you plan to link them but I don't see the correlations you're trying to make, I'm not sure if this is in one of the txt files you uploaded here?
Is urbanization just giving a number value where higher is more urbanization per state? I also think we could incorporate that with SDOH (more urbanization better SDOH) and or AVG state income (more urbanization, higher average income?). Maybe we could find some traffic safety data too (more urbanization more car deaths?)
Food prices and nutrition are hard because they're both per year but we could see are food prices trending up? When food prices go up does nutrition go down?
I don't understand the data for metro_grade and redlining, what do these columns mean?

@Aerlenbeck  I mainly explained the links in the comments above. I guess it would be simpler to just look at life expectancy. Lets just go with that. If we feel like we are running out coorelations to make, we can bring in the other data sets.

I don't understand the data for metro_grade and redlining, what do these columns mean?

metro_grade measures a degree of segregation from redlining practices https://projects.fivethirtyeight.com/redlining/

@jacob-umich I see your correlations now. I think those would all work well too. We need 15 correlations so I think we'll have enough space for everything.

@tsivitse yea I finished cleaning the main dataset. I think we decided to just use ths augmenting data set that @Aerlenbeck found. That needs to be cleaned I think. You can follow what I did for cleaning that. Then we just need to make some plots

@tsivitse @jacob-umich Yea I added a life expectancy dataset to the data folder on google drive, but I also cleaned that data in the .ipynb file (should be towards the bottom) I added to the EDA section. It's mostly just swapping state codes for their full name so the table can be merged with the CDI dataset.

@tsivitse We need more diverse plot types, if you're looking for something to think about. I am going to make bar charts (12) with a drop down menu for each SDOH question from the CDI dataset for best 5/worst 5 states for each question, but those will all be one plot type. I'm also planning to do a chloropleth plot for the life expectancy.
With the food prices/nutrition data we could probably do line/scatter plots, but I think we need 5 plots types, so at least 1 maybe 2 more if you see anything that stands out to you.

how do I access the cleaned data? I'm trying to read in temp.md into main.py but main.py seems to have an invalid syntax error, does anyone know anything about this? Also want to confirm temp.md is the cleaned file we should be working with?

Confirming if the EDA I'm working on should be in the SDOH file with your cleaned data @Aerlenbeck?

Yea you can add to that file, if I'm understanding your question correctly @tsivitse

I have compiled all the clean data into one database. The server can use this database instead of pulling from two separate ones. This task is done


Fix tellraw issue when console or datapacks use tellraw without slash

The Fabric Discord Link mod has the ability to enable/disable tellraw commands form propagating to the Discord; however, it doesn't currently account for the console or other sources such as datapacks calling tellraw without the preceeding slash. This commit simply adds an additional check to the command check to look for tellraw without the slash (in addition to the check with the slash). There is likely a more efficient way to write this code, but readability of the current solution is high.

Fixes #185


Fix filtering by unbounded numeric literal, = <numeric>::numeric

Summary of the changes / Why this improves CrateDB
Initial attempt https://github.com/crate/crate/pull/16829 with narrowed scope.
Fixes:
cr> create table t (a numeric(4,2));
CREATE OK, 1 row affected (1.642 sec)
cr> insert into t values (1.11);
INSERT OK, 1 row affected (2.536 sec)
cr> insert into t values (11.11);
INSERT OK, 1 row affected (1.567 sec)
cr> select * from t where a = 1.111::numeric;  -- 1.111 has scale = 3 so it can be turned into a MatchNoDocsQuery
+-------+
|     a |
+-------+
| 11.11 |
+-------+
SELECT 1 row in set (1.843 sec)
cr> select * from t where a =111::numeric;  -- the unscaled value of 111 was 111 which was identical to the unscaled value of 1.11.
+------+
|    a |
+------+
| 1.11 |
+------+
SELECT 1 row in set (0.004 sec)

Please see the test cases added for more.
https://github.com/crate/crate/blob/a1f77cfb706fb3fb470a58e5d0dad58eb679ade6/docs/appendices/release-notes/5.9.1.rst?plain=1#L118-L120 is enough that we do not need an extra release entry.
Checklist

[x] Added an entry in the latest docs/appendices/release-notes/<x.y.0>.rst for user facing changes
[x] Updated documentation & sql_features table for user facing changes
[x] Touched code is covered by tests
[x] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in the latest docs/appendices/release-notes/<x.y.0>.rst
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

Hi @seut, @matriv, I tried to narrow the scope from my initial PR, please take a look. Thank you

Looks good, but missing a changes entry.

Apologies for being a bit late, isn't this a valid request ^^ ?

thank you @jeeminso!
I tried it out as well, sorry for being a bit late with the review.


[#3183] Adjust rollDamage to return merged roll, rather than first

In 3.0.0 we modified Item5e#rollDamage to support separate rolls for multiple damage parts and added returnMultiple option for backward compatibility. The issue with that initial implementation is that if returnMultiple is false, it returns the first roll made rather than matching previously behavior of having all the damage parts merged into a single roll.
This change alters the bahavior of damageRoll if returnMultiple is false and there are more than one roll to create a new merged roll from the evaulated rolls and return that.
The separate rolls are still stored in the chat message and used by the system, but the combined rolls are now available to modules that rely on the dnd5e.rollDamage hook or call Item5e#rollDamage directly.

Shouldn't the returned roll be a DamageRoll rather than a Roll?
The options.damageType will end up being that of the first roll which means you can't recover the damage type for each of the terms in the merged roll. I realise there is no solution for that with the current damageRoll, but perhaps consideration for each term optionally having a damageType to allow for that.

Shouldn't the returned roll be a DamageRoll rather than a Roll?

Good catch!

The options.damageType will end up being that of the first roll which means you can't recover the damage type for each of the terms in the merged roll. I realise there is no solution for that with the current damageRoll, but perhaps consideration for each term optionally having a damageType to allow for that.

Since damage type is a new option that wasn't available with the legacy single-roll format and this is a backwards-compatibility feature, we probably don't need to handle that case. Anyone who wants to take advantage of the roll-specific damage types can set returnMultiple to be true in the pre-roll hook and get the data they need.


Turbocharged formula name completion

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[ ] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew tests with your changes locally?
[ ] Have you successfully run brew man locally and committed any changes?

This is a reworked formula name completion for the brew command that produces a name list that is identical to $(brew search) while being roughly 20-30 times faster, on average.  Note, the test below was done with 9 taps, some of them with dozens of Formulae, just to show that it is fast even with a lot of taps.
=> echo && echo "Old brew completion time:" && time brew_old_completion \
&& echo && echo "New brew completion time:" && time brew_new_completion

Old brew completion time:

real	0m4.114s
user	0m1.856s
sys	0m2.267s

New brew completion time:

real	0m0.134s
user	0m0.081s
sys	0m0.118s

It uses the high speed of find, sed, and grep to obtain and output that is functionally identical, but by traversing the Tap directory structure directly and avoids invoking brew search.  brew search is much more thorough, but in the special case of being invoked with no pattern, it falls back to a simpler output intended for use with completions,  but still takes the same time to run.  It seems like there is no advantage here to using brew's search command for completions when it can be done just as well using only default system binaries and switches that work just as readily with the BSD and GNU variants of those binaries.  I also derive the location of the file structure the same way the completion script is already doing it for completing tap names, so I think doing it for Formula as well should be ok.
I know it is not exactly the most readable solution, but regular expressions rarely are.  It isn't doing anything particularly exotic.  It uses find to traverse the tap directories, using -prune to ignore the Cask, cmd, and .git/hub directories which can contain non-formula .rb files, then produces a list of .rb files which are definitely formulas.  Sed removes the extra stuff we don't want (homebrew- prefixes, the full path, .rb extensions).  Then short names are also produced, and finally the extended names for the homebrew core tap is reduced to short names only.  This results in an identical completion list as brew search produces now.
This method is very similar to how the brew cask name completion is already handled, which also does it at the filesystem level.  This just has a little bit more sed and uses find instead of ls as it needs to traverse multiple taps with inconsistent directory structures.
If no one takes issue or otherwise finds a problem with doing it this way, I plan on porting it to fish, and possibly zsh, though zsh seems to cache the completion results so the speed up would have a minimal impact (it would mostly just speed up the first time formula name completion was used in that shell).

The speedup is really impressive, but since you use gfind, which doesn't ship by default on macOS or Linux, won't this cause errors for users who don't have findutils installed?

Whoops, that was a mistake.  In my own personal version, I use gfind because its a little faster, I must have committed it by accident.  It works with the BSD find as well, but takes about 30ms longer.
Anyway, fixed, thanks for catching that.  I also added caching of the HOMEBREW_REPO variable which trims off another ~50ms.

You'll need to run brew man and then commit the changes.

Ah, whoops  Thanks!

On Oct 24, 2020, at 7:00 PM, Jonathan Chang<EMAIL_ADDRESS>wrote:
You'll need to run brew man and then commit the changes.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/Homebrew/brew/pull/8846#issuecomment-716076645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABMM6SFVOBQ2W2V5G2D743SMN2EHANCNFSM4SC2JMIQ.

Thanks everyone for your suggestions and help.  Keep the opinions/concerns coming and I'll try to get this to a point that everyone is happy with it!

Thanks so much for your great, first contribution!  You rock, @metacollin!


Documentation is not clear about maxDuration in backoffStrategy

Summary
The documentation about maxDuration for backoffStrategy
is misleading and can be understood as the maximum waiting time between retries.
Currently documentation says:
MaxDuration is the maximum amount of time allowed for the backoff strategy

Use Cases
Some other retrying tools like Polly uses a parameter maxDelay which is similar but means the maximum waiting time between retries regardless of other parameters.

Message from the maintainers:
Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

I had the same confusion, until I learnt it the hard way. The documentation is insufficient and unclear.
Agreed with the maxDelay suggestion from @matjanos.
In fact maxDuration would make much more sense if it was moved alongside limit as a direct property of retryStrategy, and should best work with or without the backoff directive, like so:
retryStrategy:
  limit: '10'
  maxDuration: 10m
  backoff:
    duration: 10s
    factor: 2
    maxDelay: 1m # good to have

and
retryStrategy:
  limit: '10'
  maxDuration: 10m


WIP: Transition to 3.11

I think this is much more likely to work for 1.3 (or later) because we have to wait for Numba, dipy, etc. to update, but let's get the ball rolling now to see what we're waiting on.
Pending:

[x] mayavi: https://github.com/conda-forge/mayavi-feedstock/pull/70
[x] spyder-kernels: https://github.com/conda-forge/spyder-kernels-feedstock/pull/87
[x] dipy: https://github.com/conda-forge/dipy-feedstock/pull/41
[ ] numba: https://github.com/numba/numba/issues/8304

Hoo boy... I forgot about VTK. Let's be prepared to wait for that one!

... or not!
https://github.com/conda-forge/vtk-feedstock/pull/265

... or not!
https://github.com/conda-forge/vtk-feedstock/pull/265

Ha! Nice!!!!! Fingers crossed the remaining builds will succeed too

VTK all green, amazing!

Mayavi C bug :(
https://github.com/enthought/mayavi/issues/1188

Mayavi C bug :(

… we still need Mayavi? What for? the KIT GUI? Can we just not drop this?

More ball rolling
conda-forge/mayavi-feedstock#69 conda-forge/dipy-feedstock#41

FYI you really needn't (shouldn't) open all these PRs manually, a bot will perform the transition step by step and open PRs once all dependencies have been made available for 3.11.

I figured there was no harm in getting the ball rolling. Why "shouldn't"? Does it make it harder for them?

... and no I don't think we should drop inclusion of mne-kit-gui

Why "shouldn't"? Does it make it harder for them?

My understanding was always that they have limited CI resources and it's best to rely on he bot to schedule the feedstock updates and builds :) but maybe this doesn't apply anymore, I'm not sure!

Statsmodels needs to release, they plan to do it in the next couple of weeks:
https://github.com/statsmodels/statsmodels/issues/8287
Then dipy might have some stuff to fix (but hopefully not). Maybe someone can fix Mayavi in that time :)

I just checked on this, and statsmodels is 3.11-compatible now. But it looks like Dipy, numba, and mayavi are behind currently.

I just checked on this, and statsmodels is 3.11-compatible now. But it looks like Dipy, numba, and mayavi are behind currently.

Dipy 1.6 should be 3.11-compatible
I'm afraid numba and mayavi are still lagging behind…

Dipy 1.6 should be 3.11-compatible
I'm afraid numba is still lagging behind though.

Actually dipy 1.5 was 3.11-compatible on conda-forge, too. But 1.6 should land soon, anyway. But yes numba remains a problem...

Do we actually "need" Numba? In the sense of, do we actually think users who prefer the standalone installers will even take advantage of it?

It's used internally in MNE to speed some things up by an order of magnitude or so, including source-space surface-to-surface morphing and EEG electrode projection IIRC. So yes users use it without knowing it :)

So yes users use it without knowing it :)

Oh wow, I never realized!!! Interesting

If you're ever curious to see how/why, you can git grep "@jit" and you'll see we have 35 instances in MNE-Python at the moment

Closing as this is way outdated and I'm now subscribed to the relevant numba/llvmlite issues upstream


Checkout unable to give Camera permission to Tamara

What area is the issue related to?
Checkout Sheet Kit
What version of the package are you using?
2.0.1
Reproducible sample code
No response
Steps to Reproduce

Open Checkout
Go with any installment plan, in my case it's Tamara.
Tamara asks for ID Verification where the camera opens
Checkout unable to give Camera Permissions to Tamara

Expected Behavior
Camera should open as Camera permission is already given and don't ask for camera permission again and again.
Actual Behavior
Unable to give Camera permission to Tamara & asking for permissions again & again. However Camera permission is already given
Screenshots/Videos/Log output

Storefront domain
https://revibe.me/

A new version has been released which exposes an onPermissionRequest() callback in the event processor.
You should now be able to implement that to grant access to the camera (and/or microphone) in the checkout WebView.
There's a basic implementation in the MobileBuyIntegration sample app


fix enumerate_paths

see discussion in #150.

Coverage decreased (-0.1%) when pulling 84df17dd8aea3741f1728840d1709f450fa6c1f2 on yeesian:master into d018f24e0e901a5518e483ef202746e88a5ed6ff on JuliaLang:master.

Is this ready to merge?

I'm not sure if the discussion about whether parent_indices[src] should equals 0 (currently not the case) has been resolved, but @sbromberger has already moved his work over to FastGraphs.
I'm okay either way; just waiting for there to be a resolution, so we can move on with things over at https://github.com/tedsteiner/OpenStreetMap.jl/pull/58.


FIX - Networks: Control show/hide PF & LB rules tabs

Fixes #678
@rhtyd cc @svenvogel

Case 1:

Case 2:

Case 3:

Case 4:

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/744 (JID-3506)

Control the enable static NAT button

PF or LB rules:

no tier, PF or LB rules:

@blueorangutan package

@utchoang a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/744 (JID-3507)

@blueorangutan package

@davidjumani a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/744 (JID-3509)

@utchoang can you test and advise when this is ready for testing? I navigated to the IP view from VPC -> select Admin-RR -> select IP address tab and select an IP and I see this:

@rhtyd Sorry, I commit missing. Please try again. Thanks.

@blueorangutan package

@utchoang a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/744 (JID-3511)

@blueorangutan package

@rhtyd a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/744 (JID-3518)

cc @DaanHoogland

@utchoang both portforwarding and loadbalancing disable the other two options, but the enable static nat button seems to be non-func:

after enabling static nat in the old UI:

disable static nat doesn't work either. marking this as blocker

@blueorangutan package

@utchoang a Jenkins job has been kicked to build primate packages. I'll keep you posted as I make progress.

Packaging result: :heavy_check_mark:centos :heavy_check_mark:debian :heavy_check_mark:archive.
QA: http://primate-qa.cloudstack.cloud:8080/client/pr/744 (JID-3520)

great @utchoang all three work now


Server Side Rendering Incomplete on ChromePicker (and others)

Hi,
I get a warning from React (0.14) that background-image is null when rendered server side, but contains the image client-side meaning that they are rendering differently. This makes it a pain to use for isomorphic apps, as there will always be this warning here.

What's the full trace?
On Sunday, April 10, 2016, Darren<EMAIL_ADDRESS>wrote:

Hi,
I get a warning from React (0.14) that background-image is null when
rendered server side, but contains the image client-side meaning that they
are rendering differently. This makes it a pain to use for isomorphic apps,
as there will always be this warning here.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/casesandberg/react-color/issues/127

Hi Again!
It appears we have encountered the same issue (ISO incompatible).
any idea if fixing this was accomplished, still of interest or in the works?
If not overly complex we can send a PR & try to help with a brief overview of needed steps.
Thanks!

@casesandberg I did a bit of research into this. The problem lies in helpers/checkboard.js#L4 which is using a HTML5 canvas to generate a data-url.
I noticed that this replaced code which rendered a React component. Rather than reinstating this, which would take a lot of work, is there merit in a PR which introduces node-canvas for server-side rendering? Or would you be reluctant to add a server-only dependency?

I definitely don't want to install a dependency just for SSR. I will have to look into node-canvas though, because if the API is the same as the browser I can put in a prop which one could map the node-canvas dep to that would replace the browser canvas.

Just taking a quick look at the exmples it seems like it would be quite easy to do by passing node-canvas though. Any thoughts on the API:
import Canvas from 'node-canvas'
import { SketchPicker } from 'react-color'

<SketchPicker server={{ Canvas }} />
// or
<SketchPicker ssr={{ Canvas }} />
// or
<SketchPicker renderers={{ canvas: Canvas }} />

This looks great, thanks! Personally, I would prefer:
<SketchPicker renderers={{ canvas: Canvas }} />

It leaves more scope for further server changes in the future, plus I think it reads better than server/ssr.
I implemented node-canvas in my own fork: https://github.com/ryan-codingintrigue/react-color
It works OK, except I'm seeing a difference in the Data URIs generated on server/client. I've logged a bug on their repo: https://github.com/Automattic/node-canvas/issues/853.
Thanks again for looking at this. It's important for me to match my server/client in my React projects 😄

This is live in<EMAIL_ADDRESS>

Awesome, thank you so much! Just FYI, I couldn't get the Data URI returned from node-canvas to match the Data URI generated in my browser, but using this API I was able to create a stub class for the server:
const dataUrl = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAMUlEQVQ4T2NkYGAQYcAP3uCTZhw1gGGYhAGBZIA/nYDCgBDAm9BGDWAAJyRCgLaBCAAgXwixzAS0pgAAAABJRU5ErkJggg==";

export default class ServerCanvas {
    constructor() {
        this.toDataURL = () => dataUrl;
        this.getContext = () => ({
            fillRect: () => {},
            translate: () => {}
        });
    }
}

Since I'm only using the ChromePicker and one browser, this suits my requirements for allowing React to match server/client

Thats perfect 🙌

I'm having this issue as well. Would it not be possible to have the component automatically return a pre-rendered dataUrl like @ryan-codingintrigue does above instead of returning null here?
https://github.com/casesandberg/react-color/blob/0657c78ef80d1144495e45aff6555f163eb7b240/src/helpers/checkboard.js#L4-L6


Merge OpenAI Triton till June 7th

It is essential to keep up to date with OpenAI Triton, to get the latest features, and reduce the difficulty to upstream our changes to OpenAI Triton.
This ticket is continuation of:

#1095
#996
#884
#554
#208

Moved up from May 24 to May 27 Triton version from OpenAI. [https://github.com/intel/intel-xpu-backend-for-triton/pull/1227 ]


Doesnt work on some DEs

it doesnt work on some DEs like kde xv, xfce and mabye some others

I can't speak to the other DEs, but it doesn't work out of the box on KDE because by default KDE uses A-M-x for an accessibility setting. You can turn that off in the KDE system settings and then warpd should work fine.

Wayland support is currently confined to wlroots based compositors like sway and hyprland, though it should work on most X based DEs. Feel free to file issues for specific DEs that you think should work so they can be evaluated. It is worth noting that warpd is currently in maintenance mode, and efforts are being directed towards porting most of its useful functionality to keyd, which uses a lower level input mechanism and should support a larger number of environments.


Incorrect behavior when generate module which already exists

When regenerate module with name already exists

No any warnings about deletion my files (it possible I already have some code here)
Incorrect deletion from project, some files paths corrupted in compiled sources

steps:
generamba gen RootNavigation  bl_viper_default_gt

and again
generamba gen RootNavigation  bl_viper_default_gt

@MaksimBazarov thanks for reporting! I've already encountered a very similar problem, seems that my fix wasn't enough. I'll investigate this behavior (described under p.2).
And a nice idea about posting warnings about deletion - I'll plan it for one of the upcoming versions!


V1.4 finalize dg

finalize on DG
https://kaiyichen.github.io/tp/DeveloperGuide.html

Codecov Report

Merging #272 (09ef79e) into master (ee00527) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #272   +/-   ##
=========================================
  Coverage     72.87%   72.87%
  Complexity      587      587
=========================================
  Files            89       89
  Lines          1755     1755
  Branches        211      211
=========================================
  Hits           1279     1279
  Misses          408      408
  Partials         68       68

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee00527...09ef79e. Read the comment docs.


Adding an issue to test links

http://whathappens.here

http://example.com

What are you seeing that seems off? We support GitHub Flavored Markdown (https://help.github.com/articles/github-flavored-markdown/) in comments and issue descriptions. Are you seeing buggy behavior with that?


Use pkg.pr.new for snapshots

Type

[ ] Refactor
[ ] Feature
[ ] Bug Fix
[x] Optimization
[ ] Documentation Update

Description
Swaps our snapshot strategy from publishing to the NPM registry to using pkg.pr.new from the folks at Stackblitz.
This has numerous benefits, not the least of which is avoiding pollution of the NPM registry with extraneous snapshot packages.
Related

Related Issue #
Closes #

/snapshot


Working with both Windows and Linux packages

Hello, I'm working on a project that builds both for Windows and for Linux.  I'm trying to consume some components from a Windows program (as source code) in our Windows/Linux project.  This program depends on some packages that are obtained via vcpkg; the people who built that repo simply checked-in the binaries obtained via vcpkg to their repo.  I'm trying to instead get them via nuget:

Clone the vcpkg repo.
Export a nuget package of Windows binaries, and a second nuget package of Linux binaries.
Consume both nuget packages inside the Windows/Linux project, depending on the target OS.

I've had no problem getting the Windows work done.  But I can't figure out how to import the Linux binaries.  I'm a long-time Windows user and a newbie Linux user, so probably just misunderstanding something very basic.  I'm running WSL with Ubuntu 20.04.  In https://docs.microsoft.com/en-us/cpp/build/vcpkg?view=msvc-160#target-linux-from-windows-via-wsl it looks like I can clone the repo just once and use it for both OS'.  Inside WSL I can go to the vcpkg repo under /mnt/c/ (in my case, /mnt/c/repos/vcpkg) and successfully run ./bootstrap-vcpkg.sh.  But when I try the ./vcpkg.exe install command, it throws an error about the compiler.
[DEBUG] CreateProcessW("C:\repos\vcpkg\downloads\tools\cmake-3.18.4-windows\cmake-3.18.4-win32-x86\bin\cmake.exe" -DCURRENT_PORT_DIR=C:/repos/vcpkg/scripts/detect_compiler -DCURRENT_BUILDTREES_DIR=C:/repos/vcpkg/buildtrees/detect_compiler -DCURRENT_PACKAGES_DIR=C:/repos/vcpkg/packages/detect_compiler_x64-linux -DCMD=BUILD -DTARGET_TRIPLET=x64-linux "-DTARGET_TRIPLET_FILE=C:\repos\vcpkg\triplets\x64-linux.cmake" -DVCPKG_PLATFORM_TOOLSET=external -DDOWNLOADS=C:/repos/vcpkg/downloads -DVCPKG_CONCURRENCY=9 "-DGIT=C:/Program Files/Git/cmd/git.exe" -DVCPKG_ROOT_DIR=C:/repos/vcpkg -DPACKAGES_DIR=C:/repos/vcpkg/packages -DBUILDTREES_DIR=C:/repos/vcpkg/buildtrees -D_VCPKG_INSTALLED_DIR=C:/repos/vcpkg/installed -DDOWNLOADS=C:/repos/vcpkg/downloads -DVCPKG_MANIFEST_INSTALL=OFF -P "C:\repos\vcpkg\scripts\ports.cmake")
[DEBUG] -- Configuring x64-linux
[DEBUG] CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:106 (message):
[DEBUG]     Command failed: ninja -v
[DEBUG]     Working Directory: C:/repos/vcpkg/buildtrees/detect_compiler/x64-linux-rel/vcpkg-parallel-configure
[DEBUG]     Error code: 1
[DEBUG]     See logs for more information:
[DEBUG]       C:\repos\vcpkg\buildtrees\detect_compiler\config-x64-linux-out.log

the .log file:
[1/1] cmd /c "cd .. && "C:/repos/vcpkg/downloads/tools/cmake-3.18.4-windows/cmake-3.18.4-win32-x86/bin/cmake.exe" "C:/repos/vcpkg/scripts/detect_compiler" "-DCMAKE_MAKE_PROGRAM=C:/repos/vcpkg/downloads/tools/ninja/1.10.1-windows/ninja.exe" "-DCMAKE_SYSTEM_NAME=Linux" "-DBUILD_SHARED_LIBS=OFF" "-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=C:/repos/vcpkg/scripts/toolchains/linux.cmake" "-DVCPKG_TARGET_TRIPLET=x64-linux" "-DVCPKG_SET_CHARSET_FLAG=ON" "-DVCPKG_PLATFORM_TOOLSET=external" "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON" "-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-DVCPKG_APPLOCAL_DEPS=OFF" "-DCMAKE_TOOLCHAIN_FILE=C:/repos/vcpkg/scripts/buildsystems/vcpkg.cmake" "-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON" "-DVCPKG_CXX_FLAGS=" "-DVCPKG_CXX_FLAGS_RELEASE=" "-DVCPKG_CXX_FLAGS_DEBUG=" "-DVCPKG_C_FLAGS=" "-DVCPKG_C_FLAGS_RELEASE=" "-DVCPKG_C_FLAGS_DEBUG=" "-DVCPKG_CRT_LINKAGE=dynamic" "-DVCPKG_LINKER_FLAGS=" "-DVCPKG_LINKER_FLAGS_RELEASE=" "-DVCPKG_LINKER_FLAGS_DEBUG=" "-DVCPKG_TARGET_ARCHITECTURE=x64" "-DCMAKE_INSTALL_LIBDIR:STRING=lib" "-DCMAKE_INSTALL_BINDIR:STRING=bin" "-D_VCPKG_ROOT_DIR=C:/repos/vcpkg" "-D_VCPKG_INSTALLED_DIR=C:/repos/vcpkg/installed" "-DVCPKG_MANIFEST_INSTALL=OFF" "-G" "Ninja" "-DCMAKE_BUILD_TYPE=Release" "-DCMAKE_INSTALL_PREFIX=C:/repos/vcpkg/packages/detect_compiler_x64-linux""
FAILED: ../CMakeCache.txt
cmd /c "cd .. && "C:/repos/vcpkg/downloads/tools/cmake-3.18.4-windows/cmake-3.18.4-win32-x86/bin/cmake.exe" "C:/repos/vcpkg/scripts/detect_compiler" "-DCMAKE_MAKE_PROGRAM=C:/repos/vcpkg/downloads/tools/ninja/1.10.1-windows/ninja.exe" "-DCMAKE_SYSTEM_NAME=Linux" "-DBUILD_SHARED_LIBS=OFF" "-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=C:/repos/vcpkg/scripts/toolchains/linux.cmake" "-DVCPKG_TARGET_TRIPLET=x64-linux" "-DVCPKG_SET_CHARSET_FLAG=ON" "-DVCPKG_PLATFORM_TOOLSET=external" "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON" "-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-DVCPKG_APPLOCAL_DEPS=OFF" "-DCMAKE_TOOLCHAIN_FILE=C:/repos/vcpkg/scripts/buildsystems/vcpkg.cmake" "-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON" "-DVCPKG_CXX_FLAGS=" "-DVCPKG_CXX_FLAGS_RELEASE=" "-DVCPKG_CXX_FLAGS_DEBUG=" "-DVCPKG_C_FLAGS=" "-DVCPKG_C_FLAGS_RELEASE=" "-DVCPKG_C_FLAGS_DEBUG=" "-DVCPKG_CRT_LINKAGE=dynamic" "-DVCPKG_LINKER_FLAGS=" "-DVCPKG_LINKER_FLAGS_RELEASE=" "-DVCPKG_LINKER_FLAGS_DEBUG=" "-DVCPKG_TARGET_ARCHITECTURE=x64" "-DCMAKE_INSTALL_LIBDIR:STRING=lib" "-DCMAKE_INSTALL_BINDIR:STRING=bin" "-D_VCPKG_ROOT_DIR=C:/repos/vcpkg" "-D_VCPKG_INSTALLED_DIR=C:/repos/vcpkg/installed" "-DVCPKG_MANIFEST_INSTALL=OFF" "-G" "Ninja" "-DCMAKE_BUILD_TYPE=Release" "-DCMAKE_INSTALL_PREFIX=C:/repos/vcpkg/packages/detect_compiler_x64-linux""
-- The C compiler identification is unknown
CMake Error at CMakeLists.txt:11 (enable_language):
  No CMAKE_C_COMPILER could be found.

  Tell CMake where to find the compiler by setting either the environment
  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
  the compiler, or to the compiler name if it is in the PATH.

-- Configuring incomplete, errors occurred!

I've set these environment variables:
  export VCPKG_DEFAULT_TRIPLET=x64-linux
  export VCPKG_PLATFORM_TOOLSET=v142
  export CC=gcc
  export CXX=g++

But it looks like the code is running Windows tools - like that CreateProcess on "C:\repos\vcpkg\downloads\tools\cmake-3.18.4-windows\cmake-3.18.4-win32-x86\bin\cmake.exe" - that's calling out to Windows.
Did I do this wrong?  Do I need to clone the repo a second time inside WSL in order to import packages for Linux?  I've looked all over the MSDN doc above plus https://github.com/microsoft/vcpkg/blob/master/docs/index.md and haven't found the details I need for how to work with Linux in WSL.  Thanks for the help with my newbie question.

Oh, gosh, the command is still running, but I think I have the answer.  I was so close - but I shouldn't call "./vcpkg.exe install" from inside Linux.  I should call "./vcpkg install" (without the .exe extension).

@strega-nil Any ideas?
@sloh-ms By using different vcpkg binaries, ports under different platforms will be built according to different compilers.

Thanks @JackBoosY, yeah having separate compilers is going to happen for sure.  That's expected.
Reporting back: using "./vcpkg" instead of "./vcpkg.exe" appears to have worked.  I did get a number of messages during the install like
sh: 1: zip: not found
Failed to store binary cache /home/sloh/.cache/vcpkg/archives/ce/ce68db8e056a815087b2e74d682e6787ef8a14fe.zip: Invalid cross-device link

But they appear to be benign.  The binaries were built and I was able to export a nuget package containing all of the Linux binaries.  The export likewise failed to find those cached .zip files, so the whole process is probably a lot slower than it should be.  But at least it works.
I've got a separate failure on making a nuget package that I hesitate to morph this thread with.
Could not automatically acquire mono because there is no entry in /mnt/c/repos/vcpkg/scripts/vcpkgTools.xml for os=linux. You may be able to install mono via your system package manager.

Rather than solve either of the above here, I just want to end with one last question -- is my approach valid?  Am I doing a reasonable thing by sharing the same repo for building both Windows and Linux, or should I be using two separate copies of the repo?

@sloh-ms it's absolutely valid. It's not necessarily a tested configuration, but it's something I do all the time. If you run into any bugs, please open issues :)

Also, you should really install zip via sudo apt install zip, because otherwise you don't get binary caching.
On the nuget issue, yes, you should probably open an issue about that. We should be able to export nuget packages for linux.

Thank you for the confirmation!  I've already done the zip install, and am working on getting the nuget export to work.  I'll open an issue if I am stuck.


Move to CSV crate

The latest CSV crate seems to have incorporated what made quick-csv so fast as well as enables serde to serialize/deserialize rows. Regardless of serde the csv crate is far more popular and lively and probably worth switching to.

I had to throw out all my code, but this is pretty promising. However I just ran cargo bench and the results aren't good:
before
running 9 tests
test bench_agency_iterator        ... bench:         964 ns/iter (+/- 100) = 117 MB/s
test bench_calendar_date_iterator ... bench:         874 ns/iter (+/- 1) = 53 MB/s
test bench_calendar_iterator      ... bench:       2,607 ns/iter (+/- 71) = 61 MB/s
test bench_frequency_iterator     ... bench:       3,623 ns/iter (+/- 73) = 95 MB/s
test bench_route_iterator         ... bench:       3,221 ns/iter (+/- 48) = 96 MB/s
test bench_shape_iterator         ... bench:         821 ns/iter (+/- 11) = 87 MB/s
test bench_stop_iterator          ... bench:       4,794 ns/iter (+/- 90) = 124 MB/s
test bench_stop_time_iterator     ... bench:       7,378 ns/iter (+/- 129) = 151 MB/s
test bench_trip_iterator          ... bench:       4,762 ns/iter (+/- 32) = 86 MB/s

after
running 11 tests
test bench_agency_iterator          ... bench:      21,758 ns/iter (+/- 416) = 5 MB/s
test bench_calendar_date_iterator   ... bench:      21,557 ns/iter (+/- 1,031) = 2 MB/s
test bench_calendar_iterator        ... bench:      23,141 ns/iter (+/- 114) = 6 MB/s
test bench_fare_attributes_iterator ... bench:      22,173 ns/iter (+/- 106) = 4 MB/s
test bench_fare_rules_iterator      ... bench:      22,441 ns/iter (+/- 203) = 4 MB/s
test bench_frequency_iterator       ... bench:      25,033 ns/iter (+/- 150) = 13 MB/s
test bench_route_iterator           ... bench:      24,054 ns/iter (+/- 696) = 12 MB/s
test bench_shape_iterator           ... bench:      21,194 ns/iter (+/- 186) = 3 MB/s
test bench_stop_iterator            ... bench:      26,219 ns/iter (+/- 208) = 22 MB/s
test bench_stop_time_iterator       ... bench:      31,360 ns/iter (+/- 94) = 35 MB/s
test bench_trip_iterator            ... bench:      26,242 ns/iter (+/- 60) = 15 MB/s

A couple of things could be at work here:

custom field deserializers might be written or applied badly (but bench_agencies_iterator would still be relatively fast)
automatic deserialization could be too generic (so csv crate StringRecords plus the old parse functions might be better)
csv crate doesn't have the speed improvements that quick_csv has (or they don't apply when running with serde)

Overall though its a solid 300 less lines of code to maintain (400 if you measure from 0.2.0). It's also dirt simple to add support for new files now (fare_rules.txt was already supported, just added tests, fare_attributes.txt just needed custom deserializers for a few fields).

I just tweaked bench_agency_iterator to use Reader::records instead of deserializing to see if its serde support that's killing performance and I still got similar benchmark numbers.
Maybe I'm using it wrong, but csv is just fundamentally still slower than quick_csv.
Lemme know if you still want a pull request for this.

That is incredibly unfortunate, I was hoping to save maintenance costs because it would automatically match fields.

The biggest issue seems to be in our benchmarks we are creating a new CSV reader each time we read the file. I kid you not when I say creating that new csv Reader type is the bain of the benchmarks. I'm updating some code to avoid that as thats not really what I meant to benchmark to begin with.

Even more concerning is the newer CSV crate actually has a benchmark showing MBTA stop times working much faster
https://github.com/BurntSushi/rust-csv/blob/master/benches/bench.rs#L22

I tried to make a version of GTFSIterator::new that accepted a reference to csv::Reader but the de-serializing iterator moves content and the other version wants lifetimes so I gave up on trying to pull it out of the loop. (With this approach I'd also have to reset the Reader every iteration but I think thats do-able)
That is probably the right choice here though because yeah its all Reader building time. Here is the MBTA deserializing to String bench result:
test count_mbta_deserialize_owned_str      ... bench:   4,698,632 ns/iter (+/- 444,004) = 153 MB/s

If I hack up our benchmark to to use the MBTA data then we get:
test bench_stop_time_iterator       ... bench:   6,412,457 ns/iter (+/- 395,219) = 112 MB/s

So pretty comparable and considering the extra work that has to be put into enum fields I think  adequate for now.
Ideally I'd like to pull out the iterator building entirely but I don't see a way to do that (and apparently neither does BurntSushi ;)). Failing that: We're still slower than the quick_csv StopTimes iteration but I think its not a totally fair comparison (TimePoint parsing was adding, shape distance is optional) not am I 100% confident my serde deserializers are optimal. I think its still close enough, but its also worse enough that I understand if you still have concerns.
I'm trying to resolve the other major regression (leading/trailing whitespace) in upstream (BurntSushi/rust-csv#78) but I'm stuck there too XD

I think going this route is much nicer for other reasons, I think we're ok. I've simplified the benchmarks using a macro very similiar to the csv crate. Feel free to modify. You are a contributor now and should be able to do whatever you need to directly to this repo if you like :+1:

OK its pushed this plus some other changes I had queued, I couldn't find the macro you mentioned in any of your branches so I'm assuming you'll apply it from your local stuff.
Thank you for making me a contributor BTW. Unless you want to change the process I plan on submitting issues for features to get feedback and signpost where I'm going. I'll go forward changes in general if I'm not conflicted about how to approach things. I am a deep fan of code reviews so feel free to post-hoc file issues.


[feature] Allow having an additional whitelist of custom files to watch

Not the job of CW, I prefer to keep things simple.


Added the test cases for rake_new2

Description of the change
This is a fix to the issue #9 . In this I have added the edge cases for the rake_new2 module.
I have tried to add all the html tags in edge cases because the rake_new2 can take the HTML tags.
Related Issues
#9  Adding the edge cases for the rake_new2 module
What type of PR is this?

[ ] Adding tests to existing code

Added tests?

[ ] Yes

Codecov Report

Merging #53 (9434a13) into master (df045cd) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master     #53   +/-   ##
======================================
  Coverage    5.69%   5.69%
======================================
  Files           5       5
  Lines         123     123
======================================
  Hits            7       7
  Misses        116     116

Flag
Coverage Δ

unittests
5.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df045cd...9434a13. Read the comment docs.

Codecov Report

Merging #53 (9434a13) into master (df045cd) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master     #53   +/-   ##
======================================
  Coverage    5.69%   5.69%
======================================
  Files           5       5
  Lines         123     123
======================================
  Hits            7       7
  Misses        116     116

Flag
Coverage Δ

unittests
5.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df045cd...9434a13. Read the comment docs.

@BALaka-18 can you please review the pull request for the issue #9 . the competition is about to end, can you please merge the pr soon.


chore: update pubsub samples to new test runner

@bshaffer not sure how to fix the redeclared function error. Locally it seems like I can get it working by modifying this line to use require_once, but I'm not sure if there's some context I'm missing.


Aggregation on empty local table

(you don't have to strictly follow this form)
Describe the unexpected behaviour
A clear and concise description of what works not as it is supposed to.
How to reproduce
    val emptyTableData = Seq.empty[(Int, Int)].toDF("a", "b")  ;     emptyTableData.agg(sum($"a")).show()
the above code will return an empty set instead of a single row with null value
Expected behavior
like Spark, return a  single row with null value
Additional context
Add any other context about the problem here.

The empty dataset is represented by a LocalRelation in this example, and this LocalRelation will be transfomed to LocalTableScanExec in org.apache.spark.sql.execution.SparkStrategies.BasicOperators.
LocalTableScanExec will return an EmptyRDD at LocalTableScanExec.rdd, which will cause the first stage of the query to be skipped (since EmptyRDD has 0 partitions):

The absence of the first stage will cause bad consequences for Gluten-CH, because the CH native code will have no chance to consume the empty raw table and generate an "intermediate buffer", which should have happened in the first stage and will produce one buffer row (with null values filled). Spark has no concept of intermediate buffer, so it will work correctly even if the first stage is skipped.
So how to fix this?
An RDD with 0 partitions(EmptyRDD )is mainly observed in non-production scenarios. When we have a parquet file as input, the   input RDD will have 1 partition even if the parquet file has zero rows.
We'll add an extra SparkStrategy which will translate LocalRelation into a modified version of LocalTableScanExec (let's call it MyLocalTableScanExec for now). For empty inputs, MyLocalTableScanExec will generate a MyEmtpyRDD instead of EmptyRDD. MyEmtpyRDD has 1 partition and this partition will return an empty Iterator. This will change the plan to:

notice that the first stage is no longer skipped, and the result will be a single row with null values.
Similar strategies should be added for other SparkPlans that will generate EmptyRDD.


Remove defunct template

docs-metadata-release.yml was replaced with common/pipelines/templates/steps/update-docsms-metadata.yml and should have been removed at that time but was not.

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow

This file is ancient and not used anymore. Thanks!!

This file is ancient and not used anymore. Thanks!!

Are there other files related that should also be cleaned up?

update-docs-metadata.ps1 reference in this file is already removed. I think this template yml file was somehow missed in that earlier cleanup.

This file is ancient and not used anymore. Thanks!!

Are there other files related that should also be cleaned up?

Only the template yml file needs to be removed. The script the template was using, update-docs-metadata.ps1, was already removed.


systemctl-tui: init at 0.2.2

Description of changes
A fast, simple TUI for interacting with systemd services and their logs.
https://crates.io/crates/systemctl-tui
closes #262762
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 262822 run on x86_64-linux 1

1 package built:

systemctl-tui


One-pager for e-signature services

For meeting w/Paz on 8/25.

@daguar says that he has some prior art around this from California/research.

https://github.com/codeforamerica/health/wiki/Research#electronic-signatures-in-ca

thanks @daguar!

Overview and comparison spreadsheet here: https://docs.google.com/spreadsheets/d/1jPRJ_N14bETeUW0goFQybVbrAegrdiucA9kyQ4rr_MI/edit#gid=1583737287
Will be taking content from this spreadsheet and condensing it into a visually appealing 1 or 2-pager

Prettified overview: https://docs.google.com/document/d/14Q0v5qn3mdaLLYRAZZIqfnRTMRYHd-9uhnRNOaoYz9c/edit#
Now 2 pages -- first is exec summary and comparison table, 2nd is implementation recommendations.


geoWithin vs GeoWithin

Looks like the file core/attribute/GeoPredicate/geoWithin.py should be renamed GeoWithin. Doing so allows me to use:  from janusgraph_python.structure.io.GraphsonReader import JanusGraphSONReader.  Otherwise, I get the error:  ModuleNotFoundError: No module named 'janusgraph_python.core.attribute.GeoPredicate.GeoWithin'

@John-Boik : First of all thanks for starting to use the library. We need the community to help us grow :-)
Can you open an issue at official repo ?
This library is currently under review to be merged into official repo.
If you create an issue there, it will be easier for us to track as this repo will deleted in few days.
And, we can take the discussion forward from there.
Thanks

Also, this repo isnt maintained since few days. We have moved the code & most updated code janusgraph-fork
Please use that for reference.


Android: Bugfixes

So, this isn't really any full progress on new features (though some groundwork for them is included), but it does contain a few small bugfixes and the big bugfix for #713, which appears to be pretty active in the wild.  I would suggest we cut an Beta ASAP so folks in Prod can get some relief.  This has tested well for me, so I would put it on the fast track if possible.

Pushed 4.3.2b1.


Fails for anything other than "Homepage only"

I've recently started trying to use WP2Static to get a ZIP archive of the HTML for doc pages and blog posts.  However, I have only been able to get an archive with the "Homepage only" option.
"Initial crawl list" gives "6051 URLs were detected on your site that will be used to initiate the crawl. Other URLs will be discovered while crawling."
I have used Crawling settings with "Posts and pages only" and to exclude a large range of URLs, but WP2Static fails repeatedly when crawling, and sometimes at other times:
"Failed during "Analyzing site... this may take a few minutes (but it's worth it!)"
or
"Failed during Crawling"
(the fail point varies from day to day)
I enabled the Advanced / debug mode, but did not get any useful information.
P.S.  The plugin was enabled by my Wordpress admin and I only have access through the Wordpress UI.

I'm having the same issue (though using AWS S3 for storage and also tested with local directory storage). Homepage works (though I need to add a lot of URLs for inclusion that it isn't catching - e.g. font URLs from plugins). But others quit at various stages on each run - never the same stage twice - and no useful debug information or messages anywhere.

This issue prevents the WP2STATIC plugin from working at all.
I have kept trying from time to time, but the plugin just does not work.
My last tests are failing with

Failed during "Crawling discovered URLs", please consult the Help tab for where to get assistance.

... but I don't want the plugin to crawl discovered URLs, as I have a precise list of URLs that I want crawled, and the discovered URLs log adds nothing different to the initial URLs list.
I've asked my Wordpress admin to switch to a different plugin as WP2STATIC does not work at all, and my issue goes unaddressed.

@AnimatIan - sorry for dropping the ball on this issue for you here. I've generally kept up with email/chat support requests  (if sometimes delayed), but GH issues got pushed to the back.
V7 is ready for testing and will help in this case. Default detection option in core is now to detect as much as possible, with some configurability, but total detection options moving into their own add-on for the finer grained control not every user will need to be bombarded with.
Please give it a try if you're still in need of a solution or on a future site when the need arises: https://github.com/WP2Static/wp2static/releases/tag/7.0-alpha-003


Mechanically tag all new APIs as @ExperimentalOkHttpApi

In a follow-up PR I intend to remove the annotation on any API that we're willing to commit to for OkHttp 5.0.0.
In a follow-up PR I'd also like to hide all remaining APIs from Java langauge callers.

> Task :okhttp-android:compileDebugKotlin FAILED
e: file:///home/runner/work/okhttp/okhttp/okhttp-android/src/main/kotlin/okhttp3/android/AndroidAsyncDns.kt:38:2 Unresolved reference: ExperimentalOkHttpApi
e: file:///home/runner/work/okhttp/okhttp/okhttp-android/src/main/kotlin/okhttp3/android/AndroidAsyncDns.kt:40:25 This declaration needs opt-in. Its usage must be marked with<EMAIL_ADDRESS>or '@OptIn(okhttp3.ExperimentalOkHttpApi::class)'
e: file:///home/runner/work/okhttp/okhttp/okhttp-android/src/main/kotlin/okhttp3/android/AndroidAsyncDns.kt:40:34 This declaration needs opt-in. Its usage must be marked with<EMAIL_ADDRESS>or '@OptIn(okhttp3.ExperimentalOkHttpApi::class)'
e: file:///home/runner/work/okhttp/okhttp/okhttp-android/src/main/kotlin/okhttp3/android/AndroidAsyncDns.kt:42:5 This declaration needs opt-in. Its usage must be marked with

@swankjesse I've added the Gradle opt-in for kotlin("android") as well, which was failing on CI.


Extension issue

Issue Type: Bug
Extension Name: kite
Extension Version: 0.109.0
OS Version: Windows_NT x64 10.0.18363
VSCode version: 1.42.1

:warning: We have written the needed data into your clipboard. Please paste! :warning:

please provide details about the issue you're experiencing


add ability to replace jquery instance

Per the contrib doc, please make your PRs to the development branch. PRs to the master will be closed.

Per the contrib doc, please make your PRs to the development branch. PRs to the master will be closed.


Kwikmodel spike_times not recomputed after switching channel_group

When extracting spike_times from a .kwik file using KwikModel, the model always displays spike_times for the first channel_group accessed. Setting a new channel_group updates the other measurements (i.e. spike_samples) appropriately, it seems there is no functionality in spike_times(self) to force recomputation after changing channel group.
Our quick fix: removing the if statement forces recomputation on each call of spike_times:
def spike_times(self):
    """Spike times from the current channel_group.

    This is a NumPy array containing `float64` values (in seconds).

    The spike times of all recordings are concatenated. There is no gap
    between consecutive recordings, currently.
    """
    self._spike_times = (self.spike_samples.astype(np.float64) /
                             self.sample_rate)
    return self._spike_times

Thanks for the bug report. A proper fix might be to set self._spike_times = None here. Feel free to try it and propose a pull request if it works!


web: desktop navigation improvements

Improves navigation in the Tlon app on desktop by modifying the mobile tab bar to orient vertically along the left side of the page. State is not preserved between tabs (existing behavior).
https://github.com/tloncorp/landscape-apps/assets/748181/892479b3-5fdd-42db-bdd2-c93d9585bb61

Adds an AppNav component, which wraps the main view in both mobile and desktop contexts with a sidebar/bottom-bar (fixes LAND-1602)
Modifies all NavTab components to accommodate desktop click targets (and to act like a SidebarItem)
Adds the shouldApplyPaddingBottom check to all views
Relocates the "+" button to create/join a group or create a message from the navigation to the top of each respective sidebar (fixes LAND-1580)
Relocates the public profile toggle to the profile editor itself
Adds a Profile sidebar with settings-specific nested views on desktop (fixes LAND-1389)
Refactors the profile cover component to adapt to both square sidebar and ProfileEditor use. Adapts to suitable sizes in the ProfileModal depending on the presence of a cover image
Moves the DM lure link/QR code component out of a sheet and into its own view

PR Checklist:

[ ] Includes changes to desk files
[x] Tested locally using the Vite dev proxy against a livenet planet in (A) a browser and (B) the iOS simulator using the tlon-app project
[x] Includes e2e tests to verify (A) presence and placement of AppNav on mobile and desktop and (B) clicking the navigation items navigates to the appropriate app section
[ ] Comments added anywhere logic may be confusing without context

Known issue with the profile menu not scrolling on mobile - will remedy shortly

Thank you for the help @latter-bolden!


bug: Keep cli-domain-from-installer-image until we use minimal hive version

Which issue this PR addresses:
Until we're on a version of hive which supports minimal installs, we need the annotation to pull the cli-image from arosvc, not quay.io
Test plan for issue:
test in INT.
Is there any documentation that needs to be updated for this PR?
Nope

merge on CI passing, e2e doesn't cover this.


perf: Blocksync is double calling types.Block.ValidateBasic at costs to sync speed

Feature Request
Block sync is double-calling types.Block.ValidateBasic, which re-hashes every transaction in a block. In a 100 block CPU profile sync on Osmosis, over a period with especially large blocks, we found that this was the cause of some notable overhead. (18 seconds out of 185 seconds over all processes, at least 70s of which were in non-blocking operations) Under normal block sizes, this time is lower but still high/worth speeding up. (4.1 seconds for 100 blocks)

We should not be double verifying this.
Summary
We are double verifying block hashes in block sync by calling this in two different spots:
ValidateMsg here, which calls types.BlockFromProto which calls validate basic, and types.BlockFromProto right here
Problem Definition
Speed up block sync
Proposal
Do some mild API or order of operations changes to reuse the types.BlockFromProto result and only compute it once.

gonna check this out.

Got it :).
I think that this will make your flame graph much happier.

Thanks @ValarDragon ❤ Definitely something we should fix


Template logic?

Does this support template logic such as if, each, etc?

Hey @chrisabrams,
Starplate template strings are just ES6 template strings. You have access to javascript with the ${ ... } notation. In the event you construct a template string from a template string the #{ ... } notation will also work to escape javascript to prevent it from being executed. Consider the following example:
import { Template } from 'starplate'
let template = new Template(`
  hello #{name}
`)

template.render({name: 'alice'})

Using ${name}  instead of #{name} would result in immediate evaluation of the variable name where as #{name} escapes evaluation instead.
Simple approach
Logic can be achieved with ternary operators ?:.
Looping can be achieved with the methods on the Array prototype object.
Ternary logic
let template = new Template(`
   Hello #{name ? name : 'everyone'}
`)

Array looping
let template = new Template(`

    #{names.map(name =>  \`<li> #{name} </li>\`)}

`)

Advanced approach
Closures are a good way of representing advanced logic where ternary and array methods are too awkward.
let template = new Template(`

Animals

  #{(() => {}
    let animals = ['kinkajou', 'monkey', 'raccoon', 'dog']
    let excluded = new Set(['raccoon'])
    let createElement = name => '
'+ name +'
'
    return animals
    .filter(animal => false == excluded.has(animal))
    .map(animal => createElement(animal))
  )()}

`)

@chrisabrams it may be worth it for you to check out https://github.com/littlstar/stardux
You can move logic to reducers in stardux which are just redux reducers

@jwerle best answer I've ever gotten for an issue :+1:

@chrisabrams no problem !


Revert "::Rake::CommandFailedError doesn't exist."

This reverts commit c5a889ce865cc0314597e9de11e2bee366e797ac.

This fixes #5072.

@homu r+

:pushpin: Commit cc1cfb7 has been approved by indirect

:hourglass: Testing commit cc1cfb7 with merge ad3a2a2...

:broken_heart: Test failed - status

@homu retry

:hourglass: Testing commit cc1cfb7 with merge 061e68f...

:sunny: Test successful - status


fix: corrected exports in package.json

Updated package.json, export paths did not include the dist folder causing an 'ERR MODULE NOT FOUND' when trying import the register script

this is one way around it. the other way would be to add the root register.js and register.d.ts to the files array
"files": [
    "dist/*",
    "!dist/.tsbuildinfo",
    "register.js",
    "register.d.ts"
],

i am just not sure which impl @viztea will prefer

will release new versions later 👍🏼


loctool: Add support for converting string target with HTML tag nodes to plural.

Summary: Added support for converting string targets containing HTML tags to plural resources. This includes handling both simple and nested HTML tags within the strings, such as:

There is string.
There is string.

Also, please rebase onto main again to get Wadim's latest fixes for circleci builds


Issue when running linregress()

Hi, I am a newbie to Julia, and I ran into this problem when trying to do a linear regression to two data arrays. Could you please give me more hint on how to solve this? Thank you so much!

Hi, could you turn this into a minimal reproducible example (include some definitions of the variables you're using, e.g. defined using rand())?

The problem is that y isa Vector{Any}, which means that the QR factorization fails because oneunit(Any) is not defined. If you broadcast something over your y, assuming your vector contains only numbers, it should work, i.e. I would try doing
lr = linregress(time_list_05[1:1:end]*tf/length(sol_05), something.(P_05[1:1:end]))


The question about Hudi transform timestamp and decimal's type

I have used hudi for a few weeks. During these weeks, I learned a lot about hudi in detail, but there's a problem that's been bothering me for a long time. Timestamp and Decimal types from avro to parquet in spark2.1-2.3 by spark-avro project[4.0.0] are replaced to long and string. I think these two types should be correctly transformed.
So I started reading hudi's/spark-avro's codes and trying to change some codes to solve this problem. My basic environment is spark2.2.1(parquet1.8.2 and replace avro1.7 jars to 1.8.2), hadoop2.6.0 and hive1.2.1.
I had written some solutions in this issue https://github.com/uber/hudi/issues/517#issuecomment-448506773 . But it can only solve part of the problem(timestamp and date).
In parquet decimal is stored as 'fixed_len_byte_array', so we may change decimal col's schema to fixed.
Then I read spark2.4 code about avro and take some codes to spark-avro(like AvroSerializer.scala in https://github.com/apache/spark/tree/branch-2.4/external/avro/src/main/scala/org/apache/spark/sql/avro). I change createConverterToAvro's  function DecimalType case at the same time.
spark-avro's change in my fork:
https://github.com/cdmikechen/spark-avro/tree/branch-4.0
hudi's change about adding avro schema and decimal fixed type:
val genericRecords: RDD[GenericRecord] = AvroConversionUtils.createRdd(df, schema.toString(), structName, nameSpace)

def createRdd(df: DataFrame, avroSchemaAsJsonString : String, structName: String, recordNamespace: String): RDD[GenericRecord] = {
  val dataType = df.schema
  val encoder = RowEncoder.apply(dataType).resolveAndBind()
  df.queryExecution.toRdd.map(encoder.fromRow)
    .mapPartitions { records =>
    if (records.isEmpty) Iterator.empty
    else {
       val avroSchema = new Schema.Parser().parse(avroSchemaAsJsonString)
       val convertor = createConverterToAvro(avroSchema, dataType, structName, recordNamespace)
      records.map { x => convertor(x).asInstanceOf[GenericRecord] }
    }
  }
}

def createConverterToAvro(avroSchema: Schema,
                          dataType: DataType,
                          structName: String,
                          recordNamespace: String): Any => Any = {
  dataType match {
    case BinaryType => (item: Any) =>
      item match {
        case null => null
        case bytes: Array[Byte] => ByteBuffer.wrap(bytes)
      }
    case ByteType | ShortType | IntegerType | LongType |
         FloatType | DoubleType | StringType | BooleanType => identity
    case dec: DecimalType => (item: Any) =>
      Option(item).map { i =>
          val bigDecimalValue = item.asInstanceOf[java.math.BigDecimal]
          val decimalConversions = new DecimalConversion()
          decimalConversions.toFixed(bigDecimalValue, avroSchema.getField(structName).schema().getTypes.get(0),
            LogicalTypes.decimal(dec.precision, dec.scale))
        }.orNull
    case TimestampType => (item: Any) =>

The code for the current solution is still rough. I will try my best to improve this part in the next time.
But after looking at spark-avro project, I feel that we could actually combine into our project(It has only a few classes) and then modify or upgrade codes according to our situation.

@cdmikechen there is a thread on this on the mailing list. Are you still working/hitting this? If so, could you jump onto the thread on ML, so we can hash out a solution

@vinothchandar
Sorry for delay. I've changed my job in February, and my new job works for an automobile company. This company use HDP 3.1 with hadoop3 and spark2.3 and hive3.1. Because the version of hadoop and hive is too high, the new job is not using hudi for the time being.
Recently, my team is going to launch a new data lake project. My work is busy, so I don't have much time to pay attention to it in May. I'll do it in April and add some new research after that.

@cdmikechen no worries.. it would help if you can file a JIRA on hive/hadoop version gaps preventing you from using Hudi now? :)

Congrats on the new gig! :)

What is the status on this - meaning - what is required so that it is fixed  ?   Maybe I'll have a crack at it.

Consensus was that upgrading to spark 2.4 is a good step since it bundles spark-avro with it and also has few fixes around this.
There is a jira for this already. If you can drive this forward, thatd be awesome. I am parallely working on simplifying the pom for moving to spark 2.4 ...
Even if you can clearly summarize problem with a reproducible test file/data, thatd be awesome

@vinothchandar  made this https://github.com/apache/incubator-hudi/pull/960

@arw357 : ACK. I will review this code.

Closing due to inactivity


Agent log configuration problem

What did you do to encounter the bug?
deploy the community-operator chart with the following values:
community-operator-crds:
  enabled: true
operator:
  priorityClassName: infra-priority
  resources:
    limits:
      cpu: "2"
      memory: 1Gi
    requests:
      cpu: 10m
      memory: 100Mi

and deploy a mongodb cluster with the following config:
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
    name: test-mongodb
    namespace: test
spec:
    members: 1
    type: ReplicaSet
    version: 7.0.11
    security:
        authentication:
            modes:
                - SCRAM
    statefulSet:
        spec:
            template:
                metadata:
                    labels:
                        role: dbserver
                        type: mongodb
                spec:
                    agent:
                      logLevel: "DEBUG"
                      logFile: "/var/log/mongodb-mms-automation/automation-agent-test.log"
                      logRotate:
                        numTotal: 5
                        numUncompressed: 1
                        sizeThresholdMB: "1"
                        timeThresholdHrs: 1
                    containers:
                        - name: mongod
                          resources:
                            limits:
                                cpu: "1"
                                memory: 1Gi
                            requests:
                                cpu: 10m
                                memory: 100Mi
                        - name: mongodb-agent
                          resources:
                            limits:
                                cpu: "1"
                                memory: 1Gi
                            requests:
                                cpu: 10m
                                memory: 100Mi
                    topologySpreadConstraints:
                        - labelSelector:
                            matchLabels:
                                type: mongodb
                          maxSkew: 1
                          topologyKey: topology.kubernetes.io/zone
                          whenUnsatisfiable: DoNotSchedule
                    affinity:
                        nodeAffinity:
                            preferredDuringSchedulingIgnoredDuringExecution:
                                - weight: 100
                                  preference:
                                    matchExpressions:
                                        - key: test.help/type
                                          operator: In
                                          values:
                                            - application
                    serviceAccountName: test-mongodb-sa
                    tolerations:
                        - key: node.kubernetes.io/not-ready
                          operator: Exists
                          effect: NoExecute
                          tolerationSeconds: 5
                        - key: node.kubernetes.io/unreachable
                          operator: Exists
                          effect: NoExecute
                          tolerationSeconds: 5
                    priorityClassName: infra-priority
            volumeClaimTemplates:
                - kind: PersistentVolumeClaim
                  apiVersion: v1
                  metadata:
                    name: data-volume
                    namespace: test
                  spec:
                    accessModes:
                        - ReadWriteOnce
                    storageClassName: nfs-csi
                    resources:
                        requests:
                            storage: 10G
                - kind: PersistentVolumeClaim
                  apiVersion: v1
                  metadata:
                    name: logs-volume
                    namespace: test
                  spec:
                    accessModes:
                        - ReadWriteOnce
                    storageClassName: nfs-csi
                    resources:
                        requests:
                            storage: 10G
    users:
    ...

What did you expect?
I would expect the agent's log configuration to be applied
....
                spec:
                    agent:
                      logLevel: "DEBUG"
                      logFile: "/var/log/mongodb-mms-automation/automation-agent-test.log"
                      logRotate:
                        numTotal: 5
                        numUncompressed: 1
                        sizeThresholdMB: "1"
                        timeThresholdHrs: 1
....

What happened instead?
The specified configuration does not apply. For example, the log file remains the default one /var/log/mongodb-mms-automation/automation-agent.log, instead of the one specified /var/log/mongodb-mms-automation/automation-agent-test.log
Kubernetes Cluster Information
kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

helm list
NAME             	NAMESPACE	REVISION	UPDATED                                 	STATUS  	CHART                    	APP VERSION
mongodb          	test    	1       	2024-08-08 13:31:24.882820339 +0200 CEST	deployed	community-operator-0.10.0	0.10.0

I'm probably setting something wrong, but I can't figure out what it is.
Thank you.

Have the exact same problem. Have you been able to fix it ? @vramperez

I am able to run version 5.0.28 successfully.  Its definitely not relates to AVX also.

MongoDB Kubernetes Operator 0.10.0 works fine. The migration to UBI based images in 0.11.0 seems to be broken.

No news from my side. Finally what I have done is to override the agent command in the chart to be able to pass the configuration as arguments to the agent directly. I don't like that solution, but for the time being, until it is fixed I have had to opt for this workaround.