[EPIC] Fixes, Enhancements and Improvements for CatFIM

This is an EPIC card. As items are taken off this list, they will get moved into their own cards, referenced back to this as a total list of all known items, mostly are brand new so prioritization is required
Currently, the CatFIM process takes a very long time to run (~25 hrs for a full BED run) and lacks efficiency both for full runs and for partial/testing runs. Update the CatFIM scripts to incorporate more efficient multiprocessing and segmented run capabilities (i.e. running only on a list of HUCs and/or a flag that allows CatFIM to re-run without downloading all of the metadata again).
These are above and beyond finishing the AK integration
Obviously, we won't get all in, but this is a list of all possible fixes. We will extract out some for the next CatFIM release when the AK integration is done.  However, some in this list are very important.
Potential CatFIM Fixes and Upgrades:

(Lower priority, should be fixed by AWS shift) Improve efficiency for calling the WRDS API. In our "get_thresholds" in generate_categorical_fim_flows.py, it calls WRDS to get some nws_lid but only passes in lid data and not huc data. BUT... our process_generate_flow function, it calls that get_threshhold for every huc and every ahps site. This is a massive amount of duplication. We can call WRDS for each ahps or even all ahps via the specific WRDS api call, save it to our disk and use it. We can likely even load it in certain parts of the code and pass that dataframe around to other functions including into a Multi-Proc. This would be a huge performance gain.

(Lower priority, should be fixed by AWS shift) Facilitate a pre-download option for the threshold files (similar to what we've done with the metadata files). We now have the ability to save metadata file from WRDS in generate_categorical_fim_flows.py but can we make a similar system for the threshold files so we can run catfim tools in a non OWP server enviro? Assuming that the fix has not yet been made to let AWS all the WRDS api.  Can we just get a new DB or something we can put on a file server instead of making 5,000+ Web API calls which is a huge amount of overhead?

Review and maybe upgrade the HUC_messages system. The system creates a file per HUC / lid which says if if the lid was loaded or if it failed and why. It later reloads all of the files in the "huc_messages" folder, maps them to each record for the output csv with some saying why they failed. There are huge opportunities for improvement here, but it does work so it may not be worth the effort to fix. The current setup is "multi-proc" safe.  Attached is one example of one ahps site that was not fully able to load catfim and the message why. The image is in two parts, limitations of screen caps.

Examples of how huc_messages turn into status and end up in the UI.

(Priority): (1286) Add in an argument to remove intermediate extent files. In the mapping/{huc}/{lid} are a bump of inundated images changed extent files. It is done to each branch, then merged to a single "extent" file for that {huc}/{lid}/{lid}_{magnitude}_extent.tif.  We obviously need to keep the final extent files per magnitude and not all of the branch intermediary files. Note: that system is now in but the code commented for version or two for tracing reasons. Maybe uncomment in the next HV 2.1.8 release (in a few months) to clean up all of those intermediate tif files as there is A LOT.  Update: We have commented out code in there to delete them, just make it driven by an arg to delete or not (default delete). Can helps with debugging during dev.  Add an argument in for deleting or not, default to delete. Nice to have when code/debugging and most of the code is there, easy to finish.

Rob on it

Fix duplication of processing paths for stage and flow based as it calls functions in the code.

(Should be fixed by AWS shift)  Adjust and review the multi-proc job number arguments. We have three job numbers, but we have some parts that are MP, inside MP, inside MP.  ie process_generate_categorical_fim -> (MP) iterate_through_huc_stage_based -> (MP) produce_stage_based_catfim_tifs ->  (MP) produce_inundation_map_with_stage_and_feature_ids. This may not be the most efficient use of jobs or even MP but needs to be reviewed. There are others such as post_process_cat_fim_for_viz -> (MP) post_process_huc_level -> (MP) reformat_inundation_maps

Test and re-eval the "past_major_interval_cap" system. (argument of -mc / --past_major_interval_caps).  Does it still work? Do we want to keep it and maintain that code? -> Yes! (as of 9/11/24) 1283

1283 Test and eval if we want the search system (argument of -s / --search)). Does it still work? Do we want to keep it and maintain that code? -> Yes!

1287 ~Test and fix OR remove the lid choice system (arguments of -l / --lid_to_run). It has been broken since winter 2024. Could be handy but might not be worth the effort to fix it.~ -> 09/11/24 - Decided to remove this feature in lieu of finishing adding the new list of HUCs system, # 10 below.

(Priority) (no card yet) Fix Emily's list of HUCs to run argument (argument of -lh / --lst_hucs). This is very handy, new to CatFIM 2.0 but we temp disabled it, so it should be quick to re-enable. Lower priority for v2.1, but nice to get in there.  Prioritized over #9 as of 9/11/24

For Emily later.

Finish the new "step" system, which allows for us to skip flows if all of the files are already in place to go straight to inundation. Maybe consider adding more "step" points?

Review test / fix using the generate_categorical_fim_mapping.py and generate_catfim_flow.py scripts to see if they work from command line. I rebuilt the generate_categorical_fim_mapping.py part way through the rebuild but did not come back to finish it. The generate_catfim_flow.py script is definitely broken. More of a rewrite task, lower priority as of 9/11/24.

(Priority) (no card needed, ongoing) Clean up doc string notes below most functions. Most are very, very out of date. For now, either finish a doc string section to be correct or remove if not.  Important! But doesn't need to be its own issue. Good to be fixing as we go.

(Priority) 1272  Review and cleanup the lid sites rule such as special cases for lids based on crs, datum_adjust, etc. It is now it's own issue Big deal! Out-of-date hard coding could be causing a lot of the issues in the subsequent bullet points. A note from Carson: Would be good to add "and" statements to make sure that the known issue is true before applying a hard-coded fix. Pseudocode example: if LID == fhjs5 AND datum == NULL, set datum = ‘NAD83’

For Emily later.

Note from Derek:
It would be good to add the AHPS restricted sites list to the catfim folder of our online inundation-mapping repo so that the field can check and provide feedback on whether certain sites can be removed from the list. Would be good to sort the LIDs by RFC and then alphabetical order. new service / feature someday?  xls list? other?

(Priority) 1273.
Review Errors from the <IP_ADDRESS> flow based and stage based full runs.

Rob on it

(Priority) 1274 Review / Fix logic for Stage Based Sea Level sites and applicable ahps site list. Also covers moving the HV applicable ahps list over to CatFIM code .

Rob on it.

(Priority) (no card but done) Ensure that the stage_based_catfim_sites.csv file has the 'nws_lid' column renamed to 'ahps_lid'.

update. Now fixed in new dev-catfim-v2-1 branch.

(Priority) 1275 ***FIM <IP_ADDRESS> - CatFIM stage site issues - jrsu1 and okfi2 (and other misc sites) ***

For Emily later.

(Added Sep 4, 2024) 1277 CatFIM - issues with new docker image.
- update. Now fixed in new dev-catfim-v2-1 branch.

(Added Sep 4, 2024) Review sites that were dropped since <IP_ADDRESS> to <IP_ADDRESS>. They might be related # 1274, or 1277above but might not. Derek wants us to compile and review all. Rob will put a card in for it.  Related: HydroVIS has to parts of there system which can drop ahps sites as well, so we will reconcile that with the  HV team at the same time. In HV, they compare our sites coming in against a DB of their own named something like "ahps_restricted_sites" and drops more. We need to reconcile that as part of this task, plus 1274 and 1275.

update: One of the five places we drop (or change mapped = no) for sites is the HV test against the "aphs_restricted_sites". Derek said it is fine for now but will double check later to see how up to date it is. While we won't worry about the status of the data in the aphs restricted sites db, we will pull that data and logic over to FIM catFIM processing code. 1288.  All part of site reconciliation.
Update: Sep 23: For the "sites" list, it was discovered that if the t does appear that the sites that don't have all 4 thresholds were dropped in this new version.  This is likely incorrect. But watch for <IP_ADDRESS> records where their might be missing threshold stages and all categories failed to map. We will need to confirm rules of when a record shoudl be dropped in relation to partial threshholds. This logic appears to have not changed sine <IP_ADDRESS>, but may be wrong in the first place.
Update: Sep 23: Some HUC8s appear to have changed on some records. Research required.
Update: Sep 23: review actual status msg's.  ie)

<IP_ADDRESS> was: "large discrepancy in elevation estimates from gage and HAND and all categories failed to map".  <IP_ADDRESS> now shows "large discrepancy in elevation estimates from gage and HAND"
<IP_ADDRESS> was "no stage values available and all categories failed to map",  now "no stage values available"
<IP_ADDRESS>: There are a bunch saying status of "undermined". This was a backup system and any record showing that value, means their is a code logic error.
<IP_ADDRESS> was: "likely unacceptable gage datum error or accuracy code(s); please see acceptance criteria and all categories failed to map". <IP_ADDRESS> is "usgs_elev_table missing, likely unacceptable gage datum error -- more details to come in future release"

Update Sep 23:  How can there be any miss matches on AHPS id's. both <IP_ADDRESS> and <IP_ADDRESS>, have some that do not matching ahps_id record.

Update Sep 28: Fixed in dev-catfim-v2-1, other than some outlier issues mentioned above.

*(Added Sep 6, 2024): For the all four .csv's that we create, drop the geometry column and create four more files with the name "_metadata" in them. This will make for smaller versions for purely analyzing non geo columns, like status, is mapped flag, and other values in it. It is purely a debugging tool but only takes 2 line of code to be added and would help quite a bit. Might no longer be needed depending on split library outputs (see # 25 below)

(Added 9/11/24): 1282 Make a CatFIM folder to store the CatFIM scripts!

(Added 9/9/24): 1279 Hawaii points disappeared from stage-based CatFIM in release <IP_ADDRESS> . Keep an eye on other tasks relating to lid reconciliation as they might/might not be related.

Rob to check it, might be fixed in the status fixes from num 20 above.

(Added 09/23/24): Ensure we compare mapped = Yes from out site list, and ensure there is at least one rec in library file.

Emily will look into this.

(Added 10/07/24):  Priority (card coming) Split library csv output files to new folder with each HUC having it's own sites. ie) split to HUC level instead of one giant CONUS+ library csv.

Some of these tidbits were fixed in PR 1165 which is the rebuild of CatFIM. However, this Issue will remain open as there are more things to cover, each a different importance levels.

Notes from 9/11/24 meeting
1 & 2: These issues will probably go away when we move CatFIM to AWS.
4: Should be quick.
5: A simple reorganization task.
6: A larger reorganization task, but will probably not be needed when we move CatFIM to AWS.
7&8: Need too make an issue, we have decided to maintain these functions. Would be good to clarify their function in docs and add comments in the code.
9&10: Choose one because they’re very similar. I think we will choose #10 because it’s closer to finished.
11: Do it!
12: More of a rewrite task, lower priority.
13: Important! But doesn’t need to be its own issue. Rather, this documentation is something that we can and should be fixing as we go.
14: Big! Out-of-date hard coding could be causing a lot of the issues noted in the subsequent bullet points. A note from Carson: Would be good to add “and” statements to make sure that the known issue is true before applying a hard-coded fix. Pseudocode example: if LID == fhjs5 AND datum == NULL, set datum = ‘NAD83’
It would be good to add the AHPS restricted sites list to the catfim folder of our online inundation-mapping repo so that the field can check and provide feedback on whether certain sites can be removed from the list. Would be good to sort the LIDs by RFC and then alphabetical order.
15: New issue: Make a CatFIM folder to store the CatFIM

For Test and fix OR remove the lid choice system (arguments of -l / --lid_to_run). It has been broken since winter 2024. Could be handy but might not be worth the effort to fix it. -> Removed 9/11/24.  I think we decided to drop that system in favor of finishing the filter by huc system (# 10?).  I think it is not wise to leave in a feature that we don't know if it works especially if we are replacing it with # 10 hucs.  I will put in a card to drop that feature.

This might be something we tackle after addressing all of this technical debt, but I wanted to track it here since it's a CatFIM task...
It's been suggested that we map Stage-Based CatFIM for sites that are not forecast sites. There are some AHPS points that are observation-only but still have flood thresholds that we could map. Even more sites for Stage-based (yay!).

Added separate card for future enhancement:  1308  Add lake and levee protected area masking

Previous version of this card before it was reorganized on 12/4/24:

(Lower priority, should be fixed by AWS shift) Improve efficiency for calling the WRDS API. In our "get_thresholds" in generate_categorical_fim_flows.py, it calls WRDS to get some nws_lid but only passes in lid data and not huc data. BUT... our process_generate_flow function, it calls that get_threshhold for every huc and every ahps site. This is a massive amount of duplication. We can call WRDS for each ahps or even all ahps via the specific WRDS api call, save it to our disk and use it. We can likely even load it in certain parts of the code and pass that dataframe around to other functions including into a Multi-Proc. This would be a huge performance gain.

(Lower priority, should be fixed by AWS shift) Facilitate a pre-download option for the threshold files (similar to what we've done with the metadata files). We now have the ability to save metadata file from WRDS in generate_categorical_fim_flows.py but can we make a similar system for the threshold files so we can run catfim tools in a non OWP server enviro? Assuming that the fix has not yet been made to let AWS all the WRDS api.  Can we just get a new DB or something we can put on a file server instead of making 5,000+ Web API calls which is a huge amount of overhead?

Review and maybe upgrade the HUC_messages system. The system creates a file per HUC / lid which says if if the lid was loaded or if it failed and why. It later reloads all of the files in the "huc_messages" folder, maps them to each record for the output csv with some saying why they failed. There are huge opportunities for improvement here, but it does work so it may not be worth the effort to fix. The current setup is "multi-proc" safe.  Attached is one example of one ahps site that was not fully able to load catfim and the message why. The image is in two parts, limitations of screen caps.

Examples of how huc_messages turn into status and end up in the UI.

(1286)  Add in an argument to remove intermediate extent files.*** In the mapping/{huc}/{lid} are a bump of inundated images changed extent files. It is done to each branch, then merged to a single "extent" file for that {huc}/{lid}/{lid}_{magnitude}_extent.tif.  We obviously need to keep the final extent files per magnitude and not all of the branch intermediary files. Note: that system is now in but the code commented for version or two for tracing reasons. Maybe uncomment in the next HV 2.1.8 release (in a few months) to clean up all of those intermediate tif files as there is A LOT.  Update: We have commented out code in there to delete them, just make it driven by an arg to delete or not (default delete). Can helps with debugging during dev.  Add an argument in for deleting or not, default to delete. Nice to have when code/debugging and most of the code is there, easy to finish.

Update: Did not make this 2.1 version. Held back for debugging and tracing purposes. (aka.. the delete cleanup of the tif files.

Fix duplication of processing paths for stage and flow based as it calls functions in the code.

(Should be fixed by AWS shift)  Adjust and review the multi-proc job number arguments. We have three job numbers, but we have some parts that are MP, inside MP, inside MP.  ie process_generate_categorical_fim -> (MP) iterate_through_huc_stage_based -> (MP) produce_stage_based_catfim_tifs ->  (MP) produce_inundation_map_with_stage_and_feature_ids. This may not be the most efficient use of jobs or even MP but needs to be reviewed. There are others such as post_process_cat_fim_for_viz -> (MP) post_process_huc_level -> (MP) reformat_inundation_maps

Test and re-eval the "past_major_interval_cap" system. (argument of -mc / --past_major_interval_caps).  Does it still work? Do we want to keep it and maintain that code? -> Yes! (as of 9/11/24) 1283

1283 Test and eval if we want the search system (argument of -s / --search)). Does it still work? Do we want to keep it and maintain that code? -> Yes!

1287 ~Test and fix OR remove the lid choice system (arguments of -l / --lid_to_run). It has been broken since winter 2024. Could be handy but might not be worth the effort to fix it.~ -> 09/11/24 - Decided to remove this feature in lieu of finishing adding the new list of HUCs system, # 10 below.

(Priority) (Issue 1311) Fix Emily's list of HUCs to run argument (argument of -lh / --lst_hucs). This is very handy, new to CatFIM 2.0 but we temp disabled it, so it should be quick to re-enable. Lower priority for v2.1, but nice to get in there.  Prioritized over #9 as of 9/11/24

For Emily later. Issue 1311
update. Now fixed in new dev-catfim-v2-1 branch.    -

Finish the new "step" system, which allows for us to skip flows if all of the files are already in place to go straight to inundation. Maybe consider adding more "step" points?

update. Now added in new dev-catfim-v2-1 branch.

Review test / fix using the generate_categorical_fim_mapping.py and generate_catfim_flow.py scripts to see if they work from command line. I rebuilt the generate_categorical_fim_mapping.py part way through the rebuild but did not come back to finish it. The generate_catfim_flow.py script is definitely broken. More of a rewrite task, lower priority as of 9/11/24.

(Priority) (no card needed, ongoing) Clean up doc string notes below most functions. Most are very, very out of date. For now, either finish a doc string section to be correct or remove if not.  Important! But doesn't need to be its own issue. Good to be fixing as we go. More to be done but we can call this done for CatFIM 2.1.

(Priority) 1272  Review and cleanup the lid sites rule such as special cases for lids based on crs, datum_adjust, etc. It is now it's own issue Big deal! Out-of-date hard coding could be causing a lot of the issues in the subsequent bullet points. A note from Carson: Would be good to add "and" statements to make sure that the known issue is true before applying a hard-coded fix. Pseudocode example: if LID == fhjs5 AND datum == NULL, set datum = ‘NAD83’

Done in CatFIM 2.1

Note from Derek:
It would be good to add the AHPS restricted sites list to the catfim folder of our online inundation-mapping repo so that the field can check and provide feedback on whether certain sites can be removed from the list. Would be good to sort the LIDs by RFC and then alphabetical order. new service / feature someday?  xls list? other?

Done. It now exists in the repo at /tools/catfim/stage_based_ahps_restricted_sites.csv.  For future releases, it can be updated by others and updated in the repo.

(Priority) 1273.
Review Errors from the <IP_ADDRESS> flow based and stage based full runs.

Update: Oct 21, 2024: Done. Ran a set of all stage based dropped sites with newest code and reviewed it. A good handful 40-50% ?? are now fixed. They were dropped due to the bug where one missing stage failed the record. Now fixed.

(Priority) 1274 Review / Fix logic for Stage Based Sea Level sites and applicable ahps site list. Also covers moving the HV applicable ahps list over to CatFIM code .

Fixed and in 2.1 branch. Updated to a csv in code that has the list of stage_based_aphs_restricted_sites.csv that the code can use to delimited sites and use it's reason back to our outputs. This also allows RFC's to review and update it for future releases easily.

(Priority) (no card but done) Ensure that the stage_based_catfim_sites.csv file has the 'nws_lid' column renamed to 'ahps_lid'.

update. Now fixed in new dev-catfim-v2-1 branch.

(Priority) 1275 ***FIM <IP_ADDRESS> - CatFIM stage site issues - jrsu1 and okfi2 (and other misc sites) ***

For Emily later.

(Added Sep 4, 2024) 1277 CatFIM - issues with new docker image.
- update. Now fixed in new dev-catfim-v2-1 branch.

(Added Sep 4, 2024) Review sites that were dropped since <IP_ADDRESS> to <IP_ADDRESS>. They might be related # 1274, or 1277above but might not. Derek wants us to compile and review all. Rob will put a card in for it.  Related: HydroVIS has to parts of there system which can drop ahps sites as well, so we will reconcile that with the  HV team at the same time. In HV, they compare our sites coming in against a DB of their own named something like "ahps_restricted_sites" and drops more. We need to reconcile that as part of this task, plus 1274 and 1275.

update: One of the five places we drop (or change mapped = no) for sites is the HV test against the "aphs_restricted_sites". Derek said it is fine for now but will double check later to see how up to date it is. While we won't worry about the status of the data in the aphs restricted sites db, we will pull that data and logic over to FIM catFIM processing code. 1288.  All part of site reconciliation.
Update: Sep 23: For the "sites" list, it was discovered that if the t does appear that the sites that don't have all 4 thresholds were dropped in this new version.  This is likely incorrect. But watch for <IP_ADDRESS> records where their might be missing threshold stages and all categories failed to map. We will need to confirm rules of when a record shoudl be dropped in relation to partial threshholds. This logic appears to have not changed sine <IP_ADDRESS>, but may be wrong in the first place.
Update: Sep 23: Some HUC8s appear to have changed on some records. Research required.
Update: Sep 23: review actual status msg's.  ie)

<IP_ADDRESS> was: "large discrepancy in elevation estimates from gage and HAND and all categories failed to map".  <IP_ADDRESS> now shows "large discrepancy in elevation estimates from gage and HAND"
<IP_ADDRESS> was "no stage values available and all categories failed to map",  now "no stage values available"
<IP_ADDRESS>: There are a bunch saying status of "undermined". This was a backup system and any record showing that value, means their is a code logic error.
<IP_ADDRESS> was: "likely unacceptable gage datum error or accuracy code(s); please see acceptance criteria and all categories failed to map". <IP_ADDRESS> is "usgs_elev_table missing, likely unacceptable gage datum error -- more details to come in future release"

Update Sep 23:  How can there be any miss matches on AHPS id's. both <IP_ADDRESS> and <IP_ADDRESS>, have some that do not matching ahps_id record.
Update Sep 28: Fixed in dev-catfim-v2-1, other than some outlier issues mentioned above.

(Added Sep 6, 2024): For the all four .csv's that we create, drop the geometry column and create four more files with the name "_metadata" in them. This will make for smaller versions for purely analyzing non geo columns, like status, is mapped flag, and other values in it. It is purely a debugging tool but only takes 2 line of code to be added and would help quite a bit. Might no longer be needed depending on split library outputs (see # 25 below)

update Oct 31: Skipped for now.

(Added 9/11/24): 1282 Make a CatFIM folder to store the CatFIM scripts!

update. Now fixed in new dev-catfim-v2-1 branch.

(Added 9/9/24): 1279 Hawaii points disappeared from stage-based CatFIM in release <IP_ADDRESS> . Keep an eye on other tasks relating to lid reconciliation as they might/might not be related.

Rob to check it, might be fixed in the status fixes from num 20 above.
Update Oct 18: Some what put on hold. There is some weirdness in the FIM code creating the usgs_elev_tables.csv for HUCS. A separate card is created for that problem. However.. we now estimate that just two Hawaii hucs might make it through which is KMSH1 and WRSH1. Testing in progress. Update: Those failed for other reasons. None remained from 20060000. Will try other Hawaii hucs later.
Update: Oct 21: This is not a FIM bug, but a WRDS issue and is impacting the usgs_guages.gpkg that FIM uses to create the usgs_elev_tables. Topic resolved.

(Added 09/23/24): Ensure we compare mapped = Yes from out site list, and ensure there is at least one rec in library file.

Done

(Added 10/07/24):  Priority (card coming) Split library csv output files to new folder with each HUC having it's own sites. ie) split to HUC level instead of one giant CONUS+ library csv.

Update. 10/28/24: (1331)[https://github.com/NOAA-OWP/inundation-mapping/issues/1331]:  Proved we must split the outputs library files. Stage based will be a fair bit bigger than previous CatFIM runs, for various reasons. Merge all library files to be HUC2 grouped.
Update 10/31/24: Found splitting creates even more problems especially that the catfim compare tool won't work. That tool can compare one set of inputs from previous versions to this version.

(Completed 11/5/24) Fix CatFIM duplicate records (Issue 1336)[https://github.com/NOAA-OWP/inundation-mapping/issues/1336]

27). (Issue 1353)[https://github.com/NOAA-OWP/inundation-mapping/issues/1353] Ensure CatFIM key column names are consistent.

Archive - Completed or Tabled in CatFIM v2.1 :

*(Tabled in CatFIM v2.1)  1287 ~Test and fix OR remove the lid choice system (arguments of -l / --lid_to_run). It has been broken since winter 2024. Could be handy but might not be worth the effort to fix it.~ -> 09/11/24 - Decided to remove this feature in lieu of finishing adding the new list of HUCs system, # 10 below.

(Resolved in CatFIM v2.1) (Issue 1311) Fix Emily's list of HUCs to run argument (argument of -lh / --lst_hucs). This is very handy, new to CatFIM 2.0 but we temp disabled it, so it should be quick to re-enable. Lower priority for v2.1, but nice to get in there.  Prioritized over #9 as of 9/11/24

(Resolved in CatFIM v2.1) Finish the new "step" system, which allows for us to skip flows if all of the files are already in place to go straight to inundation. Maybe consider adding more "step" points?

update. Now added in new dev-catfim-v2-1 branch.

(Resolved, for now, in CatFIM v2.1) (no card needed, ongoing) Clean up doc string notes below most functions. Most are very, very out of date. For now, either finish a doc string section to be correct or remove if not.  Important! But doesn't need to be its own issue. Good to be fixing as we go. More to be done but we can call this done for CatFIM 2.1.

(Resolved in CatFIM v2.1) 1272  Review and cleanup the lid sites rule such as special cases for lids based on crs, datum_adjust, etc. It is now it's own issue Big deal! Out-of-date hard coding could be causing a lot of the issues in the subsequent bullet points. A note from Carson: Would be good to add "and" statements to make sure that the known issue is true before applying a hard-coded fix. Pseudocode example: if LID == fhjs5 AND datum == NULL, set datum = ‘NAD83’

Note from Derek:
It would be good to add the AHPS restricted sites list to the catfim folder of our online inundation-mapping repo so that the field can check and provide feedback on whether certain sites can be removed from the list. Would be good to sort the LIDs by RFC and then alphabetical order. new service / feature someday?  xls list? other?

Done. It now exists in the repo at /tools/catfim/stage_based_ahps_restricted_sites.csv.  For future releases, it can be updated by others and updated in the repo.

(Resolved in CatFIM v2.1) 1273.
Review Errors from the <IP_ADDRESS> flow based and stage based full runs.

Update: Oct 21, 2024: Done. Ran a set of all stage based dropped sites with newest code and reviewed it. A good handful 40-50% ?? are now fixed. They were dropped due to the bug where one missing stage failed the record. Now fixed.

(Resolved in CatFIM v2.1) 1274 Review / Fix logic for Stage Based Sea Level sites and applicable ahps site list. Also covers moving the HV applicable ahps list over to CatFIM code .

Fixed and in 2.1 branch. Updated to a csv in code that has the list of stage_based_aphs_restricted_sites.csv that the code can use to delimited sites and use it's reason back to our outputs. This also allows RFC's to review and update it for future releases easily.

(Resolved in CatFIM v2.1) (no card but done) Ensure that the stage_based_catfim_sites.csv file has the 'nws_lid' column renamed to 'ahps_lid'.

(Resolved in CatFIM v2.1) 1277 CatFIM - issues with new docker image.

(Resolved in CatFIM v2.1) Review sites that were dropped since <IP_ADDRESS> to <IP_ADDRESS>. They might be related # 1274, or 1277above but might not. Derek wants us to compile and review all. Rob will put a card in for it.  Related: HydroVIS has to parts of there system which can drop ahps sites as well, so we will reconcile that with the  HV team at the same time. In HV, they compare our sites coming in against a DB of their own named something like "ahps_restricted_sites" and drops more. We need to reconcile that as part of this task, plus 1274 and 1275.

update: One of the five places we drop (or change mapped = no) for sites is the HV test against the "aphs_restricted_sites". Derek said it is fine for now but will double check later to see how up to date it is. While we won't worry about the status of the data in the aphs restricted sites db, we will pull that data and logic over to FIM catFIM processing code. 1288.  All part of site reconciliation.
Update: Sep 23: For the "sites" list, it was discovered that if the t does appear that the sites that don't have all 4 thresholds were dropped in this new version.  This is likely incorrect. But watch for <IP_ADDRESS> records where their might be missing threshold stages and all categories failed to map. We will need to confirm rules of when a record shoudl be dropped in relation to partial threshholds. This logic appears to have not changed sine <IP_ADDRESS>, but may be wrong in the first place.
Update: Sep 23: Some HUC8s appear to have changed on some records. Research required.
Update: Sep 23: review actual status msg's.  ie)

<IP_ADDRESS> was: "large discrepancy in elevation estimates from gage and HAND and all categories failed to map".  <IP_ADDRESS> now shows "large discrepancy in elevation estimates from gage and HAND"
<IP_ADDRESS> was "no stage values available and all categories failed to map",  now "no stage values available"
<IP_ADDRESS>: There are a bunch saying status of "undermined". This was a backup system and any record showing that value, means their is a code logic error.
<IP_ADDRESS> was: "likely unacceptable gage datum error or accuracy code(s); please see acceptance criteria and all categories failed to map". <IP_ADDRESS> is "usgs_elev_table missing, likely unacceptable gage datum error -- more details to come in future release"

Update Sep 23:  How can there be any miss matches on AHPS id's. both <IP_ADDRESS> and <IP_ADDRESS>, have some that do not matching ahps_id record.
Update Sep 28: Fixed in dev-catfim-v2-1, other than some outlier issues mentioned above.

(Tabled in CatFIM v2.1): For the all four .csv's that we create, drop the geometry column and create four more files with the name "_metadata" in them. This will make for smaller versions for purely analyzing non geo columns, like status, is mapped flag, and other values in it. It is purely a debugging tool but only takes 2 line of code to be added and would help quite a bit. Might no longer be needed depending on split library outputs (see # 25 below)

update Oct 31: Skipped for now.

(Resolved in CatFIM v2.1): 1282 Make a CatFIM folder to store the CatFIM scripts!

(Resolved in CatFIM v2.1): 1279 Hawaii points disappeared from stage-based CatFIM in release <IP_ADDRESS> . Keep an eye on other tasks relating to lid reconciliation as they might/might not be related.

Rob to check it, might be fixed in the status fixes from num 20 above.
Update Oct 18: Some what put on hold. There is some weirdness in the FIM code creating the usgs_elev_tables.csv for HUCS. A separate card is created for that problem. However.. we now estimate that just two Hawaii hucs might make it through which is KMSH1 and WRSH1. Testing in progress. Update: Those failed for other reasons. None remained from 20060000. Will try other Hawaii hucs later.
Update: Oct 21: This is not a FIM bug, but a WRDS issue and is impacting the usgs_guages.gpkg that FIM uses to create the usgs_elev_tables. Topic resolved.

(Resolved in CatFIM v2.1): Ensure we compare mapped = Yes from out site list, and ensure there is at least one rec in library file.

(Resolved in CatFIM v2.1): Split library csv output files to new folder with each HUC having it's own sites.*** ie) split to HUC level instead of one giant CONUS+ library csv.

Update. 10/28/24: (1331)[https://github.com/NOAA-OWP/inundation-mapping/issues/1331]:  Proved we must split the outputs library files. Stage based will be a fair bit bigger than previous CatFIM runs, for various reasons. Merge all library files to be HUC2 grouped.
Update 10/31/24: Found splitting creates even more problems especially that the catfim compare tool won't work. That tool can compare one set of inputs from previous versions to this version.

(Resolved in CatFIM v2.1) Fix CatFIM duplicate records (Issue 1336)[https://github.com/NOAA-OWP/inundation-mapping/issues/1336]


update nlp-feature-extractors readme, dependencies and add a release script

Description
Several updates to the new nlp-feature-extractors package:

Update the readme to be more similar to our other packages
Update the dependencies to be compatible with our packages
Add a release script for the package and remove the old one which won't work anymore
Cleanup released package metdata to be more in-line with our other python packages

Checklist

[ ] I have added screenshots above for all UI changes.
[ ] I have added e2e tests for all UI changes.
[x] Documentation was updated if it was needed.

Codecov Report

Merging #1666 (cc5ad2e) into main (e2ca0ae) will increase coverage by 6.68%.
The diff coverage is n/a.

:exclamation: Current head cc5ad2e differs from pull request most recent head 77b8991. Consider uploading reports for the commit 77b8991 to get more accurate results

@@            Coverage Diff             @@
##             main    #1666      +/-   ##
==========================================
+ Coverage   85.61%   92.29%   +6.68%
==========================================
  Files          58       17      -41
  Lines        3017     1298    -1719
==========================================
- Hits         2583     1198    -1385
+ Misses        434      100     -334

Flag
Coverage Δ

unittests
92.29% <ø> (+6.68%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

raiutils/raiutils/common/retries.py

raiutils/raiutils/__init__.py

rai_core_flask/rai_core_flask/__init__.py

...iwidgets/raiwidgets/fairness_metric_calculation.py

raiutils/raiutils/dataset/dataset.py

raiutils/raiutils/webservice/webservice.py

raiutils/raiutils/data_processing/__init__.py

..._core_flask/rai_core_flask/environment_detector.py

...iwidgets/raiwidgets/explanation_dashboard_input.py

...sk/rai_core_flask/environments/base_environment.py

... and 31 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Help in Typl ESLint Plugin

Hi @getify
I'm working on eslint plugin for your nice type checker 🌟 .
I'm confused how can I push the errors into the eslint API 😕.
I think that doing it as so in ESLint will give the Developer good experience to integrate with Editors and CI tasks. So I went with 0 rules and I'm trying to use a processor to type check the files and I get interesting data from you checker!
I hope you can guide me a little bit on how I can continue with this approach.
Thanks in advance.

I used this.errors here because I couldn't understand how should I communicate between these two functions.

Give me more time to have a look on how I can implement this.

Thanks for your willingness to contribute! The idea of integrating with eslint is very much a good idea, but before we can do that we have a lot of work to do changing the interaction with the check() method (what is returned) and especially to work out multi-file checking in Typl. I recommend holding off on this plugin for a bit until that settles down. :)

😍 I actually added the loc in the outputErrorMessages object.

I'd like to contribute wherever I can to this awesome Repo. Is there any guidelines or some advices on how to do so?

Can you email me (my same handle on gmail) and we can discuss further?

Sure. Will send to you 👍

🌟

@Attrash-Islam
Where eslint-plugin-typl is living?

It's a private repo. Can't publish it yet since it need a lot of work and a changes in the typL core that was tested locally. Also the linter don't support the typL annotation syntax completely. I can share it with you but won't publish it publicly since it's not ready yet


Recursively find parent for callbacks

I have a fairly simple component set up like so:
<template>
    <row>
        <column>
            <vuetable url="..."></vuetable>
        </column>
    </row>
</template>

<script>
    export default {
        data () => {
            return {
                fields: [{name: "test", callback: "test"}]
            };
        },
        methods: {
            test: (value) => value.split('').reverse().join('')
        }
    }
</script>

When trying to call the callback, the following code is triggered in VueTable:
    callCallback: function(field, item) {
      if ( ! this.hasCallback(field)) return

      if(typeof(field.callback) == 'function') {
       return field.callback(this.getObjectValue(item, field.name))
      }

      let args = field.callback.split('|')
      let func = args.shift()

      if (typeof this.$parent[func] === 'function') {
        let value = this.getObjectValue(item, field.name)

        return (args.length > 0)
          ? this.$parent[func].apply(this.$parent, [value].concat(args))
          : this.$parent[func].call(this.$parent, value)
      }

      return null
    },

The relevant line here is if(typeof this.$parent[func] === 'function')... Because this.$parent refers to the column, not my custom component. And of course <column> doesn't have a method called test
This is easily resolved with something like the following:
let elem = this.$parent;
while (elem.$parent && !elem[func]) {
    elem = elem.$parent;
}

This would travel up the component tree until it found one where the callback was defined. Of course performing this while loop per row could lead to some gross performance penalties, so you may want to add caching of the callback afterwards.
Just a thought for an improvement

@stevendesu Thanks for you suggestions and contributions later. (Yeah, I've noticed) :) Even though, I do not have time to follow up on them at the moment, but I did read and really appreciate it.
When I first implemented this, I had a mixed feeling about it. Trying to contain everything within the component itself and exposes its functionalities to the externals via api and events.
As in using jquery, I see this as a convenient feature to automatically look up its parent for override-able function, but I'm trying not to go beyond its immediate parent as I usually don't have a need for that. And I really don't know if there will be any case that requires going up the chain or not. Because if uses incorrectly, it will be very hard to find the problem.
What's your opinion on this?

Thinking about it for some time, I feel like the proper solution is instead of passing the name of the callback, the actual callback should be passed in. This way the component is handed everything it needs to operate and does not need to go looking for it in the parent.
Looking at the code, this is actually how it's written -- if a function is passed instead of a name, the function is executed. I might update the tutorial, however, to make it clear that this is possible (and also the preferred method, as it would be more performant)
https://github.com/ratiw/vuetable-2-tutorial/blob/master/doc/lesson-06.md
Currently the tutorial only demonstrates passing a string for the callback


Cant start the Add-on on my Deye 12kW 3phases

Issue related to
Sunsynk / mbusd Home Assistant Add-On
Describe the issue/bug
My issue is that after a proper configuration my Addon stops (HALT).
Expected behavior
Should provide information via Entities
Your environment

Home Assistant version: 2023.8.4

Addon:

Name: sunsynk-multi
Version: 2023-08-17

Inverter:

Make: Deye
Model: 12kW 3phases EU 230V
Firmware: __

Adaptor details:

RS485 to USB

(set and found via Debug)

You configuration

DRIVER: pymodbus
INVERTERS:
  - SERIAL_NR: "12345678"
    HA_PREFIX: SS
    MODBUS_ID: 1
    DONGLE_SERIAL_NUMBER: "0"
    PORT: ""
SENSOR_DEFINITIONS: three-phase
SENSORS:
  - energy_management
  - power_flow_card
SENSORS_FIRST_INVERTER:
  - settings
MANUFACTURER: Deye
READ_ALLOW_GAP: 2
READ_SENSORS_BATCH_SIZE: 20
SCHEDULES:
  - KEY: W
    READ_EVERY: 5
    REPORT_EVERY: 60
    CHANGE_ANY: false
    CHANGE_BY: 80
    CHANGE_PERCENT: 0
NUMBER_ENTITY_MODE: auto
MQTT_HOST: core-mosquitto
MQTT_PORT: 1883
MQTT_USERNAME: mqtt-user
MQTT_PASSWORD: xyz
DEBUG: 1
DEBUG_DEVICE: /dev/serial/by-id/usb-FTDI_FT232R_USB_UART_AQ043VSX-if00-port0

Logs

2023-09-06 16:28:48,022 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day Grid Import (reason: Unit)
2023-09-06 16:28:48,022 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day Load Energy (reason: Unit)
2023-09-06 16:28:48,023 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day PV Energy (reason: Unit)
2023-09-06 16:28:48,023 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: essential_power
2023-09-06 16:28:48,023 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: grid_connected
2023-09-06 16:28:48,023 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for Grid CT power (reason: Unit)
2023-09-06 16:28:48,024 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for Grid frequency (reason: Any unit)
2023-09-06 16:28:48,024 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for Grid power (reason: Unit)
2023-09-06 16:28:48,024 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for Grid voltage (reason: Any unit)
2023-09-06 16:28:48,025 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: inverter_current
2023-09-06 16:28:48,025 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for Inverter power (reason: Unit)
2023-09-06 16:28:48,025 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: load_frequency
2023-09-06 16:28:48,026 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: non_essential_power
2023-09-06 16:28:48,026 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule no_unit used for Overall state (reason: No unit)
2023-09-06 16:28:48,026 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Priority Load (reason: RWSensor)
2023-09-06 16:28:48,027 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for PV1 current (reason: Any unit)
2023-09-06 16:28:48,027 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for PV1 power (reason: Unit)
2023-09-06 16:28:48,027 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for PV1 voltage (reason: Any unit)
2023-09-06 16:28:48,028 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Use Timer (reason: RWSensor)
2023-09-06 16:28:48,028 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Load Limit (reason: RWSensor)
2023-09-06 16:28:48,028 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 Capacity (reason: RWSensor)
2023-09-06 16:28:48,028 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 charge (reason: RWSensor)
2023-09-06 16:28:48,029 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 power (reason: RWSensor)
2023-09-06 16:28:48,029 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 Time (reason: RWSensor)
2023-09-06 16:28:48,030 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 Capacity (reason: RWSensor)
2023-09-06 16:28:48,030 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 charge (reason: RWSensor)
2023-09-06 16:28:48,031 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 power (reason: RWSensor)
2023-09-06 16:28:48,031 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 Time (reason: RWSensor)
2023-09-06 16:28:48,031 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 Capacity (reason: RWSensor)
2023-09-06 16:28:48,031 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 charge (reason: RWSensor)
2023-09-06 16:28:48,032 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 power (reason: RWSensor)
2023-09-06 16:28:48,032 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 Time (reason: RWSensor)
2023-09-06 16:28:48,033 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 Capacity (reason: RWSensor)
2023-09-06 16:28:48,033 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 charge (reason: RWSensor)
2023-09-06 16:28:48,033 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 power (reason: RWSensor)
2023-09-06 16:28:48,033 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 Time (reason: RWSensor)
2023-09-06 16:28:48,034 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 Capacity (reason: RWSensor)
2023-09-06 16:28:48,034 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 charge (reason: RWSensor)
2023-09-06 16:28:48,034 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 power (reason: RWSensor)
2023-09-06 16:28:48,035 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 Time (reason: RWSensor)
2023-09-06 16:28:48,035 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 Capacity (reason: RWSensor)
2023-09-06 16:28:48,035 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 charge (reason: RWSensor)
2023-09-06 16:28:48,036 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 power (reason: RWSensor)
2023-09-06 16:28:48,036 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 Time (reason: RWSensor)
2023-09-06 16:28:48,037 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Battery Low Capacity (reason: RWSensor)
2023-09-06 16:28:48,037 INFO    ha_addon_sunsynk_multi.sensor_options Added hidden sensors as other sensors depend on it: Rated power, Serial
2023-09-06 16:28:48,038 DEBUG   asyncio Using selector: EpollSelector
2023-09-06 16:28:48,040 INFO    ha_addon_sunsynk_multi.a_inverter Connecting to /dev/ttyUSB0
2023-09-06 16:28:48,040 INFO    sunsynk.pysunsynk PyModbus 3.4.1 Serial: /dev/ttyUSB0
2023-09-06 16:28:48,041 DEBUG   pymodbus.logging Connecting to /dev/ttyUSB0.
2023-09-06 16:28:48,041 DEBUG   pymodbus.logging Connecting comm
2023-09-06 16:28:48,058 DEBUG   pymodbus.logging Connected to comm
2023-09-06 16:28:48,058 DEBUG   pymodbus.logging callback_connected called
2023-09-06 16:28:48,060 INFO    ha_addon_sunsynk_multi.a_inverter Reading startup sensors Prog4 Time, Rated power, Prog1 Time, Serial, Prog5 Time, Prog6 Time, Prog3 Time, Battery Low Capacity, Prog2 Time
2023-09-06 16:28:48,070 DEBUG   pymodbus.logging send: 0x1 0x3 0x0 0x3 0x0 0x5 0x75 0xc9
2023-09-06 16:28:48,072 DEBUG   pymodbus.logging Adding transaction 1
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/sunsynk/src/ha_addon_sunsynk_multi/__main__.py", line 72, in <module>
    main()
  File "/sunsynk/src/ha_addon_sunsynk_multi/__main__.py", line 67, in main
    loop.run_until_complete(main_loop(loop))
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/sunsynk/src/ha_addon_sunsynk_multi/__main__.py", line 37, in main_loop
    await ist.connect()
  File "/sunsynk/src/ha_addon_sunsynk_multi/a_inverter.py", line 115, in connect
    if not await self.read_sensors_retry(sensors=list(SOPT.startup)):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sunsynk/src/ha_addon_sunsynk_multi/a_inverter.py", line 66, in read_sensors_retry
    await self.read_sensors(sensors=sensors, msg=msg)
  File "/sunsynk/src/ha_addon_sunsynk_multi/a_inverter.py", line 53, in read_sensors
    await self.inv.read_sensors(sensors)
  File "/sunsynk/src/sunsynk/sunsynk.py", line 94, in read_sensors
    r_r = await asyncio.wait_for(
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File "/sunsynk/src/sunsynk/pysunsynk.py", line 83, in read_holding_registers
    res = await self.client.read_holding_registers(  # type:ignore
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymodbus/client/base.py", line 201, in async_execute
    resp = await asyncio.wait_for(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 479, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
asyncio.exceptions.CancelledError
[16:28:51] WARNING: Halt add-on with exit code 1
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Edit: logs before i click update
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun sunsynk (no readiness notification)
s6-rc: info: service legacy-services successfully started
17:49:54 INFO    Loading HASS OS configuration
2023-09-06 17:49:54,210 DEBUG   : pymodbus driver options: {'port': '/dev/ttyUSB0', 'server_id': 1, 'timeout': 10, 'read_sensors_batch_size': 20, 'allow_gap': 2}
2023-09-06 17:49:54,214 INFO    ha_addon_sunsynk_multi.timer_schedule Schedules:
+-----------+-----+------+--------+-----------+----------+------------+
|    Key    | src | Read | Report | Change by | Change % | Change any |
+-----------+-----+------+--------+-----------+----------+------------+
| date_time |     |  60  |   60   |           |          |    True    |
|     rw    |     |  5   |  300   |           |          |    True    |
|     w     |  *  |  5   |   60   |     80    |          |            |
|    kwh    |     | 300  |  300   |           |          |            |
|  any_unit |     |  15  |  300   |           |          |            |
|  no_unit  |     |  15  |  300   |           |          |    True    |
+-----------+-----+------+--------+-----------+----------+------------+
2023-09-06 17:49:54,214 INFO    ha_addon_sunsynk_multi.sensor_options Using three phase sensor definitions.
2023-09-06 17:49:54,217 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Total Battery Charge (reason: Unit)
2023-09-06 17:49:54,217 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Total Battery Discharge (reason: Unit)
2023-09-06 17:49:54,217 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Total Grid Export (reason: Unit)
2023-09-06 17:49:54,218 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Total Grid Import (reason: Unit)
2023-09-06 17:49:54,219 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Total PV Energy (reason: Unit)
2023-09-06 17:49:54,219 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: aux_power
2023-09-06 17:49:54,220 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for Battery current (reason: Any unit)
2023-09-06 17:49:54,220 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for Battery power (reason: Unit)
2023-09-06 17:49:54,220 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for Battery SOC (reason: Any unit)
2023-09-06 17:49:54,221 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for Battery voltage (reason: Any unit)
2023-09-06 17:49:54,221 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day Battery Charge (reason: Unit)
2023-09-06 17:49:54,221 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day Battery discharge (reason: Unit)
2023-09-06 17:49:54,221 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day Grid Export (reason: Unit)
2023-09-06 17:49:54,222 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day Grid Import (reason: Unit)
2023-09-06 17:49:54,222 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day Load Energy (reason: Unit)
2023-09-06 17:49:54,223 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule kwh used for Day PV Energy (reason: Unit)
2023-09-06 17:49:54,223 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: essential_power
2023-09-06 17:49:54,223 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: grid_connected
2023-09-06 17:49:54,223 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for Grid CT power (reason: Unit)
2023-09-06 17:49:54,224 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for Grid frequency (reason: Any unit)
2023-09-06 17:49:54,224 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for Grid power (reason: Unit)
2023-09-06 17:49:54,224 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for Grid voltage (reason: Any unit)
2023-09-06 17:49:54,224 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: inverter_current
2023-09-06 17:49:54,225 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for Inverter power (reason: Unit)
2023-09-06 17:49:54,225 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: load_frequency
2023-09-06 17:49:54,225 ERROR   ha_addon_sunsynk_multi.sensor_options Unknown sensor specified: non_essential_power
2023-09-06 17:49:54,226 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule no_unit used for Overall state (reason: No unit)
2023-09-06 17:49:54,226 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Priority Load (reason: RWSensor)
2023-09-06 17:49:54,226 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for PV1 current (reason: Any unit)
2023-09-06 17:49:54,226 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule w used for PV1 power (reason: Unit)
2023-09-06 17:49:54,226 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule any_unit used for PV1 voltage (reason: Any unit)
2023-09-06 17:49:54,226 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Use Timer (reason: RWSensor)
2023-09-06 17:49:54,227 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Load Limit (reason: RWSensor)
2023-09-06 17:49:54,227 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 Capacity (reason: RWSensor)
2023-09-06 17:49:54,227 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 charge (reason: RWSensor)
2023-09-06 17:49:54,227 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 power (reason: RWSensor)
2023-09-06 17:49:54,227 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog1 Time (reason: RWSensor)
2023-09-06 17:49:54,227 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 Capacity (reason: RWSensor)
2023-09-06 17:49:54,228 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 charge (reason: RWSensor)
2023-09-06 17:49:54,228 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 power (reason: RWSensor)
2023-09-06 17:49:54,228 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog2 Time (reason: RWSensor)
2023-09-06 17:49:54,228 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 Capacity (reason: RWSensor)
2023-09-06 17:49:54,228 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 charge (reason: RWSensor)
2023-09-06 17:49:54,228 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 power (reason: RWSensor)
2023-09-06 17:49:54,228 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog3 Time (reason: RWSensor)
2023-09-06 17:49:54,229 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 Capacity (reason: RWSensor)
2023-09-06 17:49:54,229 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 charge (reason: RWSensor)
2023-09-06 17:49:54,229 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 power (reason: RWSensor)
2023-09-06 17:49:54,229 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog4 Time (reason: RWSensor)
2023-09-06 17:49:54,229 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 Capacity (reason: RWSensor)
2023-09-06 17:49:54,229 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 charge (reason: RWSensor)
2023-09-06 17:49:54,229 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 power (reason: RWSensor)
2023-09-06 17:49:54,230 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog5 Time (reason: RWSensor)
2023-09-06 17:49:54,230 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 Capacity (reason: RWSensor)
2023-09-06 17:49:54,230 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 charge (reason: RWSensor)
2023-09-06 17:49:54,230 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 power (reason: RWSensor)
2023-09-06 17:49:54,230 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Prog6 Time (reason: RWSensor)
2023-09-06 17:49:54,231 DEBUG   ha_addon_sunsynk_multi.timer_schedule Schedule rw used for Battery Low Capacity (reason: RWSensor)
2023-09-06 17:49:54,231 INFO    ha_addon_sunsynk_multi.sensor_options Added hidden sensors as other sensors depend on it: Rated power, Serial
2023-09-06 17:49:54,232 DEBUG   asyncio Using selector: EpollSelector
2023-09-06 17:49:54,233 INFO    ha_addon_sunsynk_multi.a_inverter Connecting to /dev/ttyUSB0
2023-09-06 17:49:54,234 INFO    sunsynk.pysunsynk PyModbus 3.4.1 Serial: /dev/ttyUSB0
2023-09-06 17:49:54,235 DEBUG   pymodbus.logging Connecting to /dev/ttyUSB0.
2023-09-06 17:49:54,235 DEBUG   pymodbus.logging Connecting comm
2023-09-06 17:49:54,251 DEBUG   pymodbus.logging Connected to comm
2023-09-06 17:49:54,251 DEBUG   pymodbus.logging callback_connected called
2023-09-06 17:49:54,253 INFO    ha_addon_sunsynk_multi.a_inverter Reading startup sensors Prog2 Time, Prog4 Time, Rated power, Prog1 Time, Prog6 Time, Prog3 Time, Battery Low Capacity, Serial, Prog5 Time
2023-09-06 17:49:54,262 DEBUG   pymodbus.logging send: 0x1 0x3 0x0 0x3 0x0 0x5 0x75 0xc9
2023-09-06 17:49:54,264 DEBUG   pymodbus.logging Adding transaction 1

Both debugs clearly show a timeout.
 pymodbus.logging send: 0x1 0x3 0x0 0x3 0x0 0x5 0x75 0xc9

I can fix the retry mechanism in the first log, but it won't solve your issue. Have you looked at the fault-finding page on the docs?

The traceback in your first log should be solved now. But you will still have to look at the fault finding guide to get this working

Morning - I need some help please. I have an  issue with opening the connection to the solarman dongle;  "pysolarmanv5.pysolarmanv5.NoSocketAvailableError: Cannot open connection to <IP_ADDRESS>" The connect call fails and I read that it could be something to do with the encryption on the dongle - I am trying via a RPi. If I use a the RS485 - which port to use (CAN is occupied with BMS)? Also, where can I get the applicable cable?
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
On MQTT:
023-09-26 07:48:10: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:49:10: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:49:10: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:49:52: New connection from <IP_ADDRESS>:48572 on port 1883.
2023-09-26 07:49:52: Client  closed its connection.
2023-09-26 07:50:10: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:50:10: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:51:10: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:51:10: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:51:52: New connection from <IP_ADDRESS>:53956 on port 1883.
2023-09-26 07:51:52: Client  closed its connection.
2023-09-26 07:52:10: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:52:10: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:53:11: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:53:11: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:53:52: New connection from <IP_ADDRESS>:57820 on port 1883.
2023-09-26 07:53:52: Client  closed its connection.
2023-09-26 07:54:11: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:54:11: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:55:11: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:55:11: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:55:52: New connection from <IP_ADDRESS>:37094 on port 1883.
2023-09-26 07:55:52: Client  closed its connection.
2023-09-26 07:56:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:56:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:57:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:57:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:57:52: New connection from <IP_ADDRESS>:51896 on port 1883.
2023-09-26 07:57:52: Client  closed its connection.
2023-09-26 07:58:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:58:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:59:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:59:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:59:52: New connection from <IP_ADDRESS>:45136 on port 1883.
2023-09-26 07:59:52: Client  closed its connection.
2023-09-26 08:00:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:00:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:01:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:01:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:01:52: New connection from <IP_ADDRESS>:60454 on port 1883.
2023-09-26 08:01:52: Client  closed its connection.
2023-09-26 08:02:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:02:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:03:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:03:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:03:52: New connection from <IP_ADDRESS>:55940 on port 1883.
2023-09-26 08:03:52: Client  closed its connection.
2023-09-26 08:04:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:04:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:05:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:05:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:05:52: New connection from <IP_ADDRESS>:44346 on port 1883.
2023-09-26 08:05:52: Client  closed its connection.
2023-09-26 08:06:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:06:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:07:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:07:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:07:52: New connection from <IP_ADDRESS>:57240 on port 1883.
2023-09-26 08:07:52: Client  closed its connection.
2023-09-26 08:08:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:08:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:08:28: Saving in-memory database to /data//mosquitto.db.
2023-09-26 08:09:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:09:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:09:52: New connection from <IP_ADDRESS>:40638 on port 1883.
2023-09-26 08:09:52: Client  closed its connection.
2023-09-26 08:10:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:10:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:11:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:11:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:11:52: New connection from <IP_ADDRESS>:60108 on port 1883.
2023-09-26 08:11:52: Client  closed its connection.
2023-09-26 08:12:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:12:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:13:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:13:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:13:53: New connection from <IP_ADDRESS>:51858 on port 1883.
2023-09-26 08:13:53: Client  closed its connection.
2023-09-26 08:14:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:14:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:15:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:15:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:15:53: New connection from <IP_ADDRESS>:33168 on port 1883.
2023-09-26 08:15:53: Client  closed its connection.
2023-09-26 08:16:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:16:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:17:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:17:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:17:53: New connection from <IP_ADDRESS>:57754 on port 1883.
2023-09-26 08:17:53: Client  closed its connection.
2023-09-26 08:18:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:18:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:19:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:19:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:19:53: New connection from <IP_ADDRESS>:60002 on port 1883.
2023-09-26 08:19:53: Client  closed its connection.
2023-09-26 08:20:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:20:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:21:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:21:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun sunsynk (no readiness notification)
s6-rc: info: service legacy-services successfully started
20:21:31 INFO    Loading configuration: /data/options.json
20:21:31 INFO    sunsynk library version: 0.6.4
20:21:31 INFO    Schedules:
+-----------+-----+------+--------+-----------+----------+------------+
|    Key    | src | Read | Report | Change by | Change % | Change any |
+-----------+-----+------+--------+-----------+----------+------------+
| date_time |     |  60  |   60   |           |          |    True    |
|     rw    |     |  5   |  300   |           |          |    True    |
|     w     |  *  |  20  |   60   |     80    |          |            |
|    kwh    |     | 300  |  300   |           |          |            |
|  any_unit |     |  15  |  300   |           |          |            |
|  no_unit  |     |  15  |  300   |           |          |    True    |
+-----------+-----+------+--------+-----------+----------+------------+
20:21:31 INFO    Using three phase sensor definitions.
20:21:31 ERROR   Unknown sensor specified: aux_power
20:21:31 ERROR   Unknown sensor specified: essential_power
20:21:31 ERROR   Unknown sensor specified: grid_connected
20:21:31 ERROR   Unknown sensor specified: inverter_current
20:21:31 ERROR   Unknown sensor specified: load_frequency
20:21:31 ERROR   Unknown sensor specified: non_essential_power
20:21:31 INFO    Added hidden sensors as other sensors depend on it: Rated power, Serial
20:21:31 INFO    Connecting to tcp://<IP_ADDRESS>:8899
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/pysolarmanv5/pysolarmanv5_async.py", line 68, in connect
self.reader, self.writer = await asyncio.open_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/streams.py", line 48, in open_connection
transport, _ = await loop.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
sock = await self._connect_sock(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
return await fut
^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
OSError: [Errno 113] Connect call failed ('<IP_ADDRESS>', 8899)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/main.py", line 78, in
main()
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/main.py", line 73, in main
loop.run_until_complete(main_loop(loop))
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/main.py", line 38, in main_loop
await ist.connect()
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/a_inverter.py", line 113, in connect
await self.inv.connect()
File "/usr/local/lib/python3.11/site-packages/sunsynk/solarmansunsynk.py", line 51, in connect
await self.client.connect()
File "/usr/local/lib/python3.11/site-packages/pysolarmanv5/pysolarmanv5_async.py", line 73, in connect
raise NoSocketAvailableError(f"Cannot open connection to {self.address}")
pysolarmanv5.pysolarmanv5.NoSocketAvailableError: Cannot open connection to <IP_ADDRESS>
[20:21:36] WARNING: Halt add-on with exit code 1
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Sorry, some copy error.... MQTT in the wrong place;

Add-on (multi) Deye 12Kw 3 Phase
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun sunsynk (no readiness notification)
s6-rc: info: service legacy-services successfully started
20:21:31 INFO    Loading configuration: /data/options.json
20:21:31 INFO    sunsynk library version: 0.6.4
20:21:31 INFO    Schedules:
+-----------+-----+------+--------+-----------+----------+------------+
|    Key    | src | Read | Report | Change by | Change % | Change any |
+-----------+-----+------+--------+-----------+----------+------------+
| date_time |     |  60  |   60   |           |          |    True    |
|     rw    |     |  5   |  300   |           |          |    True    |
|     w     |  *  |  20  |   60   |     80    |          |            |
|    kwh    |     | 300  |  300   |           |          |            |
|  any_unit |     |  15  |  300   |           |          |            |
|  no_unit  |     |  15  |  300   |           |          |    True    |
+-----------+-----+------+--------+-----------+----------+------------+
20:21:31 INFO    Using three phase sensor definitions.
20:21:31 ERROR   Unknown sensor specified: aux_power
20:21:31 ERROR   Unknown sensor specified: essential_power
20:21:31 ERROR   Unknown sensor specified: grid_connected
20:21:31 ERROR   Unknown sensor specified: inverter_current
20:21:31 ERROR   Unknown sensor specified: load_frequency
20:21:31 ERROR   Unknown sensor specified: non_essential_power
20:21:31 INFO    Added hidden sensors as other sensors depend on it: Rated power, Serial
20:21:31 INFO    Connecting to tcp://<IP_ADDRESS>:8899
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/pysolarmanv5/pysolarmanv5_async.py", line 68, in connect
self.reader, self.writer = await asyncio.open_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/streams.py", line 48, in open_connection
transport, _ = await loop.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1085, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1069, in create_connection
sock = await self._connect_sock(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 973, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 634, in sock_connect
return await fut
^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/selector_events.py", line 674, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
OSError: [Errno 113] Connect call failed ('<IP_ADDRESS>', 8899)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/main.py", line 78, in
main()
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/main.py", line 73, in main
loop.run_until_complete(main_loop(loop))
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/main.py", line 38, in main_loop
await ist.connect()
File "/usr/local/lib/python3.11/site-packages/ha_addon_sunsynk_multi/a_inverter.py", line 113, in connect
await self.inv.connect()
File "/usr/local/lib/python3.11/site-packages/sunsynk/solarmansunsynk.py", line 51, in connect
await self.client.connect()
File "/usr/local/lib/python3.11/site-packages/pysolarmanv5/pysolarmanv5_async.py", line 73, in connect
raise NoSocketAvailableError(f"Cannot open connection to {self.address}")
pysolarmanv5.pysolarmanv5.NoSocketAvailableError: Cannot open connection to <IP_ADDRESS>
[20:21:36] WARNING: Halt add-on with exit code 1
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

MQTT:
2023-09-26 07:52:10: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:53:11: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:53:11: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:53:52: New connection from <IP_ADDRESS>:57820 on port 1883.
2023-09-26 07:53:52: Client  closed its connection.
2023-09-26 07:54:11: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:54:11: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:55:11: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:55:11: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:55:52: New connection from <IP_ADDRESS>:37094 on port 1883.
2023-09-26 07:55:52: Client  closed its connection.
2023-09-26 07:56:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:56:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:57:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:57:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:57:52: New connection from <IP_ADDRESS>:51896 on port 1883.
2023-09-26 07:57:52: Client  closed its connection.
2023-09-26 07:58:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:58:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:59:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:59:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 07:59:52: New connection from <IP_ADDRESS>:45136 on port 1883.
2023-09-26 07:59:52: Client  closed its connection.
2023-09-26 08:00:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:00:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:01:12: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:01:12: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:01:52: New connection from <IP_ADDRESS>:60454 on port 1883.
2023-09-26 08:01:52: Client  closed its connection.
2023-09-26 08:02:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:02:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:03:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:03:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:03:52: New connection from <IP_ADDRESS>:55940 on port 1883.
2023-09-26 08:03:52: Client  closed its connection.
2023-09-26 08:04:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:04:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:05:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:05:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:05:52: New connection from <IP_ADDRESS>:44346 on port 1883.
2023-09-26 08:05:52: Client  closed its connection.
2023-09-26 08:06:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:06:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:07:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:07:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:07:52: New connection from <IP_ADDRESS>:57240 on port 1883.
2023-09-26 08:07:52: Client  closed its connection.
2023-09-26 08:08:13: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:08:13: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:08:28: Saving in-memory database to /data//mosquitto.db.
2023-09-26 08:09:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:09:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:09:52: New connection from <IP_ADDRESS>:40638 on port 1883.
2023-09-26 08:09:52: Client  closed its connection.
2023-09-26 08:10:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:10:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:11:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:11:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:11:52: New connection from <IP_ADDRESS>:60108 on port 1883.
2023-09-26 08:11:52: Client  closed its connection.
2023-09-26 08:12:14: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:12:14: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:13:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:13:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:13:53: New connection from <IP_ADDRESS>:51858 on port 1883.
2023-09-26 08:13:53: Client  closed its connection.
2023-09-26 08:14:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:14:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:15:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:15:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:15:53: New connection from <IP_ADDRESS>:33168 on port 1883.
2023-09-26 08:15:53: Client  closed its connection.
2023-09-26 08:16:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:16:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:17:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:17:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:17:53: New connection from <IP_ADDRESS>:57754 on port 1883.
2023-09-26 08:17:53: Client  closed its connection.
2023-09-26 08:18:15: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:18:15: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:19:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:19:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:19:53: New connection from <IP_ADDRESS>:60002 on port 1883.
2023-09-26 08:19:53: Client  closed its connection.
2023-09-26 08:20:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:20:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:21:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:21:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:21:53: New connection from <IP_ADDRESS>:40760 on port 1883.
2023-09-26 08:21:53: Client  closed its connection.
2023-09-26 08:22:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:22:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:23:16: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:23:16: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:23:53: New connection from <IP_ADDRESS>:45478 on port 1883.
2023-09-26 08:23:53: Client  closed its connection.
2023-09-26 08:24:17: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:24:17: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:25:17: Received PINGREQ from 0Loy1diDJ1jYSdF5mD9cav
2023-09-26 08:25:17: Sending PINGRESP to 0Loy1diDJ1jYSdF5mD9cav

@Niruservices - are you sure you have the corretc IP for your dongle?
OSError: [Errno 113] Connect call failed ('<IP_ADDRESS>', 8899)

Evening,
It is really odd – as it shows the correct MAC & shows connection speeds on the router – yet – not able to ping it either.
BR;
Nico
From: Johann Kellerman @.>
Sent: Tuesday, September 26, 2023 7:21 PM
To: kellerza/sunsynk @.>
Cc: Niruservices @.>; Mention @.>
Subject: Re: [kellerza/sunsynk] Cant start the Add-on on my Deye 12kW 3phases (Issue #180)
@Niruservices https://github.com/Niruservices  - are you sure you have the corretc IP for your dongle?
OSError: [Errno 113] Connect call failed ('<IP_ADDRESS>', 8899)
—
Reply to this email directly, view it on GitHub https://github.com/kellerza/sunsynk/issues/180#issuecomment-1735969335 , or unsubscribe https://github.com/notifications/unsubscribe-auth/BCZGBF7IEKCTOYDRCNC65DTX4MFGFANCNFSM6AAAAAA4NTSTBQ .
You are receiving this because you were mentioned.  https://github.com/notifications/beacon/BCZGBF4T26I52256N25JTFLX4MFGFA5CNFSM6AAAAAA4NTSTBSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTHPDFDO.gif Message ID: @.*** @.***> >

Morning,
BTW, if I use RS485 – which bus to select py or U?
BR;
Nico
From: Nico Steenkamp @.>
Sent: Tuesday, September 26, 2023 7:33 PM
To: 'kellerza/sunsynk' @.>; 'kellerza/sunsynk' @.>
Cc: 'Mention' @.>
Subject: RE: [kellerza/sunsynk] Cant start the Add-on on my Deye 12kW 3phases (Issue #180)
Evening,
It is really odd – as it shows the correct MAC & shows connection speeds on the router – yet – not able to ping it either.
BR;
Nico
From: Johann Kellerman @.*** @.> >
Sent: Tuesday, September 26, 2023 7:21 PM
To: kellerza/sunsynk @. @.> >
Cc: Niruservices @. @.> >; Mention @. @.***> >
Subject: Re: [kellerza/sunsynk] Cant start the Add-on on my Deye 12kW 3phases (Issue #180)
@Niruservices https://github.com/Niruservices  - are you sure you have the corretc IP for your dongle?
OSError: [Errno 113] Connect call failed ('<IP_ADDRESS>', 8899)
—
Reply to this email directly, view it on GitHub https://github.com/kellerza/sunsynk/issues/180#issuecomment-1735969335 , or unsubscribe https://github.com/notifications/unsubscribe-auth/BCZGBF7IEKCTOYDRCNC65DTX4MFGFANCNFSM6AAAAAA4NTSTBQ .
You are receiving this because you were mentioned.  https://github.com/notifications/beacon/BCZGBF4T26I52256N25JTFLX4MFGFA5CNFSM6AAAAAA4NTSTBSWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTHPDFDO.gif Message ID: @.*** @.***> >

You can use either, whichever is the most reliable for you.
I have found that the recent pymodbus is fairly reliable for direct serial.
umodbus has always worked well (via some gateway like mbusd, but not direct serial)

@lordvalium did you manage to get it running?

no unfortunately not @kellerza
Have bought that addon https://www.amazon.de/dp/B08DXRKFZR but its not recognized in Homeassistant


Discussion : why ZXing 3.3.0 ?

Your project's pom.xml declare a dependency to ZXing version 3.3.0 (whose javase module uses JAI Image IO 1.3.1)
ZXing updates are available :

3.3.3 was release in may 2018, it uses JAI Image IO 1.4.0.
3.4.0 was release in may 2019, it uses JAI Image IO 1.4.0.

Is there any rational behind your choice that we should be aware of ?

No particular reason, to be honest I probably just copied the version number from a blog post! Happy to have it upgraded to the latest if you want to make a quick PR.

I'll do that as soon as I can.

See #17

Merged


Broken formatting in DispatcherOperationStatus Enum

https://docs.microsoft.com/en-gb/dotnet/api/system.windows.threading.dispatcheroperationstatus?view=netframework-4.7.1
This page currently ends with the title: XAML Text Usage. Not quite sure what's meant to be there - MSDN appears to be the same.
https://msdn.microsoft.com/en-us/library/system.windows.threading.dispatcheroperationstatus(v=vs.110).aspx

Thanks @ChrisMaddock. I think we should just remove this. Not sure what was the original intention either. @rpetrusha do you know? Previous versions also don't show this information. Might be an old issue with previous migrations.


build(build-tools): Squelch warnings about oclif updates

oclif outputs warnings whenever a new release is available, and they release frequently. This leads to a lot of warning spam when building build-tools, because we build the readme and command manifest as part of the build.
The only way I can find to squash the warnings is to set an env variable before running the commands. This works, but it makes incremental support for the tasks in fluid-build more difficult, because the cross-env args would need to be parsed out to map the command to the right task. However, since the commands aren't cached by fluid-build today anyway, there's no incremental build impact as it stands, and this improves the build experience for build-tools.

Since this was opened, the tasks this applies to are now cached in fluid-build, so this solution no longer works. Closing.


fix(router) do not force an extra segment in the path

Issues resolved
Fixes the issue/todo identified in #3749
Read carefully
The way this is handled is by changing the way the router constructs the outgoing path. The path is constructed from these elements in order:

the upstream prefix (the path property in the service entity)
the path match (based on the paths property in the route entity)

if matching on path
if not stripping the path

the request postfix (the part after the matched path, or if not matched by path, the full request path)

In all cases, the initial / of the incoming requested path will be ignored/removed. This allows the user to control the presence of the / in the outgoing path.
The reason for this change is because in the existing code, different inputs lead to the same output, always injecting the extra /. See these lines for example. The first set of 5 tests, yield the exact same upstream path as the second set of 5 lines, despite the upstream_path being different. The updated tests (now a block of 12) start here.
This PR also adds quite a number of test cases that were not previously covered.
The bad news is that this is a breaking change, since it changes the way the upstream path is constructed. The good news is that in the entire Kong test suite there was only 1 occurrence of a test failing because of this.

@Tieske Why does this PR have a "do not merge" label? This label signifies that a PR is rejected. If you are worried that this PR gets merged without proper review/testing, that should not happen as per in the first place as per our process.

If good to merge on your end too, let's proceed with a merge commit

yay 🎉   @thibaultcha please go ahead and merge as you see fit


sub resource test failed in Windows

Describe the bug
The test loader::resource::file_resource::tests::test_sub fails in Windows
To Reproduce
Steps to reproduce the behavior:

Checkout the current version Windows
Run 'cargo test'

Expected behavior
The test should be green.
Screenshots
Desktop (please complete the following information):

OS: Windows
Version: 0.3.0

fluff.txt ❤

Issue is correct behavior, as the paths under Windows should be \ instead auf /. Nevertheless FileResource does not create a valid filepath, if you build on Windows, but enter a Unix path.


Craft 4

Will Craft 4 support be added ?

I'm actually planning an update to this plugin. Preserve pagination for example is no longer a required function, but fetching of the querystring array is still useful for some projects. Craft 4 update coming soon!


[Docs]: 1-Click- Update Installation guides.

Is there an existing issue for this?

[X] I have searched the existing issues

Engineering Ticket Link
NA
Release Date
NA
Release Number
NA
First Draft
No response
Loom video
No response
Discord/slack/intercom Link if needed
No response
PRD
No response
Test plan/cases
No response
Use cases or user requests

To update all the installation guides based on the 1-click project.

@pranayagarwal96 - Please add engineering related info like release date, engineering ticket etc.


Add TMS trailing slash at end of path, rather than end of url.

This is PR into #849, so merge that first.
TileMapServiceRasterOverlay attempts to add a / to the end of the URL it's given when that URL doesn't end in .xml and it doesn't already end in a slash. But the code was looking at the last character of the entire URL, rather than the last character of the path. So if the URL had query parameters, it didn't work at all.

As we discussed offline, https://sandcastle.cesium.com/CesiumUnminified/Assets/Textures/NaturalEarthII/tilemapresource.xml?some=parameter should demonstrate the bug in that it will become https://sandcastle.cesium.com/CesiumUnminified/Assets/Textures/NaturalEarthII/tilemapresource.xml?some=parameter/ before this PR (note the trailing slash at the end). But because the sandcastle.cesium.com web server ignores that query parameter, it doesn't matter if there's an extra slash on the end of it or not, so nothing breaks. In my case the query parameter was an API key, so the extra slash made it invalid.

I'll add a quick test for it, one moment... (mentioning it so you don't write that issue)