Copy from web does not work with 1.0.6

Just tried with the new 1.0.6 release to copy simple text content from a tldraw.com web page to the plugin hosted in Obsidian.
With right click > paste, or ctrl+v nothing happens. In the console the following error is shown:
plugin:tldraw:32891 Error migrating store Incompatible schema?
migrateStoreSnapshot @ plugin:tldraw:32891
putContentOntoCurrentPage @ plugin:tldraw:54074
pasteTldrawContent @ plugin:tldraw:76585
handleClipboardThings @ plugin:tldraw:76855
await in handleClipboardThings (async)
handlePasteFromClipboardApi @ plugin:tldraw:76763
onPaste @ plugin:tldraw:76969
eval @ plugin:tldraw:78391
Promise.then (async)
onSelect @ plugin:tldraw:78390
onSelect @ plugin:tldraw:82867
menuItem.addEventListener.once @ plugin:tldraw:81845
eval @ plugin:tldraw:72229
flushSync3 @ plugin:tldraw:20874
flushSync$1 @ plugin:tldraw:23451
$8927f6f2acc4f386$export$6d1a0317bde7de7f @ plugin:tldraw:72228
handleSelect @ plugin:tldraw:81850
handleEvent @ plugin:tldraw:71878
callCallback2 @ plugin:tldraw:5602
invokeGuardedCallbackDev @ plugin:tldraw:5627
invokeGuardedCallback @ plugin:tldraw:5661
invokeGuardedCallbackAndCatchFirstError @ plugin:tldraw:5664
executeDispatch @ plugin:tldraw:8944
processDispatchQueueItemsInOrder @ plugin:tldraw:8964
processDispatchQueue @ plugin:tldraw:8973
dispatchEventsForPlugins @ plugin:tldraw:8981
eval @ plugin:tldraw:9105
batchedUpdates$1 @ plugin:tldraw:20837
batchedUpdates @ plugin:tldraw:5507
dispatchEventForPluginEventSystem @ plugin:tldraw:9104
dispatchEventWithEnableCapturePhaseSelectiveHydrationWithoutDiscreteEventReplay @ plugin:tldraw:7406
dispatchEvent @ plugin:tldraw:7400
dispatchDiscreteEvent @ plugin:tldraw:7377
plugin:tldraw:54076 Uncaught (in promise) Error: Could not put content: could not migrate content
    at Editor.putContentOntoCurrentPage (plugin:tldraw:54076:13)
    at pasteTldrawContent (plugin:tldraw:76585:10)
    at handleClipboardThings (plugin:tldraw:76855:7)
    at async handlePasteFromClipboardApi (plugin:tldraw:76763:10)

Thank you for this plugin, very valuable! 🙏

#18 was closed, but it doesn't seem that the problem was specifically fixed. I'll note down my observations until I get around to it or someone else does:
Firefox:
Copy/pasting from or to Firefox doesn't work, at least on version 127.0. This seems to be a Firefox specific feature so keep that in mind if you use Firefox. There is an issue on the main Tldraw repo: https://github.com/tldraw/tldraw/issues/3949
Chromium based browsers:
Copy and pasting from the tldraw obsidian plugin to tldraw in the chromium based browser does work.


is root node the source node for the first edge?

I have tried the code for a directed graph with 1 and only 1 unspecified root. That is to say, we only know its existence, but we don't know which one it is.
I have tested for some simple graphs, it seems the source node for the first edge is always the root node. Is my assumption right?

sorry, I have tried more graphs, it turns out root node can be any other nodes.


Threat doc classified as SFN-DNS has no domain name

When logstash encounters a THREAT DNS event, it should put the domain name in the SFN.domain_name field in the ES threat* index.  However, if the event DNS lookup is to a domain with underscores ( _ ) in it, it does not create that field and SFN code throws an error.
Example domain that broke it:  _adsp._domainkey.ecopromconsalting.ru

This is fixable by using the DATA grok regex pattern rather than the HOSTNAME.  See this for more explanation on the patterns used by grok.
This will only be done on the dns-cloud as we are already seeing underscores in domains in a few customers that are running the SFN4.0 alpha.
This will NOT be changed in content or EDL based parsers because we never see this as a problem.  Probably because nobody is looking for it and it doesn't wind up in either the content or the EDLs.

This was fixed by changing the grok patterns in the logstash gtp.conf file.   It is fixed in v3.5


Part: Create Note Error

This is weird - I have (I think) only one part out of about 100 for which I can create a note.
Seemingly all others throw an error when I try and access the Notes tab for the part.
Sucess shown in Pic - and error log for rest of the parts.
(So I can sort of see that its having a problem (applying .strip) in the markdown display of an empty field, but I get confused when trying to chase it further through the code - would love to be able to help - but not yet :) )
At a total loss as to how one part manages to work - must be something different about my part creation sequence??

Environment:
Request Method: GET
Request URL: http://<IP_ADDRESS>:8000/part/45/notes

Django Version: 3.0.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'build.apps.BuildConfig',
 'common.apps.CommonConfig',
 'company.apps.CompanyConfig',
 'label.apps.LabelConfig',
 'order.apps.OrderConfig',
 'part.apps.PartConfig',
 'report.apps.ReportConfig',
 'stock.apps.StockConfig',
 'users.apps.UsersConfig',
 'django_filters',
 'dbbackup',
 'rest_framework',
 'rest_framework.authtoken',
 'corsheaders',
 'crispy_forms',
 'import_export',
 'django_cleanup',
 'qr_code',
 'mptt',
 'markdownx',
 'markdownify',
 'django_tex',
 'django_admin_shell']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'InvenTree.middleware.AuthRequiredMiddleware']

Template error:
In template /home/richard/inventree/InvenTree/templates/base.html, error at line 0
   'NoneType' object has no attribute 'strip'
   1 : {% load static %}
   2 : {% load i18n %}
   3 : {% load inventree_extras %}
   4 :
   5 : <!DOCTYPE html>
   6 : <html lang="en">
   7 : <head>
   8 :
   9 : <!-- Required meta tags -->
   10 : <meta charset="utf-8">

Traceback (most recent call last):
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/core/handlers/base.py", line 145, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/core/handlers/base.py", line 143, in _get_response
    response = response.render()
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/response.py", line 105, in render
    self.content = self.rendered_content
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/response.py", line 83, in rendered_content
    return template.render(context, self._request)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/backends/django.py", line 61, in render
    return self.template.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 171, in render
    return self._render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 936, in render
    bit = node.render_annotated(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
    return self.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 936, in render
    bit = node.render_annotated(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
    return self.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 936, in render
    bit = node.render_annotated(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
    return self.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/loader_tags.py", line 150, in render
    return compiled_parent._render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
    return self.nodelist.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 936, in render
    bit = node.render_annotated(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
    return self.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 936, in render
    bit = node.render_annotated(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
    return self.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/loader_tags.py", line 62, in render
    result = block.nodelist.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 936, in render
    bit = node.render_annotated(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
    return self.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/defaulttags.py", line 309, in render
    return nodelist.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 936, in render
    bit = node.render_annotated(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
    return self.render(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 986, in render
    output = self.filter_expression.resolve(context)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/django/template/base.py", line 697, in resolve
    new_obj = func(obj, *arg_vals)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/markdownify/templatetags/markdownify.py", line 46, in markdownify
    html = markdown.markdown(text, extensions=extensions)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/markdown/core.py", line 386, in markdown
    return md.convert(text)
  File "/home/richard/inventree-env/lib/python3.8/site-packages/markdown/core.py", line 247, in convert
    if not source.strip():

Exception Type: AttributeError at /part/45/notes
Exception Value: 'NoneType' object has no attribute 'strip'

Further Info..
Not sure how this happens, but by looking at the database, I can see that some items have 'no entry' in the notes field, but the problematic ones seem to have "NULL".  You cant strip the leading/trailing from a NULL :)
Ones in Green I can add Notes to, Red I cant. Not sure how I managed to have some one way - some the other - perhaps to do with the entry method as they seem to be in a block (my IPN numbers are all clustered together)

@watarat When you create a new part and leave the notes field blank, does it show notes as empty string in database?
The notes model field is set with null=True so it shouldn't be...
Also, it is based on django-markdownx which seems like it can handle NULL based values. So there shouldn't be any problem either way...

Ok found it, the template was trying to render the markdown even though part.notes was NULL

Thanks @eeintech for the fix 😀
Part->Create doesnt allow entering a comment, so I'm still confused as to how I managed to get a few parts with 'empty text' rather than NULL?

Part->Create doesnt allow entering a comment, so I'm still confused as to how I managed to get a few parts with 'empty text' rather than NULL?

If your database is older than end of September, previously the null=True constraint was not set on the notes field, so older part would have an empty string and new one have the notes field set to NULL. I do have a similar database as yours :wink:


The set handler examples on CRDT objects.

The index page should have info on how to use CRDT change handlers.

Closed by 6cb03ebaa43b14055389484d5035f856c5f29136.


type:datetime editing toIsoString transforms time to UTC

The datetime editing input requires an IsoString input for the current value. This causes a shift from local time to UTC.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toISOString

Resolved in PR.


simplify/rewrite dbs3upload

Current code was not compatible with the Tier0. It somehow worked so far, but never really correctly (mostly causing internal multiplication of work) and with the huge backlog we accumulated due to the few deadlocks and component restarts it completely broke it. I tried to keep the changes minimal. Didn't quite succeed, but this is about as minimal as I can get it.
I needed to remove the workflow parameter from DBSBufferBlock as workflow is not a property of a block. Replaced it with a workflows set, which is filled when files are added to the DBSBufferBlock. Then for block closing I look at whether any of the workflows in a block is completed.
Dataset_algo is also not a property of a block, but it was used as a key for a block cache and to split blocks into smaller groups to process. Removed the whole cache based on dataset_algo as it wasn't needed and replaced the splitting to use datasetpath instead. That actually is a property of a block, but I had to add it to DBSBufferBlock as it wasn't stored.
As I had to touch some queries for these datasetpath and workflow changes anyways, optimized these queries when I saw something obvious. Saw plenty more to be optimized there, but constrained myself...
Reworked the transaction code in DBSBufferUtil, read methods don't need transactions, they just slow things down. Removed them. Left the transactions intact for writes.
Tested this code in a short Tier0 replay, both in live mode and in the mode where DBSUpload component is restarted and has to pick up from open blocks stored in the database. Worked.
Currently running a larger Tier0 replay with Express and Bulk processing and automatic transfers from T0_CH_CERN_Disk to T2_CH_CERN and automatic deletions from T0_CH_CERN_DIsk. If that works too I am going to pull the trigger and deploy this in the production Tier0.

Tier0 deployment does not need this to be merged btw. I still would appreciate any comments on the code and the deployment plans. @ticoann @amaltaro @ericvaandering

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/660/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py decreased from 9.57 to 9.38
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors
Score for src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py decreased from 9.40 to 9.35
Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py decreased from 9.50 to 8.57

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py got worse in pylint. Report follows:

Line 81 in uploadWorker W0622 Redefining built-in 'input'
Line 90 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 116 in uploadWorker W0703 Catching too general exception Exception
Line 113 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 121 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 122 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 123 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 137 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 149 in DBSUploadPoller R0902 Too many instance attributes (21/7)
Line 245 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 256 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 316 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 317 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 328 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 335 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 348 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 355 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 390 in DBSUploadPoller.loadFiles W1201 Specify string format arguments as logging function parameters
Line 452 in DBSUploadPoller.isBlockOpen R0201 Method could be a function
Line 566 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 567 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 589 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 607 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 611 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 507 in DBSUploadPoller.inputBlocks R0912 Too many branches (16/12)
Line 681 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 700 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 624 in DBSUploadPoller.retrieveBlocks R0912 Too many branches (13/12)
Line 728 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
Line 738 in DBSUploadPoller.checkBlocks W0703 Catching too general exception Exception
Line 759 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
plus 80 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 50 in FindDASToUpload.makeDAS W0612 Unused variable 'junk'
Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

Pre-python 2.6 idioms found in changed files. Please consider updating the code. Suggested patch follows:
--- src/python/WMComponent/DBS3Buffer/DBSBufferBlock.py (original)
+++ src/python/WMComponent/DBS3Buffer/DBSBufferBlock.py (refactored)
@@ -105,8 +105,7 @@
             msg += "Ignoring this file for now!\n"
             logging.error(msg)
             logging.debug("Block length: %i" % len(self.files))
-            l = [x['id'] for x in self.files]
-            l.sort()
+            l = sorted([x['id'] for x in self.files])
             logging.debug("First file: %s    Last file: %s" % (l[0], l[-1]))
             return

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock changed from success to error
WMComponent_t.PhEDExInjector_t.PhEDExInjectorPoller_t.PhEDExInjectorPollerTest:test_CustodialSiteA changed from success to error
WMComponent_t.PhEDExInjector_t.PhEDExInjectorSubscriber_t.PhEDExInjectorSubscriberTest:testNormalModeSubscriptions changed from failure to error
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testEndPolicyNegotiating changed from failure to error
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testInvalidSpecs changed from success to error

Larger replay has problem, seems I need to keep debugging this. Also will address the problems found in the jenkins test. Fixed all unit test problems apart from the last two as I don't understand why it fails there now. If someone has a suggestion...

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/661/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py decreased from 9.57 to 9.38
Score for test/python/WMComponent_t/DBS3Buffer_t/DBSUpload_t.py decreased from 9.52 to 9.39
Errors for test/python/WMComponent_t/DBS3Buffer_t/DBSUpload_t.py increased from 2 to 3
Score for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorPoller_t.py decreased from 9.65 to 9.35
Errors for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorPoller_t.py increased from 0 to 1
Score for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorSubscriber_t.py decreased from 9.38 to 9.10
Errors for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorSubscriber_t.py increased from 1 to 2
Score for src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py decreased from 9.40 to 9.26
Errors for src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py increased from 0 to 1
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors
Score for test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py decreased from 9.93 to 9.82
Errors for test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py increased from 0 to 1
Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py decreased from 9.50 to 8.57

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

test/python/WMComponent_t/DBS3Buffer_t/DBSUpload_t.py got worse in pylint. Report follows:

Line 2 E0012 Bad option value 'W6501'
Line 29 E0611 No name 'DBSBufferBlock' in module 'WMComponent.DBS3Buffer.DBSBufferBlock'
Line 165 in DBSUploadTest.createFilesWithChildren R0201 Method could be a function
Line 267 in DBSUploadTest.verifyData W0622 Redefining built-in 'file'
Line 238 in DBSUploadTest.verifyData W0612 Unused variable 'junk'
Line 238 in DBSUploadTest.verifyData W0612 Unused variable 'primary'
Line 239 in DBSUploadTest.verifyData W0612 Unused variable 'procVer'
Line 232 in DBSUploadTest.verifyData R0912 Too many branches (16/12)
Line 415 in DBSUploadTest.testDualUpload W0622 Redefining built-in 'file'
Line 392 in DBSUploadTest.testDualUpload E0602 Undefined variable 'config'
Line 612 in DBSUploadTest.testCloseSettingsPerWorkflow W0702 No exception type(s) specified
Line 463 in DBSUploadTest.testCloseSettingsPerWorkflow R0912 Too many branches (15/12)
Line 463 in DBSUploadTest.testCloseSettingsPerWorkflow R0915 Too many statements (109/100)
Line 37 in DBSUploadTest R0904 Too many public methods (24/20)
Line 14 W0611 Unused import logging
Line 21 W0611 Unused import Configuration
plus 120 comments on code style

test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorPoller_t.py got worse in pylint. Report follows:

Line 17 E0611 No name 'DBSBufferBlock' in module 'WMComponent.DBS3Buffer.DBSBufferBlock'
Line 28 in PhEDExInjectorPollerTest R0902 Too many instance attributes (10/7)
Line 78 in PhEDExInjectorPollerTest.stuffDatabase W0613 Unused argument 'spec'
Line 250 in PhEDExInjectorPollerTest.retrieveReplicaInfoForBlock W1201 Specify string format arguments as logging function parameters
Line 271 in PhEDExInjectorPollerTest.testPoller W0622 Redefining built-in 'file'
Line 263 in PhEDExInjectorPollerTest.testPoller W0101 Unreachable code
Line 28 in PhEDExInjectorPollerTest R0904 Too many public methods (21/20)
Line 165 in PhEDExInjectorPollerTest.stuffDatabase W0201 Attribute 'blockAName' defined outside init
Line 166 in PhEDExInjectorPollerTest.stuffDatabase W0201 Attribute 'blockBName' defined outside init
plus 104 comments on code style

test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorSubscriber_t.py got worse in pylint. Report follows:

Line 14 E0611 No name 'DBSBufferBlock' in module 'WMComponent.DBS3Buffer.DBSBufferBlock'
Line 32 in PhEDExInjectorSubscriberTest R0902 Too many instance attributes (9/7)
Line 263 in PhEDExInjectorSubscriberTest.testNormalModeSubscriptions E1101 Instance of 'PhEDEx' has no 'subRequests' member
Line 185 in PhEDExInjectorSubscriberTest.stuffDatabase W0201 Attribute 'blockAName' defined outside init
Line 186 in PhEDExInjectorSubscriberTest.stuffDatabase W0201 Attribute 'blockBName' defined outside init
Line 26 W0611 Unused import Workflow
Line 24 W0611 Unused import Fileset
Line 23 W0611 Unused import File
Line 25 W0611 Unused import Subscription
plus 86 comments on code style

src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py got worse in pylint. Report follows:

Line 41 E0611 No name 'DBSBufferBlock' in module 'WMComponent.DBS3Buffer.DBSBufferBlock'
Line 90 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 116 in uploadWorker W0703 Catching too general exception Exception
Line 113 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 121 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 122 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 123 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 137 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 149 in DBSUploadPoller R0902 Too many instance attributes (21/7)
Line 246 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 257 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 317 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 318 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 329 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 336 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 349 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 356 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 391 in DBSUploadPoller.loadFiles W1201 Specify string format arguments as logging function parameters
Line 453 in DBSUploadPoller.isBlockOpen R0201 Method could be a function
Line 567 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 568 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 590 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 608 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 612 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 508 in DBSUploadPoller.inputBlocks R0912 Too many branches (16/12)
Line 682 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 701 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 625 in DBSUploadPoller.retrieveBlocks R0912 Too many branches (13/12)
Line 729 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
Line 739 in DBSUploadPoller.checkBlocks W0703 Catching too general exception Exception
Line 760 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
plus 77 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py got worse in pylint. Report follows:

Line 18 E0611 No name 'DBSBufferBlock' in module 'WMComponent.DBS3Buffer.DBSBufferBlock'
Line 56 in DBSBufferFileTest.testCreateDeleteExists R0201 Method could be a function
Line 85 in DBSBufferFileTest.testCreateTransaction R0201 Method could be a function
Line 119 in DBSBufferFileTest.testDeleteTransaction R0201 Method could be a function
Line 159 in DBSBufferFileTest.testGetParentLFNs R0201 Method could be a function
Line 226 in DBSBufferFileTest.testLoad R0201 Method could be a function
Line 270 in DBSBufferFileTest.testAddChild R0201 Method could be a function
Line 319 in DBSBufferFileTest.testAddChildTransaction R0201 Method could be a function
Line 387 in DBSBufferFileTest.testSetLocation R0201 Method could be a function
Line 420 in DBSBufferFileTest.testSetLocationTransaction R0201 Method could be a function
Line 473 in DBSBufferFileTest.testLocationsConstructor R0201 Method could be a function
Line 525 in DBSBufferFileTest.testAddRunSet R0201 Method could be a function
Line 789 in DBSBufferFileTest.testSetLocationByLFN W0612 Unused variable 'myThread'
Line 824 in DBSBufferFileTest.testAddCKSumByLFN W0612 Unused variable 'checksums'
Line 849 in DBSBufferFileTest.testBulkLoad W0612 Unused variable 'bulkLoad'
Line 20 in DBSBufferFileTest R0904 Too many public methods (36/20)
plus 408 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 50 in FindDASToUpload.makeDAS W0612 Unused variable 'junk'
Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

Pre-python 2.6 idioms found in changed files. Please consider updating the code. Suggested patch follows:
--- test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py (original)
+++ test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py (refactored)
@@ -250,18 +250,18 @@
         assert testFileA == testFileC, \
                "ERROR: File load by ID didn't work"

-        assert type(testFileB["id"]) == int or type(testFileB["id"]) == long, \
+        assert isinstance(testFileB["id"], int) or isinstance(testFileB["id"], long), \
                "ERROR: File id is not an integer type."
-        assert type(testFileB["size"]) == int or type(testFileB["size"]) == long, \
+        assert isinstance(testFileB["size"], int) or isinstance(testFileB["size"], long), \
                "ERROR: File size is not an integer type."
-        assert type(testFileB["events"]) == int or type(testFileB["events"]) == long, \
+        assert isinstance(testFileB["events"], int) or isinstance(testFileB["events"], long), \
                "ERROR: File events is not an integer type."

-        assert type(testFileC["id"]) == int or type(testFileC["id"]) == long, \
+        assert isinstance(testFileC["id"], int) or isinstance(testFileC["id"], long), \
                "ERROR: File id is not an integer type."
-        assert type(testFileC["size"]) == int or type(testFileC["size"]) == long, \
+        assert isinstance(testFileC["size"], int) or isinstance(testFileC["size"], long), \
                "ERROR: File size is not an integer type."
-        assert type(testFileC["events"]) == int or type(testFileC["events"]) == long, \
+        assert isinstance(testFileC["events"], int) or isinstance(testFileC["events"], long), \
                "ERROR: File events is not an integer type."

         testFileA.delete()

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testAddCKSumByLFN was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testAddChild was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testAddChildTransaction was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testAddParents was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testAddRunSet was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testBulkLoad was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testCountFilesDAO was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testCreateDeleteExists was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testCreateTransaction was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testDeleteTransaction was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testGetChildrenDAO was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testGetParentLFNs was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testGetParentStatusDAO was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testLoad was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testLocationsConstructor was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testProperties was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetLocation was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetLocationByLFN was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetLocationTransaction was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success
WMComponent_t.DBS3Buffer_t.DBSUpload_t.DBSUploadTest:testCloseSettingsPerWorkflow was deleted. Prior status was success
WMComponent_t.PhEDExInjector_t.PhEDExInjectorPoller_t.PhEDExInjectorPollerTest:test_CustodialSiteA was deleted. Prior status was success
WMComponent_t.PhEDExInjector_t.PhEDExInjectorSubscriber_t.PhEDExInjectorSubscriberTest:testNormalModeSubscriptions was deleted. Prior status was failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testContinuousSplittingSupport changed from success to failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testDataDirectiveFromQueue changed from success to error
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testDatasetLocation changed from success to failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testLumiSplitTier1ReRecoWorkload changed from success to failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testMultiTaskProcessingWorkload changed from success to failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testPileupData changed from success to failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testRunWhitelist changed from success to error
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testTier1ReRecoWorkload changed from success to failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testWhiteBlackLists changed from success to failure
WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t.BlockTestCase:testWithMaskedBlocks changed from success to error
WMCore_t.WorkQueue_t.Policy_t.Start_t.Dataset_t.DatasetTestCase:testDataDirectiveFromQueue changed from error to success
WMCore_t.WorkQueue_t.Policy_t.Start_t.Dataset_t.DatasetTestCase:testMultiTaskProcessingWorkload changed from error to success
WMCore_t.WorkQueue_t.Policy_t.Start_t.Dataset_t.DatasetTestCase:testParentProcessing changed from failure to success
WMCore_t.WorkQueue_t.Policy_t.Start_t.Dataset_t.DatasetTestCase:testTier1ReRecoWorkload changed from error to success
WMCore_t.WorkQueue_t.Policy_t.Start_t.Dataset_t.DatasetTestCase:testWhiteBlackLists changed from error to success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testBlackList changed from failure to success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testCloseWorkOpenBlocks changed from failure to success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testGlobalDatasetSplitting changed from error to failure
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testParentProcessing changed from failure to success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testProcessing changed from failure to success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testSplittingLargeInputs changed from failure to success

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/662/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py decreased from 9.57 to 9.38
Score for src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py decreased from 9.40 to 9.38
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors
Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py decreased from 9.50 to 8.57

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py got worse in pylint. Report follows:

Line 90 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 116 in uploadWorker W0703 Catching too general exception Exception
Line 113 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 121 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 122 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 123 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 137 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 149 in DBSUploadPoller R0902 Too many instance attributes (21/7)
Line 246 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 257 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 317 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 318 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 329 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 336 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 349 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 356 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 391 in DBSUploadPoller.loadFiles W1201 Specify string format arguments as logging function parameters
Line 453 in DBSUploadPoller.isBlockOpen R0201 Method could be a function
Line 567 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 568 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 590 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 608 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 612 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 508 in DBSUploadPoller.inputBlocks R0912 Too many branches (16/12)
Line 682 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 701 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 625 in DBSUploadPoller.retrieveBlocks R0912 Too many branches (13/12)
Line 729 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
Line 739 in DBSUploadPoller.checkBlocks W0703 Catching too general exception Exception
Line 760 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
plus 77 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 50 in FindDASToUpload.makeDAS W0612 Unused variable 'junk'
Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

Pre-python 2.6 idioms found in changed files. Please consider updating the code. Suggested patch follows:
--- test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py (original)
+++ test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py (refactored)
@@ -250,18 +250,18 @@
         assert testFileA == testFileC, \
                "ERROR: File load by ID didn't work"

-        assert type(testFileB["id"]) == int or type(testFileB["id"]) == long, \
+        assert isinstance(testFileB["id"], int) or isinstance(testFileB["id"], long), \
                "ERROR: File id is not an integer type."
-        assert type(testFileB["size"]) == int or type(testFileB["size"]) == long, \
+        assert isinstance(testFileB["size"], int) or isinstance(testFileB["size"], long), \
                "ERROR: File size is not an integer type."
-        assert type(testFileB["events"]) == int or type(testFileB["events"]) == long, \
+        assert isinstance(testFileB["events"], int) or isinstance(testFileB["events"], long), \
                "ERROR: File events is not an integer type."

-        assert type(testFileC["id"]) == int or type(testFileC["id"]) == long, \
+        assert isinstance(testFileC["id"], int) or isinstance(testFileC["id"], long), \
                "ERROR: File id is not an integer type."
-        assert type(testFileC["size"]) == int or type(testFileC["size"]) == long, \
+        assert isinstance(testFileC["size"], int) or isinstance(testFileC["size"], long), \
                "ERROR: File size is not an integer type."
-        assert type(testFileC["events"]) == int or type(testFileC["events"]) == long, \
+        assert isinstance(testFileC["events"], int) or isinstance(testFileC["events"], long), \
                "ERROR: File events is not an integer type."

         testFileA.delete()

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/663/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/664/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/665/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/666/ for details

retest this

please retest this

retest this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/667/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/668/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/670/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/672/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/676/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/677/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/678/ for details

restest this please

retest this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/683/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/684/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/685/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/687/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/688/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/689/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py decreased from 9.57 to 9.38
Score for src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py decreased from 9.40 to 9.38
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors
Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py decreased from 9.50 to 8.57

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py got worse in pylint. Report follows:

Line 90 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 116 in uploadWorker W0703 Catching too general exception Exception
Line 113 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 121 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 122 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 123 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 137 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 149 in DBSUploadPoller R0902 Too many instance attributes (21/7)
Line 246 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 257 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 317 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 318 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 329 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 336 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 349 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 356 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 391 in DBSUploadPoller.loadFiles W1201 Specify string format arguments as logging function parameters
Line 453 in DBSUploadPoller.isBlockOpen R0201 Method could be a function
Line 567 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 568 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 590 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 608 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 612 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 508 in DBSUploadPoller.inputBlocks R0912 Too many branches (16/12)
Line 682 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 701 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 625 in DBSUploadPoller.retrieveBlocks R0912 Too many branches (13/12)
Line 729 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
Line 739 in DBSUploadPoller.checkBlocks W0703 Catching too general exception Exception
Line 760 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
plus 77 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 50 in FindDASToUpload.makeDAS W0612 Unused variable 'junk'
Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success

test this please

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/696/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/698/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py decreased from 9.57 to 9.38
Score for src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py decreased from 9.40 to 9.38
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors
Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py decreased from 9.50 to 8.57

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

src/python/WMComponent/DBS3Buffer/DBSUploadPoller.py got worse in pylint. Report follows:

Line 90 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 116 in uploadWorker W0703 Catching too general exception Exception
Line 113 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 121 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 122 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 123 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 137 in uploadWorker W1201 Specify string format arguments as logging function parameters
Line 149 in DBSUploadPoller R0902 Too many instance attributes (21/7)
Line 246 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 257 in DBSUploadPoller.close W0703 Catching too general exception Exception
Line 317 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 318 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 329 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 336 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 349 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 356 in DBSUploadPoller.loadBlocks W1201 Specify string format arguments as logging function parameters
Line 391 in DBSUploadPoller.loadFiles W1201 Specify string format arguments as logging function parameters
Line 453 in DBSUploadPoller.isBlockOpen R0201 Method could be a function
Line 567 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 568 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 590 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 608 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 612 in DBSUploadPoller.inputBlocks W1201 Specify string format arguments as logging function parameters
Line 508 in DBSUploadPoller.inputBlocks R0912 Too many branches (16/12)
Line 682 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 701 in DBSUploadPoller.retrieveBlocks W1201 Specify string format arguments as logging function parameters
Line 625 in DBSUploadPoller.retrieveBlocks R0912 Too many branches (13/12)
Line 729 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
Line 739 in DBSUploadPoller.checkBlocks W0703 Catching too general exception Exception
Line 760 in DBSUploadPoller.checkBlocks W1201 Specify string format arguments as logging function parameters
plus 77 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 50 in FindDASToUpload.makeDAS W0612 Unused variable 'junk'
Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testParentProcessing changed from failure to error
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testProcessingWithContinuousSplitting changed from failure to error

Tier0 deployment does not need this to be merged btw. I still would appreciate any comments on the code and the deployment plans. @ticoann @amaltaro @ericvaandering

Hi Dirk, sorry I haven't review the code yet. How was the testing going with large data? As far as for the merging to branch it won't be merged until next week since we need to run agent test as well.
I will try to review the code, it helps that you summarized the changes you made. Thanks for that.

@tiocann, I made some further changes to clean up transaction handling and hopefully fix the crash that caused the whole mess with the backlog in the first place. That crash is still there btw, we saw it in the Tier0 over the weekend.
I am going to deploy this change in the Tier0 either way, but do you prefer it as part of this pull request or should I make a new one ? If the latter I need to wait until this is merged (there are conflicts between them, I can make a clean new pull request that also works together with this one).

Hi Dirk, you can add that here and part of same request. Thanks,

Ok, thanks, will do once the replay testing the new code is done.

Additional changes are to remove transactions within transactions (which some calls to DBSBufferUtil were doing). No idea what the system is actually doing then, but the crashes we saw were in one of these calls. I not only cleaned up that call but also other areas with the same pattern.

Also remove a bunch of methods from DBSBufferUtil that weren't used anywhere and remove the transaction from a simple SELECT call (not needed for that).

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/748/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/750/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py changed from 9.50 to 8.57 with 0 (2) total errors (warnings)
Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py changed from 9.57 to 9.38 with 0 (1) total errors (warnings)
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/DBSBufferBlock.py follows:

Line 227 in DBSBufferBlock.setProcessingVer W0612 Unused variable 'junk'
Line 293 in DBSBufferBlock.setDataset W0612 Unused variable 'junk'
Line 273 in DBSBufferBlock.setDataset W0613 Unused argument 'valid'
Line 296 in DBSBufferBlock.setDataset W0612 Unused variable 'ex'
plus 6 total warnings
plus 63 comments on code style

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/DBSBufferUtil.py follows:

Line 22 in DBSBufferUtil.init W0613 Unused argument 'dbfactory'
Line 8 W0611 Unused import logging
Line 11 W0611 Unused import WMFactory
plus 4 total warnings
plus 32 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 50 in FindDASToUpload.makeDAS W0612 Unused variable 'junk'
Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/MySQL/LoadDBSFilesByDAS.py follows:

Line 11 W0611 Unused import logging
plus 3 total warnings
plus 27 comments on code style

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/MySQL/LoadFilesByBlock.py follows:

Line 11 W0611 Unused import logging
plus 1 total warnings
plus 24 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py follows:

Line 789 in DBSBufferFileTest.testSetLocationByLFN W0612 Unused variable 'myThread'
Line 824 in DBSBufferFileTest.testAddCKSumByLFN W0612 Unused variable 'checksums'
Line 849 in DBSBufferFileTest.testBulkLoad W0612 Unused variable 'bulkLoad'
plus 3 total warnings
plus 408 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/DBS3Buffer_t/DBSUpload_t.py follows:

Line 2 E0012 Bad option value 'W6501'
Line 238 in DBSUploadTest.verifyData W0612 Unused variable 'junk'
Line 238 in DBSUploadTest.verifyData W0612 Unused variable 'primary'
Line 239 in DBSUploadTest.verifyData W0612 Unused variable 'procVer'
Line 392 in DBSUploadTest.testDualUpload E0602 Undefined variable 'config'
Line 14 W0611 Unused import logging
Line 21 W0611 Unused import Configuration
plus 8 total warnings
plus 120 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorPoller_t.py follows:

Line 78 in PhEDExInjectorPollerTest.stuffDatabase W0613 Unused argument 'spec'
plus 6 total warnings
plus 104 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorSubscriber_t.py follows:

Line 263 in PhEDExInjectorSubscriberTest.testNormalModeSubscriptions E1101 Instance of 'PhEDEx' has no 'subRequests' member
Line 26 W0611 Unused import Workflow
Line 24 W0611 Unused import Fileset
Line 23 W0611 Unused import File
Line 25 W0611 Unused import Subscription
plus 6 total warnings
plus 86 comments on code style

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success
WMCore_t.WebTools_t.NestedModel_t.NestedModelTest:testInnerPingPass changed from success to failure
WMCore_t.WorkQueue_t.WorkQueue_t.WorkQueueTest:testResubmissionWorkflow changed from failure to success

Previously working unit tests have failed!

Needs to be rebased

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/766/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/769/ for details

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/773/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py changed from 9.50 to 8.57 with 0 (2) total errors (warnings)
Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py changed from 9.57 to 9.38 with 0 (1) total errors (warnings)
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/DBSBufferBlock.py follows:

Line 227 in DBSBufferBlock.setProcessingVer W0612 Unused variable 'junk'
Line 293 in DBSBufferBlock.setDataset W0612 Unused variable 'junk'
Line 273 in DBSBufferBlock.setDataset W0613 Unused argument 'valid'
Line 296 in DBSBufferBlock.setDataset W0612 Unused variable 'ex'
plus 6 total warnings
plus 63 comments on code style

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/DBSBufferUtil.py follows:

Line 22 in DBSBufferUtil.init W0613 Unused argument 'dbfactory'
Line 8 W0611 Unused import logging
Line 11 W0611 Unused import WMFactory
plus 4 total warnings
plus 32 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 50 in FindDASToUpload.makeDAS W0612 Unused variable 'junk'
Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/MySQL/LoadDBSFilesByDAS.py follows:

Line 11 W0611 Unused import logging
plus 3 total warnings
plus 27 comments on code style

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/MySQL/LoadFilesByBlock.py follows:

Line 11 W0611 Unused import logging
plus 1 total warnings
plus 24 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/DBS3Buffer_t/DBSBufferFile_t.py follows:

Line 789 in DBSBufferFileTest.testSetLocationByLFN W0612 Unused variable 'myThread'
Line 824 in DBSBufferFileTest.testAddCKSumByLFN W0612 Unused variable 'checksums'
Line 849 in DBSBufferFileTest.testBulkLoad W0612 Unused variable 'bulkLoad'
plus 3 total warnings
plus 408 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/DBS3Buffer_t/DBSUpload_t.py follows:

Line 2 E0012 Bad option value 'W6501'
Line 238 in DBSUploadTest.verifyData W0612 Unused variable 'junk'
Line 238 in DBSUploadTest.verifyData W0612 Unused variable 'primary'
Line 239 in DBSUploadTest.verifyData W0612 Unused variable 'procVer'
Line 392 in DBSUploadTest.testDualUpload E0602 Undefined variable 'config'
Line 14 W0611 Unused import logging
Line 21 W0611 Unused import Configuration
plus 8 total warnings
plus 120 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorPoller_t.py follows:

Line 78 in PhEDExInjectorPollerTest.stuffDatabase W0613 Unused argument 'spec'
plus 6 total warnings
plus 104 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorSubscriber_t.py follows:

Line 263 in PhEDExInjectorSubscriberTest.testNormalModeSubscriptions E1101 Instance of 'PhEDEx' has no 'subRequests' member
Line 26 W0611 Unused import Workflow
Line 24 W0611 Unused import Fileset
Line 23 W0611 Unused import File
Line 25 W0611 Unused import Subscription
plus 6 total warnings
plus 86 comments on code style

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/776/ for details

Summary of pylint changes for pull request 6288:

Score for src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py changed from 9.50 to 9.29 with 0 (1) total errors (warnings)
Score for src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py changed from 9.57 to 9.38 with 0 (1) total errors (warnings)
New file src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py with score 10.00, 0 warnings, and 0 errors

Abbreviated pylint report for src/python/WMComponent/DBS3Buffer/DBSBufferBlock.py follows:

Line 294 in DBSBufferBlock.setDataset W0612 Unused variable 'junk'
plus 3 total warnings
plus 62 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/FindDASToUpload.py got worse in pylint. Report follows:

Line 40 in FindDASToUpload.makeDAS R0201 Method could be a function
Line 58 in FindDASToUpload.execute W0221 Arguments number differs from overridden method
plus 8 comments on code style

src/python/WMComponent/DBS3Buffer/MySQL/LoadBlocks.py got worse in pylint. Report follows:

Line 38 in LoadBlocks.execute W0222 Signature differs from overridden method
plus 9 comments on code style

src/python/WMComponent/DBS3Buffer/Oracle/FindDASToUpload.py got worse in pylint. Report follows:

plus 1 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/DBS3Buffer_t/DBSUpload_t.py follows:

Line 2 E0012 Bad option value 'W6501'
Line 236 in DBSUploadTest.verifyData W0612 Unused variable 'junk'
Line 236 in DBSUploadTest.verifyData W0612 Unused variable 'primary'
Line 237 in DBSUploadTest.verifyData W0612 Unused variable 'procVer'
Line 390 in DBSUploadTest.testDualUpload E0602 Undefined variable 'config'
plus 6 total warnings
plus 120 comments on code style

Abbreviated pylint report for test/python/WMComponent_t/PhEDExInjector_t/PhEDExInjectorSubscriber_t.py follows:

Line 259 in PhEDExInjectorSubscriberTest.testNormalModeSubscriptions E1101 Instance of 'PhEDEx' has no 'subRequests' member
plus 2 total warnings
plus 86 comments on code style

Unit test changes for pull request 6288:

WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testSetBlock was added. Status is success
WMComponent_t.DBS3Buffer_t.DBSBufferFile_t.DBSBufferFileTest:testXSetBlock was deleted. Prior status was success

So this is rebased and the last jenkins test worked. @ticoann , can you review this please ?

@amaltaro , please include this in your testing.

Ok. I guess @ticoann and I will cut a new tag today after we merge all these stuff.


PP-4276: Use direct debit connector's DropwizardJUnitRunner

This commit largely copies's dd-connector's DropwizardJUnitRunner which spins
up the dropwizard app and postgres container only once for all tests that use
it. This commit reduces the build time from ~20minutes to <10minutes.
In the ChargingITestBase we have had to make the
[worldpay|smartpay|epdq]MockClients static as some tests fail when all the
tests are being run. Probably some weird connection pooling issue.
@oswaldquek, @georgeracu

@IreneLau-GDS With regards to the truncate cascade comment, originally I didn't want to do this. In an ideal world I would re-create the database (using a template) for each test. That's what we do in dd-connector. However we can't re-create the database like we do in dd-connector as we use JPA (dd-connector using JDBI) and JPA doesn't like the fact we re-create the database. To get around this we truncate the database here. I appreciate it's not ideal but applying the concept of YAGNI, the tests work fine and if we encounter a problem in the future because of this we should deal with it then.


test: Add Java test [TSI-2113]

Adds basic test for Java library
https://phrase.atlassian.net/browse/TSI-2113

@theSoenke  I was looking into moving .gitignore chunks into appropriate subdirectories, but they already contain generated .gitignore files, so we'd have to customize corresponding mustache templates, which felt a bit wrong. also, having all these in a single place makes more sense IMO.


[HUDI-5239] support HoodieJavaWriteClient compact

Change Logs
Describe context and summary for this change. Highlight if any code was copied.
Impact
no
Risk level (write none, low medium or high below)
Risk level: none
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

[ ] Read through contributor's guide
[ ] Change Logs and Impact were stated clearly
[ ] Adequate tests were added if applicable
[ ] CI passed

@ymZhao1001 can you add unit tests pls?


Getting Wrong data for TATASTEEL

Hi,
I am getting wrong prices for TATASTEEL while using capital_market.price_volume_data.
this issue is related to stock splits,price are not adjusted when stock-split happened.
same issue can be seen in RAMASTEEL also.
here is my code:

Actual Data from Yahoo Finance:

Note : I have nselib with version 1.0

Please have a look,
Thanks !

Hi Avani,
You are right, This library is to get the data , what ever present in the NSE sight (https://www.nseindia.com/).
I do not want to manipulate the data in this library, If you want history dates you can use yfinance library (https://pypi.org/project/yfinance/), the history data already available there.
Thanks,
Tanmay

Thanks.

Hi Avni,
nselib is extracting data, what is available in nse india site. Please
refer the image.
[image: image.png]
Thanks,
Tanmay
On Tue, Sep 3, 2024 at 2:25 PM Avani-a2p @.***> wrote:

Hi,
I am getting wrong prices for TATASTEEL while using
capital_market.price_volume_data.
here is my code:
image.png (view on web)
https://github.com/user-attachments/assets/a3637a00-d0b6-4a8e-aaef-f9fc30fd7ffe
Actual Data from Yahoo Finance:
image.png (view on web)
https://github.com/user-attachments/assets/68dda228-eb03-401c-9670-1414ec52c550
Note : I have nselib with version 1.0
image.png (view on web)
https://github.com/user-attachments/assets/c1b81902-0f3d-4248-bbe4-485c652734df
Thanks !
—
Reply to this email directly, view it on GitHub
https://github.com/RuchiTanmay/nselib/issues/32, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AVPTOM3HDX2NUHHKREF7BDDZUV2PVAVCNFSM6AAAAABNRUYIKKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGUYDEMZQGIZTQMQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

--
Thanks & Regards
Tanmay

Hi Tanmay,
I am not able to access the image.
well, i understood your point. Thank you so much for your suggestions,
capital_market.equity_list and yf.download are exactly what i wanted.
both are time efficient and accurate !
Thanks again.


Flaky Tests: AdminApiSchemaTest#testSchemaInfoApi

Motivation
I see AdminApiSchemaTest#testSchemaInfoApi failing in this way:
2021-10-22T01:01:52,307+0000 [jersey-client-async-executor-5] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [http://localhost:38587/admin/v2/schemas/schematest/test/test-INT32/schema] Failed to perform http post request: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?) (through reference chain: org.apache.pulsar.common.protocol.schema.PostSchemaPayload["properties"]->java.util.HashMap["null"])

It only fails with INT32 default shema.
I found another test (SchemaTest#testNullKeyValueProperty) which sets a null-key property to INT32 instance; so if it runs before, it makes AdminApiSchemaTest#testSchemaInfoApi fail.
Modifications
Changed the SchemaTest#testNullKeyValueProperty test in order to leave the default INT32 implementation unchanged.
Alternatively, I could made the properties field final, at least for defaults SchemaInfos but this would result in a breaking change.
This change is a trivial rework / code cleanup without any test coverage.
Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?

[x] no-need-doc

@lhotari @codelipenghui PTAL


Filter fix

fixed issue Els described where record count was giving 1 more record than expected.  This was due to the filtered dataframe being overwritten during iteration.

I updated the multi-filter test case--it fails for the way filter was written as it would only filter on the second key:value in the filter and return 2; if it filters on both parts of the filter, it will return 1 which this fix does now.


[#1271] improvement(server): change transportTime and processTime summary to Thread Pool Instead of block

Change transportTime and processTime summary to Thread Pool Instead of block
Closes #1271
What changes were proposed in this pull request?
Change transportTime and processTime summary to Thread Pool Instead of block
Why are the changes needed?
Not block GRPC Thread
Fix: #1271
Does this PR introduce any user-facing change?
No.
How was this patch tested?
No

We're currently processing your upload.  This comment will be updated when the results are available.

What will it happen if we don't have this pr?

@lifeSo Could you change your title? We should use [#xxxx] or [MINOR] as the start of our title.

@lifeSo Could you change your title? We should use [#xxxx] or [MINOR] as the start of our title.

Ok, I changed it.

What will it happen if we don't have this pr?

GRPC Thread will block on the metric calc.
Here is the jstack from PRD, We find many GRPC Thread blocks.
So, if we don't have this pr, the GRPC Thread will block.

What will it happen if we don't have this pr?

GRPC Thread will block on the metric calc. Here is the jstack from PRD, We find many GRPC Thread blocks. So, if we don't have this pr, the GRPC Thread will block.

Will it cause many requests timeout?

What will it happen if we don't have this pr?

GRPC Thread will block on the metric calc. Here is the jstack from PRD, We find many GRPC Thread blocks. So, if we don't have this pr, the GRPC Thread will block.

Will it cause many requests timeout?

It will cause the GRPC block, then GRPC request queue come with many request task.
Some client will read shuffle data fail

What will it happen if we don't have this pr?

GRPC Thread will block on the metric calc. Here is the jstack from PRD, We find many GRPC Thread blocks. So, if we don't have this pr, the GRPC Thread will block.

Will it cause many requests timeout?

It will cause the GRPC block, then GRPC request queue come with many request task. Some client will read shuffle data fail

OK, I know.


V4/trip style

Motivation
The legend on trips graphs looked a little wonky.
Changes
Redid style for legend and download button

Testing Instructions
👀

Based on just the screenshots, make sure the padding on the download button and the legend leave enough room on the left and right on both travel times and dwells

Look good? @devinmatte

How does dwells look? The download button looked a little close to the left in the first screenshot?

Oh yea, good call. Just fixed that


Test-Proxy should exponentially back off when in record mode

On error code 429, we should retry, but actually back off between attempts to see if we can mitigate throttling errors.

429 also sometimes come with a Retry-After header which you can try and use.


Basic fields and output panel

I have started working on fields and graphical output. As of now the interface displays a basic plot with minimum styling. I will be updating this section according to the points discussed in the e-meet.
User can create plot for any multi-dimensional variable by selecting x and y. For example: Selectiing variable temp wirh x as lat and y as lon . Then clicking on the Plot button produces the output.

Please ping us when you'd like a review of the work here.

Present Status:

@martindurant @rsignell-usgs I think most of the work in this section is complete. Please have a look.

We can't plot the variables which are Data Coordinates beacause of:
Python failed with the following traceback:
/home/hdsingh/Desktop/xrviz/xrviz/sigslot.py _emit L63
/home/hdsingh/Desktop/xrviz/xrviz/dashboard.py create_plot L59
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/hvplot/__init__.py quadmesh L579
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/hvplot/__init__.py __call__ L183
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/hvplot/__init__.py _get_converter L191
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/hvplot/converter.py __init__ L234
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/hvplot/converter.py _process_data L491
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/hvplot/util.py process_xarray L252
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/xarray/core/dataarray.py to_dataset L358
/home/hdsingh/anaconda3/envs/check3/lib/python3.7/site-packages/xarray/core/dataarray.py _to_dataset_whole L315
ValueError: cannot create a Dataset from a DataArray with the same name as one of its coordinates

Also we can't plot the variables which are one dimensional. So I have disabled plot button for these variables.

We can't plot the variables which are Data Coordinates

Not sure what you did to get that traceback, but the following do work
In [2]: ds = xr.open_dataset('great_lakes.nc')

In [3]: ds['lat']
Out[3]:
<xarray.DataArray 'lat' (ny: 25, nx: 61)>
array([[43.14354, 43.1445 , 43.14544, ..., 43.14364, 43.14264, 43.14162],
       [43.18853, 43.1895 , 43.19042, ..., 43.18863, 43.18763, 43.18661],
       [43.23352, 43.23448, 43.23541, ..., 43.23362, 43.23262, 43.23159],
       ...,
       [44.13329, 44.13425, 44.13518, ..., 44.13338, 44.13239, 44.13136],
       [44.17828, 44.17924, 44.18017, ..., 44.17837, 44.17738, 44.17635],
       [44.22327, 44.22423, 44.22516, ..., 44.22336, 44.22237, 44.22134]],
      dtype=float32)
Dimensions without coordinates: ny, nx
Attributes:
    long_name:      Latitude
    units:          degrees_north
    standard_name:  latitude

In [4]: ds['nx']
Out[4]:
<xarray.DataArray 'nx' (nx: 61)>
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,
       36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
       54, 55, 56, 57, 58, 59, 60])
Dimensions without coordinates: nx

plotting lat as a function of nx, ny seems like a valid thing for a user to want to do.
Agreed that we can't make a 2D map out of a 1D array.

This works well:
ds = xr.open_dataset('great_lakes.nc')
ds.lat.hvplot.quadmesh()

however when we set lat as Data Coord:
ds = xr.open_dataset('great_lakes.nc')
ds.set_coords(['lat',],inplace = True)
ds.lat.hvplot.quadmesh()

we get the error mentioned above.

Similarly, for another dataset:
url = 'http://gamone.whoi.edu/thredds/dodsC/coawst_4/use/fmrc/coawst_4_use_best.ncd'
ds = xr.open_dataset(url)

which as lat_u,lat_v,lon_u as one of the data coords none of the following work:
ds.lat_u.hvplot.quadmesh()
ds.lat_v.hvplot.quadmesh()
ds.lon_u.hvplot.quadmesh()

However for h which is not a data coord:
ds.h.hvplot.quadmesh() a graph is created.

OK, so it makes sense that you would not want to do set_coords in the case that you want to visualise those coords. I don't imagine it takes any computation on xarray's part, we can leave the logic on whether to use set_coords until after the user has chosen their fields, no?

Generally, I think this is great! It is showing the whole workflow as an application.
Running examples/control_section.py, I am seeing errors like
holoviews.core.data.interface.DataError: xarray Dataset must define coordinates for all defined kdims, [Dimension('time'), Dimension('lat'), Dimension('index')] coordinates not found.

if I select "time" as one of the axes to plot. Presumably we should be able to infer that that selection combination is inconsistent.
The coordinate variables nx, ny do not show up in the list of fields to plot agains, but they should. In the example script, there is a .set_coords(['lat', 'lon']), which must be optional, and done within the interface code itself. This is important, because I might want to be able to plot a variable as a function of nx/ny (without mapping coordinates) or even ny/time is a valid choice.
I'll comment soon on specific points in the code.

Thanks a lot for the review!

I have changed the plotting function as discussed. Now the data is selected first using data.sel and then plotted. This has solved the errors being faced earlier (incompatible dimensions). Code cleanup, rearrangement and commenting yet needs to be done, however the plotting interface works fine.

@rsignell-usgs @martindurant I have made improvements according to the conversation above. Please review the changes. Thanks!

OK, let's merge this and move on. I may yet come back to some of the choices here, but it's better if you can get on with your other PRs.


Align project dependencies

Currently the different projects call Module.Initialize down their stack. But the dependencies defined in the gir file does not match the called initialize function calls. Additionally ensure that the project dependencies align to the gir file dependencies.

I had a try at this, see https://github.com/gircore/gir.core/pull/875.


optimize align for scalars at least

What happened?
Here's a simple rescaling calculation:
import numpy as np
import xarray as xr

ds = xr.Dataset(
    {"a": (("x", "y"), np.ones((300, 400))), "b": (("x", "y"), np.ones((300, 400)))}
)
mean = ds.mean() # scalar
std = ds.std() # scalar
rescaled = (ds - mean) / std

The profile for the last line shows 30% (!!!) time spent in align (really reindex_like) except there's nothing to reindex when only scalars are involved!

This is a small example inspired by a ML pipeline where this normalization is happening very many times in a tight loop.
cc @benbovy
What did you expect to happen?
A fast path for when no reindexing needs to happen.

Maybe it would be enough to refactor Aligner.reindex_all() like this:
    def reindex_all(self) -> None:
        results = []

        for obj, matching_indexes in zip(
            self.objects, self.objects_matching_indexes
        ):
            if any(self.reindex[k] for k in matching_indexes):
                results.append(self._reindex_one(obj, matching_indexes))
            else:
                results.append(obj.copy(deep=self.copy))

        self.results = tuple(results)

Maybe it would be enough to refactor Aligner.reindex_all() like this

It depends on the profiling results below align in the call stack: if most of the time is spent in Aligner.normalize_index or Aligner.find_matching_*, optimizing this case would require a bit more work. @dcherian could you paste another screenshot with the root set at that level, please?

Does this help?

Yes, thanks. dataset.copy() is 50% of the align time. Looks like the code snippet in my previous comment won't improve the performance issue much or at all? Should we rather look into if/how to optimize copy instead?

I guess that in this case we could skip copying objects that don't need reindexing, but is it always safe or desirable? Do we have an option for this (or skip alignment altogether)?


add '.'

https://github.com/HP4k1h5/ephemeris/blob/7ea70128501ca0336467b7be4f583c070f8402ea/autoload/ephemeris/ind.vim#L19

closed by commit ee869767d7141929f5f249be1dcc0f24529d0510


PROG: Fix transport and work area

Fixes issue with incomplete deletion of inactive objects
Removes entries for deleted inactive programs/includes from work area

Closes #4151

Adds R3TR PROG to transport instead of several LIMUs

Test with https://github.com/abapGit-tests/PROG
Before:

After:

thanks 👍


Wrong hex and rgb information

Hi,
I just made a small Node with a Hue Light and Hue Group and somehow the output rgb and hex code is wrong.
For example:
Hue Group/Light input is rgb(255,0,0) or hex code: #ff0000 the reading output is rgb(255,254,88).

Hi,
yes this is correct. The reason is that directly after you update a node / light, the module tries to get the current light bulb color and you often get the color of the transition to your wished color. In your case it was a yellowish color at the time of the request.
There is also to mention, that converting the current exact color into hex / rgb is not possible, since every light bulb / model has its own light color "limits". The algorithm behind of it tries to convert your "red" into the closest possible color that your light bulb can display.
That means that you unfortunately will never get the exact hex / rgb code back from your light bulb that you initially pushed to the node until Philips makes these conversions for each light bulb specifications directly on their bridge and offers them in their API.


security fix for missing env variable replacement

Pull Request type

[x] Bugfix

NOTE: Please remember to run ./gradlew spotlessApply to fix any format violations.
Changes in this PR
The current implementation of the Conductor properties file reader, didn't substitute environment variables.
This is a major security and design flaw leading to bad setups and insecure installations in the field.
Alternatives considered
The properties system is a bit outdated and requires some work. For now we go with the security fix.

This approach is useful when constructing strings with env variables which is not the use case here.  A typical k8s based deployment environment supports sending environment variables which is a recommended way to handle deployments for production use cases.

This approach is useful when constructing strings with env variables which is not the use case here. A typical k8s based deployment environment supports sending environment variables which is a recommended way to handle deployments for production use cases.

I agree and that is what I am using in my Docker containers.
https://github.com/egandro/conductor-containers/blob/e4583d0dbbeab494a2440a6d4ad7cc9912362ecc/docker/bin/startup.sh#L23
I specifically disable the CONDUCTOR_CONFIG_FILE and go with a --spring.config.location=/app/config/application.properties.
However - you need to make a decission. If you want to have  CONDUCTOR_CONFIG_FILE either fix it the way I suggest - or - explicitly use a piece of documentation and make it a major security issue to use in production enviornment (that part of the documentation is really missing) the docker-compose files in this repo are even encuraging people going the super super insecure was - if - they are not Java Spring experts.
Fix it - document it - or delete it! Even for development mode it make no sense to have a settings and some "advanced light settings overirde - in a bad security style" enabled.


Exposing Cheating in SQL/SPLdepth: The Impact of Overlapping Training and Test Splits

From here, it can be known that this work also use the 70k+ training split. But I need to point out that the 70k training split has a critical issue: there is overlap between the training and test splits!
First, please check 2011_09_30/2011_09_30_drive_0027_sync 656 l, which appears in both the 70k training split and the Eigen test split.
Second, in the traditional Eigen split, there is no overlap in sequences between training and test sets. For example, 2011_10_03_drive_0027_sync is part of the test set and never appears in the training set. However, in the 70k split, this sequence appears 8995 times in the training set!
In other words, SQLdepth is not maintaining the proper separation and is effectively cheating.
Since the SQLdepth author released the training set in April, many people have already identified this error, but for various reasons, they did not report it. As a result, numerous papers have been based on this flawed work. I hope that everyone will be cautious and not continue to trust this repository.

Hi @ghost, thanks for pointing that out—it's helpful to know. I was also curious about why SQLdepth used 70k+ samples for the unsupervised setup but reduced it to 20k+ for fine-tuning. I agree it could affect unsupervised training settings, but considering that the metric depth training maintains a proper split, I think its impact on performance is likely minimal.


update README

Add how to download space icon to README

Codecov Report

Merging #28 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #28   +/-   ##
=======================================
  Coverage   84.90%   84.90%
=======================================
  Files          20       20
  Lines        1424     1424
=======================================
  Hits         1209     1209
  Misses        113      113
  Partials      102      102

Flag
Coverage Δ

#unittests
84.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 831352a...e41aab5. Read the comment docs.


Various logging tweaks

Do not leak CRYPTO_RWLOCK
Introduce ability to set different levels of logging verbosity for
using OpenSSL ERR and stderr/logfile. Set default level of logging for
OpenSSL ERR to ERROR to avoid problems in OpenSSL unit tests.
Update Engine id and name

    _traceLogFile = fopen(_traceLogFilename, "a");

_traceLogFilename may be (usually is) NULL here. It seems to sort of work anyway - I guess fopen just returns NULL - but valgrind catches this as as an error. Can you change this to only call fopen when _traceLogFileName != NULL?

Refers to: SymCryptEngine/src/scossl_helpers.c:160 in 2291285. [](commit_id = 22912858fce46cd2fe3b61ea70b7a946bf739ad2, deletion_comment = False)


ENHANCING THE USERINTERFACE OF THE MAIN LANDING PAGE AND MAKE IT ATTRACTIVE

### ENHANCING THE USERINTERFACE OF THE MAIN LANDING PAGE AND MAKE IT ATTRACTIVE
I can make the page very attractive and can make it interacting and can provide the next level gaming experience or the environment throught the webpage .

As a experienced frontend developer I humbly request you to assign this task to me I can resolve it very effectively.


Fixed issue where BOM wasn't being stripped from transcripts opened v…

…ia the File menu.
Fixes #1256

Pull Request Test Coverage Report for Build 2343

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 55.414%

Totals

Change from base Build 2335:
0.0%

Covered Lines:
4894

Relevant Lines:
8322

💛  - Coveralls


Current weather data not updating

Hi,
I'm using your api and there is an issue as the current weather data is not updated since yesterday, 14:30 UTC.
Thank you for your work and this great api!

Thanks for the report, see https://github.com/jdemaeyer/brightsky/issues/152#issuecomment-1703752844 :)
Follow-up: https://github.com/jdemaeyer/brightsky/issues/154


ListBox: random paddings and underrendered items at the end of the list

Fast up and down scrolling causes random rendering behavior. On the screenshots you could see that the ListBox is scrolled to the end, but there is either some unwanted padding at the end or an under-rendered item. Most of the time ListBox renders as intended.

@x2bool do you have a repository exhibiting this behavior I can take a look at?

@grokys A private repo on GitLab. I could give you access if you have an account there. #1395 and #1394 also reproducible on this repo.

Yep, I do - same username as on here!

Granted the access to you. The project to run is called "Epigram". Look for MessageExplorerControl. Navigate between chats using the left panel, and sooner or later this bug will reproduce.


Error Completing call

Did a call,
pulled up existing case
hit save
it focused to call status
I put in status other
added text
did other stuff for 3 or 4 minutes
then hit complete call
got this error:

will assume fixed.. not happing now


ttnn.pad doesn't work for some inputs

ttnn.pad is not working for some inputs.
Repro steps:
Build with debug.
Run the following pytest:
import pytest
import torch
import ttnn
from tests.ttnn.utils_for_testing import assert_with_pcc
from models.utility_functions import torch_random

def test_pad3(device):
    N = 8
    C = 1
    H = 64
    W = 2048
    input_shape = (N, C, H, W)
    torch_input_tensor = torch.ones(input_shape)
    a = ttnn.from_torch(torch_input_tensor, dtype=ttnn.float32, layout=ttnn.TILE_LAYOUT, device=device)
    grid_size = (8, 4)
    xt = ttnn.interleaved_to_sharded(
            a,
            grid_size,
            [N * C * H, W // (grid_size[0] * grid_size[1])],
            ttnn.TensorMemoryLayout.WIDTH_SHARDED,
            ttnn.ShardOrientation.COL_MAJOR,
        )
    c = ttnn.pad(xt, [8, 1, 64, 2048], [0, 0, 0, 0], 3)

Expected:

no failures

Actual:
E       RuntimeError: TT_ASSERT @ ../ttnn/cpp/ttnn/operations/data_movement/pad/pad.cpp:129: !input_tensor.is_sharded() || output_w == output_memory_config.shard_spec->shape[1]
E       info:
E       output_w != output_memory_config.shard_spec().shape[1]
E       backtrace:
E        --- /proj_sw/user_dev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x627ff7) [0x79e986e2cff7]
E        --- void tt::assert::tt_assert<>(char const*, int, char const*, bool, char const*, fmt::v11::basic_format_string<char>)
E        --- /proj_sw/user_dev/bbradel/tt-metal/ttnn/ttnn/_ttnn.so(+0x97212f) [0x79e98717712f]
E        --- ttnn::operations::data_movement::ExecutePad::invoke(unsigned char, tt::tt_metal::Tensor const&, std::__1::array<unsigned int, 4ul> const&, std::__1::array<unsigned int, 4ul> const&, float, bool, std::__1::optional<tt::tt_metal::MemoryConfig> const&)
...

The sharded padding might not work for arbitrary size, will look into it.