<jupyter_start><jupyter_text>What is PIC-SURE PIC-SURE is a an ecosystem designed to handle very large amounts of data in a secure way.  This reduces the administrative load on researchers and allows them to better focus on Data Science discoveries.  PIC-SURE is composed of two major parts: 1) the PIC-SURE framework which handles communication, security, auditing and other system tasks, and 2) Datasets which are also described as "data resources within the PIC-SURE service".  The PIC-SURE also allows some "data wrangling" on the server, easing the amount of energy required by a researcher to load, process, and trim the data down to the details needed to conduct research.***What PIC-SURE does is allow you to filter a dataset to only the rows and columns that you are interested in***![title](../resources/HPDS_Diagram.gif)  PIC-SURE Connector and Adapter LibrariesAll operations on PIC-SURE rely upon two component libraries: The Connection Library, and a Datasource Adapter Library.  The connection library manages network communication and authentication.  Within a PIC-SURE instance exists one or more Datasources.  Each Datasources is of a specific type and are run using different server code.  For each type of Datasource there is a Datasource Adapter Library which is written specifically to communicate with it's particular type of Datasource.In Python, the conneciton library is imported using the following statement:    import PicSureClient    One Datasource Adapter that can be imported is for HPDS-typed datasets.  This can be imported using the following statement:    import PicSureHpdsLib<jupyter_code># Install a pip package in the current Jupyter kernel
import sys
!{sys.executable} -m pip install --upgrade pip
!{sys.executable} -m pip install git+https://github.com/hms-dbmi/pic-sure-python-client.git    
!{sys.executable} -m pip install git+https://github.com/hms-dbmi/pic-sure-python-adapter-hpds.git
!{sys.executable} -m pip install pandas
!{sys.executable} -m pip install matplotlib<jupyter_output>Collecting git+https://github.com/hms-dbmi/pic-sure-python-client.git
  Cloning https://github.com/hms-dbmi/pic-sure-python-client.git to /tmp/pip-req-build-05fun733
  Running command git clone -q https://github.com/hms-dbmi/pic-sure-python-client.git /tmp/pip-req-build-05fun733
Requirement already satisfied (use --upgrade to upgrade): PicSureClient==0.1.0 from git+https://github.com/hms-dbmi/pic-sure-python-client.git in /opt/conda/lib/python3.5/site-packages
Building wheels for collected packages: PicSureClient
  Building wheel for PicSureClient (setup.py) ... [?25l- \ done
[?25h  Stored in directory: /tmp/pip-ephem-wheel-cache-j_regwbx/wheels/b4/c1/d9/744d4e1c1ba5f0c5d847deba1a7c8f96f47d4b36c610dda114
Successfully built PicSureClient
Collecting git+https://github.com/hms-dbmi/pic-sure-python-adapter-hpds.git
  Cloning https://github.com/hms-dbmi/pic-sure-python-adapter-hpds.git to /tmp/pip-req-build-jcg_x9aa
  Running command git clone -q https://github.com/hms-dbmi/pic-sure-p[...]<jupyter_text>Using the Connection Library to connect to a PIC-SURE endpointThe first step in using any PIC-SURE Resource is to establish a connection to the endpoint.  That is demonstrated the following code which: 1) imports the connection library, creates a reference to the library's communication client object, [then use the "help()" function on the client to see how to get the connection object].<jupyter_code>import PicSureClient
client = PicSureClient.Client()
client.help()
# Now create a connection object by passing the PIC-SURE endpoint URL 
# and the security token to the client object's connect() function

connection = client.connect("http://localhost/PIC-SURE", "your_security_token_here")<jupyter_output><empty_output><jupyter_text>Connect to the NHANES data resource using the HPDS AdapterAfter a valid connection is established to the PIC-SURE endpoint a Resource Adapter can be loaded and configured by passing the connection object to it's adapter object. Once the Adapter is connected, a PIC-SURE resource object is retreved by indicating its UUID to the Adapter's *useResource()* function.<jupyter_code>import PicSureHpdsLib
#adapter = PicSureHpdsLib.Adapter(connection)
adapter = PicSureHpdsLib.BypassAdapter("http://pic-sure-hpds-nhanes:8080/PIC-SURE")
adapter.help()<jupyter_output>[HELP] PicSureHpdsLib.Adapter(picsure_connection)
            .version()                      gives version information for library
            .list()                         lists available resources
            .useResource(resource_uuid)     returns an object for selected resource<jupyter_text>Use adapter.list() to discover what data resource are available<jupyter_code>adapter.list()<jupyter_output>+--------------------------------------+------------------------------------------------------
|  Resource UUID                       |  Resource Name                                  
+--------------------------------------+------------------------------------------------------
| cfee4932-4b61-4f3e-856c-118dc13855d6 | PhenoCube v1.0-SNAPSHOT
| Description: [Resource accessed directly (bypassing PIC-SURE framework)]
+--------------------------------------+------------------------------------------------------<jupyter_text>Get a data resource reference to be able to work with a dataset<jupyter_code>resource = adapter.useResource("91ad021c-00cb-4f44-98fb-72e0983fdd73")
resource.help()<jupyter_output>[HELP] PicSureHpdsLib.useResource(resource_uuid)
            .dictionary()       Used to access data dictionary of the resource
            .query()            Used to query against data in the resource
            
        [ENVIRONMENT]
              Endpoint URL: http://pic-sure-hpds-nhanes:8080/PIC-SURE/
             Resource UUID: 91ad021c-00cb-4f44-98fb-72e0983fdd73
StarcoderdataJupyterStructuredCleanDedup