<jupyter_start><jupyter_text>Slicing data in pandasThis is second in the series on indexing and selecting data in pandas. If you haven't read it yet, see [the first post](xxx) that covers the basics of selecting based on index or relative numerical indexing. In this post, I'm going to review slicing, which is a core Python topic, but has a few subtle issues related to pandas. Then, using slicing, we'll review the selection methods from the last post, but now with slicing applied. What is a slice, and what is slicing?A slice is defined in the [Python glossory](https://docs.python.org/3/glossary.htmlterm-slice).> An object usually containing a portion of a sequence. A slice is created using the subscript notation, [] with colons between numbers when several are given, such as in variable_name[1:3:5]. The bracket (subscript) notation uses slice objects internally.Slicing is defined in the [Python reference](https://docs.python.org/3/reference/expressions.htmlslicing).> a slicing selects a range of items in a sequence object (e.g., a string, tuple or list). Slicings may be used as expressions or as targets in assignment or del statements. A slicing has three parts: start, stop, and step. A ```slice``` class is a Python builtin. If you instantiante a ```slice``` object explicitly, you can supply a single value (the stop), or you can supply the start and stop and optionally, the step. In terms of what is selected in the sequence, the start is inclusive, the stop is not, and the step determines how you count from start to stop. You can use slice objects or slice notation, but unless you need to store the slice object for re-use, you usually see slice notation in code.So let's look at few examples to clear this up. We'll start with a string as a sequence object.<jupyter_code>w = "abcdefg"

# all of these select the same slice of w, "abc"

w[slice(0, 3, 1)]  # This is slice(start=0, stop=3, step=1)
w[0:3:1]           # This is the same, but in slice notation
w[slice(3)]        # start=0, step=1 are the defaults
w[:3]              # same here

# select the entire sequence. start=0, stop=last element (inclusive), step=1
w[:]<jupyter_output><empty_output><jupyter_text>Now slices can be used very effectively and concisely to get views of data. I'll just cover a few examples here.<jupyter_code>w[::-1] # reverse a string (start=0, stop=end, stepping backwards)
w[1:3] # a portion of a string<jupyter_output><empty_output><jupyter_text>Now strings are special, you can't modify them directly or delete their values. Same with tuples.<jupyter_code>try:
    w[2] = 'z'
except TypeError as te:
    print(te)
    
try:
    t = (0, 1, 2)
    t[0] = -1
except TypeError as te:
    print(te)<jupyter_output>'str' object does not support item assignment
'tuple' object does not support item assignment<jupyter_text>We'll switch to using a list to show some more applications of slicing.<jupyter_code>l = [0, 1, 2, 3, 4, 5, 6]

# just as with strings, these all work on a list, and return [0, 1, 2]
l[slice(0,3,1)]
l[0:3:1]
l[slice(3)]
l[:3]

# this returns a full copy
l[:]
# using step
l[::2]
# can also assign
l[1::2] = [-1, -1, -1]
l<jupyter_output><empty_output><jupyter_text>On to pandasOK, after a quick overview of slicing, let's move onto pandas and see why we're talking about slicing at all. Just like the last post, I'll grab a dataset from the [Chicago Data Portal](https://data.cityofchicago.org), but this time we'll grab the [list of Chicago public libraries](https://data.cityofchicago.org/Education/Libraries-Locations-Hours-and-Contact-Information/x8fc-8rcq).<jupyter_code>import pandas as pd
import numpy as np

# you should be able to grab this dataset as an unauthenticated user, but you can be rate limited
df = pd.read_json("https://data.cityofchicago.org/resource/x8fc-8rcq.json")
df.dtypes
# trim down the columns
df = df[['name_', 'hours_of_operation', 'address', 'city', 'state', 'zip', 'phone', 'website', 'location']]
df.head(3)<jupyter_output><empty_output><jupyter_text>Slicing in pandasNow in pandas (and the underlying NumPy data structures), slicing is also supported, and you can use it via the indexing operator (```[]```), ```.loc```, and ```.iloc```.  Just like in the last post, we'll walk through these three methods and look at slicing for both ```Series``` and ```DataFrame``` objects. Also, as last time, to prevent confusion between location based indexing and label indexing, I'm going to update the index of our sample data to be string based for the initial explanations.<jupyter_code># this will make our examples a bit more obvious
df.index = "ID-" + df.index.astype(str)
# Use the name_ column for a sample Series.
s = df['name_']<jupyter_output><empty_output><jupyter_text>Slicing with ```[]```Just like basic indexing, we'll see that slicing with ```.loc``` and ```.iloc``` is preferred to using ```[]```. Since the behavior of ```[]``` can depend on the index and arugments, it's better to use the more explicit indexers. But it is supported, there's a just a few things to keep in mind. SeriesJust like standard Python, you can use integer slicing on Series objects, with ```start:stop:step```.<jupyter_code>s[:3]            # These all slice the first 3 elements in the Series
s[0:3]
s[0:3:1]
s[slice(0,3,1)]<jupyter_output><empty_output><jupyter_text>In a ```Series```, we also can slice on our index, which in this case is an index of objects (strings). Note that the slice start and stop are inclusive, which is different than regular Python slicing!<jupyter_code>s['ID-4':'ID-9']<jupyter_output><empty_output><jupyter_text>You can specify a step as well.<jupyter_code>s['ID-10':'ID-15':2]<jupyter_output><empty_output><jupyter_text>Note that your labels have to be in the index or you'll get a ```KeyError```.<jupyter_code>try:
    s['ID-10':'ID-999']
except KeyError as ke:
    print(ke)<jupyter_output>'ID-999'<jupyter_text>You can also (potentially) assign/update a slice. Note that using ```[]``` is not the right way to do this, you'll get a ```SettingWithCopyWarning```. More on this in future posts (or read the pandas docs for details).<jupyter_code>s['ID-79':] = "Changed"
s.tail()<jupyter_output><empty_output><jupyter_text>DataFrameIn a ```DataFrame```, slicing with ```[]``` will **slice on the rows**. If you remember back to the last past, selection on a ```DataFrame``` using ```[]``` selects the columns first, so this is a bit confusing. Just remember, if you see a ```:```, it's slicing on rows.<jupyter_code>df[1:3]<jupyter_output><empty_output><jupyter_text>You can also use the step.<jupyter_code>df['ID-3':'ID-7':2]<jupyter_output><empty_output><jupyter_text>Slicing with ```.loc```.Now, just like with basic selection, ```.loc``` and ```.iloc``` are meant for label and position respectively. With ```.loc``` you cannot pass in locations unless those match the index you are using. In our case, we'll get a ```TypeError```. The issue is our index doesn't contain these values, and so we'll be rejected.Note that with ```.loc```, we will get both the start and stop values if they are in the index, just like with ```[]```. SeriesFor our example data, you can't slice with integers (since they aren't in the index), but we can slice by labels.<jupyter_code>try:
    s.loc[1:3]
except TypeError as te:
    print(te)
    
s.loc['ID-1':'ID-3']<jupyter_output>cannot do slice indexing on Index with these indexers [1] of type int<jupyter_text>Assignment or updates using slices (can) work with ```.loc```. We'll talk about updates in more detail in the future.<jupyter_code>s.loc['ID-79':] = "Changed again"
s.tail()<jupyter_output><empty_output><jupyter_text>DataFrameAnd things are similar for a ```DataFrame```.<jupyter_code>try:
    df.loc[1:3]
except TypeError as te:
    print(te)
    
df.loc['ID-1':'ID-3']<jupyter_output>cannot do slice indexing on Index with these indexers [1] of type int<jupyter_text>Now with a ```DataFrame```, you can slice both on the index and the columns, by label.<jupyter_code>df.loc['ID-10':'ID-20':2, "name_":"address"]<jupyter_output><empty_output><jupyter_text>I'll also point out a common idiom in pandas code to use a full slice (```:```) to select the entire object when not using slicing to limit selection.<jupyter_code>df['name_']           # this is one way to select a single column, for example
df.loc[:,'name_']     # but this is how you select that column using .loc<jupyter_output><empty_output><jupyter_text>Slicing with ```.iloc```Remember, ```.iloc``` is for strictly integer based indexing, so as you'd expect, it doesn't work with labels. But also note that ```.iloc``` works like standard Python slicing, i.e. the stop value is not included. Series<jupyter_code>try:
    s.iloc["ID-1":"ID-3"]  # not this way with .iloc
except TypeError as te:
    print(te)
s.iloc[1:3]                # this way<jupyter_output>cannot do positional indexing on Index with these indexers [ID-1] of type str<jupyter_text>DataFrameSimilarly, ```DataFrame``` selects from the index as the first argument, and the columns with the second argument.<jupyter_code>df.iloc[1:3]
df.iloc[1:3, 0:2]
df.iloc[1:10:3, 2:-4]<jupyter_output><empty_output><jupyter_text>Some special casesThere's always some special cases to consider, so I'll go through those last.  Changing behavior for integer based indexesYou'll remember that I changed the index on our first test ```DataFrame``` to be a string based index instead of the standard ```RangeIndex``` of sequential integer values. This following example shows why, but you should now understand why it works this way.<jupyter_code>df2 = pd.DataFrame({"a": [0, 1, 2, 3], "b": [4, 5, 6, 7], "c": [8, 9, 10, 11]})
df2
df2.loc[0:2]  # Remember, .loc is label based
df2.iloc[0:2]  # And .iloc is position based
df2[0:2]   # So here [] behaves like position based, even though we may be trying to use our index
df2.index = ["w", "x", "y", "z"]
df2["w":"y"]  # now, [] behaves like it's label based.<jupyter_output><empty_output><jupyter_text>Hopefully this example reinforces for you why you'll want to use ```.loc``` and ```.iloc``` in production code, especially when you're taking values to select rows or columns from unknown sources! Label slicing and sorted indexesWe saw that using ```.loc``` will include both the start and stop values of a slice, which is not the way standard Python works. Well, you can end up with indexes that are not sorted and will behave slightly differently than expected. Essentially, what you need to notice is that with a sorted index, all values in between the start and stop will be included (including the value for the stop label)<jupyter_code>s2 = pd.Series([1, 2, 3, 4, 5], index=[5, 2, 1, 9, 7])
s3 = pd.Series([1, 2, 3, 4, 5], index=[0, 1, 2, 3, 4])

s3.loc[0:3] # works just as you'd expect for .loc (not standard python slicing, includes end label)
s3.loc[0:9]  # no exception raised, even though 9 is not in the index, and includes all values between them
try:
    s2.loc[0:9]   # hmmm, this won't work
except KeyError as ke:
    print(ke)
s2.sort_index().loc[0:9] # this works though.<jupyter_output><empty_output><jupyter_text>The reason for this functionality is that it is too expensive to do the sorting first, and it may not return the values you'd expect anyway since sorting may not make sense for your index. Just be aware of this behavior when you are dealing with non-sorted indexes. You'll realize that it's important to understand the indexes on your data and ensure they are accurately reflecting your data and what you're trying to accomplish with indexing and slicing it.  The Ellipsis and NumPy slicingThis is a little known part of Python, but in working through these examples I did come across the ```Ellipsis``` keyword in looking at the slicing documentation. ```Ellipsis``` is just a singleton (like ```None```) and is intended to be used in extended slicing syntax by user-defined container data types. It is represented by the ellipsis literal, ```...```.It doesn't appear to be supported much by pandas beyond ```Series```, but it is more useful in NumPy. The way to think of it is as a special value that will insert as many full slices as needed to extend the slice at the point of insertion in an index in all directions. This makes more sense with an example.<jupyter_code>...   # the singleton
m = np.arange(27).reshape((3,3,3))  # 3 x 3 x 3 multi-dimensional array
print("Full array:\n", m)
print("First element in last dimension:\n", m[:,:,0])
print("With ellipsis:\n", m[...,0])
print("Last element in last dimension:\n", m[...,-1])<jupyter_output>Full array:
 [[[ 0  1  2]
  [ 3  4  5]
  [ 6  7  8]]

 [[ 9 10 11]
  [12 13 14]
  [15 16 17]]

 [[18 19 20]
  [21 22 23]
  [24 25 26]]]
First element in last dimension:
 [[ 0  3  6]
 [ 9 12 15]
 [18 21 24]]
With ellipsis:
 [[ 0  3  6]
 [ 9 12 15]
 [18 21 24]]
Last element in last dimension:
 [[ 2  5  8]
 [11 14 17]
 [20 23 26]]
StarcoderdataJupyterStructuredCleanDedup