<jupyter_start><jupyter_text>pandas 是基于 Numpy 构建的，相比numpy专注于计算，pandas更多的用在数据处理方面 其有三种格式 ![1.png](attachment:1.png)  Series 由一组数据和与之相关的索引组成。<jupyter_code># 创建一个Series：
import pandas as pd
s = pd.Series([1,3,5,7,6,8])
s
s.index
# 给series加上列名
s.name = 'x'
s<jupyter_output><empty_output><jupyter_text>Dataframe DataFrame 是一个类似Excel表格的数据结构，其带有索引 创建一个DataFrame<jupyter_code>import pandas as pd
d = {'one': [1, 2, 3], 'two': [1, 2, 3]}
df = pd.DataFrame(d,index=['a', 'b', 'c'])
df
#创建一个空的 DataFrame
df = pd.DataFrame()
print(df)
#查看行
df.index
# 查看列
df.columns
#查看元素
df.values
# 获取表里面有多少数据
df.size
# 一般性的 从api 获取的股票数据，都是dataframe结构
import tushare as ts
df = ts.get_hist_data('000001','2020-01-01')
type(df)<jupyter_output><empty_output><jupyter_text>从数据分析操作的角度来学习dataframe 查看前几行,查看后几行<jupyter_code>import tushare as ts
df = ts.get_k_data('000001','2020-01-01')
df.head(10),df.tail(10)
#查看 DataFrame 的列名
list(df.columns)
#查看DataFrame 信息
df.info()
# 将series转为dataframe
import pandas as pd
s = pd.Series([1,3,5,7,6,8])
s.name = 'x'
df= s.to_frame()
df<jupyter_output><empty_output><jupyter_text>在dataframe当中选取数据，主要通过下面几个函数<jupyter_code>import tushare as ts
df = ts.get_k_data('000001','2020-01-01')
df.head(30)<jupyter_output><empty_output><jupyter_text>df.loc[行标签,列标签]<jupyter_code># 选取0，1两行数据
df.loc[0:1]
#选取 某一列的数据
df.loc[0:5,'close']
# 选取某个值
df.loc[0,'close']<jupyter_output><empty_output><jupyter_text>df.loc 的第一个参数是行标签，第二个参数为列标签，两个参数既可以是列表也可以  是单个字符，如果两个参数都为列表则返回的是 DataFrame，否则，则为 Series。<jupyter_code>df.loc[0:5,'open':'low']<jupyter_output><empty_output><jupyter_text>df.iloc[行位置,列位置]<jupyter_code>#选取第一行及第五行的数据
df.iloc[[0,5],:] 
#选取第一行到第五行（不包含）的数据
df.iloc[0:5,:] 
#选取所有记录的第二列的值，返回的为一个Series
df.iloc[0:5,1] 
#选取第二行，第二列的值，返回的为单个值
df.iloc[1,1]<jupyter_output><empty_output><jupyter_text>获取某行有一个简单的写法<jupyter_code>df.iloc[0]  #第一行
df.iloc[-1]  #最后一行<jupyter_output><empty_output><jupyter_text>更广义的切片方式是使用.ix，它自动根据给到的索引类型判断是使用位置还是标签进行切片 df.ix[]  但这个被抛弃了<jupyter_code>df.ix[0:5,'open':'low']<jupyter_output>d:\vn studio\lib\site-packages\ipykernel_launcher.py:1: DeprecationWarning: 
.ix is deprecated. Please use
.loc for label based indexing or
.iloc for positional indexing

See the documentation here:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated
  """Entry point for launching an IPython kernel.<jupyter_text>使用isin()方法来过滤在指定列中的数据<jupyter_code># 选取 high 列中数为 1 和 774.00 的数。
df[df['high'].isin([17.31,16.34])]<jupyter_output><empty_output><jupyter_text>增 插入一列<jupyter_code>import tushare as ts
df = ts.get_k_data('000001','2020-01-01')
del df['code']
df.head(5)
df.insert(0,'codel','000001')
df.head(5)<jupyter_output><empty_output><jupyter_text>二维数组转为dataframe<jupyter_code>import numpy as np
import pandas as pd
a = [[1,2,3],[2,3,4],[2,5,3]]
array1 = np.array(a)
df = pd.DataFrame(array1)
df.columns = ['x','y','z']
print(df)<jupyter_output>x  y  z
0  1  2  3
1  2  3  4
2  2  5  3<jupyter_text>二维数组作为新列添加到dataframe<jupyter_code>import numpy as np
import pandas as pd
df1 = pd.DataFrame({'a':['a','b','c']})
a = [[1,2,3],[2,3,4],[2,5,3]]
array1 = np.array(a)
df2 = pd.DataFrame(array1)
df2.columns = ['x','y','z']
df = pd.concat([df1,df2],axis=1)
print(df)<jupyter_output>a  x  y  z
0  a  1  2  3
1  b  2  3  4
2  c  2  5  3<jupyter_text>遍历dataframe<jupyter_code>for i in df.index:
    print(i)
for index,row in df.iterrows():
    print(index)
    print(row)
for i in range(len(df)):
    print(df.iloc[i,0])<jupyter_output>a
b
c<jupyter_text>删<jupyter_code># 对于列的删除只需要对列进行切片，保留需要的即可
import tushare as ts
df1 = ts.get_k_data('000001','2020-08-01','2020-08-07')
df1

# 删除 volume这一列
df1.pop('volume')
del df1['low']
df2 = df1.drop(['close', 'code'], axis=1)
df2
# 删除某一行
df3 = df2.drop([140,141], axis=0)#等效于df3 = df2.drop(index=[140,141], axis=0)
df3
# 删除含有某个值的那一行
df4 = df3.drop(index=(df3.loc[df3['high']==13.96].index), axis=0)
df4<jupyter_output><empty_output><jupyter_text>改<jupyter_code>修改的部分：先定位到某个位置
直接给这个位置赋值就可以了<jupyter_output><empty_output><jupyter_text>从部分dataframe一部分获取最大值<jupyter_code>import tushare as ts
df = ts.get_k_data('000001','2020-01-01')
max = df.loc[1:120,'close'].max()
max<jupyter_output><empty_output><jupyter_text>获取最大值的索引<jupyter_code>index1 = df.loc[1:120,'close'][df.loc[1:120,'close']==max].index
index1<jupyter_output><empty_output><jupyter_text>判断dataframe是否为空<jupyter_code>import pandas as pd
df = pd.DataFrame()
df
if df.empty==True:
    print(1)<jupyter_output>1<jupyter_text>多个条件筛选<jupyter_code>import tushare as ts
df = ts.get_hist_data('000001','2020-08-01')
df
a = df.p_change>0
b = df.ma5>df.ma10
c = df.ma5>df.ma20
d = df.turnover>0.8
df = df[a & (b | c) & d]
df<jupyter_output><empty_output><jupyter_text>排序<jupyter_code>type(df.index)
# 日期是索引，按日期排序
df.sort_index()
import tushare as ts
df = ts.get_k_data('000001','2020-08-01')
df
type(df.date[140])

df.date = pd.to_datetime(df.date)
#日期不是索引，将df按照日期排序
df.sort_values(by='date',ascending=False,inplace=True)
df<jupyter_output><empty_output><jupyter_text>再重新设置索引<jupyter_code>df.reset_index(drop = True,inplace=True)
df
df.set_index('date',inplace=True)
df<jupyter_output><empty_output><jupyter_text>concat 可以沿一条轴将多个对象堆叠到一起。分别获取两个DataFrame。<jupyter_code>import tushare as ts
df1 = ts.get_k_data('000001','2020-08-01','2020-08-07')
df1
df2 = ts.get_k_data('000001','2020-08-10','2020-08-15')
df2
df3 = pd.concat([df1,df2],axis=0)
df3
df4 = pd.concat([df1,df2],axis=1)
df4<jupyter_output><empty_output><jupyter_text>只增加一行<jupyter_code>s = df1.iloc[0]
df1 = df1.append(s, ignore_index=False) # ignore_index=False 表示索引不变
df1<jupyter_output><empty_output><jupyter_text>查看重复数据：<jupyter_code>df1.duplicated()<jupyter_output><empty_output><jupyter_text>移除重复数据：<jupyter_code>df1.drop_duplicates()
df1['high_cahnge']=df1.high-df1.high.shift(1)
df1
df1['low_cahnge']=df1.low-df1.low.shift(-2)
df1
#去掉包含缺失值的行
df1.dropna()<jupyter_output><empty_output><jupyter_text>对缺失值进行填充为30<jupyter_code>df1.fillna(value=30)<jupyter_output><empty_output><jupyter_text>筛选排除掉空值<jupyter_code>df1 = df1.loc[df1['high_cahnge'].isna()==False] 
df1<jupyter_output><empty_output><jupyter_text>保留两位小数<jupyter_code>df1.loc[:,'high_cahnge'] = round(df1.loc[:,'high_cahnge'],1)
df1<jupyter_output>d:\vn studio\lib\site-packages\pandas\core\indexing.py:543: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item] = s<jupyter_text>计算波动率 某个变量的波动率定义为这一变量在单位时间内连续复利回报率的标准差。    对数收益率的历史移动标准差，也就是历史移动波动率      1.先计算某只股票的对数收益  2.由对数收益率计算波动率 滑动计算  rolling()   DataFrame.rolling(window, min_periods=None, freq=None, center=False, win_type=None, on=None, axis=0, closed=None)  window：表示时间窗的大小<jupyter_code>import tushare as ts
import numpy as np
df1 = ts.get_k_data('000001','2020-01-01','2020-08-07')
df1.head(5)
# 1.先计算某只股票的对数收益
df1['log_return']=np.log(df1['close']/df1['close'].shift(1))
df1.head(5)
# 2.由对数收益率计算波动率  月和年的话你可以用日波动率乘以30开方和365开方
df1['volatility'] = df1['log_return'].rolling(window=5).std()*np.sqrt(5)
df1.head(10)<jupyter_output><empty_output><jupyter_text>Groupby<jupyter_code>DataFrame.groupby(by=None, axis=0, level=None, as_index=True, sort=True, group_keys=True, squeeze=False, **kwargs)
import pandas as pd
 
df = pd.DataFrame({'code' : ['000001', '000002', '000003', '000001', '000002', '000003', '000001', '000002', '000003'],
          'time' : ['2020-01-01', '2020-01-01', '2020-01-01', '2020-01-02', '2020-01-02', '2020-01-02', '2020-01-03', '2020-01-03', '2020-01-03'],
          'close' : [4.5, 2.9, 3.8, 3.7, 4.0, 4.1, 1.9, 4.1, 3.2],
         'tag' : [1.5, 1.9, 2.8, 1.7, 4.1, 2.5, 1.1, 3.4, 1.2]
         })
df
df.groupby('code').mean()
df.groupby('time').max()
df.groupby(['code','time'])['close'].mean()<jupyter_output><empty_output><jupyter_text>Resample 重采样<jupyter_code>DataFrame.resample(rule, how=None, axis=0, fill_method=None, closed=None, label=None, convention='start', kind=None, loffset=None, limit=None, base=0, on=None, level=None)<jupyter_output><empty_output><jupyter_text>采样间隔![image.png](attachment:cb0c8f7c-6d91-4e38-8a07-2a2cf2ebc44d.png)<jupyter_code>import tushare as ts
import numpy as np
df1 = ts.get_hist_data('000001','2020-08-01','2020-08-07',ktype = '60')
df1.head(5)
df1.index= pd.DatetimeIndex(df1.index)
df1.resample('D').first()  #mean(),max(),min(),last()<jupyter_output><empty_output><jupyter_text>面试题<jupyter_code>#  iid_list 是 tick 级数据，按照合约（InstrumentID）分成不同的 df（以 SysTime 为轴，按照分钟对LastPrice的最大值的最小值进行重采样）
import pandas as pd
df = pd.read_csv('iid_list.csv')
df.head(10)
df.groupby('TradingDay').describe()
df.groupby('InstrumentID').describe()
df1 = df[df['InstrumentID']=='IC1909']
df1.head(10)
# df1['SysTime']= pd.DatetimeIndex(df1['SysTime'])
df1['SysTime']= pd.to_datetime(df1['SysTime'])
df2 = df1.set_index('SysTime',inplace=False)
# df.set_index('date',inplace=True)
# type(df2.index[0])
s1=df2['LastPrice'].resample('T').max()
s1.name = 'max'
s2=df2['LastPrice'].resample('T').min()
s2.name = 'min'
df3 = s1.to_frame()
df3['min'] = s2
df3<jupyter_output>d:\vn studio\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
StarcoderdataJupyterStructuredCleanDedup