需求¶

存储py对象时，有几个要注意

实际存储路径，格式，备份
存储具体的子目录，tags
存储的key和value

查询时，按照子目录、key、tag其中组合查找

实现考虑¶

文件也是个流，os中持久的流，文件路径和文件名是流的id
其他存储也应该是个持久化的流，需要一个流的id，
文件流是顺序的字符串流，读取也只能按照顺序读
对象流是顺序的key，value顺序集合，按照key读取，不按照时间读取
钉钉和微信的wenhook也是流
本地文件方便备份和复制，系统路径不需要考虑路径
普通对象可以不考虑目录，合并成key。特俗分类对象数据需要有目录分类清晰
value是dataframe时的优化

deva实践¶

In [1]:

from deva import *

NODB全局命名对象数据库¶

支持独特数据表名
支持日志
支持自定义存储的文件路径，默认在库目录

创建和插入数据¶

In [2]:

testdb = NODB('testdb',log=log)
{'key':'value'}>>testdb
('key1','value1')>>testdb
'todayvlue'>>testdb

[2020-02-11 13:17:58.073039] INFO: deva.log: {'key': 'value'}
[2020-02-11 13:17:58.076541] INFO: deva.log: ('key1', 'value1')
[2020-02-11 13:17:58.085674] INFO: deva.log: todayvlue

Out[2]:

'todayvlue'

全局命名统一¶

In [3]:

ndb = NODB('testdb',log=log)
ndb is testdb

Out[3]:

True

像字典一样查看和删除数据¶

In [4]:

testdb.items()>>ls
# testdb.tablename

Out[4]:

[('tablename', 'testdb'),
 ('key', 'value'),
 ('key1', 'value1'),
 ('2020-02-11', 'todayvlue')]

In [5]:

testdb.get('key')

Out[5]:

'value'

In [6]:

del testdb['key1']

当前目录下保持数据库文件¶

In [7]:

tmpdb = NODB('tmp1',fname='tmp1',log=log)

tmpdb['tablename']

Out[7]:

'tmp1'

In [8]:

123>>tmpdb

[2020-02-11 13:17:58.255343] INFO: deva.log: 123

Out[8]:

In [9]:

123>>tmpdb
2345>>tmpdb
33>>tmpdb
{'name':'tal','desc':'好未来','df':[1,2,3]}>>tmpdb
('hello','moto')>>tmpdb
{'key':'value'}>>tmpdb

[2020-02-11 13:17:58.287090] INFO: deva.log: 123
[2020-02-11 13:17:58.294454] INFO: deva.log: 2345
[2020-02-11 13:17:58.305012] INFO: deva.log: 33
[2020-02-11 13:17:58.308525] INFO: deva.log: {'name': 'tal', 'desc': '好未来', 'df': [1, 2, 3]}
[2020-02-11 13:17:58.323626] INFO: deva.log: ('hello', 'moto')
[2020-02-11 13:17:58.328127] INFO: deva.log: {'key': 'value'}

Out[9]:

{'key': 'value'}

HDFStore单独对DF数据存储优化¶

In [10]:

import pandas as pd
from sqlalchemy import create_engine
import sqlalchemy

engine = create_engine('mysql+pymysql://root:root@localhost/shejishi?charset=utf8')
df = pd.read_sql('select * from pins limit 3', engine)
# df=df.set_index('id')
# make sure emp_master_backup table has been created
# so the table schema is what we want
# df.to_sql('emp_backup', engine, index=True ,if_exists='replace')



from deva.store import HDFStore
hs = HDFStore()
hs.daily_set('/stock/tal/',df)
# hs.daily_read('/stock/tal')

Out[10]:

<deva.store.HDFStore at 0x10ff6bda0>

In [11]:

hs.keys()

Out[11]:

['/mykey',
 '/stock/tal/ts20200116',
 '/stock/tal/ts20200131',
 '/stock/tal/ts20200201',
 '/stock/tal/ts20200211']

In [12]:

hs.daily_get('/stock/tal')

Out[12]:

	id	url	desc	cn_desc	last_modify_time
0	391	https://i.pinimg.com/736x/e2/2a/31/e22a310d4e7...	We are a proud custom home builder in the Nash...	我们是纳什维尔地区一个自豪的定制住宅建筑商。点击照片，浏览我们豪宅的社区！	NaN
1	392	https://i.pinimg.com/736x/c2/b2/f6/c2b2f6dd736...	Weekend lounging in this perfect, mod Boho Bed...	周末在这个完美的模型Boho卧室休息。中性色彩，绿色植物和大量自然光！爱！	NaN
2	393	https://i.pinimg.com/736x/d7/88/19/d788193ece5...	Want to transform your home? Work with our bud...	想改造你的家？与我们的预算友好的室内设计师一起在线工作。	NaN

In [13]:

hs.keys()

Out[13]:

['/mykey',
 '/stock/tal/ts20200116',
 '/stock/tal/ts20200131',
 '/stock/tal/ts20200201',
 '/stock/tal/ts20200211']

In [14]:

hs.set('mykey',df)
hs.get('mykey')

Out[14]:

	id	url	desc	cn_desc	last_modify_time
0	391	https://i.pinimg.com/736x/e2/2a/31/e22a310d4e7...	We are a proud custom home builder in the Nash...	我们是纳什维尔地区一个自豪的定制住宅建筑商。点击照片，浏览我们豪宅的社区！	NaN
1	392	https://i.pinimg.com/736x/c2/b2/f6/c2b2f6dd736...	Weekend lounging in this perfect, mod Boho Bed...	周末在这个完美的模型Boho卧室休息。中性色彩，绿色植物和大量自然光！爱！	NaN
2	393	https://i.pinimg.com/736x/d7/88/19/d788193ece5...	Want to transform your home? Work with our bud...	想改造你的家？与我们的预算友好的室内设计师一起在线工作。	NaN

文件流¶

写入¶

In [15]:

123123>>write_to_file('tmp.txt')

Out[15]:

读取¶

In [16]:

'tmp.txt'>>read>>ls

Out[16]:

['123123\n', '123123\n']

In [17]:

range(10)>>head(3)>>tee>>ls

0
1
2

Out[17]:

[0, 1, 2]

函数流¶

range sum 等常见函数同时支持pipe¶

In [18]:

range<<6>>ls

Out[18]:

[0, 1, 2, 3, 4, 5]

In [19]:

range<<6>>sum

Out[19]:

钉钉和微信流¶

In [20]:

range<<10>>tee>>sum>>Dtalk(webhook='https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=f5110ae1-bb4a-4e58-a87c-6ccafa3c8d25')

Out[20]:

In [21]:

cth = Dtalk(webhook='https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=f5110ae1-bb4a-4e58-a87c-6ccafa3c8d25')

需求¶