关于JWT(Json Web Token)是一种较新的用户认证方式,官网在这里,网上有篇中文解释写的很好,点此跳转

用户认证(Authentication)和用户授权(Authorization)是两个不同的概念,认证解决的是“有没有”的问题,而授权解决的是“能不能”的问题。

一般用到JWT认证的情况大多都是配合REST框架使用,比如我大Django的Django-REST-framework框架,就已经有了现成的三方库django-rest-framework-jwt。不过这个库默认只支持基于Header传递信息,所以改成基于Cookie方式还需要我们来手动处理一下。

阅读全文

最近一直在思考如何更好的组织Django中的静态资源,比如JS、CSS一类,如何结合前端构建工具写出更好的代码以及结构呢?

首先需要解决的一个问题就是某些时候需要把JS代码写在模板里来获取后台传递过来的变量,比如:

1
2
3
4
5
6
7
8
9
<div>
<h1>Test</h1>
<div id="my-test" ></div>
</div>
<script>
$(function(){
$('#my-test').html("{{ some_var_from_view }}")
});
</script>

这么写代码的话,别扭不说,前端的那些构建工具比如webpack,gulp的使用范围也将大大降低。

首先说结论,想完全剥离JS和模板而又需要使用模板渲染的数据,我是没想到什么好办法。如果读者有好办法希望赐教。
既然不能完全剥离,那么就进最大的努力分离JS所需的数据和代码吧。

阅读全文

有些时候我们会有这种需求:用户上传一个格式固定excel表格到网站上,然后程序负债解析内容并进行处理。
举一个简单的栗子,比如我们有这样一个HTML:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
</head>
<body>
<p>上传EXCEL表格</p>
<form class="" action="" method="post" enctype="multipart/form-data" >
{% csrf_token %}
<input type="file" name="excel">
<input type="submit" value="上传">
</form>
</body>
</html>

forms.py文件内容如下,编写一个简单的判断后缀的验证:

1
2
3
4
5
6
7
8
9
10
11
12
# coding=utf-8
from django import forms
from django.utils.translation import gettext as _
from django.core.exceptions import ValidationError
def validate_excel(value):
if value.name.split('.')[-1] not in ['xls','xlsx']:
raise ValidationError(_('Invalid File Type: %(value)s'),params={'value': value},)
class UploadExcelForm(forms.Form):
excel = forms.FileField(validators=[validate_excel]) #这里使用自定义的验证

处理excel表格我这里使用xlrd库,使用pip安装即可。此时处理POST请求时有2种方法:

  1. 将用户上传的excel存储到磁盘中再读取交给xlrd处理。
  2. 直接在内存中读取用户上传的excel读取交给xlrd处理。

这里我使用第二个办法——在不修改django默认settings.py配置情况下,用户上传的文件其实是InMemoryUploadedFile类型,这个类型有一个read()方法,所以views.py中可以内存直接读取内容而不用写磁盘再读取了:

1
2
3
4
5
6
7
8
9
10
11
def post(self, request, *args, **kwargs):
form = UploadExcelForm(request.POST, request.FILES)
if form.is_valid():
wb = xlrd.open_workbook(
filename=None, file_contents=request.FILES['excel'].read()) # 关键点在于这里
table = wb.sheets()[0]
row = table.nrows
for i in xrange(1, row):
col = table.row_values(i)
print col
return HttpResponse("ok")

其他文件类型同理,如果不需要保存用户上传的文件到硬盘其实都可以这么处理。这里记录2个和django处理excel有关的资源:

  1. django-excel 判断用户excel格式的三方库
  2. https://assist-software.net/blog/how-export-excel-files-python-django-application 讲解如何导出excel的文章

评论和分享

自定义Django用户模型

发布在 Django

Django最方便的一点可以说就是自带的用户系统了,不过某些情况下自带的用户系统不太符合项目需求,比如你想添加几个字段怎么办?当然可以使用自定Model然后外键关联User类来实现,不过一方面关联查询的效率比直接查询效率要低,另一方面想删除系统自带用户系统的某些字段怎么办呢?

所以,自定义用户模型可以说是一种很常见的需求。这里以Django1.9为例,记录一下自定义用户模型的方法。

阅读全文

今天调试Django项目时候,使用Logging记录异常并发送邮件给网站管理人员,测试时候始终报错:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Traceback (most recent call last):
File "/usr/lib64/python2.7/wsgiref/handlers.py", line 85, in run
self.result = application(self.environ, self.start_response)
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 177, in __call__
response = self.get_response(request)
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/core/handlers/base.py", line 230, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/core/handlers/base.py", line 284, in handle_uncaught_exception
'request': request
File "/usr/lib64/python2.7/logging/__init__.py", line 1185, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib64/python2.7/logging/__init__.py", line 1278, in _log
self.handle(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 1288, in handle
self.callHandlers(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 1328, in callHandlers
hdlr.handle(record)
File "/usr/lib64/python2.7/logging/__init__.py", line 751, in handle
self.emit(record)
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/utils/log.py", line 117, in emit
self.send_mail(subject, message, fail_silently=True, html_message=html_message)
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/utils/log.py", line 120, in send_mail
mail.mail_admins(subject, message, *args, connection=self.connection(), **kwargs)
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/core/mail/__init__.py", line 97, in mail_admins
mail.send(fail_silently=fail_silently)
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/core/mail/message.py", line 292, in send
return self.get_connection(fail_silently).send_messages([self])
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 100, in send_messages
new_conn_created = self.open()
File "/home/xsy/.virtualenvs/dz_pro/lib/python2.7/site-packages/django/core/mail/backends/smtp.py", line 67, in open
self.connection.login(self.username, self.password)
File "/usr/lib64/python2.7/smtplib.py", line 607, in login
(code, resp) = self.docmd(encode_cram_md5(resp, user, password))
File "/usr/lib64/python2.7/smtplib.py", line 571, in encode_cram_md5
response = user + " " + hmac.HMAC(password, challenge).hexdigest()
File "/usr/lib64/python2.7/hmac.py", line 78, in __init__
self.outer.update(key.translate(trans_5C))
TypeError: character mapping must return integer, None or unicode

根据错误提示,追进了hmac.py中看了源码,发现key其实就是一个字符串,是配置在settings中的EMAIL_HOST_USER以及EMAIL_HOST_PASSWORD,而translate函数则是根据给出的参数表将字符串进行映射加密的,而且是最简单的凯撒加密法。

阅读全文

Django常用三方库

发布在 Django

更新于2017.01.10

首先安利本书《two scoops of django1.8》目前还没有中文版,不过是我看过关于django的书中收获最大的一本。本来有机会和作者面基,可惜我大天朝的签证太难办了,只得作罢。

这里推荐三方库的大部分出自这本书,移除了部分失效或者不维护的库,添加了一些其他库,强烈建议大家阅读原书。

下面的模块有些是django模块,有些则是python模块,可以脱离django直接使用。

话说那些复制到别地方然后标注“原创”的,敢加个本文链接不?手工整理很累的好不?

阅读全文

最近使用django+requests+eventlet做了个小程序,eventlet用来替代原生的多线程,最后发现有关manage.py的功能全都不能用了,报错信息类似:

1
django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 139911009593152 and this is thread id 51055504.

产生这个问题的原因在于我的monky_patch()是在爬虫模块中执行,而希望这个爬虫模块能够保持独立性不希望和django融合的太深,所以解决问题只需根据实际需求在manage.py或settings.py或wsgi.py中先把monkey_patch()执行了即可。

阅读全文

之前的博客中简单的介绍了celery的安装配置以及如何在python程序中使用,这里记录一下我使用django结合celery以及rabbitmq提供web服务,同时使用flower进行监控的过程。至于这几样东西是什么、怎么安装这里就不再细说了。

阅读全文

django性能分析

发布在 Django

一般情况我们使用django-debug-toolbar就能够看到每个步骤的耗时等信息,不过如果需要调试某个接口就不那么直观了,这种情况下我们可以使用下面的中间件来解决问题:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
# Orignal version taken from http://www.djangosnippets.org/snippets/186/
# Original author: udfalkso
# Modified by: Shwagroo Team and Gun.io
import sys
import os
import re
import hotshot, hotshot.stats
import tempfile
import StringIO
from django.conf import settings
words_re = re.compile( r'\s+' )
group_prefix_re = [
re.compile( "^.*/django/[^/]+" ),
re.compile( "^(.*)/[^/]+$" ), # extract module path
re.compile( ".*" ), # catch strange entries
]
class ProfileMiddleware(object):
"""
Displays hotshot profiling for any view.
http://yoursite.com/yourview/?prof
Add the "prof" key to query string by appending ?prof (or &prof=)
and you'll see the profiling results in your browser.
It's set up to only be available in django's debug mode, is available for superuser otherwise,
but you really shouldn't add this middleware to any production configuration.
WARNING: It uses hotshot profiler which is not thread safe.
"""
def process_request(self, request):
if (settings.DEBUG or request.user.is_superuser) and 'prof' in request.GET:
self.tmpfile = tempfile.mktemp()
self.prof = hotshot.Profile(self.tmpfile)
def process_view(self, request, callback, callback_args, callback_kwargs):
if (settings.DEBUG or request.user.is_superuser) and 'prof' in request.GET:
return self.prof.runcall(callback, request, *callback_args, **callback_kwargs)
def get_group(self, file):
for g in group_prefix_re:
name = g.findall( file )
if name:
return name[0]
def get_summary(self, results_dict, sum):
list = [ (item[1], item[0]) for item in results_dict.items() ]
list.sort( reverse = True )
list = list[:40]
res = " tottime\n"
for item in list:
res += "%4.1f%% %7.3f %s\n" % ( 100*item[0]/sum if sum else 0, item[0], item[1] )
return res
def summary_for_files(self, stats_str):
stats_str = stats_str.split("\n")[5:]
mystats = {}
mygroups = {}
sum = 0
for s in stats_str:
fields = words_re.split(s);
if len(fields) == 7:
time = float(fields[2])
sum += time
file = fields[6].split(":")[0]
if not file in mystats:
mystats[file] = 0
mystats[file] += time
group = self.get_group(file)
if not group in mygroups:
mygroups[ group ] = 0
mygroups[ group ] += time
return "<pre>" + \
" ---- By file ----\n\n" + self.get_summary(mystats,sum) + "\n" + \
" ---- By group ---\n\n" + self.get_summary(mygroups,sum) + \
"</pre>"
def process_response(self, request, response):
if (settings.DEBUG or request.user.is_superuser) and 'prof' in request.GET:
self.prof.close()
out = StringIO.StringIO()
old_stdout = sys.stdout
sys.stdout = out
stats = hotshot.stats.load(self.tmpfile)
stats.sort_stats('time', 'calls')
stats.print_stats()
sys.stdout = old_stdout
stats_str = out.getvalue()
if response and response.content and stats_str:
response.content = "<pre>" + stats_str + "</pre>"
response.content = "\n".join(response.content.split("\n")[:40])
response.content += self.summary_for_files(stats_str)
os.unlink(self.tmpfile)
return response

把这个文件保存为middleware.py,然后在settings.py中的MIDDLEWARE_CLASSES添加刚才写好的中间件
'yourproject.yourapp.middleware.ProfileMiddleware',
添加完成后,在原本要访问的url后添加’?prof’就可以看到性能分析结果了:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<pre> 202188 function calls (182154 primitive calls) in 2.948 seconds
Ordered by: internal time, call count
ncalls tottime percall cumtime percall filename:lineno(function)
302 2.444 0.008 2.454 0.008 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/MySQLdb/cursors.py:277(_do_query)
2 0.086 0.043 0.104 0.052 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/MySQLdb/connections.py:62(__init__)
2 0.017 0.009 0.017 0.009 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/MySQLdb/connections.py:287(set_character_set)
2 0.016 0.008 0.016 0.008 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/backends/mysql/base.py:299(_set_autocommit)
302 0.008 0.000 2.488 0.008 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/backends/utils.py:76(execute)
19410 0.008 0.000 0.008 0.000 /usr/lib64/python2.7/copy.py:267(_keep_alive)
10237 0.008 0.000 0.008 0.000 /usr/lib64/python2.7/json/encoder.py:33(encode_basestring)
600 0.008 0.000 0.012 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/query.py:246(clone)
6 0.007 0.001 0.007 0.001 /usr/lib64/python2.7/urlparse.py:336(unquote)
500 0.007 0.000 2.712 0.005 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/query.py:229(iterator)
1152 0.005 0.000 0.029 0.000 /usr/lib64/python2.7/copy.py:253(_deepcopy_dict)
1 0.005 0.005 0.014 0.014 /usr/lib64/python2.7/json/encoder.py:212(iterencode)
302 0.005 0.000 0.010 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/MySQLdb/cursors.py:117(_do_get_result)
300 0.005 0.000 0.121 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/compiler.py:351(as_sql)
300 0.004 0.000 0.033 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/query.py:1114(build_filter)
302 0.004 0.000 0.004 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/MySQLdb/cursors.py:313(_get_result)
1 0.003 0.003 0.017 0.017 /usr/lib64/python2.7/json/encoder.py:186(encode)
1300 0.003 0.000 0.004 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/backends/mysql/operations.py:176(get_db_converters)
600 0.003 0.000 0.006 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/query.py:1334(names_to_path)
300 0.003 0.000 0.004 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/query.py:110(__init__)
302 0.003 0.000 0.003 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/MySQLdb/cursors.py:82(_warning_check)
200 0.003 0.000 0.004 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/base.py:388(__init__)
302 0.003 0.000 2.469 0.008 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/MySQLdb/cursors.py:139(execute)
300 0.003 0.000 0.022 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/compiler.py:155(get_select)
300 0.003 0.000 0.007 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/compiler.py:472(get_default_columns)
908 0.003 0.000 0.003 0.000 /home/xsy/.virtualenvs/calc/lib64/python2.7/encodings/utf_8.py:15(decode)
3500 0.003 0.000 0.005 0.000 /home/xsy/.virtualenvs/calc/lib/python2.7/site-packages/django/db/models/sql/compiler.py:325(quote_name_unless_alias)
<pre> ---- By file ----

简单解释下主要的信息,ncalls是函数调用的次数,percall指调用一次所需时间,cumtime是运行时占用的总时间。比如percall时间很短而cumtime时间很长,则代表循环次数过多了,试着从算法角度进行优化;如果从输出中看到大量时间消耗和“sql”有关的,则试着从数据库相关代码优化;如果看见大量时间消耗和”template”有关,则尝试从模板渲染处着手优化。

评论和分享

Roy.S

微信公众号:hi-roy


野生程序员


China