Get complete error traceback without terminating program

   For some try except cases its difficult to get the appropriate error traceback only using 'Exception' keyword in try..except.
But for such cases python has a module named 'traceback' which can be used in such situations.

Consider the following piece of code from myscript.py

try:
    test_variable = tuple(range(0,10))
    test_variable.append(10)
except Exception as error:
    print "[ - ] Error:", error

Output would look like:

     [ - ] Error: AttributeError: 'tuple' object has no attribute 'append'

Now issue with above error response is vagueness in details regarding the error. In order to get complete error traceback we can use python 'traceback' module.

Now the code would look like:

import traceback

try:
    test_variable = tuple(range(0,10))
    test_variable.append(10)
except:
    print "[ - ] Error:", traceback.print_exc()

Output will be:

[ - ] Error: Traceback (most recent call last):
File "/home/myscript.py", line 5, in <module>
    a.append(15)
AttributeError: 'tuple' object has no attribute 'append'

We can also use python sys module to get the exception info of the error and print it. For example:

import traceback
import sys

try:
    exc_info = sys.exc_info()
    test_variable = tuple(range(0,10))
    test_variable.append(10)
except:
    traceback.print_exception(*exc_info)
    del exc_info

Output will be same as one shown above.

For more usages regarding trackback module can be found here.

Django Order by field ignoring NULL

   Order_by is a commonly used django query to order the queryset based on the order_by field name. Basic syntax of order_by is shown below.

MyModel.objects.all().order_by('field_name')

Consider the following models.py

from django.db import models

class SelectedProducts(models.model):
    '''
    Model for storing the product basic updation details.
    '''
    name = models.CharField(max_length=250)
    price = models.FloatField(default=0.0)
    count = models.IntegerField(blank=True, null=True, max_length=5)

  def __unicode__(self):
       return self.name

Now consider the case where you have 3 entries in your table.

Name         Price           Count

abc              15.0               5
lmo            10.25              2
xyz             12.50           (None)

According to the following query:

     SelectedProducts.objects.all().order_by('count')

Output:
[ <objects:xyz>, <objects:lmo>, <objects:abc>]

Since the count of xyz is None, it will be shown first.
Now, to show model instance with count 'None' at the end, you can use following methods:

1. Method: Using 'extra'

Based on this method code will  be:

    selected = SelectedProducts.objects.all().order_by('count')
    q = selected.extra(select={'count_null': 'count is null'})
    q = q.extra(order_by=['count_null'])
    print q

Output:
[ <objects:lmo>, <objects:abc>, <objects:xyz>]

In some cases the above approach returns sql syntax error.

2. Method: Using 'aggregate'

Based on this method code will  be:

    from django.db.models import Count
    q = SelectedProducts.objects.all().annotate(null_count=Count('count')).order_by('-null_count', 'count')
    print q

Output:
[ <objects:lmo>, <objects:abc>, <objects:xyz>]

Django Admin Action

   If you are looking for a way to update for change many objects at once via admin panel then the best way is to use Django's admin action. By setting a custom action you can change the object list in the admin by selecting the objects you want to change and go to the bottom of the page where you can see a drop down. Click on the drop down and it will show a list of admin action including the custom action that you have created. By default you can see 'Delete Selected User' action.

Creating Sample Model

Lets create a sample table in models.py to.

 
from django.db import models

class ProductsUpdated(models.model):
    '''
    Model for storing the product basic updation details.
    '''
    name = models.CharField(max_length=250)
    price = models.FloatField(default=0.0)
    updated = models.BooleanField(default=False)

  def __unicode__(self):
       return self.name

Writing Custom Action

In admin.py

 
from django.contrib import admin

def updated(modeladmin, request, queryset):
    '''
    Custom action method
    '''
    queryset.update(updated=True)           # Updating the 'úpdated' field
updated.short_description="Items Updated"   # Description thats shown in the bottom drop down

Now call the action method in the admin of the table ProductsUpdated.
The whole code would look like this:

from django.contrib import admin
from myapp.models import ProductsUpdated


def updated(modeladmin, request, queryset):
    queryset.update(updated=True)
updated.short_description =  "Items Updated"


def ProductsUpdatedAdmin(admin.ModelAdmin):
    list_display = [ 'title', 'úpdated']
    action = [updated]

admin.site.register(ProductsUpdated, ProductsUpdatedAdmin)

Django Complex Query

    Recently I got a chance to work on database query handling in a project. There were lot of objects filtering using field names, some advanced filtering etc. There I met a simple piece of code that was doing something amazing, our Django's Q object. Normally when we use a get or filter on object we specify some parameters inside the filter or get or ... So what this Q object does is it gives you the power to add logical operation to our filtering query. We can do our logical AND or OR operation in Q object. I'll list the syntax of Q object operation one by one.

First of all you need to import the Q object. (obviously...)

from django.db.models import Q 

Consider our models.py as

class Author(models.Model):
    name = models.CharField(max_length=200)
    
    def __unicode__(self):
        return self.name


class Store(models.Model):
    name = models.CharField(max_length=200)
    
    def __unicode__(self):
        return self.name


class Books(models.Model):
    author_name = models.ForeignKey(Author)
    name = models.CharField(max_length=200)
    store_name = models.ForeignKey(Store)

    def __unicode__(self):
        return self.name

Now our logical operations:

AND Operation:

    Search for book written by specific author and in specific store.

Books.objects.filter(Q(author_name__contains='John Doe') & Q(store_name__contains='amazon'))

OR Operation:

Search for book written by specific author or books in specific store.

Books.objects.filter(Q(author_name__contains='John Doe') | Q(store_name__contains='amazon'))

NOT Operation:

Search for book not in specific store.

Books.objects.filter(~Q(store_name__contains='amazon'))

Now suppose you have a list of store names and authors you need to query and its not good to use above approach since length will be high and makes things more complex. For these kinds of situation you can use the following approach:

To check books present in all store. (AND Operation)

my_stores = [list of store names]
q_object = Q()
for store in my_stores:
    q_object.add(Q(store_name__contains=store), Q.AND)
Books.objects.filter(q_object)

To check books present in any one of the store. (OR Operation)

my_stores = [list of store names]
q_object = Q()
for store in my_stores:
    q_object.add(Q(store_name__contains=store), Q.OR)
Books.objects.filter(q_object)

Note: By default Q object operation will be AND

SwampDragon push notification

Push notification, one of the complex problem that a occurs for a web developer. Old time we need to install number of packages and make numerous changes in django files especially wsgi.py file to make it work. But now we have SwampDragon to help us out. SwampDragon made push notification too easy (not that easy...) implement compared to former method.

SwampDragon

SwampDragon helps to create near real-time functionality for Django via web socket support.Swamp dragon consists of:

• Redis. Basically a very quick, persistent kind of memcache. Supports pubsub.
• Tornado. A python webserver. Non-blocking IO. Lots of connections are no problem.
• Django.

Below I shown the simple work flow using swampdragon and django:
 

How To 

 

Below I have explained how to send notification to one or more users using same channel with the help of swamdragon and django.

Hope that you have already created your django app.

After that install the following packages:

To install SwampDragon:

 
pip install swampdragon

or

pip install -e git://github.com/jonashagstedt/swampdragon.git#egg=swampdragon

Install redis server (If you don't have redis)

 
sudo apt-get install redis-server

After installation redis server will start automatically. If not use the following command

 
sudo service redis-server start

Now redis server will be up and running. In settings.py file add swampdragon to installed app.
Also add SWAMP_DRAGON_CONNECTION and DRAGON_URL.

 
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'swampdragon',
) 
SWAMP_DRAGON_CONNECTION = ('swampdragon.connections.sockjs_connection.DjangoSubscriberConnection', '/data')

DRAGON_URL = 'http://localhost:9999/'

Create router.py file in the app folder. Router.py helps in routing incoming connection to specified endpoint. 

 
from swampdragon import route_handler
from swampdragon.route_handler import BaseRouter

class MyRouter(BaseRouter):
    route_name = 'my-route'
    valid_verbs = [ 'subscribe']

    def get_subscription_channels(self, **kwargs):
        return ['notification']

route_handler.register(MyRouter)

Now create a js file. Let it be router.js .
Add the channel subscription and all display method for incoming message, inside the router.js file.

 
swampdragon.ready(function () {
//subscribes the channel and router
    swampdragon.subscribe('my-route', 'notification', null, function (context, data) {
        // any thing that happens after successfully subscribing
    }, function (context, data) {
        // any thing that happens if subscribing failed
    });
});
//Extracts message in the channel
swampdragon.onChannelMessage(function (channel, data) {
            InsertMsg2Html( data.data.message);
});

//inserts the received message from the channel to html 
function InsertMsg2Html(msg) {
    var messages = document.getElementById("notification");

    var span = document.createElement("span");
    messages.insertBefore(span, notification.firstChild);
    div.innerHTML = "<strong>" + msg + "</strong> " ;
}

In your template add the following lines:

    <script type="text/javascript" src="http://localhost:9999/settings.js"></script>
    <script type="text/javascript" src="js/sockjs-0.3.4.min.js"></script>
    <script type="text/javascript" src="js/swampdragon.js"></script>
    <script type="text/javascript" src="/js/router.js"></script>
    <script type="text/javascript" src="js/datamapper.js"></script>
    <script type="text/javascript" src="js/services.js"></script>

Now in views.py 

from swampdragon.pubsub_providers.data_publisher import publish_data

data = { 'message': 'You have successfully send a request'}
publish_data(channel='notification', data=data)

Here publish_data method will publish the data to the specified channel. onChannelMessage mentioned in the router.js gets the published message. Remaining logic to what to do with the incoming message can be changed depending up on the user requirement.

Django Profiler

Recently I have used many profilers in python which got me thinking about using profiler in a django app. So I started searching for ways to use a profiler in django. I have tried integrating many python profiler in django app's views.py file but it didn't go well as I expected. But then I tried to remove the profile added in the py file and used python's builtin profiler cProfile. I used the following command to execute python profiler along with the django runserver.

 
   $ python -m cProfile manage.py runserver

At first I couldn't figure out the way to get the profiler output while running the django dev server. As soon as I terminate the server the profiler output began to display in the terminal. Yes I got an output from profile but not the expected way. The present generated profiler output was the profiler output of the all the django process from the beginning of the server starting which is not easy to go through especially the list was too long. So I continued the R&D and finally reached at a py file which is added as a middleware file in django. Way the profiler work is like this after starting the server you can access the profiler data by typing ?prof which will open the profiler output of the current page in the browser. The profiler code is mentioned below.

 
try:
    import cProfile as profile
except ImportError:
    import profile

try:
    from cStringIO import StringIO
except:
    from io import StringIO

import pstats
from django.conf import settings


class ProfilerMiddleware(object):
    """
    Simple profile middleware to profile django views. To run it, add ?prof to
    the URL like this:

        http://localhost:8000/view/?prof

    Optionally pass the following to modify the output:

    ?sort => Sort the output by a given metric. Default is time.
        See http://docs.python.org/2/library/profile.html#pstats.Stats.sort_stats
        for all sort options.

    ?count => The number of rows to display. Default is 100.

    This is adapted from an example found here:
    http://www.slideshare.net/zeeg/django-con-high-performance-django-presentation.
    """
    def can(self, request):
        return settings.DEBUG and 'prof' in request.GET and \
            request.user is not None and request.user.is_staff

    def process_view(self, request, callback, callback_args, callback_kwargs):
        if self.can(request):
            self.profiler = profile.Profile()
            args = (request,) + callback_args
            try:
                return self.profiler.runcall(callback, *args, **callback_kwargs)
            except:
                # we want the process_exception middleware to fire
                # https://code.djangoproject.com/ticket/12250
                return

    def process_response(self, request, response):
        if self.can(request):
            self.profiler.create_stats()
            io = StringIO()
            stats = pstats.Stats(self.profiler, stream=io)
            stats.strip_dirs().sort_stats(request.GET.get('sort', 'time'))
            stats.print_stats(int(request.GET.get('count', 100)))
            response.content = '<pre>%s</pre>' % io.getvalue()
        return response

Steps:

• Copy paste the code to file and save it as .py file.
• Add the file to the MIDDLEWARE_CLASSES section in djnago settings.py file.
• Run the django server.
• To see the profiler add '?prof' after the url in the browser. (eg: http://12.0.0.1:8000/home/?prof )

For other profilers for django app:
     Profiling Django
     Django Live Profiler

Python Profilers

Code optimization is one of the obstacles that every developer now a days come across. There are lot of tools present now a days to optimize code. You can find python builtin or 3rd party modules for code optimization. Below I have mentioned a few codes optimization tools that I have used.

Builtin Modules

• Profile

  
  
  How To :  First of all import profile into you python code. Then at the main method use the profile.run() method. On executing the py file we will provide the profiler output. An example code is shown below:
  
  

   
  import profile
  
  
  def add(first, second):
      return first + second
  
  
  profile.run(' print add(1,2); print ')
  
  
  

        Profile module also provides runtx( ) for custom profiler generation.
  

• cProfile

  
  
  How To :This module can be used in two ways. One is to import the module and call it as cProfile.run( <your method> )Second is to call the cProfile module at the time of file execution.
  
  

    
     python -m cProfile pythonfile.py
  
  

• pstat

  
  
  How To: Standard report produced by the profile module is not very flexible. So depending upon the need the user can save the output of the run() or runtx() and process it using the Stats class from pstat.

Third Party Modules

• Line Profiler

  
         How To :  First of all install line profile using the following command.
  

   
     pip install line_profiler
  
  

  Then add the @profile above your method definition. Finally execute the code using the following command.

  
    $ kernprof.py -l -v fib.py