Django Order by field ignoring NULL

   Order_by is a commonly used django query to order the queryset based on the order_by field name. Basic syntax of order_by is shown below.

MyModel.objects.all().order_by('field_name')

Consider the following models.py

from django.db import models

class SelectedProducts(models.model):
    '''
    Model for storing the product basic updation details.
    '''
    name = models.CharField(max_length=250)
    price = models.FloatField(default=0.0)
    count = models.IntegerField(blank=True, null=True, max_length=5)

  def __unicode__(self):
       return self.name

Now consider the case where you have 3 entries in your table.

Name         Price           Count

abc              15.0               5
lmo            10.25              2
xyz             12.50           (None)

According to the following query:

     SelectedProducts.objects.all().order_by('count')

Output:
[ <objects:xyz>, <objects:lmo>, <objects:abc>]

Since the count of xyz is None, it will be shown first.
Now, to show model instance with count 'None' at the end, you can use following methods:

1. Method: Using 'extra'

Based on this method code will  be:

    selected = SelectedProducts.objects.all().order_by('count')
    q = selected.extra(select={'count_null': 'count is null'})
    q = q.extra(order_by=['count_null'])
    print q

Output:
[ <objects:lmo>, <objects:abc>, <objects:xyz>]

In some cases the above approach returns sql syntax error.

2. Method: Using 'aggregate'

Based on this method code will  be:

    from django.db.models import Count
    q = SelectedProducts.objects.all().annotate(null_count=Count('count')).order_by('-null_count', 'count')
    print q

Output:
[ <objects:lmo>, <objects:abc>, <objects:xyz>]

Django Admin Action

   If you are looking for a way to update for change many objects at once via admin panel then the best way is to use Django's admin action. By setting a custom action you can change the object list in the admin by selecting the objects you want to change and go to the bottom of the page where you can see a drop down. Click on the drop down and it will show a list of admin action including the custom action that you have created. By default you can see 'Delete Selected User' action.

Creating Sample Model

Lets create a sample table in models.py to.

 
from django.db import models

class ProductsUpdated(models.model):
    '''
    Model for storing the product basic updation details.
    '''
    name = models.CharField(max_length=250)
    price = models.FloatField(default=0.0)
    updated = models.BooleanField(default=False)

  def __unicode__(self):
       return self.name

Writing Custom Action

In admin.py

 
from django.contrib import admin

def updated(modeladmin, request, queryset):
    '''
    Custom action method
    '''
    queryset.update(updated=True)           # Updating the 'úpdated' field
updated.short_description="Items Updated"   # Description thats shown in the bottom drop down

Now call the action method in the admin of the table ProductsUpdated.
The whole code would look like this:

from django.contrib import admin
from myapp.models import ProductsUpdated


def updated(modeladmin, request, queryset):
    queryset.update(updated=True)
updated.short_description =  "Items Updated"


def ProductsUpdatedAdmin(admin.ModelAdmin):
    list_display = [ 'title', 'úpdated']
    action = [updated]

admin.site.register(ProductsUpdated, ProductsUpdatedAdmin)

Django Complex Query

    Recently I got a chance to work on database query handling in a project. There were lot of objects filtering using field names, some advanced filtering etc. There I met a simple piece of code that was doing something amazing, our Django's Q object. Normally when we use a get or filter on object we specify some parameters inside the filter or get or ... So what this Q object does is it gives you the power to add logical operation to our filtering query. We can do our logical AND or OR operation in Q object. I'll list the syntax of Q object operation one by one.

First of all you need to import the Q object. (obviously...)

from django.db.models import Q 

Consider our models.py as

class Author(models.Model):
    name = models.CharField(max_length=200)
    
    def __unicode__(self):
        return self.name


class Store(models.Model):
    name = models.CharField(max_length=200)
    
    def __unicode__(self):
        return self.name


class Books(models.Model):
    author_name = models.ForeignKey(Author)
    name = models.CharField(max_length=200)
    store_name = models.ForeignKey(Store)

    def __unicode__(self):
        return self.name

Now our logical operations:

AND Operation:

    Search for book written by specific author and in specific store.

Books.objects.filter(Q(author_name__contains='John Doe') & Q(store_name__contains='amazon'))

OR Operation:

Search for book written by specific author or books in specific store.

Books.objects.filter(Q(author_name__contains='John Doe') | Q(store_name__contains='amazon'))

NOT Operation:

Search for book not in specific store.

Books.objects.filter(~Q(store_name__contains='amazon'))

Now suppose you have a list of store names and authors you need to query and its not good to use above approach since length will be high and makes things more complex. For these kinds of situation you can use the following approach:

To check books present in all store. (AND Operation)

my_stores = [list of store names]
q_object = Q()
for store in my_stores:
    q_object.add(Q(store_name__contains=store), Q.AND)
Books.objects.filter(q_object)

To check books present in any one of the store. (OR Operation)

my_stores = [list of store names]
q_object = Q()
for store in my_stores:
    q_object.add(Q(store_name__contains=store), Q.OR)
Books.objects.filter(q_object)

Note: By default Q object operation will be AND

SwampDragon push notification

Push notification, one of the complex problem that a occurs for a web developer. Old time we need to install number of packages and make numerous changes in django files especially wsgi.py file to make it work. But now we have SwampDragon to help us out. SwampDragon made push notification too easy (not that easy...) implement compared to former method.

SwampDragon

SwampDragon helps to create near real-time functionality for Django via web socket support.Swamp dragon consists of:

• Redis. Basically a very quick, persistent kind of memcache. Supports pubsub.
• Tornado. A python webserver. Non-blocking IO. Lots of connections are no problem.
• Django.

Below I shown the simple work flow using swampdragon and django:
 

How To 

 

Below I have explained how to send notification to one or more users using same channel with the help of swamdragon and django.

Hope that you have already created your django app.

After that install the following packages:

To install SwampDragon:

 
pip install swampdragon

or

pip install -e git://github.com/jonashagstedt/swampdragon.git#egg=swampdragon

Install redis server (If you don't have redis)

 
sudo apt-get install redis-server

After installation redis server will start automatically. If not use the following command

 
sudo service redis-server start

Now redis server will be up and running. In settings.py file add swampdragon to installed app.
Also add SWAMP_DRAGON_CONNECTION and DRAGON_URL.

 
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'swampdragon',
) 
SWAMP_DRAGON_CONNECTION = ('swampdragon.connections.sockjs_connection.DjangoSubscriberConnection', '/data')

DRAGON_URL = 'http://localhost:9999/'

Create router.py file in the app folder. Router.py helps in routing incoming connection to specified endpoint. 

 
from swampdragon import route_handler
from swampdragon.route_handler import BaseRouter

class MyRouter(BaseRouter):
    route_name = 'my-route'
    valid_verbs = [ 'subscribe']

    def get_subscription_channels(self, **kwargs):
        return ['notification']

route_handler.register(MyRouter)

Now create a js file. Let it be router.js .
Add the channel subscription and all display method for incoming message, inside the router.js file.

 
swampdragon.ready(function () {
//subscribes the channel and router
    swampdragon.subscribe('my-route', 'notification', null, function (context, data) {
        // any thing that happens after successfully subscribing
    }, function (context, data) {
        // any thing that happens if subscribing failed
    });
});
//Extracts message in the channel
swampdragon.onChannelMessage(function (channel, data) {
            InsertMsg2Html( data.data.message);
});

//inserts the received message from the channel to html 
function InsertMsg2Html(msg) {
    var messages = document.getElementById("notification");

    var span = document.createElement("span");
    messages.insertBefore(span, notification.firstChild);
    div.innerHTML = "<strong>" + msg + "</strong> " ;
}

In your template add the following lines:

    <script type="text/javascript" src="http://localhost:9999/settings.js"></script>
    <script type="text/javascript" src="js/sockjs-0.3.4.min.js"></script>
    <script type="text/javascript" src="js/swampdragon.js"></script>
    <script type="text/javascript" src="/js/router.js"></script>
    <script type="text/javascript" src="js/datamapper.js"></script>
    <script type="text/javascript" src="js/services.js"></script>

Now in views.py 

from swampdragon.pubsub_providers.data_publisher import publish_data

data = { 'message': 'You have successfully send a request'}
publish_data(channel='notification', data=data)

Here publish_data method will publish the data to the specified channel. onChannelMessage mentioned in the router.js gets the published message. Remaining logic to what to do with the incoming message can be changed depending up on the user requirement.

Django Profiler

Recently I have used many profilers in python which got me thinking about using profiler in a django app. So I started searching for ways to use a profiler in django. I have tried integrating many python profiler in django app's views.py file but it didn't go well as I expected. But then I tried to remove the profile added in the py file and used python's builtin profiler cProfile. I used the following command to execute python profiler along with the django runserver.

 
   $ python -m cProfile manage.py runserver

At first I couldn't figure out the way to get the profiler output while running the django dev server. As soon as I terminate the server the profiler output began to display in the terminal. Yes I got an output from profile but not the expected way. The present generated profiler output was the profiler output of the all the django process from the beginning of the server starting which is not easy to go through especially the list was too long. So I continued the R&D and finally reached at a py file which is added as a middleware file in django. Way the profiler work is like this after starting the server you can access the profiler data by typing ?prof which will open the profiler output of the current page in the browser. The profiler code is mentioned below.

 
try:
    import cProfile as profile
except ImportError:
    import profile

try:
    from cStringIO import StringIO
except:
    from io import StringIO

import pstats
from django.conf import settings


class ProfilerMiddleware(object):
    """
    Simple profile middleware to profile django views. To run it, add ?prof to
    the URL like this:

        http://localhost:8000/view/?prof

    Optionally pass the following to modify the output:

    ?sort => Sort the output by a given metric. Default is time.
        See http://docs.python.org/2/library/profile.html#pstats.Stats.sort_stats
        for all sort options.

    ?count => The number of rows to display. Default is 100.

    This is adapted from an example found here:
    http://www.slideshare.net/zeeg/django-con-high-performance-django-presentation.
    """
    def can(self, request):
        return settings.DEBUG and 'prof' in request.GET and \
            request.user is not None and request.user.is_staff

    def process_view(self, request, callback, callback_args, callback_kwargs):
        if self.can(request):
            self.profiler = profile.Profile()
            args = (request,) + callback_args
            try:
                return self.profiler.runcall(callback, *args, **callback_kwargs)
            except:
                # we want the process_exception middleware to fire
                # https://code.djangoproject.com/ticket/12250
                return

    def process_response(self, request, response):
        if self.can(request):
            self.profiler.create_stats()
            io = StringIO()
            stats = pstats.Stats(self.profiler, stream=io)
            stats.strip_dirs().sort_stats(request.GET.get('sort', 'time'))
            stats.print_stats(int(request.GET.get('count', 100)))
            response.content = '<pre>%s</pre>' % io.getvalue()
        return response

Steps:

• Copy paste the code to file and save it as .py file.
• Add the file to the MIDDLEWARE_CLASSES section in djnago settings.py file.
• Run the django server.
• To see the profiler add '?prof' after the url in the browser. (eg: http://12.0.0.1:8000/home/?prof )

For other profilers for django app:
     Profiling Django
     Django Live Profiler

Converting rml file to pdf

PDF generation using reportlab's rml file

What is RML?
RML is the Report Markup Language - a member of the XML family of languages, and the XML dialect used by rml2pdf to produce documents in Adobe's Portable Document Format (PDF).RML documents can be written automatically by a program or manually using any word processor that can output text files (e.g. using a "Save as Text" option from the save menu). Since RML documents are basic text files, they can be created on the fly by scripts in Python, Perl, or almost any other language. RML makes creating documents in PDF as simple as creating a basic web page - RML is as easy to write as HTML, and uses "tags" just like HTML. It is much easier than trying to write PDF programmatically.
User guide for rml can be found here.


Installation:
 You will have to download and install reportlab plus. You can find the installation steps here.


(Assuming that you have already created a django app)

Now consider the case that I have an xml file which I need to convert to pdf. For that first you will have to create an rml file depending upon your need and save with .rml extension.

RML sample files can be found here.

In views.py import the required modules.

from rlextra.rml2pdf import rml2pdf
import xml.etree.ElementTree as ElementTree
from xml.dom import minidom

Next we write a function to obtain the rml file. The function at first extracts data from the xml file (as dict) the pass it on to the rml through context. Finally the function returns an rml in the specified encoded format  .

def getRML():
    xmldict = {}
    tree = ElementTree.parse('path to your xml file')
    root = tree.getroot()
    xmldict = XmlDictConfig(root)
    t = Template(open('path your rml file').read())
    c = Context({"xmldata": xmldict}) 
    rml = t.render(c)
    #django templates are unicode, and so need to be encoded to utf-8
    return rml.encode('utf16')

Last step is the pdf generation process.

class ConvertView(View):
   

    def get(self, request, *args, **kwargs):
        rml = getRML()
        buf = cStringIO.StringIO()
        
        #create the pdf
        rml2pdf.go(rml, outputFileName='output.pdf')
        buf.reset()
        pdfData = buf.read()
        
        #send the response
        response = HttpResponse(mimetype='application/pdf')
        response.write(pdfData)
        response['Content-Disposition'] = 'attachment; filename=output.pdf'
        return response

Now in urls.py :

from app.views import ConvertView

urlpatterns = patterns('',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^convert/$', ConvertView.as_view(), name="convert2pdf"),
      
)

Finally call the 'convert' url to obtain the pdf document.

PDF generation using easy pdf

 

PDF generation using easy pdf

Easy pdf is a pdf rendering django package. You can install it with the following command:

$ pip install django-easy-pdf
$ pip install "xhtml2pdf>=0.0.6" "reportlab>=2.7,<3"
 

Easy pdf documentation can be found here

Now in your views.py file:

import easy_pdf
from easy_pdf.views import PDFTemplateView 



class MyView(PDFTemplateView):
     template_name = "converter.html"
     pdf_filename = "Output.pdf"


    def post(self, request, args, kwargs):
          objects = MyReport.objects.all()           
          context = { 'objects'  : objects}
          return  easy_pdf.rendering.render_to_pdf_response(request,  self.template_name, context, filename='Output.pdf', encoding=u'utf-8', **kwargs)

Now in urls.py :

from app.views import MyView

urlpatterns = patterns('',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^convert/$', MyView.as_view(), name="convert2pdf"),
      
)

Finally call the 'convert' url to obtain the pdf document.