Celery: an overview and how it works

I don’t know about other programming languages, but if you are using Python or Django, you must have heard about Celery quite a few times, and if not, you better look into it. As stated on the project Celery website:
Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
In case of a web service (most common use-case), asynchronous task queues are utilities to push (time-consuming) tasks in background while timely sending back the response for a user request. These delegated tasks can be anything from sending few notifications, dispatching emails, update system logs, or update internal ERP. Having the aforementioned tasks in line with the request processing, can delay the response back to the user to a large extent.

Continue reading “Celery: an overview and how it works”

Deploy Django with NginX, Gunicorn, PostgreSQL, virtualenv

Step 0 – Update and upgrade

We are using Ubuntu 16.04 LTS for this tutorial
ubuntu version
apt-get update update the list of available packages and their versions, but it does not install or upgrade any packages. apt-get upgrade actually installs newer versions of the packages you have. After updating the lists, the package manager knows about available updates for the software you have installed.
$ sudo apt-get update
$ sudo apt-get upgrade

Continue reading “Deploy Django with NginX, Gunicorn, PostgreSQL, virtualenv”

DB partition trigger with PostgreSQL

Database partitioning is about logically splitting one large table into smaller physical pieces, such that improving query performance.  DB partitioning is a good alternate for indexing multiple columns, reducing index size, hence the memory in use. Few common pros of database partitioning:
  • Improved performance – data operations (CRUD) can be performed on a smaller volume of data, for example, in case of collecting data overtime, putting old data in separate partition might help with performance.
  • Bulk create and delete can be efficient by adding or removing separate partitions.
  • Time based partition can be helpful in cleaning old seldom-used data i.e. month based partition we can simply set a cron job for cleaning 12 month old partition, without effecting the table portion heavily in use for ADD, UPDATE, etc.
  • Improved scalability – In case of very large tables, you can partition and have them hosted on a separate server.
There are 2 main approaches to database partitioning:
  • Horizontal partitioning (Sharding) – a table is split horizontally, such that each partition is a subset of the table, having the same schema (i.e. number of fields/columns).
  • Vertical partitioning – a table is split on the fields/columns, such that each subset has separate schema. A common use-case for vertical partitioning is to partition table fields on the basis of pattern of use i.e. frequently accessed fields are to be grouped together, and the less frequently accessed are put in a separate partition.

Continue reading “DB partition trigger with PostgreSQL”