Step 0 – Update and upgrade
We are using Ubuntu 16.04 LTS for this tutorial
apt-get update update the list of available packages and their versions, but it does not install or upgrade any packages. apt-get upgrade actually installs newer versions of the packages you have. After updating the lists, the package manager knows about available updates for the software you have installed.
$ sudo apt-get update
$ sudo apt-get upgrade
Continue reading “Deploy Django with NginX, Gunicorn, PostgreSQL, virtualenv”
With the adaptation of micro-service architecture i.e. various components as independent services, docker community came up with docker-compose (previously FIg). Using a single (YAML) configuration file (docker-compose.yml) to specify all the components, which docker compose build and spawn as independent services i.e. docker containers.
Use-case: You have a web project, with web application developed using Django, using a Postgres database, redis as caching engine, and NginX for serving over the web. Using docker-compose you can deploy this stack with a single command:
docker-compose build --no-cache && docker-compose up
This blog post is about using docker compose, for deploying your Django application with Postgres, Redis, and Nginx. It is presumed, you already have your Django project, and want to deploy your full stack.
High level steps
Install and start Docker compose
Setup project – presume you already have a Django project.
Create Dockerfile(s) and docker-compose.yml
Build service images – docker-compose build
Create database and database migrations – docker-compose run web python manage.py migrate
Start services containers – docker-compose up
Continue reading “Deploy Django using Docker Compose”
Database partitioning is about logically splitting one large table into smaller physical pieces, such that improving query performance. DB partitioning is a good alternate for indexing multiple columns, reducing index size, hence the memory in use. Few common pros of database partitioning:
- Improved performance – data operations (CRUD) can be performed on a smaller volume of data, for example, in case of collecting data overtime, putting old data in separate partition might help with performance.
- Bulk create and delete can be efficient by adding or removing separate partitions.
- Time based partition can be helpful in cleaning old seldom-used data i.e. month based partition we can simply set a cron job for cleaning 12 month old partition, without effecting the table portion heavily in use for ADD, UPDATE, etc.
- Improved scalability – In case of very large tables, you can partition and have them hosted on a separate server.
There are 2 main approaches to database partitioning:
- Horizontal partitioning (Sharding) – a table is split horizontally, such that each partition is a subset of the table, having the same schema (i.e. number of fields/columns).
- Vertical partitioning – a table is split on the fields/columns, such that each subset has separate schema. A common use-case for vertical partitioning is to partition table fields on the basis of pattern of use i.e. frequently accessed fields are to be grouped together, and the less frequently accessed are put in a separate partition.
This blog post is about setting automatic horizontal partitioning (month based) on a table in PostgreSQL.
The high-level steps are:
- Create table, or select an existing one.
- Execute partitioning function or procedure
- Table trigger – to call the partition procedure.
- View for parent-child tables (optional)
Step 1 – Create Table
CREATE TABLE partition_test(id BIGINT, created_datetime DATE);
Step 2 – Trigger function
CREATE OR REPLACE FUNCTION test_partition_function() RETURNS trigger AS
partition_date := to_char(NEW.created_datetime,'YYYY_MM');
partition := TG_RELNAME || '_' || partition_date;
IF NOT EXISTS(SELECT relname FROM pg_class WHERE relname=partition) THEN
RAISE NOTICE 'A partition has been created %',partition;
EXECUTE 'CREATE TABLE ' || partition || ' () INHERITS (' || TG_RELNAME || ');';
EXECUTE 'INSERT INTO ' || partition || ' SELECT(' || TG_RELNAME || ' ' || quote_literal(NEW) || ').* RETURNING id;';
LANGUAGE plpgsql VOLATILE
Step 3 – Table trigger
CREATE TRIGGER partition_test_trg
AFTER INSERT ON partition_test
FOR EACH ROW EXECUTE PROCEDURE test_partition_function();
Step 4 – Viewing the partition (optional)
CREATE VIEW show_partitions AS
SELECT nmsp_parent.nspname AS parent_schema,
parent.relname AS parent,
nmsp_child.nspname AS child_schema,
child.relname AS child
JOIN pg_class parent ON pg_inherits.inhparent = parent.oid
JOIN pg_class child ON pg_inherits.inhrelid = child.oid
JOIN pg_namespace nmsp_parent ON nmsp_parent.oid = parent.relnamespace
JOIN pg_namespace nmsp_child ON nmsp_child.oid = child.relnamespace
WHERE parent.relname='partition_test' ;
Step 5 – Verification (optional)
Let’s test using the show_partitions view, if we have any partitions yet
select * from show_partitions;
insert in partition_test
insert into partition_test values (1, '2018-01-19');
Few more inserts for the same month
insert into partition_test values (2, '2018-01-22');
insert into partition_test values (3, '2018-01-29');
Continue reading “DB partition trigger with PostgreSQL”
PostgreSQL (aka Postgres), is an open-source SQL relational database management system (RDBMS). These RDBMS are the key component for the majority of all the web sites and web applications, providing an efficient way to persist, organize, and access information.
In the blog, I’ll demonstrate, how to install Postgres 9.6 for Ubuntu 14.04, 16.04 LTS.
Step 1: Add Postgresql repository
$ sudo add-apt-repository "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -sc)-pgdg main"
$ wget -q -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
Continue reading “Install PostgreSQL 9.6 on Ubuntu 14.04 or 16.04 LTS”