An Intro To Dynamic Programming, Pt II: Edit Distance

 

Medium

 

Dakota Lillie
Dakota Lillie
Jun 7, 2018 · 10 min read

Last time, we covered the basic principles of dynamic programming and examined how we could use it to greatly enhance the runtime efficiency for calculating the value of the nth item in the fibonacci sequence. This time, we’ll outline a general approach for handling dynamic programming problems, and take a look at how we can use this approach to tackle a new problem — finding the minimum edit distance between two strings. Onward!


The 5 Step Process

According to this excellent series of videos from MIT OpenCourseWare, anytime you need to solve a DP problem, you can break it down into the following five steps.

  1. Define what subproblems need to be solved.
  2. Determine what you need to guess.
  3. Express the relationship between subproblems.
  4. Recurse and memoize.
  5. Solve the original problem.

Let’s go over these steps one by one, and see how they apply to our fibonacci algorithm from before.

(1) Define what subproblems need to be solved

The general rule when defining subproblems is that each subproblem needs to have the same structure as the original problem. There are two main approaches to this: prefixes/suffixes, or subsequences. These approaches both involve breaking down the original problem into smaller subproblems, but they differ in terms of how they make the subproblem smaller.

Prefix/suffix subproblems are made smaller by popping items off either the beginning or the end of the input sequence. This is essentially what we did with the fibonacci algorithm — while we weren’t working with a proper input sequence per se (like a string or array), we accomplished something similar by calculating fib() with progressively smaller values of n. Each recursive call to fib() could thus be viewed as operating on a prefix of the original problem.

Subsequence subproblems, on the other hand, are made smaller by removing items from both ends of the original sequence. We won’t look at any problems that use this approach here, but a good example of it is parenthesization for matrix multiplication.

Also involved in this step is determining the number of subproblems which will need to be solved. Generally, prefixes/suffixes will give you nsubproblems, while subsequences result in n² subproblems. In the case of fibonacci, since we were using quasi-prefixes, there were subproblems.

(2) Determine what you need to guess

DP is oftentimes used for optimization problems, and optimization typically involves picking either the minimum or maximum value from a group of potential options. Since presumably we don’t already know which option is best, the only way to find out is to try, or “guess”, them all. By analyzing the problem, we should be able to deduce the number of guesses that will need to be made per subproblem. The fibonacci algorithm did not involve guessing, but the edit distance problem we’ll soon be looking at will.

(3) Express the relationship between subproblems

The value of each subproblem can be calculated as some combination of the results of other subproblems, and this is where we explicitly define how that will work. With fibonacci, this was easy… it comes from the definition of the fibonacci sequence itself. For some value where 0 ≤ k ≤ nfib(k) = fib(k-1) + fib(k-2).

This is also where we determine how much time it will take to calculate the solution for each subproblem will take once all the requisite recursive calls have been resolved. In the case of the fibonacci sequence, this is constant, or O(1) time, because once the values of the subproblems are calculated, all you need to do is add them together.

(4) Recurse and memoize

Once we know how the subproblems relate to each other, the next step is to flush out the algorithm by adding memoization (and base cases, if taking the recursive approach). This should hopefully be fairly straight forward, if the previous steps have been properly followed. But there’s also one important thing we need to check — namely, we need to ensure that our subproblem dependency DAG is properly acyclic.

DAG is a directed acyclic graphDirected means that the edges (or “connections”) between the vertices (or “nodes”) go in a specific direction, and acyclic means that, well, there are no cycles in the graph. If you recall, we actually already constructed a subproblem dependency DAG for the fibonacci sequence when we built out its recursion tree:

The reason it’s important that the DAG lives up to its name and doesn’t have any cycles is because if you try perform DP on a graph with cycles, the algorithm’s time and space complexity become infinite. Consider the following graph:

What happens if we try to run some dynamic programming on this graph? Well, the value of A depends on B, which depends on the value of C, which in turn depends on the value of A again, and… oh no. Because there’s a cycle, no value ever gets memoized for lookup, and we end up with an infinite loop.

Of course, it’s possible to take an cyclic graph and make it acyclic by “exploding” it into multiple, limited iterations. This is the approach taken by the Bellman Ford algorithm, for instance.

(5) Solve the origin problem

This step can oftentimes be trivial, but sometimes there’s just a little bit of extra work that needs to happen to actually derive the solution to the original problem from the subproblems. This wasn’t required with the fibonacci sequence, but sometimes it is necessary, so it’s an important step to keep in mind.

Here we can also examine the overall running time of our algorithm, which is typically the number of subproblems multiplied by the time it takes to solve each subproblem. In the case of the fibonacci algorithm, we had nsubproblems each of which took constant time to resolve, for a total running time of O(n).

That’s it! Now that we’ve articulated a general formula for DP problems, let’s see if we can apply this methodology to an example case: finding the minimum edit distance between two strings.


Edit Distance

The premise is this: given two strings, we want to find the minimum number of edits that it takes to transform one string into the other. There are three edit operations we can use to transform a string: we can insert a character, remove a character, or substitute a character. Each operation has an associated cost — insertions and removals each cost 1, and substitutions cost either 1 or 2, depending on whether or not you consider a substitution to essentially be just a removal followed by an insertion. When the substitution cost is 2, the result is known as the Levenshtein distance, named after Vladimir Levenshtein, the author of the original algorithm.

For example, the edit distance between ‘hello’ and ‘hail’ is 3 (or 5, if using Levenshtein distance):

  • Substitute ‘e’ for ‘a’ → hallo
  • Substitute the following ‘i’ for ‘l’ → hailo
  • Remove the final ‘o’ → hail

As another example, the edit distance between ‘intrinsic’ and ‘intrusive’ is 4 (or 6, in Levenshtein distance):

  • Substitute the second ‘i’ in ‘intrinsic’ for a ‘u’ → intrunsic
  • Remove the following ’n’ → intrusic
  • Substitute the final ‘c’ for a ‘v’ → intrusiv
  • Insert an ‘e’ at the end → intrusive

Let’s see how we can express the solution to this in code, using the steps outlined above.

(1) Define what subproblems need to be solved

When we recurse over the input sequence (or work through it iteratively with the bottom-up approach), we’re essentially going to be saying “figure out what to do with the first (or last) character, then compute the result by adding together the cost of that and everything that comes after (or before)”. From this, we can deduce that we’re going to be working with prefixes/suffixes. What makes this problem interesting is that there are actually two strings we need to work through. So the total number of subproblems will be n * m, where is the length of the first string and is the length of the second.

(2) Determine what you need to guess

For each subproblem, we’ll need to make a choice — whether to insert, remove, or substitute a character. This is presuming, of course, that the characters we’re comparing are not already equal, in which case we want to do nothing.

(3) Express the relationship between subproblems

Let’s look at how we could formally express the ideas we’ve already come up with in code:

This is taking a suffix approach, and using Levenshtein distance as indicated by a cost of 2 for substitution. From this, we can infer that the time it will take to resolve each subproblem is constant, as all we’ll need to do is take the minimum of 3 values. When I was first looking at this, it was difficult for me to intuit how inserting a character in one string translates to incrementing the index for the other string (and so on for the other operations). So let me try and explain it for those who might be in the same boat.

First of all, there’s no need for explicit string comparisons here. When ‘inserting’ a character, it’s presumed that the character inserted into strA at indexA will be the one in strB at indexB. Once an ‘insertion’ has been made, we have to figure out what to compare next. We’ve already handled the character in strB at indexB, so we can advance the index there. But we haven’t addressed the character in strA at indexA, so we don’t advance indexA. This is perhaps a little confusing, since we’re used to insertions altering indexes… but remember, strA was not actually altered in any way, so the character pointed at by indexA remains the same.

Likewise, we’re able to emulate ‘removing’ the character in strA at indexAby advancing the iterator for that string and not the other. Since substitutions are essentially a removal followed by an insertion, that operation ends up advancing both indexes. And of course, if the characters at the given indexes in both strings are the same, there’s no operation to be performed so we can advance both indexes in that case as well. Given that no edits are actually being made to the strings, incrementing the indexes is mostly about determining which characters need to be ‘handled’ next.

(4) Recurse and memoize

Now the fun part — adding base cases and flushing out the algorithm. Looking at how we’ve laid out the structure of our solution, it’s fairly evident that there are no circular dependencies, so our DAG should indeed be acyclic (if this isn’t clear to you, I’d recommend graphing out the recursion tree like we did with the fibonacci algorithm).

Here’s the naive solution, without memoization, written out in Java:

import java.util.*;

class NaiveEditDistance {

  static int min(int insert, int remove, int replace) {
    Integer[] values = { insert, remove, replace };
    return Collections.min(Arrays.asList(values));
  }

  static int calculateEditDistance(String str1, String str2, int i, int j) {
    
    // If i is 0, we're essentially looking at an empty substring of str1.
    // To convert an empty substring to any other string, you just insert 
    // every character from the other string. Since the only operation to be 
    // performed is insertion, the cost is equal to the length of the other 
    // string.
    if (i == 0) {
      return j;
    }

    // This goes the other way as well of course — if you need to get from
    // some substring to an empty string, The shortest path is to simply 
    // remove every character in the substring.
    if (j == 0) {
      return i;
    }

    // If the characters in the strings at a particular value of i and j match,
    // then we don't need to perform any operation and the cost is 0. Since
    // Java is a 0-indexed language, we need to subtract 1, otherwise we'd
    // get a StringIndexOutOfBoundsException.
    if (str1.charAt(i - 1) == str2.charAt(j - 1)) {
      return calculateEditDistance(str1, str2, i - 1, j - 1);
    }

    // Otherwise, we need to recurse. Note that we're using Levenshtein 
    // distance here.
    return min(1 + calculateEditDistance(str1, str2, i, j - 1),      // insert
               1 + calculateEditDistance(str1, str2, i - 1, j),      // remove
               2 + calculateEditDistance(str1, str2, i - 1, j - 1)); // substitute
  }

  public static void main(String[] args) {
    String str1 = "intrinsic";
    String str2 = "intrusive";
    int minDistance = calculateEditDistance(
        str1, str2, str1.length(), str2.length()
      );
    System.out.println(minDistance);
  }
}
view rawNaiveEditDistance.java hosted with ❤ by GitHub

The output of this gist should be 6. Note that we’ve switched it around here — instead of using suffixes, we’re using prefixes. This is simply because it makes handling the base cases easier… otherwise, it’s pretty much exactly the same deal as before, just in reverse.

However, this isn’t very efficient — we’re unnecessarily recursing over subproblems we’ve already calculated the optimal value for. Let’s add some memoization to improve things:

import java.util.*;

class DPEditDistance {

  static int min(int insert, int remove, int replace) {
    Integer[] values = { insert, remove, replace };
    return Collections.min(Arrays.asList(values));
  }

  static int calculateEditDistance(String str1, String str2, int i, int j, int[][] memo) {

    // A value of greater than -1 in the memo grid means that the value for
    // this combination of i and j has already been calculated and can be
    // promptly returned.
    if (memo[i][j] > -1) return memo[i][j];

    if (i == 0) {
      memo[i][j] = j;
      return j;
    }

    if (j == 0) {
      memo[i][j] = i;
      return i;
    }

    if (str1.charAt(i - 1) == str2.charAt(j - 1)) {
      memo[i][j] = calculateEditDistance(str1, str2, i - 1, j - 1, memo);
      return memo[i][j];
    }

    memo[i][j] = min(1 + calculateEditDistance(str1, str2, i, j - 1, memo),      // insert
                     1 + calculateEditDistance(str1, str2, i - 1, j, memo),      // remove
                     2 + calculateEditDistance(str1, str2, i - 1, j - 1, memo)); // substitute
    return memo[i][j];
  }

  public static void main(String[] args) {
    String str1 = "intrinsic";
    String str2 = "intrusive";
    
    // When generating the memo, we need an extra row and column to accomodate
    // empty substrings.
    int[][] memo = new int[str1.length() + 1][str2.length() + 1];

    // int arrays in Java are initialized with 0 values, but 0 in this
    // context means that no changes are needed. We need another value to
    // represent an unmemoized value, so we'll fill the memo with -1 values 
    // instead.
    for (int[] row : memo)
      Arrays.fill(row, -1);

    int minDistance = calculateEditDistance(
        str1, str2, str1.length(), str2.length(), memo
      );
    System.out.println(minDistance);
  }
}
view rawDPEditDistance.java hosted with ❤ by GitHub

This works fine but is awfully verbose. We can tidy things up a bit by switching from the recursive to the iterative approach, using bottom-up DP:

import java.util.*;

class BottomUpEditDistance {

  static int min(int insert, int remove, int replace) {
    Integer[] values = { insert, remove, replace };
    return Collections.min(Arrays.asList(values));
  }

  static int calculateEditDistance(String str1, String str2) {

    int[][] memo = new int[str1.length() + 1][str2.length() + 1];

    for (int i = 0; i <= str1.length(); i++) {
      for (int j = 0; j <= str2.length(); j++) {
        if (i == 0) {
          memo[i][j] = j;
        }
        else if (j == 0) {
          memo[i][j] = i;
        }
        else if (str1.charAt(i - 1) == str2.charAt(j - 1)) {
          memo[i][j] = memo[i - 1][j - 1];
        }
        else {
          memo[i][j] = min(1 + memo[i][j - 1],
                           1 + memo[i - 1][j],
                           2 + memo[i - 1][j - 1]);
        }
      }
    }

    return memo[str1.length()][str2.length()];
  }

  public static void main(String[] args) {
    String str1 = "intrinsic";
    String str2 = "intrusive";

    int minDistance = calculateEditDistance(str1, str2);
    System.out.println(minDistance);
  }
}
view rawBottomUpEditDistance.java hosted with ❤ by GitHub

To gain an intuition for how we’re building up this memo matrix, I’d recommend taking a look at this excellent video here.

(5) Solve the origin problem

Fortunately, once we’ve tabulated the results from all our subproblems we don’t really have to do any extra work to solve our original problem. But we can still calculate the overall time and space complexity of our algorithm. Once again, for dynamic programming problems this is usually the number of subproblems multiplied by the time taken per subproblem. As we already covered in the previous steps, the number of subproblems is n * m, where nis the length of the first string and is the length of the second, and each subproblem takes constant time to resolve. So the overall time complexity of this algorithm is n * m, and the same goes for the space complexity, since we need n * m space in our memo table. Not so bad!


Conclusion

We’ve successfully examined how we can use a standardized methodology to tackle a non-trivial dynamic programming question. But of course, there is so much more to explore about this topic! If you’re interested, I would highly recommend this video series from MITOpenCourseWare. Of course, reading and watching about dynamic programming will only get you so far… to really learn these things, you have to practice. For that, I’ve found that the exercises in Cracking the Coding Interview are a truly fantastic resource.


Sources

Cracking the Coding Interview: 189 Programming Questions and Solutions
I am not a recruiter. I am a software engineer. And as such, I know what it's like to be asked to whip up brilliant…
www.amazon.com
 
 

 
 

 
 
 
 
Dakota Lillie

WRITTEN BY

How You Can Find the IP Address of Any Website in Just a Few Clicks

Lifewire

Each website on the internet has at least one Internet Protocol address assigned to it. Knowing a website's IP address can be useful to:

  • Bypass website-blocking tools
  • Block specific sites, such as when administering a home network
  • Narrow the geographic location of a web server

Finding IP addresses can be complicated. Web browsers don't usually display them. Furthermore, large websites use a pool of IP addresses rather than just one, meaning that the address used one day might change the next. Two people in different parts of the world often obtain different IP addresses for the same site even if they use the same lookup methods.

Using Ping

The ping utility looks up IP addresses of websites and any other kind of running network device. Ping attempts to contact the site by name and reports back the IP address it finds, along with other information about the connection. Ping is a Command Prompt command in Windows. For example, to find the IP address of Example.com on a desktop computer, use the command line interface instead of the graphical interface, and enter the command:

ping example.com
Ping command IP address screenshot

The command returns a result similar to the following, which contains the IP address:

Pinging example.com [​151.101.193.121] with 32 bytes of data: . . .

Both the Google Play and Apple App stores contain many apps that can generate these same pings from a mobile device.

Many large websites do not return connection information in response to ping commands as a security measure, but you can usually still obtain the site's IP address. The ping method fails if the website is temporarily unreachable or if the computer used to perform the ping is not connected to the internet.

Using the Internet WHOIS System

An alternative method for finding website IP addresses relies on the internet WHOIS system. WHOIS is a database that tracks website registration information including owners and IP addresses.

WHOIS lookup screenshot

WHOIS doesn't always provide IP address information. You may want to try a different method first.

To look up website IP addresses with WHOIS, simply visit one of the many public sites such as whois.net or networksolutions.com that offer WHOIS database query services. Searching for a particular site name produces a result similar to the following:

Current Registrar: REGISTER.COM, INC.
IP Address: 207.241.148.80 (ARIN & RIPE IP search) . . .

In the WHOIS method, the IP addresses are stored statically in a database and therefore do not require the website to be online or reachable over the internet.

WhatsMyIPAddress.com

WhatsMyIPAddress IP search screenshot

WhatsMyIPAddress.com is a popular website that lets you look up your own public IP address. It also has a simple tool to let you look up the IP of websites. Open your browser and visit the site. Searching is as simple as entering the name of the site you want into the search field and running the search. You'll instantly see the IP addresses used by your site.

Using IP Address Lists

Popular websites publish their IP address information, which is accessible through standard web searches — so if you are looking for an IP address for Facebook, for example, you can find it online with a simple search.

Was this page helpful?

  • How to Find Out Who Owns an IP Address

    Wondering Who Owns an IP Address? Here's How to Find Out

  • Youtube Homepage

    How Do You Open YouTube With an IP Address?

  • Terminal app on macOS showing ping command and arp command

    Is There Any Way to Find a MAC Address by Using an IP Address?

  • Find the IP Address for a Domain or the Domain Name of an IP Address

  • Computer Hacker Using Glass Tablet Seen Through Interface

    Follow This Three Step Method to Ping Computers and Websites

  • communications satellite in orbit around Earth

    Does IP Address Location (Geolocation) Really Work?

  • Network connections

    What Is an IP Address & What Are the Different Kinds of IP Addresses?

  • A Brief Overview of DNS Caching and How It Makes the Internet Better

  • Hands typing on keyboard of a laptop

    DNS Blacklists Are Effective Tools in the War Against Spam

  • An illustration of a laptop with the address 192.168.1.100 on a network.

    What Does the 192.168.1.100 IP Address Mean?

  • Man pressing a lock icon in superimposed network illustration

    What is the Significance of the IP Address 192.168.0.100?

  • White router on tabletop in a home

    Need to Know Your IP Address? Here's How to Find It

  • A man in a server rack

    Check Whether You Can Connect to a Network With the Ping Command

  • Man looking at wall of servers

    MAC Addresses Can Be Turned Into IP Addresses (Sort Of)

  • Teenage girl using laptop on bed

    A Beginner's Guide to Domain Name Systems (DNS)

  • An injet printer connected to a network

    Your Printer Has Its Own IP Address

Dipanjan (DJ) Sarkar at Medium

Medium

AI for Social Good Series— Part 1

Detecting Malaria with Deep Learning

Scalable Log Analytics with Apache Spark — A Comprehensive Case-Study

Data Analysis & Visualization at…

Build your own Robust Deep Learning Environment in Minutes

A guide to the less desirable aspects of…

Get Smarter with Data Science — Tackling Real Enterprise Challenges

Take your Data Science Projects…

Explainable Artificial Intelligence (Part 3)

Hands-on Machine Learning Model Interpretation

Deep Transfer Learning for Natural Language Processing — Text Classification with Universal Embeddings

A Comprehensive Hands-on Guide to Transfer Learning with Real-World Applications in Deep Learning

Explainable Artificial Intelligence (Part 2)

Model Interpretation Strategies

SQL at Scale with Apache Spark SQL and DataFrames — Concepts, Architecture and Examples

A Comprehensive Guide to the Grammar of Graphics for Effective Visualization of Multi-dimensional Data

Learn effective strategies for leveraging a layered Grammar of Graphics framework for effective data…

Interactive Data Visualization with D3.js

Unlock the pathway to data-driven visualizations

The content for this post has been contributed by Quincy Smith, who is a part of the marketing team at Springboard an online training company seeking…

A Practitioner's Guide to Natural Language Processing (Part I) — Processing & Understanding Text

Proven and tested hands-on strategies to tackle NLP tasks

Explainable Artificial Intelligence (Part 1)

The Importance of Human Interpretable Machine Learning

Understanding Feature Engineering (Part 4)

A hands-on intuitive approach to Deep Learning Methods for Text Data — Word2Vec, GloVe and FastText

Newer, advanced strategies for taming…

Understanding Feature Engineering (Part 3)

Traditional Methods for Text Data

Traditional strategies for taming unstructured, textual data

The Art of Effective Visualization of Multi-dimensional Data

Strategies for Effective Data Visualization

Understanding Feature Engineering (Part 2)

Categorical Data

Strategies for working with…

Understanding Feature Engineering (Part 1)

Continuous Numeric Data

Strategies for…

Jaime Zornoza at Medium

Medium

Probability Learning I : Bayes’ Theorem

Learn about one of the fundamental theorems of probability…

Targeted Sentiment analysis vs Traditional Sentiment analysis

Deep Learning for NLP: Creating a Chatbot with Keras!

Learn how to use Keras to build a Recurrent…

Deep Learning for NLP: ANNs, RNNs and LSTMs explained!

Learn about Artificial Neural Networks, Deep…

Visualisation of Information from Raw Twitter Data — Part 2

Want to find out user activity, see if certain users are Bots, make a Time Series of the Tweet Publications and much more? Read on then!

Visualisation of Information from Raw Twitter Data — Part 1

Lets explore what kind of information we can easily retrieve from raw Twitter data!

Downloading Data From Twitter Using the REST API

A Practitioner's Guide to Natural Language Processing (Part I) — Processing & Understanding Text

Medium

Introduction

Unstructured data, especially text, images and videos contain a wealth of information. However, due to the inherent complexity in processing and analyzing this data, people often refrain from spending extra time and effort in venturing out from structured datasets to analyze these unstructured sources of data, which can be a potential gold mine.

 

Natural Language Processing (NLP) is all about leveraging tools, techniques and algorithms to process and understand natural language-based data, which is usually unstructured like text, speech and so on. In this series of articles, we will be looking at tried and tested strategies, techniques and workflows which can be leveraged by practitioners and data scientists to extract useful insights from text data. We will also cover some useful and interesting use-cases for NLP. This article will be all about processing and understanding text data with tutorials and hands-on examples.

Outline for this Series

The nature of this series will be a mix of theoretical concepts but with a focus on hands-on techniques and strategies covering a wide variety of NLP problems. Some of the major areas that we will be covering in this series of articles include the following.

  1. Processing & Understanding Text
  2. Feature Engineering & Text Representation
  3. Supervised Learning Models for Text Data
  4. Unsupervised Learning Models for Text Data
  5. Advanced Topics

Feel free to suggest more ideas as this series progresses, and I will be glad to cover something I might have missed out on. A lot of these articles will showcase tips and strategies which have worked well in real-world scenarios.

What this article covers

This article will be covering the following aspects of NLP in detail with hands-on examples.

  1. Data Retrieval with Web Scraping
  2. Text wrangling and pre-processing
  3. Parts of Speech Tagging
  4. Shallow Parsing
  5. Constituency and Dependency Parsing
  6. Named Entity Recognition
  7. Emotion and Sentiment Analysis

This should give you a good idea of how to get started with analyzing syntax and semantics in text corpora.

Motivation

Formally, NLP is a specialized field of computer science and artificial intelligence with roots in computational linguistics. It is primarily concerned with designing and building applications and systems that enable interaction between machines and natural languages that have been evolved for use by humans. Hence, often it is perceived as a niche area to work on. And people usually tend to focus more on machine learning or statistical learning.

 

When I started delving into the world of data science, even I was overwhelmed by the challenges in analyzing and modeling on text data. However, after working as a Data Scientist on several challenging problems around NLP over the years, I’ve noticed certain interesting aspects, including techniques, strategies and workflows which can be leveraged to solve a wide variety of problems. I have covered several topics around NLP in my books “Text Analytics with Python” (I’m writing a revised version of this soon) and “Practical Machine Learning with Python”.

However, based on all the excellent feedback I’ve received from all my readers (yes all you amazing people out there!), the main objective and motivation in creating this series of articles is to share my learnings with more people, who can’t always find time to sit and read through a book and can even refer to these articles on the go! Thus, there is no pre-requisite to buy any of these books to learn NLP.

Getting Started

When building the content and examples for this article, I was thinking if I should focus on a toy dataset to explain things better, or focus on an existing dataset from one of the main sources for data science datasets. Then I thought, why not build an end-to-end tutorial, where we scrape the web to get some text data and showcase examples based on that!

The source data which we will be working on will be news articles, which we have retrieved from inshortsa website that gives us short, 60-word news articles on a wide variety of topics, and they even have an app for it!

Inshorts, news in 60 words !
Edit description
inshorts.com

In this article, we will be working with text data from news articles on technology, sports and world news. I will be covering some basics on how to scrape and retrieve these news articles from their website in the next section.

Standard NLP Workflow

I am assuming you are aware of the CRISP-DM model, which is typically an industry standard for executing any data science project. Typically, any NLP-based problem can be solved by a methodical workflow that has a sequence of steps. The major steps are depicted in the following figure.

 
A high-level standard workflow for any NLP project

We usually start with a corpus of text documents and follow standard processes of text wrangling and pre-processing, parsing and basic exploratory data analysis. Based on the initial insights, we usually represent the text using relevant feature engineering techniques. Depending on the problem at hand, we either focus on building predictive supervised models or unsupervised models, which usually focus more on pattern mining and grouping. Finally, we evaluate the model and the overall success criteria with relevant stakeholders or customers, and deploy the final model for future usage.

Scraping News Articles for Data Retrieval

We will be scraping inshorts, the website, by leveraging python to retrieve news articles. We will be focusing on articles on technology, sports and world affairs. We will retrieve one page’s worth of articles for each category. A typical news category landing page is depicted in the following figure, which also highlights the HTML section for the textual content of each article.

 
The landing page for technology news articles and its corresponding HTML structure

Thus, we can see the specific HTML tags which contain the textual content of each news article in the landing page mentioned above. We will be using this information to extract news articles by leveraging the BeautifulSoup and requests libraries. Let’s first load up the following dependencies.

import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import os
%matplotlib inline

We will now build a function which will leverage requests to access and get the HTML content from the landing pages of each of the three news categories. Then, we will use BeautifulSoup to parse and extract the news headline and article textual content for all the news articles in each category. We find the content by accessing the specific HTML tags and classes, where they are present (a sample of which I depicted in the previous figure).

  npl_strategy_1

It is pretty clear that we extract the news headline, article text and category and build out a data frame, where each row corresponds to a specific news article. We will now invoke this function and build our dataset.

news_df = build_dataset(seed_urls)
news_df.head(10)
 
Our news dataset

We, now, have a neatly formatted dataset of news articles and you can quickly check the total number of news articles with the following code.

news_df.news_category.value_counts()Output:
-------
world 25
sports 25
technology 24
Name: news_category, dtype: int64

Text Wrangling & Pre-processing

There are usually multiple steps involved in cleaning and pre-processing textual data. I have covered text pre-processing in detail in Chapter 3 of ‘Text Analytics with Python’ (code is open-sourced). However, in this section, I will highlight some of the most important steps which are used heavily in Natural Language Processing (NLP) pipelines and I frequently use them in my NLP projects. We will be leveraging a fair bit of nltk and spacy, both state-of-the-art libraries in NLP. Typically a pip install <library> or a conda install <library> should suffice. However, in case you face issues with loading up spacy’s language models, feel free to follow the steps highlighted below to resolve this issue (I had faced this issue in one of my systems).

# OPTIONAL: ONLY USE IF SPACY FAILS TO LOAD LANGUAGE MODEL
# Use the following command to install spaCy
> pip install -U spacy
OR> conda install -c conda-forge spacy# Download the following language model and store it in disk
https://github.com/explosion/spacy-models/releases/tag/en_core_web_md-2.0.0
# Link the same to spacy
> python -m spacy link ./spacymodels/en_core_web_md-2.0.0/en_core_web_md en_core
Linking successful
./spacymodels/en_core_web_md-2.0.0/en_core_web_md --> ./Anaconda3/lib/site-packages/spacy/data/en_core
You can now load the model via spacy.load('en_core')

Let’s now load up the necessary dependencies for text pre-processing. We will remove negation words from stop words, since we would want to keep them as they might be useful, especially during sentiment analysis.

 IMPORTANT NOTE: A lot of you have messaged me about not being able to load the contractions module. It’s not a standard python module. We leverage a standard set of contractions available in the contractions.py file in my repository.Please add it in the same directory you run your code from, else it will not work.

import spacy
import pandas as pd
import numpy as np
import nltk
from nltk.tokenize.toktok import ToktokTokenizer
import re
from bs4 import BeautifulSoup
from contractions import CONTRACTION_MAP
import unicodedata
nlp = spacy.load('en_core', parse=True, tag=True, entity=True)
#nlp_vec = spacy.load('en_vecs', parse = True, tag=True, #entity=True)
tokenizer = ToktokTokenizer()
stopword_list = nltk.corpus.stopwords.words('english')
stopword_list.remove('no')
stopword_list.remove('not')

Removing HTML tags

Often, unstructured text contains a lot of noise, especially if you use techniques like web or screen scraping. HTML tags are typically one of these components which don’t add much value towards understanding and analyzing text.

nlp_strategy_2
'Some important text'

It is quite evident from the above output that we can remove unnecessary HTML tags and retain the useful textual information from any document.

Removing accented characters

Usually in any text corpus, you might be dealing with accented characters/letters, especially if you only want to analyze the English language. Hence, we need to make sure that these characters are converted and standardized into ASCII characters. A simple example — converting éto e.

nlp s 3
'Some Accented text'

The preceding function shows us how we can easily convert accented characters to normal English characters, which helps standardize the words in our corpus.

Expanding Contractions

Contractions are shortened version of words or syllables. They often exist in either written or spoken forms in the English language. These shortened versions or contractions of words are created by removing specific letters and sounds. In case of English contractions, they are often created by removing one of the vowels from the word. Examples would be, do not to don’t and I would to I’d. Converting each contraction to its expanded, original form helps with text standardization.

We leverage a standard set of contractions available in the contractions.pyfile in my repository.

nlp s 4
'You all cannot expand contractions I would think'

We can see how our function helps expand the contractions from the preceding output. Are there better ways of doing this? Definitely! If we have enough examples, we can even train a deep learning model for better performance.

Removing Special Characters

Special characters and symbols are usually non-alphanumeric characters or even occasionally numeric characters (depending on the problem), which add to the extra noise in unstructured text. Usually, simple regular expressions (regexes) can be used to remove them.

nlp s 5
'Well this was fun What do you think '

I’ve kept removing digits as optional, because often we might need to keep them in the pre-processed text.

Stemming

To understand stemming, you need to gain some perspective on what word stems represent. Word stems are also known as the base form of a word, and we can create new words by attaching affixes to them in a process known as inflection. Consider the word JUMP. You can add affixes to it and form new words like JUMPSJUMPED, and JUMPING. In this case, the base word JUMP is the word stem.

 
Word stem and its inflections (Source: Text Analytics with Python, Apress/Springer 2016)

The figure shows how the word stem is present in all its inflections, since it forms the base on which each inflection is built upon using affixes. The reverse process of obtaining the base form of a word from its inflected form is known as stemming. Stemming helps us in standardizing words to their base or root stem, irrespective of their inflections, which helps many applications like classifying or clustering text, and even in information retrieval. Let’s see the popular Porter stemmer in action now!

nlp s 6
'My system keep crash hi crash yesterday, our crash daili'

The Porter stemmer is based on the algorithm developed by its inventor, Dr. Martin Porter. Originally, the algorithm is said to have had a total of five different phases for reduction of inflections to their stems, where each phase has its own set of rules.

Do note that usually stemming has a fixed set of rules, hence, the root stems may not be lexicographically correct. Which means, the stemmed words may not be semantically correct, and might have a chance of not being present in the dictionary (as evident from the preceding output).

Lemmatization

Lemmatization is very similar to stemming, where we remove word affixes to get to the base form of a word. However, the base form in this case is known as the root word, but not the root stem. The difference being that the root word is always a lexicographically correct word (present in the dictionary), but the root stem may not be so. Thus, root word, also known as the lemma, will always be present in the dictionary. Both nltk and spacyhave excellent lemmatizers. We will be using spacy here.

nlp s 7
'My system keep crash ! his crash yesterday , ours crash daily'

You can see that the semantics of the words are not affected by this, yet our text is still standardized.

Do note that the lemmatization process is considerably slower than stemming, because an additional step is involved where the root form or lemma is formed by removing the affix from the word if and only if the lemma is present in the dictionary.

Removing Stopwords

Words which have little or no significance, especially when constructing meaningful features from text, are known as stopwords or stop words. These are usually words that end up having the maximum frequency if you do a simple term or word frequency in a corpus. Typically, these can be articles, conjunctions, prepositions and so on. Some examples of stopwords are aantheand the like.

nlp s 8
', , stopwords , computer not'

There is no universal stopword list, but we use a standard English language stopwords list from nltk. You can also add your own domain-specific stopwords as needed.

Bringing it all together — Building a Text Normalizer

While we can definitely keep going with more techniques like correcting spelling, grammar and so on, let’s now bring everything we learnt together and chain these operations to build a text normalizer to pre-process text data.

nlp s 9

Let’s now put this function in action! We will first combine the news headline and the news article text together to form a document for each piece of news. Then, we will pre-process them.

nlp s 10
 
{'clean_text': 'us unveils world powerful supercomputer beat china us unveil world powerful supercomputer call summit beat previous record holder china sunway taihulight peak performance trillion calculation per second twice fast sunway taihulight capable trillion calculation per second summit server reportedly take size two tennis court', 'full_text': "US unveils world's most powerful supercomputer, beats China. The US has unveiled the world's most powerful supercomputer called 'Summit', beating the previous record-holder China's Sunway TaihuLight. With a peak performance of 200,000 trillion calculations per second, it is over twice as fast as Sunway TaihuLight, which is capable of 93,000 trillion calculations per second. Summit has 4,608 servers, which reportedly take up the size of two tennis courts."}

Thus, you can see how our text pre-processor helps in pre-processing our news articles! After this, you can save this dataset to disk if needed, so that you can always load it up later for future analysis.

news_df.to_csv('news.csv', index=False, encoding='utf-8')

Understanding Language Syntax and Structure

For any language, syntax and structure usually go hand in hand, where a set of specific rules, conventions, and principles govern the way words are combined into phrases; phrases get combines into clauses; and clauses get combined into sentences. We will be talking specifically about the English language syntax and structure in this section. In English, words usually combine together to form other constituent units. These constituents include words, phrases, clauses, and sentences. Considering a sentence,“The brown fox is quick and he is jumping over the lazy dog”, it is made of a bunch of words and just looking at the words by themselves don’t tell us much.

 
A bunch of unordered words don’t convey much information

Knowledge about the structure and syntax of language is helpful in many areas like text processing, annotation, and parsing for further operations such as text classification or summarization. Typical parsing techniques for understanding text syntax are mentioned below.

  • Parts of Speech (POS) Tagging
  • Shallow Parsing or Chunking
  • Constituency Parsing
  • Dependency Parsing

We will be looking at all of these techniques in subsequent sections. Considering our previous example sentence “The brown fox is quick and he is jumping over the lazy dog”, if we were to annotate it using basic POS tags, it would look like the following figure.

 
POS tagging for a sentence

Thus, a sentence typically follows a hierarchical structure consisting the following components,

sentence → clauses → phrases → words

Tagging Parts of Speech

Parts of speech (POS) are specific lexical categories to which words are assigned, based on their syntactic context and role. Usually, words can fall into one of the following major categories.

  • N(oun): This usually denotes words that depict some object or entity, which may be living or nonliving. Some examples would be fox , dog , book , and so on. The POS tag symbol for nouns is N.
  • V(erb): Verbs are words that are used to describe certain actions, states, or occurrences. There are a wide variety of further subcategories, such as auxiliary, reflexive, and transitive verbs (and many more). Some typical examples of verbs would be running , jumping , read , and write . The POS tag symbol for verbs is V.
  • Adj(ective): Adjectives are words used to describe or qualify other words, typically nouns and noun phrases. The phrase beautiful flower has the noun (N) flower which is described or qualified using the adjective (ADJ) beautiful . The POS tag symbol for adjectives is ADJ .
  • Adv(erb): Adverbs usually act as modifiers for other words including nouns, adjectives, verbs, or other adverbs. The phrase very beautiful flower has the adverb (ADV) very , which modifies the adjective (ADJ) beautiful , indicating the degree to which the flower is beautiful. The POS tag symbol for adverbs is ADV.

Besides these four major categories of parts of speech , there are other categories that occur frequently in the English language. These include pronouns, prepositions, interjections, conjunctions, determiners, and many others. Furthermore, each POS tag like the noun (N) can be further subdivided into categories like singular nouns (NN), singular proper nouns(NNP), and plural nouns (NNS).

The process of classifying and labeling POS tags for words called parts of speech tagging or POS tagging . POS tags are used to annotate words and depict their POS, which is really helpful to perform specific analysis, such as narrowing down upon nouns and seeing which ones are the most prominent, word sense disambiguation, and grammar analysis. We will be leveraging both nltk and spacy which usually use the Penn Treebank notation for POS tagging.

 
nlp s 10
POS tagging a news headline

We can see that each of these libraries treat tokens in their own way and assign specific tags for them. Based on what we see, spacy seems to be doing slightly better than nltk.

Shallow Parsing or Chunking

Based on the hierarchy we depicted earlier, groups of words make up phrases. There are five major categories of phrases:

  • Noun phrase (NP): These are phrases where a noun acts as the head word. Noun phrases act as a subject or object to a verb.
  • Verb phrase (VP): These phrases are lexical units that have a verb acting as the head word. Usually, there are two forms of verb phrases. One form has the verb components as well as other entities such as nouns, adjectives, or adverbs as parts of the object.
  • Adjective phrase (ADJP): These are phrases with an adjective as the head word. Their main role is to describe or qualify nouns and pronouns in a sentence, and they will be either placed before or after the noun or pronoun.
  • Adverb phrase (ADVP): These phrases act like adverbs since the adverb acts as the head word in the phrase. Adverb phrases are used as modifiers for nouns, verbs, or adverbs themselves by providing further details that describe or qualify them.
  • Prepositional phrase (PP): These phrases usually contain a preposition as the head word and other lexical components like nouns, pronouns, and so on. These act like an adjective or adverb describing other words or phrases.

Shallow parsing, also known as light parsing or chunking , is a popular natural language processing technique of analyzing the structure of a sentence to break it down into its smallest constituents (which are tokens such as words) and group them together into higher-level phrases. This includes POS tags as well as phrases from a sentence.

 
An example of shallow parsing depicting higher level phrase annotations

We will leverage the conll2000 corpus for training our shallow parser model. This corpus is available in nltk with chunk annotations and we will be using around 10K records for training our model. A sample annotated sentence is depicted as follows.

nlp s 11
10900 48
(S
Chancellor/NNP
(PP of/IN)
(NP the/DT Exchequer/NNP)
(NP Nigel/NNP Lawson/NNP)
(NP 's/POS restated/VBN commitment/NN)
(PP to/TO)
(NP a/DT firm/NN monetary/JJ policy/NN)
(VP has/VBZ helped/VBN to/TO prevent/VB)
(NP a/DT freefall/NN)
(PP in/IN)
(NP sterling/NN)
(PP over/IN)
(NP the/DT past/JJ week/NN)
./.)

From the preceding output, you can see that our data points are sentences that are already annotated with phrases and POS tags metadata that will be useful in training our shallow parser model. We will leverage two chunking utility functions, tree2conlltags , to get triples of word, tag, and chunk tags for each token, and conlltags2tree to generate a parse tree from these token triples. We will be using these functions to train our parser. A sample is depicted below.

nlp s 12
[('Chancellor', 'NNP', 'O'),
('of', 'IN', 'B-PP'),
('the', 'DT', 'B-NP'),
('Exchequer', 'NNP', 'I-NP'),
('Nigel', 'NNP', 'B-NP'),
('Lawson', 'NNP', 'I-NP'),
("'s", 'POS', 'B-NP'),
('restated', 'VBN', 'I-NP'),
('commitment', 'NN', 'I-NP'),
('to', 'TO', 'B-PP'),
('a', 'DT', 'B-NP'),
('firm', 'NN', 'I-NP'),
('monetary', 'JJ', 'I-NP'),
('policy', 'NN', 'I-NP'),
('has', 'VBZ', 'B-VP'),
('helped', 'VBN', 'I-VP'),
('to', 'TO', 'I-VP'),
('prevent', 'VB', 'I-VP'),
('a', 'DT', 'B-NP'),
('freefall', 'NN', 'I-NP'),
('in', 'IN', 'B-PP'),
('sterling', 'NN', 'B-NP'),
('over', 'IN', 'B-PP'),
('the', 'DT', 'B-NP'),
('past', 'JJ', 'I-NP'),
('week', 'NN', 'I-NP'),
('.', '.', 'O')]

The chunk tags use the IOB format. This notation represents Inside, Outside, and Beginning. The B- prefix before a tag indicates it is the beginning of a chunk, and I- prefix indicates that it is inside a chunk. The O tag indicates that the token does not belong to any chunk. The B- tag is always used when there are subsequent tags of the same type following it without the presence of O tags between them.

We will now define a function conll_tag_ chunks() to extract POS and chunk tags from sentences with chunked annotations and a function called combined_taggers() to train multiple taggers with backoff taggers (e.g. unigram and bigram taggers)

nlp s 1

We will now define a class NGramTagChunker that will take in tagged sentences as training input, get their (word, POS tag, Chunk tag) WTC triples, and train a BigramTagger with a UnigramTagger as the backoff tagger. We will also define a parse() function to perform shallow parsing on new sentences

The UnigramTagger , BigramTagger , and TrigramTagger are classes that inherit from the base class NGramTagger , which itself inherits from the ContextTaggerclass , which inherits from the SequentialBackoffTagger class .

We will use this class to train on the conll2000 chunked train_data and evaluate the model performance on the test_data

nlp s 14
ChunkParse score:
IOB Accuracy: 90.0%%
Precision: 82.1%%
Recall: 86.3%%
F-Measure: 84.1%%

Our chunking model gets an accuracy of around 90% which is quite good! Let’s now leverage this model to shallow parse and chunk our sample news article headline which we used earlier, “US unveils world’s most powerful supercomputer, beats China”.

chunk_tree = ntc.parse(nltk_pos_tagged)
print(chunk_tree)
Output:
-------
(S
(NP US/NNP)
(VP unveils/VBZ world's/VBZ)
(NP most/RBS powerful/JJ supercomputer,/JJ beats/NNS China/NNP))

Thus you can see it has identified two noun phrases (NP) and one verb phrase (VP) in the news article. Each word’s POS tags are also visible. We can also visualize this in the form of a tree as follows. You might need to install ghostscript in case nltk throws an error.

nlp s 15
 
Shallow parsed news headline

The preceding output gives a good sense of structure after shallow parsing the news headline.

Constituency Parsing

Constituent-based grammars are used to analyze and determine the constituents of a sentence. These grammars can be used to model or represent the internal structure of sentences in terms of a hierarchically ordered structure of their constituents. Each and every word usually belongs to a specific lexical category in the case and forms the head word of different phrases. These phrases are formed based on rules called phrase structure rules.

Phrase structure rules form the core of constituency grammars, because they talk about syntax and rules that govern the hierarchy and ordering of the various constituents in the sentences. These rules cater to two things primarily.

  • They determine what words are used to construct the phrases or constituents.
  • They determine how we need to order these constituents together.

The generic representation of a phrase structure rule is S → AB , which depicts that the structure consists of constituents and , and the ordering is followed by . While there are several rules (refer to Chapter 1, Page 19: Text Analytics with Python, if you want to dive deeper), the most important rule describes how to divide a sentence or a clause. The phrase structure rule denotes a binary division for a sentence or a clause as S → NP VP where is the sentence or clause, and it is divided into the subject, denoted by the noun phrase (NP) and the predicate, denoted by the verb phrase (VP).

A constituency parser can be built based on such grammars/rules, which are usually collectively available as context-free grammar (CFG) or phrase-structured grammar. The parser will process input sentences according to these rules, and help in building a parse tree.

 
An example of constituency parsing showing a nested hierarchical structure

We will be using nltk and the StanfordParser here to generate parse trees.

Prerequisites: Download the official Stanford Parser from here, which seems to work quite well. You can try out a later version by going to this website and checking the Release History section. After downloading, unzip it to a known location in your filesystem. Once done, you are now ready to use the parser from nltk , which we will be exploring soon.

The Stanford parser generally uses a PCFG (probabilistic context-free grammar) parser. A PCFG is a context-free grammar that associates a probability with each of its production rules. The probability of a parse tree generated from a PCFG is simply the production of the individual probabilities of the productions used to generate it.

nlp s 16
(ROOT
(SINV
(S
(NP (NNP US))
(VP
(VBZ unveils)
(NP
(NP (NN world) (POS 's))
(ADJP (RBS most) (JJ powerful))
(NN supercomputer))))
(, ,)
(VP (VBZ beats))
(NP (NNP China))))

We can see the constituency parse tree for our news headline. Let’s visualize it to understand the structure better.

from IPython.display import display
display(result[0])
 
Constituency parsed news headline

We can see the nested hierarchical structure of the constituents in the preceding output as compared to the flat structure in shallow parsing. In case you are wondering what SINV means, it represents an Inverted declarative sentence, i.e. one in which the subject follows the tensed verb or modal. Refer to the Penn Treebank reference as needed to lookup other tags.

Dependency Parsing

In dependency parsing, we try to use dependency-based grammars to analyze and infer both structure and semantic dependencies and relationships between tokens in a sentence. The basic principle behind a dependency grammar is that in any sentence in the language, all words except one, have some relationship or dependency on other words in the sentence. The word that has no dependency is called the root of the sentence. The verb is taken as the root of the sentence in most cases. All the other words are directly or indirectly linked to the root verb using links , which are the dependencies.

Considering our sentence “The brown fox is quick and he is jumping over the lazy dog” , if we wanted to draw the dependency syntax tree for this, we would have the structure

 
A dependency parse tree for a sentence

These dependency relationships each have their own meaning and are a part of a list of universal dependency types. This is discussed in an original paper, Universal Stanford Dependencies: A Cross-Linguistic Typology by de Marneffe et al, 2014). You can check out the exhaustive list of dependency types and their meanings here.

If we observe some of these dependencies, it is not too hard to understand them.

  • The dependency tag det is pretty intuitive — it denotes the determiner relationship between a nominal head and the determiner. Usually, the word with POS tag DET will also have the det dependency tag relation. Examples include fox → the and dog → the.
  • The dependency tag amod stands for adjectival modifier and stands for any adjective that modifies the meaning of a noun. Examples include fox → brown and dog → lazy.
  • The dependency tag nsubj stands for an entity that acts as a subject or agent in a clause. Examples include is → fox and jumping → he.
  • The dependencies cc and conj have more to do with linkages related to words connected by coordinating conjunctions . Examples include is → and and is → jumping.
  • The dependency tag aux indicates the auxiliary or secondary verb in the clause. Example: jumping → is.
  • The dependency tag acomp stands for adjective complement and acts as the complement or object to a verb in the sentence. Example: is → quick
  • The dependency tag prep denotes a prepositional modifier, which usually modifies the meaning of a noun, verb, adjective, or preposition. Usually, this representation is used for prepositions having a noun or noun phrase complement. Example: jumping → over.
  • The dependency tag pobj is used to denote the object of a preposition . This is usually the head of a noun phrase following a preposition in the sentence. Example: over → dog.

Spacy had two types of English dependency parsers based on what language models you use, you can find more details here. Based on language models, you can use the Universal Dependencies Scheme or the CLEAR Style Dependency Scheme also available in NLP4J now. We will now leverage spacy and print out the dependencies for each token in our news headline.

nlp s 17
[]<---US[compound]--->[]
--------
['US']<---unveils[nsubj]--->['supercomputer', ',']
--------
[]<---world[poss]--->["'s"]
--------
[]<---'s[case]--->[]
--------
[]<---most[amod]--->[]
--------
[]<---powerful[compound]--->[]
--------
['world', 'most', 'powerful']<---supercomputer[appos]--->[]
--------
[]<---,[punct]--->[]
--------
['unveils']<---beats[ROOT]--->['China']
--------
[]<---China[dobj]--->[]
--------

It is evident that the verb beats is the ROOT since it doesn’t have any other dependencies as compared to the other tokens. For knowing more about each annotation you can always refer to the CLEAR dependency scheme. We can also visualize the above dependencies in a better way.

nlp s 18
 
News Headline dependency tree from SpaCy

You can also leverage nltk and the StanfordDependencyParser to visualize and build out the dependency tree. We showcase the dependency tree both in its raw and annotated form as follows.

nlp s 19
(beats (unveils US (supercomputer (world 's) (powerful most)))  
China)
 
Dependency Tree visualizations using nltk’s Stanford dependency parser

You can notice the similarities with the tree we had obtained earlier. The annotations help with understanding the type of dependency among the different tokens.

Named Entity Recognition

In any text document, there are particular terms that represent specific entities that are more informative and have a unique context. These entities are known as named entities , which more specifically refer to terms that represent real-world objects like people, places, organizations, and so on, which are often denoted by proper names. A naive approach could be to find these by looking at the noun phrases in text documents. Named entity recognition (NER) , also known as entity chunking/extraction , is a popular technique used in information extraction to identify and segment the named entities and classify or categorize them under various predefined classes.

SpaCy has some excellent capabilities for named entity recognition. Let’s try and use it on one of our sample news articles.

nlp s 20
[(US, 'GPE'), (China, 'GPE'), (US, 'GPE'), (China, 'GPE'), 
(Sunway, 'ORG'), (TaihuLight, 'ORG'), (200,000, 'CARDINAL'),
(second, 'ORDINAL'), (Sunway, 'ORG'), (TaihuLight, 'ORG'),
(93,000, 'CARDINAL'), (4,608, 'CARDINAL'), (two, 'CARDINAL')]
 
Visualizing named entities in a news article with spaCy

We can clearly see that the major named entities have been identified by spacy. To understand more in detail about what each named entity means, you can refer to the documentation or check out the following table for convenience.

 
Named entity types

Let’s now find out the most frequent named entities in our news corpus! For this, we will build out a data frame of all the named entities and their types using the following code.

nlp s 21

We can now transform and aggregate this data frame to find the top occuring entities and types.

nlp s 22
 
Top named entities and types in our news corpus

Do you notice anything interesting? (Hint: Maybe the supposed summit between Trump and Kim Jong!). We also see that it has correctly identified ‘Messenger’ as a product (from Facebook).

We can also group by the entity types to get a sense of what types of entites occur most in our news corpus.

nlp s 23
 
Top named entity types in our news corpus

We can see that people, places and organizations are the most mentioned entities though interestingly we also have many other entities.

Another nice NER tagger is the StanfordNERTagger available from the nltkinterface. For this, you need to have Java installed and then download the Stanford NER resources. Unzip them to a location of your choice (I used E:/stanford in my system).

Stanford’s Named Entity Recognizer is based on an implementation of linear chain Conditional Random Field (CRF) sequence models. Unfortunately this model is only trained on instances of PERSONORGANIZATION and LOCATION types. Following code can be used as a standard workflow which helps us extract the named entities using this tagger and show the top named entities and their types (extraction differs slightly from spacy).

nlp s24
 
Top named entities and types from Stanford NER on our news corpus

We notice quite similar results though restricted to only three types of named entities. Interestingly, we see a number of mentioned of several people in various sports.

Emotion and Sentiment Analysis

Sentiment analysis is perhaps one of the most popular applications of NLP, with a vast number of tutorials, courses, and applications that focus on analyzing sentiments of diverse datasets ranging from corporate surveys to movie reviews. The key aspect of sentiment analysis is to analyze a body of text for understanding the opinion expressed by it. Typically, we quantify this sentiment with a positive or negative value, called polarity. The overall sentiment is often inferred as positiveneutral or negative from the sign of the polarity score.

Usually, sentiment analysis works best on text that has a subjective context than on text with only an objective context. Objective text usually depicts some normal statements or facts without expressing any emotion, feelings, or mood. Subjective text contains text that is usually expressed by a human having typical moods, emotions, and feelings. Sentiment analysis is widely used, especially as a part of social media analysis for any domain, be it a business, a recent movie, or a product launch, to understand its reception by the people and what they think of it based on their opinions or, you guessed it, sentiment!

 

Typically, sentiment analysis for text data can be computed on several levels, including on an individual sentence level, paragraph level, or the entire document as a whole. Often, sentiment is computed on the document as a whole or some aggregations are done after computing the sentiment for individual sentences. There are two major approaches to sentiment analysis.

  • Supervised machine learning or deep learning approaches
  • Unsupervised lexicon-based approaches

For the first approach we typically need pre-labeled data. Hence, we will be focusing on the second approach. For a comprehensive coverage of sentiment analysis, refer to Chapter 7: Analyzing Movie Reviews SentimentPractical Machine Learning with Python, Springer\Apress, 2018. In this scenario, we do not have the convenience of a well-labeled training dataset. Hence, we will need to use unsupervised techniques for predicting the sentiment by using knowledgebases, ontologies, databases, and lexicons that have detailed information, specially curated and prepared just for sentiment analysis. A lexicon is a dictionary, vocabulary, or a book of words. In our case, lexicons are special dictionaries or vocabularies that have been created for analyzing sentiments. Most of these lexicons have a list of positive and negative polar words with some score associated with them, and using various techniques like the position of words, surrounding words, context, parts of speech, phrases, and so on, scores are assigned to the text documents for which we want to compute the sentiment. After aggregating these scores, we get the final sentiment.

 

Various popular lexicons are used for sentiment analysis, including the following.

This is not an exhaustive list of lexicons that can be leveraged for sentiment analysis, and there are several other lexicons which can be easily obtained from the Internet. Feel free to check out each of these links and explore them. We will be covering two techniques in this section.

Sentiment Analysis with AFINN Lexicon

The AFINN lexicon is perhaps one of the simplest and most popular lexicons that can be used extensively for sentiment analysis. Developed and curated by Finn Årup Nielsen, you can find more details on this lexicon in the paper, “A new ANEW: evaluation of a word list for sentiment analysis in microblogs”, proceedings of the ESWC 2011 Workshop. The current version of the lexicon is AFINN-en-165. txt and it contains over 3,300+ words with a polarity score associated with each word. You can find this lexicon at the author’s official GitHub repository along with previous versions of it, includingAFINN-111. The author has also created a nice wrapper library on top of this in Python called afinn, which we will be using for our analysis.

The following code computes sentiment for all our news articles and shows summary statistics of general sentiment per news category.

 
nlp s 25

We can get a good idea of general sentiment statistics across different news categories. Looks like the average sentiment is very positive in sports and reasonably negative in technology! Let’s look at some visualizations now.

nlp s 26
 
Visualizing news sentiment polarity

We can see that the spread of sentiment polarity is much higher in sportsand world as compared to technology where a lot of the articles seem to be having a negative polarity. We can also visualize the frequency of sentiment labels.

nlp s 27
 
Visualizing sentiment categories per news category

No surprises here that technology has the most number of negative articles and world the most number of positive articles. Sports might have more neutral articles due to the presence of articles which are more objective in nature (talking about sporting events without the presence of any emotion or feelings). Let’s dive deeper into the most positive and negative sentiment news articles for technology news.

nlp s 28
 

Looks like the most negative article is all about a recent smartphone scam in India and the most positive article is about a contest to get married in a self-driving shuttle. Interesting! Let’s do a similar analysis for world news.

nlp s 29
 

Interestingly Trump features in both the most positive and the most negative world news articles. Do read the articles to get some more perspective into why the model selected one of them as the most negative and the other one as the most positive (no surprises here!).

Sentiment Analysis with TextBlob

TextBlob is another excellent open-source library for performing NLP tasks with ease, including sentiment analysis. It also an a sentiment lexicon (in the form of an XML file) which it leverages to give both polarity and subjectivity scores. Typically, the scores have a normalized scale as compare to Afinn. The polarity score is a float within the range [-1.0, 1.0]. The subjectivity is a float within the range [0.0, 1.0] where 0.0 is very objective and 1.0 is very subjective. Let’s use this now to get the sentiment polarity and labels for each news article and aggregate the summary statistics per news category.

nlp s 30
 

Looks like the average sentiment is the most positive in world and least positive in technology! However, these metrics might be indicating that the model is predicting more articles as positive. Let’s look at the sentiment frequency distribution per news category.

nlp s 31
 
Visualizing sentiment categories per news category

There definitely seems to be more positive articles across the news categories here as compared to our previous model. However, still looks like technology has the most negative articles and world, the most positive articles similar to our previous analysis. Let’s now do a comparative analysis and see if we still get similar articles in the most positive and negative categories for world news.

nlp s 32
 

Well, looks like the most negative world news article here is even more depressing than what we saw the last time! The most positive article is still the same as what we had obtained in our last model.

Finally, we can even evaluate and compare between these two models as to how many predictions are matching and how many are not (by leveraging a confusion matrix which is often used in classification). We leverage our nifty model_evaluation_utils module for this.

nlp s 33
 
Comparing sentiment predictions across models

In the preceding table, the ‘Actual’ labels are predictions from the Afinnsentiment analyzer and the ‘Predicted’ labels are predictions from TextBlob. Looks like our previous assumption was correct. TextBlob definitely predicts several neutral and negative articles as positive. Overall most of the sentiment predictions seem to match, which is good!

Conclusion

This was definitely one of my longer articles! If you are reading this, I really commend your efforts for staying with me till the end of this article. These examples should give you a good idea about how to start working with a corpus of text documents and popular strategies for text retrieval, pre-processing, parsing, understanding structure, entities and sentiment. We will be covering feature engineering and representation techniques with hands-on examples in the next article of this series. Stay tuned!


All the code and datasets used in this article can be accessed from my GitHub

The code is also available as a Jupyter notebook

I often mentor and help students at Springboard to learn essential skills around Data Science. Thanks to them for helping me develop this content. Do check out Springboard’s DSC bootcamp if you are interested in a career-focused structured path towards learning Data Science.

Data Science Career Track | Springboard
Data Science Career Track is your springboard to a data science career. Online, mentor-guided bootcamp, designed to get…
www.springboard.com
 

A lot of this code comes from the research and work that I had done during writing my book “Text Analytics with Python”. The code is open-sourced on GitHub(Python 3.x edition coming by end of this year!)

Text Analytics with Python - A Practical Real-World Approach to Gaining Actionable Insights from…
Derive useful insights from your data using Python. You will learn both basic and advanced concepts, including text and…
www.springer.com
 

“Practical Machine Learning with Python”, my other book also covers text classification and sentiment analysis in detail. The code is open-sourced on GitHub for your convenience.

Practical Machine Learning with Python - A Problem-Solver's Guide to Building Real-World…
Master the essential skills needed to recognize and solve complex problems with machine learning and deep learning in…
www.springer.com
 

If you have any feedback, comments or interesting insights to share about my article or data science in general, feel free to reach out to me on my LinkedIn social media channel.

Dipanjan Sarkar - Data Scientist - Intel Corporation | LinkedIn
View Dipanjan Sarkar's profile on LinkedIn, the world's largest professional community. Dipanjan has 6 jobs listed on…
www.linkedin.com
 

Thanks to Durba for editing this article.

Towards Data Science

Sharing concepts, ideas, and codes.

 

 
 

Applause from you and 1,050 others

 
 
 
 
Dipanjan (DJ) Sarkar

WRITTEN BY

Data Scientist @RedHat, Author, Consultant, Mentor @Springboard, Editor @TDataScience. Feel free to connect with me at https://www.linkedin.com/in/dipanzan

Probability Learning I : Bayes’ Theorem

Medium

Learn about one of the fundamental theorems of probability with an easy everyday example.

Jaime Zornoza
Jaime Zornoza

Aug 6 · 8 min read

This post assumes you have some basic knowledge of probability and statistics. If you don’t, do not be afraid, I have gathered a list of the best resources I could find to introduce you to these subjects, so that you can read this post, understand it, and enjoy it to its fullest.

In it, we will talk about one of the most famous and utilised theorems of probability theory: Bayes’ Theorem. Never heard about it? Then you are in for a treat! Know what it is already? Then read on to consolidate your knowledge with an easy, everyday example, so that you too can explain it in simple terms to others.

In the following posts we will learn about some simplifications of Baye’s theorem that are more practical, and about other probabilistic approaches to machine learning like Hidden Markov Models.

Lets go!

Introduction to probability:

In this section I have listed three very good and concise (the two first mainly, the third one is a bit more extensive) sources to learn the basics of probability that you will need to understand this post. Don’t be afraid, the concepts are very simple and with a quick read you will most definitely understand them.

If you already have a grasp of basic probability, feel free to skip this section.

Okay, now you are ready to continue on with the rest of the post, sit back, relax, and enjoy.

Bayes’ Theorem:

Who was Bayes?

Thomas Bayes (1701 — 1761) was an English theologian and mathematician that belonged to the Royal Society (the oldest national scientific society in the world and the leading national organisation for the promotion of scientific research in Britain), where other eminent individuals have enrolled, like Newton, Darwin or Faraday. He developed one of the most important theorems of probability, which coined his name: Bayes’ Theorem, or the Theorem of Conditional Probability.

Portrait of the Reverend Thomas Bayes, father of Bayes’ Theorem

The Theorem: Conditional Probability

To explain this theorem, we will use a very simple example. Imagine you have been diagnosed with a very rare disease, which only affects 0.1% of the population; that is, 1 in every 1000 persons.

The test you have taken to check for the disease correctly classifies 99% of the people who have the disease, incorrectly classifying healthy individuals with a 1% chance.

I am doomed! Is this disease fatal doctor?

Is what most people would say. However, after this test, what are the chances we have of actually suffering the disease?

99% for sure! I better get my stuff in order.

Upon this thought, Bayes’ mentality ought to prevail, as it is actually very far off from reality. Lets use Bayes’ Theorem to gain some perspective.

Bayes’ Theorem, or as I have called it before, the Theorem of Conditional Probability, is used for calculating the probability of a hypothesis (H) being true (ie. having the disease) given that a certain event (E) has happened (being diagnosed positive of this disease in the test). This calculation is described using the following formulation:

Bayes’ formula for conditional probability

The term on the left of the equal sign P(H |E )is the probability of having the disease (H) given that we have been diagnosed positive (E) in a test for such disease, which is what we actually want to calculate. The vertical bars (|) in a probability term denote a conditional probability (ie, the probability of A given B would be P(A|B)).

The left term of the numerator on the right side P(E|H) is the probability of the event, given that the hypothesis is true. In our example this would be the probability of being diagnosed positive in the test, given that we have the disease.

The term next to it; P(H) is the prior probability of the hypothesis before any event has taken place. In this case it would be the probability of having the disease before any test has been taken.

Lastly, the term on the denominator; P(E) is the probability of the event, that is, the probability of being diagnosed positive for the disease. This term can be further decomposed into the sum of two smaller terms: having the disease and testing positive plus not having the disease and testing positive too.

Deconstruction of the probability of testing positive on the test

In this formula P(~H) denotes the prior probability of not having the disease, where ~ means negation or not. The following picture describes each of the terms involved in the overall calculation of the conditional probability:

Description of each of the terms involved on the formulation of by Bayes’ Theorem

Remember, for us the hypothesis or assumption H is having the disease, and the event or evidence E is being diagnosed positive in the test for such disease.

If we use the first formula we saw (the complete formula for calculating the conditional probability of having the disease and being diagnosed positive), decompose the denominator, and insert in the numbers, we get the following calculation:

Calculation of the conditional probability

The 0,99 comes from the 99% probability of being diagnosed positive given that we have the disease, the 0,001 comes from the 1 in 1000 chance of having the disease, the 0,999 comes from the probability of not having this disease, and the last 0,01 comes from the probability of being diagnosed positive by the test even tough we do not have the disease. The final result of this calculation is:

Result of the calculation

9%! There is only a 9% chance that we have the disease! “How can this be?” you are probably asking yourself. Magic? No, my friends, it is not magic, it is just probability: common sense applied to mathematics. Like described in the book Thinking, Fast and Slow by Daniel Kahneman, the human mind is very bad at estimating and calculating probabilities, like it has been shown with the previous example, so we should always hold back our intuition, take a step back and use all the probability tools at our disposal.

Imagine now, that after being diagnosed positive on the first test, we decide to take another test with the same conditions, in a different clinic to double-check the results, and unluckily we get a positive diagnosis again, indicating that the second test also says that we have the disease.

What is the actual probability of having the disease now? Well, we can use exactly the same formula as before, but replacing the initial prior probability (0.1% chance of having the disease) with the posterior probability obtained the previous time (9% chance after being diagnosed positive by the test once), and their complementary terms.

If we crunch the numbers we get:

Calculation of the conditional probability after the second positive

Results after the second positive

Now we have a much higher chance, 91% of actually having the disease. As bad as it might look tough, after two positive tests it is still not completely certain that we have the disease. Certainty seems to escape the world of probability.

The intuition behind the theorem

The intuition behind this famous theorem is that we can never be fully certain of the world, as it is a changing being, change is in the nature of reality. However, something that we can do, which is the fundamental principle behind this theorem, is update and improve our knowledge of reality as we get more and more data or evidence.

This can be illustrated with a very simple example. Imagine the following situation: you are in the edge square shaped garden, sitting on a chair, looking outside the garden. On the opposite side, lies a servant that throws a initial blue ball inside the square. After that, he keeps throwing other yellow balls inside the square, and telling you where they land relatively to the initial green ball.

Video of this mental experiment with our good old Bayes sitting on the edge of his garden with his back towards a servant that is throwing the balls.

As more and more yellow balls land, and you get informed of where they land relative to the first blue ball, you gradually increase your knowledge of where the blue ball might be, leaving out certain parts of the garden: as we get more evidence (more yellow balls) we update our knowledge (the position of the blue ball).

In the example above, with only 3 yellow balls thrown, we could already start to build a certain idea that the blue ball lies somewhere within the top left corner of the garden.

When Bayes’ first formulated this theorem, he didn’t publish it at first, thinking it was nothing extraordinary, and the papers where the theorem was formulated were found after his death.

Today Bayes’ theorem is not only one of the bases of modern probability, but a highly used tool in many intelligent systems like spam filters, and many other text and non text related problem solvers.

In the following post we will see what these applications are, and how Bayes’ theorem and its variations can be applied to many real world use cases. To check it out follow me on Medium, and stay tuned!

That is all, I hope you liked the post. Feel Free to connect with me on LinkedIn or follow me on Twitter at @jaimezorno. Also, you can take a look at my other posts on Data Science and Machine Learning here. Have a good read!

Towards Data Science

Sharing concepts, ideas, and codes.

pygame install notes

Error:
In file included from src/scrap.c:28:
/Library/Frameworks/SDL.framework/Versions/Current/Headers/SDL_syswm.h:58:10: fatal error: 'X11/Xlib.h' file not found
#include
^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1

Repair:
https://stackoverflow.com/questions/14321038/x11-xlib-h-no-such-file-or-directory-on-mac-os-x-mountain-lion
sudo ln -s /opt/X11/include/X11 /usr/local/include/X11

Notes from previous parts of the same install:

Last login: Thu Jun 1 12:40:56 on ttys000
Zanes-MacBook-Air-2:~ Jasper$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:
/usr/local/bin/brew
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
/usr/local/Homebrew

Press RETURN to continue or any other key to abort
==> Searching online for the Command Line Tools
==> /usr/bin/sudo /usr/bin/touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
Password:
^Z
[1]+ Stopped /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Zanes-MacBook-Air-2:~ Jasper$
Zanes-MacBook-Air-2:~ Jasper$ ls
Applications Documents Movies Public test1.py
Battleshipinprogress.py Downloads Music SomeLocalDir
Desktop Library Pictures Untitled.py
Zanes-MacBook-Air-2:~ Jasper$ brew install mercurial
Updating Homebrew...
==> Installing dependencies for mercurial: readline, sqlite, gdbm, openssl, python
==> Installing mercurial dependency: readline
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
Error: Failure while executing: git config --local --replace-all homebrew.private true
Zanes-MacBook-Air-2:~ Jasper$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall)"
Warning: This script will remove:
/Library/Caches/Homebrew/
/Users/Jasper/Library/Caches/Homebrew/
/usr/local/Cellar/
/usr/local/Homebrew/
/usr/local/Homebrew/.github/
/usr/local/Homebrew/.gitignore
/usr/local/Homebrew/.travis.yml
/usr/local/Homebrew/.yardopts
/usr/local/Homebrew/CODEOFCONDUCT.md
/usr/local/Homebrew/CONTRIBUTING.md
/usr/local/Homebrew/LICENSE.txt
/usr/local/Homebrew/Library/
/usr/local/Homebrew/Library/Homebrew/test/.bundle/
/usr/local/Homebrew/README.md
/usr/local/Homebrew/bin/brew
/usr/local/Homebrew/completions/
/usr/local/Homebrew/docs/
/usr/local/Homebrew/manpages/
/usr/local/bin/brew -> /usr/local/Homebrew/bin/brew
/usr/local/etc/bash_completion.d/brew -> /usr/local/Homebrew/completions/bash/brew
/usr/local/share/doc/homebrew -> /usr/local/Homebrew/docs
/usr/local/share/man/man1/brew-cask.1 -> /usr/local/Homebrew/manpages/brew-cask.1
/usr/local/share/man/man1/brew.1 -> /usr/local/Homebrew/manpages/brew.1
/usr/local/share/zsh/site-functions/_brew -> /usr/local/Homebrew/completions/zsh/_brew
/usr/local/share/zsh/site-functions/_brew_cask -> /usr/local/Homebrew/completions/zsh/_brew_cask
/usr/local/var/homebrew/locks/update
Are you sure you want to uninstall Homebrew? [y/N] y
==> Removing Homebrew installation...
==> Removing empty directories...
==> /usr/bin/sudo /usr/bin/find /usr/local/Cellar /usr/local/Homebrew /usr/local/Frameworks /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/opt /usr/local/sbin /usr/local/share /usr/local/var -name .DS_Store -delete
==> /usr/bin/sudo /usr/bin/find /usr/local/Cellar /usr/local/Homebrew /usr/local/Frameworks /usr/local/bin /usr/local/etc /usr/local/include /usr/local/lib /usr/local/opt /usr/local/sbin /usr/local/share /usr/local/var -depth -type d -empty -exec rmdir {} ;
==> Homebrew uninstalled!
The following possible Homebrew files were not deleted:
/usr/local/bin/
/usr/local/etc/
/usr/local/share/
/usr/local/var/
You may wish to remove them yourself.
Zanes-MacBook-Air-2:~ Jasper$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:
/usr/local/bin/brew
/usr/local/share/doc/homebrew
/usr/local/share/man/man1/brew.1
/usr/local/share/zsh/site-functions/_brew
/usr/local/etc/bash_completion.d/brew
/usr/local/Homebrew
==> The following new directories will be created:
/usr/local/Cellar
/usr/local/Homebrew
/usr/local/Frameworks
/usr/local/include
/usr/local/lib
/usr/local/opt
/usr/local/sbin
/usr/local/share/zsh
/usr/local/share/zsh/site-functions

Press RETURN to continue or any other key to abort
==> /usr/bin/sudo /bin/mkdir -p /usr/local/Cellar /usr/local/Homebrew /usr/local/Frameworks /usr/local/include /usr/local/lib /usr/local/opt /usr/local/sbin /usr/local/share/zsh /usr/local/share/zsh/site-functions
Password:
==> /usr/bin/sudo /bin/chmod g+rwx /usr/local/Cellar /usr/local/Homebrew /usr/local/Frameworks /usr/local/include /usr/local/lib /usr/local/opt /usr/local/sbin /usr/local/share/zsh /usr/local/share/zsh/site-functions
==> /usr/bin/sudo /bin/chmod 755 /usr/local/share/zsh /usr/local/share/zsh/site-functions
==> /usr/bin/sudo /usr/sbin/chown Jasper /usr/local/Cellar /usr/local/Homebrew /usr/local/Frameworks /usr/local/include /usr/local/lib /usr/local/opt /usr/local/sbin /usr/local/share/zsh /usr/local/share/zsh/site-functions
==> /usr/bin/sudo /usr/bin/chgrp admin /usr/local/Cellar /usr/local/Homebrew /usr/local/Frameworks /usr/local/include /usr/local/lib /usr/local/opt /usr/local/sbin /usr/local/share/zsh /usr/local/share/zsh/site-functions
==> /usr/bin/sudo /bin/mkdir -p /Users/Jasper/Library/Caches/Homebrew
==> /usr/bin/sudo /bin/chmod g+rwx /Users/Jasper/Library/Caches/Homebrew
==> /usr/bin/sudo /usr/sbin/chown Jasper /Users/Jasper/Library/Caches/Homebrew
==> /usr/bin/sudo /bin/mkdir -p /Library/Caches/Homebrew
==> /usr/bin/sudo /bin/chmod g+rwx /Library/Caches/Homebrew
==> /usr/bin/sudo /usr/sbin/chown Jasper /Library/Caches/Homebrew
==> Searching online for the Command Line Tools
==> /usr/bin/sudo /usr/bin/touch /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
^[^[^[Didn't get a response from the Apple Software Update server.
==> Installing
==> /usr/bin/sudo /usr/sbin/softwareupdate -i
Software Update Tool

: No such update
No updates are available.
==> /usr/bin/sudo /bin/rm -f /tmp/.com.apple.dt.CommandLineTools.installondemand.in-progress
==> /usr/bin/sudo /usr/bin/xcode-select --switch /Library/Developer/CommandLineTools
xcode-select: error: invalid developer directory '/Library/Developer/CommandLineTools'
Failed during: /usr/bin/sudo /usr/bin/xcode-select --switch /Library/Developer/CommandLineTools
Zanes-MacBook-Air-2:~ Jasper$ code-select --install
-bash: code-select: command not found
Zanes-MacBook-Air-2:~ Jasper$ xcode-select --install,
xcode-select: error: invalid argument '--install,'
Usage: xcode-select [options]

Print or change the path to the active developer directory. This directory
controls which tools are used for the Xcode command line tools (for example,
xcodebuild) as well as the BSD development commands (such as cc and make).

Options:
-h, --help print this help message and exit
-p, --print-path print the path of the active developer directory
-s , --switch set the path for the active developer directory
--install open a dialog for installation of the command line developer tools
-v, --version print the xcode-select version
-r, --reset reset to the default command line tools path
Zanes-MacBook-Air-2:~ Jasper$ xcode-select --install
xcode-select: note: install requested for command line developer tools
Zanes-MacBook-Air-2:~ Jasper$ xcode-select --install,
xcode-select: error: invalid argument '--install,'
Usage: xcode-select [options]

Print or change the path to the active developer directory. This directory
controls which tools are used for the Xcode command line tools (for example,
xcodebuild) as well as the BSD development commands (such as cc and make).

Options:
-h, --help print this help message and exit
-p, --print-path print the path of the active developer directory
-s , --switch set the path for the active developer directory
--install open a dialog for installation of the command line developer tools
-v, --version print the xcode-select version
-r, --reset reset to the default command line tools path
Zanes-MacBook-Air-2:~ Jasper$ brew install mercurial
-bash: /usr/local/bin/brew: No such file or directory
Zanes-MacBook-Air-2:~ Jasper$ ls
Applications Documents Movies Public test1.py
Battleshipinprogress.py Downloads Music SomeLocalDir
Desktop Library Pictures Untitled.py
Zanes-MacBook-Air-2:~ Jasper$ python -version
Unknown option: -e
usage: /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
Zanes-MacBook-Air-2:~ Jasper$ python3 -version
Unknown option: -e
usage: /Library/Frameworks/Python.framework/Versions/3.5/Resources/Python.app/Contents/MacOS/Python [option] ... [-c cmd | -m mod | file | -] [arg] ...
Try `python -h' for more information.
Zanes-MacBook-Air-2:~ Jasper$ brew install mercurial
-bash: /usr/local/bin/brew: No such file or directory
Zanes-MacBook-Air-2:~ Jasper$ hg
-bash: hg: command not found
Zanes-MacBook-Air-2:~ Jasper$ brew install git
-bash: /usr/local/bin/brew: No such file or directory
Zanes-MacBook-Air-2:~ Jasper$ brew info
-bash: /usr/local/bin/brew: No such file or directory
Zanes-MacBook-Air-2:~ Jasper$ show brew
-bash: show: command not found
Zanes-MacBook-Air-2:~ Jasper$ /Developer/usr/bin/xcodebuild -version
-bash: /Developer/usr/bin/xcodebuild: No such file or directory
Zanes-MacBook-Air-2:~ Jasper$ cd /Developer/usr/bin/
-bash: cd: /Developer/usr/bin/: No such file or directory
Zanes-MacBook-Air-2:~ Jasper$ cd ~/Developer/usr/bin/xcodebuild -version
-bash: cd: /Users/Jasper/Developer/usr/bin/xcodebuild: No such file or directory
Zanes-MacBook-Air-2:~ Jasper$ ls
Applications Documents Movies Public test1.py
Battleshipinprogress.py Downloads Music SomeLocalDir
Desktop Library Pictures Untitled.py
Zanes-MacBook-Air-2:~ Jasper$ pwd
/Users/Jasper
Zanes-MacBook-Air-2:~ Jasper$ cd ..
Zanes-MacBook-Air-2:Users Jasper$ ls
Guest Jasper Shared ZaneScott matthewscott
Zanes-MacBook-Air-2:Users Jasper$ cd ..
Zanes-MacBook-Air-2:/ Jasper$ ls
Applications bin private
Developer cores sbin
Library dev tmp
Network etc usr
System home var
User Information installer.failurerequests vm
Users net
Volumes opt
Zanes-MacBook-Air-2:/ Jasper$ cd /Developer/usr/bin/
-bash: cd: /Developer/usr/bin/: No such file or directory
Zanes-MacBook-Air-2:/ Jasper$ cd Developer/
Zanes-MacBook-Air-2:Developer Jasper$ cd usr
-bash: cd: usr: No such file or directory
Zanes-MacBook-Air-2:Developer Jasper$ ls
Python
Zanes-MacBook-Air-2:Developer Jasper$ cd ..
Zanes-MacBook-Air-2:/ Jasper$ ls
Applications bin private
Developer cores sbin
Library dev tmp
Network etc usr
System home var
User Information installer.failurerequests vm
Users net
Volumes opt
Zanes-MacBook-Air-2:/ Jasper$ cd Developer/
Zanes-MacBook-Air-2:Developer Jasper$ ls
Python
Zanes-MacBook-Air-2:Developer Jasper$ cd ..
Zanes-MacBook-Air-2:/ Jasper$ ls
Applications bin private
Developer cores sbin
Library dev tmp
Network etc usr
System home var
User Information installer.failurerequests vm
Users net
Volumes opt
Zanes-MacBook-Air-2:/ Jasper$ cd bin
Zanes-MacBook-Air-2:bin Jasper$ ls
[ date expr ln pwd sync
bash dd hostname ls rm tcsh
cat df kill mkdir rmdir test
chmod domainname ksh mv sh unlink
cp echo launchctl pax sleep wait4path
csh ed link ps stty zsh
Zanes-MacBook-Air-2:bin Jasper$ xcodebuild -version
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
Zanes-MacBook-Air-2:bin Jasper$ xcode-select
xcode-select: error: no command option given
Usage: xcode-select [options]

Print or change the path to the active developer directory. This directory
controls which tools are used for the Xcode command line tools (for example,
xcodebuild) as well as the BSD development commands (such as cc and make).

Options:
-h, --help print this help message and exit
-p, --print-path print the path of the active developer directory
-s , --switch set the path for the active developer directory
--install open a dialog for installation of the command line developer tools
-v, --version print the xcode-select version
-r, --reset reset to the default command line tools path
Zanes-MacBook-Air-2:bin Jasper$ xcode-select -v
xcode-select version 2347.
Zanes-MacBook-Air-2:bin Jasper$ xcode-build
-bash: xcode-build: command not found
Zanes-MacBook-Air-2:bin Jasper$ xcode-select --install
xcode-select: note: install requested for command line developer tools
Zanes-MacBook-Air-2:bin Jasper$ xcode-select --install
xcode-select: note: install requested for command line developer tools
Zanes-MacBook-Air-2:bin Jasper$ ls
[ date expr ln pwd sync
bash dd hostname ls rm tcsh
cat df kill mkdir rmdir test
chmod domainname ksh mv sh unlink
cp echo launchctl pax sleep wait4path
csh ed link ps stty zsh
Zanes-MacBook-Air-2:bin Jasper$ pwd
/bin

Zanes-MacBook-Air-2:bin Jasper$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
==> This script will install:

Zanes-MacBook-Air-2:bin Jasper$ brew info xlib
Error: No available formula with the name "xlib"
==> Searching for a previously deleted formula...
Error: No previously deleted formula found.
Zanes-MacBook-Air-2:bin Jasper$ brew search xlib
No formula found for "xlib".
Zanes-MacBook-Air-2:bin Jasper$ brew search xlib
^C
Zanes-MacBook-Air-2:bin Jasper$ pip3 install pygame

Linux - Resources *****

 

 

Introduction

When managing a linux/unix operating system from the command line, users are interacting with the system via shell. This article will explore some of the basic features of the bash shell prompt. Default bash command line prompt on many linux systems does not have a color and display information about user's username, hostname and current working directory. As you well see in the following sections of this document this default settings can be easily changed by exporting a bash prompt PS{n} variables. Bash prompt can be modified to suit users needs and can display time, load, number of users using the system, uptime and more.

Bash prompt variables

As anything else in the linux system also bash prompt can be customized. Bash prompt can be customized by changing the values of bash PS1, PS2, PS3, PS4 variables. To keep the things simple, this article will be concerned just with PS1 and PS2 variables. Use echo command to see the current value of PS1 and PS2 variables.

echo "Bash PS1 variable:"  $PS1
echo "Bash PS2 variable:"  $PS2

Bash prompt variables

Bash PS1 prompt variable

PS1 is a primary prompt variable. Currently it holds \u@\h:\w\$ special bash characters. This is the default structure of the bash prompt on many Linux systems and is displayed every time you log in using a terminal.Please see the following section "Bash prompt special characters" for explanation of \u, \h, \w and \$ symbols. Here is a classical bash prompt with default settings:

Bash PS1 prompt variable

Bash PS2 prompt variable

PS2 bash shell variable is a secondary prompt. This prompt is displayed for if the shell waits for a user input, for example your forget to insert second quotation.

Bash PS2 prompt variable

Linux Bash prompt

Bash prompt special characters

Bash prompt can be customized by use of special characters. This table contains bash prompt special characters and its explanation.

Bash special character Bash special character explanation Bash special character Bash special character explanation
\a an ASCII bell character (07) \d the date in "Weekday Month Date" format (e.g., "Tue May 26")
! begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
\D{format} the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results in a locale-specific time representation. The braces are required

Bash prompt customization

After user login to the systems user environment variables initialized from various files like:

  • global system files /etc/profile or /etc/bashrc
  • user files ~/.bash_profile , ~/.bash_login , ~/.profile , ~/.bashrc or ~/.bash_logout

It is important to know that all users environment variable have a life time equal to the terminal session. When the terminal session is closed the user's variables including bash shell variables defined during a terminal session are emptied and a again redefined when new terminal session is created either via logo in shell or interactive shell. Lets define two variables to prove this statement.

Permanent bash variable definition

First permanent variable we define in one of the bash initialization files ~/.bash_profile and second variable we define on the shell prompt. Let's define permanent user variable:

Define Permanent bash user variable

What happened here, is that user "prompt" modified its own .bash_profile initialization file located in his/her home directory by appending a VAR1 variable definition. When user "prompt" logged out and logged in again the $var1 variable is initialized and available for the new terminal session.
On the same principles we can define our bash prompt. The best place to do it is that bash initialization file .~/bashrc. open up your ~/.bashrc file and add/edit the line defining a PS1 variable to something like:

PS1='MY NEW BASH PROMPT@\t:\w$ '

NOTE: Your ~/.barshrc file may differ from the example below !

define new bash prompt
After the user "prompt" restarts a terminal session he/she is welcomed by new bash prompt. This changes to the bash prompt are permanent.
HINT: Use "source" command to re-read a bash initialization .bashrc file instead of restarting a terminal session.

source .bashrc

or similarly

. .bashrc

bash prompt definition

Temporary bash variable definition

Temporary bash variable definition is defined only for a lifetime of the current terminal session. This is tome by an export command. bash temporary variable
As you can see the variable $VAR2 is not defined when user closes his/her terminal session. The permanent variable $VAR1 is always defined from the bash initialization file: ~/.bash_profile . As we can use an export command to define new bash variables we can also use it to modify a bash prompt $PS1 variable. To change a current bash prompt to display only time we could do:

export PS1="\t: "

define bash temporary prompt

Changing foreground and background bash prompt colors

Syntax for changing colors in the bash is as follows:

3[ -> Indicates the beginning of color in the text
 x;yzm - Indicates color code
 3[00m - Indicates the end of color in the text

Bash color codes:

Bash color codes

export PS1="3[01;31mBASH IN RED3[00m: "

bash prompt and red color

Bash Prompt Examples

To get you started with your new bash prompt here are couple examples:

Display current Time

export PS1="\u@\h \t:$ "

Display current Time with bash prompt

Counting Files in the Current Directory

This bash prompt displays current number of files and directories in the current directory.

export PS1="\u@\h [$(ls | wc -l)]:$ "

Counting Files in the Current Directory in bash prompt

 


 

Through the Wormhole

 

A daily journal of 3000 pics a day! How to process and select.

 

https://www.youtube.com/channel/UCVRl6Rj03lfF8v5nX7eJwyA

 

 

Through the Wormhole - Top Documentary Films All episodes from Through the Wormhole series gathered in one place. Hosted by Morgan Freeman, Through the Wormhole will explore the deepest mysteries of...

 

 

Data Compression

 

 

Data Compression The textbook Algorithms, 4th Edition by Robert Sedgewick and Kevin Wayne surveys the most important algorithms and data structures in use today. The broad perspective taken makes it an appropriate introduction to the field.

 

 

Data compression: reduces the size of a file to save space when storing it and to save time when transmitting it. Moore's law: # transistor on a chip doubles every 18-24 months. Parkinson's law: data expands to fill available space. Text, images, sound, video, etc. Wikipedia providespublic dumps of all content for academic research and republishing. Uses bzip and SevenZip's LZMA. Can take a week to compress of 300GB of data.

Ancient ideas.

Morse code, decimal number system, natural language, rotary phones (lower numbers were quicker to dial, so New York was 212 and Chicago 312).

Binary input and output streams.

We use BinaryStdIn.java, BinaryStdOut.java, BinaryDump.java, HexDump.java, and PictureDump.java.

Fixed length codes.

Need ceil(lg R) bits to specify one of R symbols. Genome.java. Uses Alphabet.java.

Run length encoding.

RunLength.java.

Variable-length codes.

Desire unique decodable codes. One way to achieve this is to append a special stop symbol to each codeword. Better approach: prefix-free codes: no string is a prefix of another. For example, { 01, 10, 0010, 1111 } is prefix free, but { 01, 10, 0010, 1010 } is not because 10 is a prefix of 1010.

Give fax machine example.

Huffman codes.

Specific way to construct optimal prefix-free codes. Invented by David Huffman while a student at MIT in 1950. Huffman.javaimplements Huffman algorithm.

Property A. No prefix free code uses fewer bits.

LZW compression.

Using prefix match code from TST.java, LZW.java implements LZW compression.

Real world: Pkzip = LZW + Shannon-Fano, GIF, TIFF, V.42bis modem, Unix compress. Practical issues:

  • Encode everything in binary.
  • Limit the number of elements in the symbol table (GIF = throw away and start over, Unix compress = throw away when not effective).
  • Initially dictionary has 512 elements (with 256 elements filled in for ASCII characters), so we transmit 9 bits per integer. When it fills up, we double it to 1024 and start transmitting 10 bits per integer.
  • Only traverse the tree once (might break our string table abstraction).

Practical issues: limit the number of elements in the symbol table.

Summary.

Huffman: variable length code for fixed length symbols. LZW: fixed length code for variable length strings.

Universal compression algorithm.

Impossible to compress all files (proof by simple counting argument). Intuitive argument: compress life work of Shakespeare, then compress result, then compress result again. If each file strictly shrinks, eventually you will be left with one bit.

References.

Guy Blelloch of CMU has an excellent chapter on data compression.

Error correction / detection.

Suppose channel for sending information is noisy and each bit gets flipped with probability p. Send each bit 3 times; to decode take the majority of the 3 bits. Decoded bit is correct with probability 3p^2 - 2p^3. This is less than p (if p < 1/2). Can reduce probability of decoding the bit incorrectly by sending each bit k times, but this is wasteful in terms of the transmission rate.

Reed-Solomon codes.

Reference. Used in mass storage systems (CDs and DVDs) and satellite transmissions (Voyager space probe, Mars Pathfinder) when the errors are bursty. Think of data to send as a degree d polynomial. Only need d+1 points to uniquely specify the polynomial. Send more points to enable error correction / detection. If code we want to send is a0, a1, ..., am-1 (each elements over finite field K), think of it as the polynomial p(x) = a0 + a1x + ... + am-1 x^m-1. Send p(0), p(b), p(b^2), ..., where b is a generator of multiplicative cyclic group over K.

Shannon's coding theorem.

Roughly speaking, if channel capacity is C, then we can send bits at a rate slightly less than C with an encoding scheme that will reduce probability of a decoding error to any desired level. Proof is nonconstructive.

Q+A

Exercises

  1. Which of the following codes are prefix free? Uniquely decodable? For those that are uniquely decodable, give the encoding of 1000000000000.
        code 1    code 2    code 3   code 4
    A     0         0          1       1
    B     100       1         01      01
    C     10       00        001     001
    D     11       11       0001     000
    
  2. Given an example of a uniquely-decodable code that is not prefix free.Solution. Any suffix-free code is uniquely decodable, e.g., { 0, 01 }.
  3. Given an example of a uniquely-decodable code that is not prefix free or suffix free.Solution. { 0011, 011, 11, 1110 } or { 01, 10, 011, 110 }.
  4. Are { 1, 100000, 00 }, { 01, 1001, 1011, 111, 1110 }, and { 1, 011, 01110, 1110, 10011 } uniquely decodable? If not, find a string with two encodings. Solution. The first set of codewords is uniquely decodable. The second set of codewords is not uniquely decodable because 111-01-1110-01 and 1110-111-1001 are two decodings of 11101111001. The third set of codewords ins not uniquely decodable because 01110-1110-011 and 011-1-011-10011 are two decodings of 011101110011.
  5. Test for uniquely decodability. Implement the Sardinas-Patterson algorithm for testing whether a set of codewords is uniquely decodable: Add all of the codewords to a set. Examine all pairs of codewords to see if any one is a prefix of another; if so, extract the dangling suffix (i.e., the part of the longer string that is not a prefix of the shorter one). If the dangling suffix is a codeword, then the code is not uniquely decodable; otherwise, add the dangling suffix to the list (provided it is not already there). Repeat this process with the larger list until there are no remaining new dangling suffix.The algorithm is finite because all dangling suffixes added to the list are suffixes of a finite set of codewords, and a dangling suffix can be added at most once.
    • { 0, 01, 11 }. The codeword 0 is a prefix of 01, so add the dangling suffix 1. { 0, 01, 11, 1 }. The codeword 0 is a prefix of 01, but the dangling suffix 1 is already in the list; the codeword 1 is a prefix of 11, but the dangling suffix 1 is already in the list. There are no other dangling suffixes, so conclude that the set is uniquely decodable.
    • { 0, 01, 10 }. The codeword 0 is a prefix of 01, so add the dangling suffix 1 to the list. { 0, 01, 10, 1 }. The codeword 1 is a prefix of 10, but the dangling suffix 0 is a codewords. So, conclude that the code is not uniquely decodeable.
  6. Kraft-McMillan inequality. Conside a code C with N codewords of lengths n1, n2, ..., nN. Prove that if the code is uniquely decodable, then K(C) = sum_i = 1 to N 2^(-ni) ≤ 1.
  7. Kraft-McMillan construction. Suppose that we have a set of integers n1, n2, ..., nN that satisfy the inequality sum_i = 1 to N 2^(-ni) ≤ 1. Prove that it is always possible to find a prefix-free code with codewords lengths n1, n2, ..., nN. Thus, by restricting attention to prefix-free codes (instead of uniquely decodable codes), we do not lose much.
  8. Kraft-McMillan equality for optimal prefix-free codes. Prove that if C is an optimal prefix-free code then the Kraft-McMillan inequality is an equality: K(C) = sum_i = 1 to N 2^(-ni) = 1.
  9. Suppose that all of the symbol probabilities are negative powers of 2. Describe the Huffman code.
  10. Suppose that all of the symbol frequencies are equal. Describe the Huffman code.
  11. Find a Huffman code where the length of a symbol with probability pi is greater than ceil(-lg pi).Solution. .01 (000), .30 (001), .34 (01), .35 (1). The codeword 001 has length greater than ceil(-lg .30).
  12. True or false. Any optimal prefix-free code can be obtained via Huffman's algorithm.Solution. False. Consider the following set of symbols and frequencies (A 26, B 24, C 14, D 13, E 12, F 11).
           C1   C2   C3
    A 26   01   10   00
    B 24   10   01   01
    C 14  000  111  100
    D 13  001  110  101
    E 12  110  001  110
    F 11  111  000  111
    

    In any Huffman code, the codings for A and B must begin with different bits, but the code C3 does not have this property (yet it is an optimal prefix-free code).

  13. What is the LZW encoding of the following inputs?
    • T O B E O R N O T T O B E
    • Y A B B A D A B B A D A B B A D O O
    • A A A A A A A A A A A A A A A A A A A A A
  14. Characterize the tricky situation in LZW coding.Solution. Whenever it encounteres cScSc, where c is a symbol, S is a string, cS is in the dictionary but cSc is not.
  15. As a function of N, how many bits are needed to encode N copies of the symbol A? N copies of the sequence ABC?
  16. Let F(i) be the ith Fibonacci number. Consider N symbols, where the ith symbol has frequency F(i). Note that F(1) + F(2) + ... + F(N) = F(N+2) - 1. Describe the Huffman code.Solution. Longest codeword has length N-1.
  17. Show that there are at least 2^(N-1) different Huffman codes corresponding to a given set of N symbols.Solution. There are N-1 internal nodes and each one has an arbitrary choice to assign its left and right children.
  18. Give a Huffman code where the frequency of 0s in the output is much much higher than the frequency of 1s.Solution. If the character 'A' occurs one million times and the character 'B' occurs once, the code word for 'A' will be 0 and the codeword for 'B' will be 1.
  19. Prove the following facts about Huffman tries.
    • The two longest codewords have the same length.
    • If the frequency of symbol i is strictly larger than the frequency of symbol j, then the length of the codeword for symbol i is less than or equal to the length of the codeword for symbol j.
  20. Describe how to transmit a Huffman code (or optimal prefix-free code) on a set of symbols { 0, 1, ..., N-1 } using 2N - 1 + N ceil(lg N) bits.Hint: use 2N-1 bits to specify the structure of the corresponding trie.
  21. Suppose that in an extended ASCII file (8-bit characters), the maximum character frequency is at most twice the minimum character frequency. Prove that and fixed-length 8-bit extended ASCII code is optimal.
  22. Shannon-Fano coding. Prove that the following top-down version of Huffman's algorithm is not optimal. Split the set of codewords C into two subsets C1 and C2 with (almost) equal frequencies. Recursively build the tree for C1 and C2, starting all codewords for C1 with 0 and all codewords for C2 with 1. To implement the first step, Shannon and Fano propose sorting the codewords by frequency and breaking the set up into two subarrays as best as possible.Solution. S 32, H 25, A 20, N 18, O 5.
  23. LZMW coding (Miller-Wegman 1985). LZ variant: search input for longest string already in the dictionary (the current match); add concatenation of previous match to current match to the dictionary. Dictionary entries grow more rapidly. Can also delete low-frequency entries when the dictionary fills up. Hard to implement.
  24. LZAP coding. Similar to LZMW: instead of adding just the concatenation of the previous match with the current match, add the concatenation of the previous match with all prefixes of the current match. Easier than LZMW to implement, but even more dictionary entries.
  25. Identify an optimal code that is not prefix-free.Hint: only need 3 symbols with equal frequencies.
  26. Identify two optimal prefix-free codes for the same input that have a different distribution of codeword lengths.Hint: only need 4 symbols.
  27. Minimum variance Huffman coding. Due to the nondeterminism associated with tiebraking, Huffman's algorithm may produce codes with different distributions of codeword lengths. When transmitting the compressed stream as it is being generated, it is desirable to transmit bits at a (near) constant rate. Find Huffman code that minimize sum_i (p_i (l_i - l_average(T)) ^2).Solution. When combining tries, break ties by picking the earliest produced trie with the smallest probability.
  28. Two-queue algorithm for Huffman coding. Prove that the following algorithm computes a Huffman code (and runs in linear time if the input symbols are already sorted by frequency). Maintain two FIFO queues: the first queue contains the input symbols, in ascending order of frequency, the second queue contains the internal nodes with combined weights. As long as there is more than one node in the two queues, dequeue the two nodes with the smallest weight by examining the fronts of both queues. Create a new internal node (left and right child = two nodes, weight = sum of weight of two nodes) and enqueue on the second queue.To obtain a minimum variance Huffman code, break ties by choosing nodes from the first queue.

    Hint: prove that the second queue is sorted in ascending order of frequency.

  29. Sibling property. A binary tree has the sibling property if (i) every node (except the root) has a sibling and (ii) the binary tree can be listed in non-increasing order of probability such that, in the list, all siblings are adjacent. Prove that a binary tree represents a Huffman tree if and only if it has the sibling property.
  30. Relative coding. Instead of compressing each pixel in an image, consider the difference between a pixel and the previous one and encode the difference. Intuition: usually the pixels don't change much. Use with LZW over color table alphabet.
  31. Variable-width LZW codes. Increase the width of the table from p to p+1 after 2^p th codeword is inserted into table. Used with color table alphabet.
  32. Adaptive Huffman coding. One-pass algorithm and don't need to send prefix-free code. Build Huffman tree based on frequency of characters read in so far. Update tree after reading in each character. Encoder and decoder need to coordinate on tie-breaking conventions.
  33. Shannon entropy. The entropy H of a discrete random variable X with possible values x1, ..., xN that occur with probability p1, ..., pN is defined as H(X) = -p1 lg p1 - p2 lg p2 - ... - pN lg pN, where 0 lg 0 = 0 is consistent with the limit.
    • What is the entropy of a fair coin?
    • What is the entropy of a coin where both sides are heads?
    • What is the entropy of a six-sided die?Solution. -lg (1/6) which is about 2.584962.
    • What is the entropy of the sum of two fair dice?
    • Given a random variable that takes on N values. What distribution maximizes the entropy?

    The entropy is a fundamental concept in information theory. Shannon's source coding theorem asserts that to compress the data from a stream of independent and identically distributed random variables requires at least H(X) bits per symbol in the limit. For example, to send the results of a sequence of fair die tosses requires at least 2.584962 bits per die toss.

  34. Empirical entropy. The empirical entropy of a piece of text is obtained by computing the frequency of occurrence of each symbol and using these as the probabilities for a discrete random variable. Compute the empirical entropy of your favorite novel. Compare it to the data compression rate achieved by a Huffman code.
  35. Shannon experiment. Perform the following experiment. Give a subject a sequence of k letters from a piece of text (or Leipzig corpus) and ask them to predict the next letter. Estimate the fraction of times the subject gets the answer right for k = 1, 2, 5, 100.
  36. True or false. Fixed-length codes are uniquely decodable.Solution. True, they are prefix free.
  37. Give two different Huffman trees the string ABCCDD, with different heights.
  38. Prefix-free codes. Design an efficient algorithm to determine if a set of binary code words is prefix-free. Hint: use a binary trie or sort.
  39. Uniquely decodable code. Devise a uniquely decodable code that is not a prefix free code. Hint: suffix free codes = reverse of prefix free codes. Reverse of suffix free code is prefix free code -> can decode by reading compressed message in reverse order. Not very convenient.
  40. Huffman tree. Modify Huffman.java so that the encoder prints out the lookup table instead of the preorder traversal, and modify the decoder so that it constructs the tree by reading in the lookup table.
  41. True or false. In an optimal prefix-free ternary code, the three symbols that occur least frequently have the same length.Solution. False.
  42. Ternary Huffman codes. Generalize the Huffman algorithm to codewords over the ternary alphabet (0, 1, and 2) instead of the binary alphabet. That is, given a bytestream, find a prefix-free ternary code that uses as few trits (0s, 1s, and 2s) as possible. Prove that it yields optimal prefix-free ternary code.Solution. Combine smallest 3 probabilities at each step (instead of smallest 2). This works when there are 3 + 2k symbols for some integer k. To reduce to this case, add 1 or 2 dummy symbols of probability 0. (Alternatively, combine fewer than 3 symbols in the first step if the number of symbols is not 3 + 2k.) Ex: { 0.1, 0.2, 0.2, 0.5 }.
  43. Nonbinary Huffman codes. Extend the Huffman algorithm to codewords over the m-ary alphabet (0, 1, 2, ..., m-1) instead of the binary alphabet.
  44. Consider the following 21 character message that consists of 3 a's, 7c's, 6 t's and 5 g's.
    a a c c c c a c t t g g g t t t t c c g g 
    

    Are the following 43 bits a possible Huffman encoding of the message above?

    0000001111000101010010010010101010111001001 
    

    Justify your answer as concisely and rigorously as possible.Solution. A Huffman encoding for a message produces an encoding that uses the fewest bits among any prefix free code. The 2-bit binary code a = 00, c = 01, g = 10, t = 11 is a prefix free code that uses 21 * 2 = 42 bits. Thus, a Huffman code would use fewer than 43 bits.

  45. A binary tree is full if every node that is not a leaf has two children. Prove that any binary tree corresponding to an optimal prefix-free code is full.Hint: if an internal node has only one child, replace that internal node with its unique child.
  46. Move-to-front coding (Bentley, Sleator, Tarjan, and Wei 1986). Write a program MoveToFront that implements move-to-front encoding and decoding. Maintain alphabet of symbols in a list, where frequently occurring symbols are towards the front. A symbol is encoded as the number of symbols that precede it in the list. After encoding a symbol, move it to the front of the list. reference
  47. Move-ahead-k coding. Same as move-to-front coding, but move symbol k positions toward the front.
  48. Wait-c-and-move. Same as move-to-front coding, but move symbol to the front only after it has been encountered c times since the last time it was moved to the front.
  49. Double Huffman compression. Find an input for which applying the compress() method in Huffman.java twice leads to a strictly smaller output than applying compress() only once.
  50. Merging k sorted arrays. You have k sorted lists, of lenths n1, n2, ..., nk. Supposet that the only operation you can perform to combine lists is a 2-way merge: given one sorted array of length n1 and another sorted array of length n2, replace them with a sorted array of length n = n1 + n2. Moreover, the 2-way merge operation takes exactly n units of time. What is the optimal way to merge the k sorted arrays?Solution. Sort the list lengths so that n1 < n2 < ... < nk. Repetedly take the two smallest lists and apply the 2-way merge operation. The proof of optimality is the same as the proof of optimality of Huffman codes: repeatedly applying 2-way merge operations induces a binary tree in which each leaf node corresponds to one of the original sorted lists and each internal node corresponds to a 2-way merge operation. The contribution of any original list to the overall cost is the length of the list multiplied by its tree depth (because that is the number of times its elements are involved in a 2-way merge).

 

An Idiot's Guide to C++ Templates - Part 2

 

Let's dwell deeper into C++ templates!

Elevating...

In first part of the series, I elaborated following aspects of templates in C++.

  • The syntax of C++ templates
  • Function Templates and Class Templates
  • Templates taking one or more arguments
  • Templates taking non-type integral arguments, and taking default arguments.
  • The two phase compilation process
  • Generic Pair and Array class that can hold any data-type(s).

In this part, I would try to impart more intriguing concepts of templates, its importance and binding with the other features of C++ language, and would also touch upon STL. No, you need not to know STL at all, and I would not dwell deep into STL. I request you to refresh your template understanding by reading first part, before you jump into this one!

Since my idea remain to elaborate concepts in better and elaborative manner, articles become exhaustive. It takes time and effort, and therefore not everything can be explained in one go. Hence, I write, publish, update - in gradual manner. I request your feedback considering this.

Requirements from the Underlying Type

There are class templates and function templates, and they work on given type (template argument type). For example, a function template would sum up two values (or entire array), for type T - But this function template would require operator+ to be present and accessible for the give type (type T). Similarly, a class would require the target type to have constructor, assignment operator and other set of required operators.

Requirements: Function Templates

Let me start explaining this topic in little simple and elegant manner. Following function would display value of given type on console (using std::cout):

template<typename T>
void DisplayValue(T tValue)
{
   std::cout << tValue;
}

Following set of calls would succeed:

DisplayValue(20); // <int>
DisplayValue("This is text"); // <const char*>
DisplayValue(20.4 * 3.14); // <double>

Since ostream (type of cout) has operator << overloaded for all basic types; hence it works for int, char* and double types. There is an implicit call to one of those overloads of operator<<.

Now, let us define a new structure, having two members in it:

struct Currency 
{
   int Dollar;
   int Cents;
};

And have an attempt to use this type against DisplayValue function template:

Currency c;
c.Dollar = 10;
c.Cents = 54;
 
DisplayValue(c);

For this call, you will be bombarded with host of errors from your compiler, because the following line fails to compile for the instantiation of DisplayValue for type Currency:

std::cout << tValue; // tValue is now of Currency type

Visual C++ will start reporting errors starting with:

error C2679: binary '<<' : no operator found which takes a right-hand operand of type 'Currency' (or there is no acceptable conversion)

GCC compiler would start reporting with:

In function 'void DisplayValue(T) [with T = Currency]':
16: instantiated from here
2: error: no match for 'operator<<' in 'std::cout << tValue'

Errors do differ, but they mean the same thing: None of the overloads of overloaded operator ostream::operator << can be called for Currency. Both of the compilers do report this simple error in at least 100 lines! The error is not entirely because of ostream, nor because of Currency, and neither because of templates - but due to assortment of all this into one. At this moment, you have different options available:

  • Don't call DisplayValue for type Currency, and write another function DisplayCurrencyValueinstead. This is what most programmers would do, after finding their inabilty to solve the problem with original DisplayValue with Currency type. Doing this defeats the whole purpose and power of templates in C++. Don't do it!
  • Modify ostream class, and add new member (i.e. operator<<) that takes Currency type. But you don't have liberty to do so, since ostream is in one of the C++ standard header. However, with global function, which would take ostream and Currency types, you can do it.
  • Modify your own class, Currency, so that cout<<tValue would succeed.

In short, you need to facilitate either of following:

  • ostream::operator<<(Currency value); (Simplified syntax)
  • ostream::operator<<(std::string value);
  • ostream::operator<<(double value);

First version is very much possible, but syntax is slightly complicated. The definition of custom function would take ostream as well as Currency types, so that cout<<currency_object; would work.

Second version demands understanding of std::string, and would be slower and complex.

Third version fits the requirement at this moment, and is simplest of other two. It means that Currency be converted to double, whenever demanded, and the converted data will be passed to cout call.

And here is the solution:

struct Currency
{
  int Dollar;
  int Cents;

  operator double()
  {
    return Dollar + (double)Cents/100;
  }
};

Notice that entire result would come out into a double value. So, for Currency{12, 72} object, this overloaded operator (function operator double()) would return 12.72. Compiler will now be happy, since there is a possible conversion from Currency to one of the types that ostream::operator<< takes.

And therefore, the following call, for type Currency:

std::cout << tValue;

would be expanded as:

std::cout << tValue.operator double(); 
// std::cout << (double)tValue;

And hence the following call would work:

Currency c; 
DisplayValue(c);

You see that a simple function call has invoked multiple operations with the help of compiler:

  • Instantiating DisplayValue for type Currency.
  • Calling the copy-constructor for class Currency, since type T is passed by value.
  • In an attempt to find best match for cout<<operator call, the conversion operator Currency::operator double is invoked.

Just add your own copy-constructor code for Currency class, and do step-in debugging and see how a simple call is invoking multiple operations!

Alright! Now, let's revisit PrintTwice function template from previous part of this article-series:

template<typename TYPE>
void PrintTwice(TYPE data)
{
    cout<<"Twice: " << data * 2 << endl;
}

The important part is marked in bold. When you call it as:

PrintTwice(c); // c is of Currency type

The expression data * 2 in PrintTwice will work, since Currency type facilitates this possibility. The cout statement would be rendered, by the compiler, as:

cout<<"Twice: " << data.operator double() * 2 << endl;

When you remove the overloaded operator double() from class Currency, compiler would complain that it doesn't have operator*, or any possibility where this expression can be evaluated. If you change the class (struct) Currency as:

struct Currency
{
  int Dollar;
  int Cents;

  /*REMOVED : operator double();*/
 
  double operator*(int nMultiplier)
  {
    return (Dollar+(double)Cents/100) * nMultiplier;
  }
};

Compiler will be happy. But doing this will cause DisplayValue to fail for Currency instantiation. The reasons is simple: cout<<tValue won't be valid then.

What if you provide both double conversion as well as multiplication operator? Would the call to PrintTwicefail for Currency type? A possible answer would be yes, that the compilation would fail, since compiler would have ambiguity at following the call:

cout<<"Twice: " << data * 2 << endl; 
// Ambiguity: What to call - conversion operator or operator* ?

But no, compilation will not fail, and it would make a call to operator*. Any other possible conversion routines may be called, if compiler doesn't find best candidate. Most of the stuff explained here comes from C++ rulebook, and not explicitly under template umbrella. I explained it for better understanding.

It is generally advised that class shouldn't expose conversion operators too much, instead they should provide relevant functions for them. An example is string::c_str(), which returns C-style string pointer; but another implementations (like CString) would provide necessary conversions implicitly by providing conversion-operators. But for most cases it is recommended that classes should not expose unnecessary operators just to make it work with any code. It should provide only reasonable implicit conversions.

Same thing goes with templates and underlying types, the underlying type may provide conversion operator (like conversion to int or string), but shouldn't provide conversions in excess. Therefore, for Currency, only double (and/or to string) conversion would suffice - there is no need to give (binary) operator* overloaded.

Moving on. The following function template:

template<typename T>
T Add(T n1, T n2)
{
   return n1 + n2;
}

would require T to have operator+ which takes argument of same type, and returns same type. In Currencyclass, you would implement it as:

Currency operator+(Currency); // No consts, for simplification

Interestingly, if you do not implement operator+ in Currency , and keep the double conversion operator present in class; and call function Add as:

Currency c1, c2, c3; 
c3 = Add(c1,c2);

You would get somewhat weird error:

error C2440: 'return' : cannot convert from 'double' to 'Currency'

Reason is two fold:

  • n1 + n2 is causing operator double() be called for both objects passed (implicit conversion). Therefore, n1+n2 becomes a simple double+double expression, resulting into double as final value.
  • Since the final value is of type double, and the same is to be returned from function Add; compiler would try to convert this double to Currency. Since there is no constructor available that takes double, hence the error.

For this situation, Currency may provide a constructor (a conversion constructor) that takes double. It makes sense, since Currency now provides both TO and FROM double conversions. Providing operator+ just for the sake of Add function template doesn't make much of sense.

But, I am not at all against on providing many or all required implicit conversions or overloaded operators. In reality, you should provide all required stuff from the underlying type, if the template class/function is designed such way. A class template performing complex number calculation, for example, should be provided with all mathematical operators from the underlying type.


I see most of you are just reading this article, without attempting to do anything. So, here is exercise for you. Find out the type requirements, and fulfill those requirements for Currency class for the following function template (taken as-is from first part):

template<typename T>
double GetAverage(T tArray[], int nElements)
{
   T tSum = T(); // tSum = 0
   for (int nIndex = 0; nIndex < nElements; ++nIndex)
   {
     tSum += tArray[nIndex];
   }
   
   // Whatever type of T is, convert to double
   return double(tSum) / nElements;
  }

Hey, don't be lazy... Come on! Fulfill the requirements of GetAverage for Currency type. Templates are not theoretical, but very much practical! Don't read ahead, until you understand every bit of text till here.

Requirements: Class Templates

As you should know, class templates would be more prevalent than function templates. There are more class templates in STL, Boost and other standard libraries than function templates. Believe me, there would be more class templates who would craft, than number of function templates you would code.

Class templates would ask for more requirements from the underlying type. Though, it would be mostly dependent on what you demand from the class template itself. For example, if you do not call min/maxequivalent method from class- template instantiation, the underlying type need not to provide relevant relational operators.

Most class templates would demand following from the underlying type:

  • Default constructor
  • Copy constructor
  • Assignment operator

Optionally, depending on the class template itself (i.e. the purpose of class template), it may also ask for:

  • Destructor
  • Move constructor and Move assignment operator
    (On top of R- Value references)

Only for the sake of simplicity, I would not cover the second group of basic requirements, at least not in this article. Let me start off with the first group of basic-requirements understanding from underlying type, followed by explication of more type requirements.

Note that all required-methods, when provided by the underlying type, must be accessible (i.e. public) from the class template. For instance, a protected assignment operator from Currency class won't help for some collection class that demands it. The assignment of one collection to another collection (both of Currency) will not compile due to protected nature of Currency.

Let's revisit set of classes I used in first part, for this sub-section. Following is a class template Item, for holding any type:

template<typename T>
class Item
{
   T Data;
public:
   Item() : Data( T() ) {}
   void SetData(T nValue)
   {
      Data = nValue;
   }
   T GetData() const
   {
    return Data;
   }
 
   void PrintData()
   {
      cout << Data;
   }
};

The mandatory requirement, as soon as you instantiate Item for particular type, is having default constructor in type T. Therefore, when you do:

Item<int> IntItem;

The type int must have default constructor (i.e. int::int()), since this class' template constructor would be calling default constructor for underlying type:

Item() : Data( T() ) {}

We all know that int type has its default constructor, which initializes the variable with zero. When you instantiate it for another type, and that type is not having default constructor, the compiler would be upset. To understand this, let's craft a new class:

class Point
{
   int X, Y;
public:
   // No default constructor 
   Point(int x, int y);
};

As you know, following call would fail to compile (commented code would compile).

Point pt; // ERROR: No default constructor available
// Point pt(12,40);

Similarly, following instantiation would also fail:

Item<Point> PointItem;

When you use Point directly, without using non-default constructor, the compiler will report it, and you would able to correct it - since the source (line) would be reported correctly.

But, when you use it against Item class template, the compiler would the report the error near the Itemimplementation. For PointItem example mentioned above, Visual C++ reports it at following line:

Item () : Data( T() ) 
{}

And the error is reported as:

error C2512: 'Point::Point' : no appropriate default constructor available
: while compiling class template member function 'Item<T>::Item(void)'
with
[
T=Point
]
: see reference to class template instantiation 'Item<T>' being compiled
with
[
T=Point
]

Which is very small error message, as far as C++ template error reporting is concerned. Most often, the first error would tell the whole story, and the last error reference ("see reference to class...") shows the actual cause of the error. Depending on your compiler, the error may be easy to understand, or may be quite elusive to get to the actual cause of the error. The quickness and ability of finding and fixing the bug/error would depend on your experience with template programming.

So, to use Point class, as a template type parameter for Item, you must provide the default constructor in it. Yes, a single constructor taking zero, one or two parameter would also work:

Point(int x = 0, int y = 0 );

But, the constructor must be public (or accessible, by other means).

Alright. Let's see if SetData and GetData methods would work for type Point. Yes, both of them would work since Point has a (compiler provided) copy constructor and assignment operator.

If you implement assignment operator in class Point (the underlying type for Item), that particular implementation will be called by Item (yes, the compiler will generate the relevant code). The reason is simple: Item::SetData is assigning the value:

void SetData(T nValue) // Point nValue
{
   Data = nValue; // Calling assignement operator.
}

If you put the implementation of assignment operator of Point class in private/protected area:

class Point
{
...
private:
void operator=(const Point&);
};

And make a call to Item::SetData:

Point pt1(12,40), pt2(120,400);
Item<Point> PointItem;
 
PointItem.SetData(pt1);

It would cause a compiler error, starting with something like:

error C2248: 'Point::operator =' : cannot access private member declared in class 'Point'
99:7: error: 'void Point::operator=(const Point&)' is private

The compiler would point you the location of error (inside SetData), as well as the actual source of error (call from main). Actual error messages, and sequence of messages shown would depend on compiler, but most modern compiler will attempt to give detailed error so that you can find the actual source of error.

This example shows that special method of underlying type T are called, depending on how/what call is being made. Remember that special members (constructors, assignment operators etc) may be called implicitly - while returning from function, passing value to function, using expressions and so on. It is therefore recommended that underlying type (type T) should have these special methods implemented in accessible region of class.

This example also demonstrates how different classes and functions are involved in one function call. In this case, only Point class, the class template Item and function main are involved.

Similarly, when you make a call to Item<Point>::GetItem method, the copy constructor would be called (of Point class). The reason is simple - GetItem is returning a copy of data stored. A call to GetItem may not always call copy-constructor, as RVO/NRVO, move-semantics etc may come into picture. But you should always make the copy-constructor of underlying type accessible.

The method Item<>::PrintData would not succeed for Point type, but would succeed for Currency, as underlying type for Item. Class Point doesn't have conversion, or any possible call, to the cout<< call. Please do yourself a favor - make Point class cout-able!

Separation of Declaration and Implementation

Till now I have shown the entire implementation of template-code in one source-file. Treat the source-file as one header file, or being implemented in same file containing main function. As any C/C++ programmer would know, we put declarations (or say, interfaces) in header file, and the respective implementation in one or more source files. The header file would be included by both - respective implementation file, and by one or more client of that interface.

At the compilation unit level, the given implementation file would get compiled, and an object file for the same would be generated. When linking (i.e. while generating the final executable/DLL/SO), the linker would gather all those generated object files and would produce final binary image. All good, all fine, unless linker gets upset by missing symbols or duplicate symbols being defined.

Come to templates, it is not exactly the same way. For better understanding let me first throw you some code:

Sample.H

template<typename T>
void DisplayValue(T tValue);

Sample.CPP

template<typename T>
void DisplayValue(T tValue)
{
    std::cout << tValue;
}

Main.cpp

#include "Sample.H"
 
int main()
{
  DisplayValue(20);
  DisplayValue(3.14);
}

Depending on the compiler and IDE you use, you would put both CPP files for the build. Surprisingly, you will encounter linker errors like:

unresolved external symbol "void __cdecl DisplayValue<double>(double)" (??$DisplayValue@N@@YAXN@Z)
unresolved external symbol "void __cdecl DisplayValue<int>(int)" (??$DisplayValue@H@@YAXH@Z)
(.text+0xee): undefined reference to `void DisplayValue<int>(int)'
(.text+0xfc): undefined reference to `void DisplayValue<double>(double)'

When you closely look at the errors, you would find out that linker could not find implementation for following routines:

void DisplayValue<int> (int);
void DisplayValue<double> (double);

Despite the fact that you have provided implementation of template function DisplayValue via source file Sample.CPP.

Well, here is the secret. You know that template function gets instantiated only when you make a call to it, with particular data-type(s). The compiler compiles Sample.CPP file separately having definition of DisplayValue. The compilation of Sample.CPP is done in separate translation unit. Compilation of Main.cpp is done in another translation unit. These translation units produce two object files (say Sample.obj and Main.obj).

When the compiler works on Sample.CPP, it does not find any references/calls to DisplayValue, and it doesn't instantiate DisplayValue for any type. Reason is simple, as explained earlier - On-Demand-Compilation. Since the translation-unit for Sample.CPP doesn't demand any instantiation (for any data-type), it doesn't do second-phase compilation for function template. No object code is generated for DisplayValue<>.

In another translation-unit, Main.CPP gets compiled and object code gets generated. While compiling this unit, the compiler sees a valid interface declaration for DisplayValue<>, and performs its job without any issue. Since, we have called DisplayValue with two different types, compiler intelligently produces following declarations by itself:

void DisplayValue<int>(int tValue);
void DisplayValue<double>(double tValue);

And as per its normal behavior, the compiler assumes definitions of these symbols in some other translation-unit (i.e. object code), and delegates further responsibility to the linker. This way, Sample.obj and Main.obj files get generated, but none of them do contain implementation of DisplayValue - and hence the linker produces set of errors.

What's the solution for this?

The simplest solution, which works for all modern compiler is using the Inclusion Model. Another model, not supported by most major compiler vendors is Separation Model.

Till now, whenever I explained about template stuff with code written in same file, I used inclusion model. In simple terms, you put all template related code in one file (generally a header file). The client would just include the given header file, and entire code would be compiled in one translation unit. Yet, it would follow on-demand-compilation process.

For the example given above, Sample.H would contain the definition (implementation) of DisplayValue:

Sample.H

// BOTH: Interface and Implementation
template<typename T> 
void DisplayValue(T tValue) 
{
   std::cout << tValue;
}

Main.CPP would just include this header file. The compiler will be happy, and the linker will also be happy. If you prefer, you may put all declarations, followed by definitions of all functions later in the same file. For example:

template<typename T>
void DisplayValue(T tValue);
 
template<typename T>
void DisplayValue(T tValue)
{
   std::cout << tValue;
}

It has following advantages:

  • Logical grouping of all declarations, and all implementations.
  • No compiler errors if a template function A needs to use B, and B also needs to use A. You would have already declared the prototypes for the other function.
  • Non-inlining of class methods. Till now, I have elaborated entire template class within the class' declaration body. Separating out the method implementation, is discussed later.

Since you can logically divide interface and implementation, you can also figuratively divide them into Dot-H and Dot-CPP files:

template<typename T> 
void DisplayValue(T tValue);

#include "Sample.CPP"

Sample.H is giving prototype for DisplayValue, and at the end of file, it is including Sample.CPP. Don't worry, it is perfectly valid C++ and would work with your compiler. Note that, your project/build now must not add Sample.CPP for compilation process.

The client (Main.CPP) will include the header, which is adding the code of Sample.CPP into it. In this case, just one translation unit (for Main.cpp) will do the trick.

Separating Class Implementation

The section demands more attention from the readers. Implementing a method outside a class requires complete type-specification. For example, let's implement Item::SetData outside the class definition.

template<typename T>
class Item
{
   ...
  void SetData(T data);
};
 
template<typename T>
void Item<T>::SetData(T data)
{
   Data = data;
}

Note the expression Item<T> while mentioning the class for which method is being defined. Implementing SetData method with Item::SetData will not work, since Item is not a simple class, but a class template. The symbol Item is not a type, but some instantiation of Item<> is a type, and therefore the expression Item<T>.

For instance, when you instantiate Item with type short, and use SetData method for it,

Item<short> si;
si.SetData(20);

the compiler would generate source-code like this:

void Item<short>::SetData(short data)
{
   Data = data;
}

Here, the formed class name is Item<short> and SetData is being defined for this class type.

Let's implement other methods outside the class body:

template<typename T>
T Item<T>::GetData() const
{
   return Data;
}
 
template<typename T>
void Item<T>::PrintData()
{
   cout << Data;
}

Clearly notice that template<typename T> is required in all cases, and Item<T> is also required. When implementing GetData, the return type is T itself (as it should be). In PrintData implementation, though T is not used, Item<T> specification is needed anyway.

And finally, here is the constructor implemented outside class:

template<typename T>
Item<T>::Item()  /*: Data( T() ) */
{
}

Here, the symbol Item<T> is the class, and Item()is the method of this class (i.e. the constructor). We need not to (or cannot, depending on compiler) use Item<T>::Item<T>() for the same. Constructor (and destructor) are special member methods of class, and not class types, and therefore they shouldn't be used as type, in this context.

Only for simplicity I commented default initialization of Data with the default constructor-call for type T. You should uncomment the commented part, and understand the meaning.

If the template class has one or more default types/non-types as template parameters, we just need to specify them while declaring the class:

template<typename T = int> // Default to int
class Item
{
... 
void SetData(T data);
};
 
void Item<T>::SetData() { }

We cannot/need-not specify the default template parameters, at the implementation stage:

void Item<T = int>::SetData() {} // ERROR

That would simply be an error. The rule and reasoning is very much similar to a C++ function taking default argument(s). We only specify default parameter(s) when declaring the interface of a function, and not when (separately) implementing the function. Eg:

void Allocate(int nBytes = 1024);
void Allocate(int nByte /* = 1024* / )  // Error, if uncommented.
{ }

Implementing Method Templates Outside Class

For this, first consider a simple example by a code snippet:

Item<int> IntItem;
Item<short> ShortItem; 
IntItem.SetData(4096);
 
ShortItem = IntItem;

The important line of discussion is the one which is marked bold. It tries to assign Item<int> object to Item<short> instance, which is not possible, since these two are different types. Of course, we could have used SetData on one and GetData on another object. But what if we needed the assignment to work?

For this, you can implement a custom assignment operator, which would itself be on top of templates. That would be classified as method template, and it was already covered in first part. This discussion is only for outside-class implementation. Anyway, here is the in-class implementation:

template<typename U>
void operator = (U other)
{
 Data = other.GetData();
}

Where U is the type of other class (another instantiation of Item). I didn't use const and reference specification for other argument, only for simplicity. When ShortItem = IntItem takes place, following code is generated:

void operator = (Item<int> other) 
{
   Data = other.GetData();
}

Note that other.GetData() returns int, and not short, since source object other is of type Item<int>. If you call this assignment operator with non-convertible types (such as int* to int), it would cause compiler error as these two types are not implicitly convertible. One should not use any type of typecasting within template code, for these kind of conversions. Let the compiler report error to the client of your template.

One more interesting thing worth mentioning here. If you code above' assignment operator like this:

template<typename U>
void operator = (U other)
{
 Data = other.Data;
}

It simply won't compile - the compiler would complain that Data is private! You would wonder why?
Reason is quite simple: this class (Item<short>), and the other class (Item<int>) are actually two different classes and have no connection between them. By standard C++ rule, only same class can access private data of current class. Since Item<int> is another class, it doesn't give private access to Item<short> class, and hence the error! That's the reason I had to use GetData method instead!

Anyway, here is how we implement a method template outside the class declaration.

template<typename T>
template<typename U>
void Item<T>::operator=( U other )
{
   Data = other.GetData();
}

Note that we need to use template specification with template keyword two times - one for class template, and one for method template. Following will not work:

template<typename T, class U>
void Item<T>::operator=( U other )

Reason is simple - class template Item does not take two template-arguments; it takes only one. Or, if we take it other way around - method template (i.e. assignment operator) doesn't take two template arguments. Class and methods are two separate template-entities, and need to be classified individually.

You should also notice that <<code>class T> comes first, followed by <class U>. Studying it by the left-to-right parsing logic of the C++ language, we see that class Item comes first then the method. You may treat the definition as (see tabs):

template<typename T>
    template<typename U>
void Item<T>::operator=( U other )
{ }

The separation of two-template specification is definitely not dependent on the original parameter names used in class and method declarations. By this, I mean U and T can exchange positions, and it would compile fine. You can also name the way you like - other than T or U.

template<typename U>
template<typename T>
void Item<U>::operator=( T other )
{ }

The order of arguments must match, however. But, as you can understand, using the same name is recommended for readability.

Read and understand enough? Well, then it is time to test yourself by writing some template code! I just need the following to work:

const int Size = 10;
Item<long> Values[Size]; 
 
for(int nItem = 0; nItem < Size; ++nItem)
    Values[nItem].SetData(nItem * 40);
 
Item<float> FloatItem;
FloatItem.SetAverageFrom(Values, Size);

The method-template SetAverageFrom would calculate the average from the Item<> array being passed. Yes, the argument (Values) may be of any underlying type Item-array. Implement it outside class body! Irrespective of who you are - A super-genuis in C++ templates or if you think this task as Rocket-Science tough, you must do it - Why fool yourself?

Additionally, what would you do if Values is an array of underlying type Currency?


Most of the template implementations would be using inclusion model only, that too only in one header file, and all inline code! STL, for example, uses header-only, inline implementation technique. Few libraries are using include-other-stuff technique - but they require only header to be included by client, and they operate on top of inclusion model only.

For most template related code, inlining doesn't harm for few reasons.

One, the template code (class, function, entire-library) is generally short and concise, like implementing a class template less which calls operator < on the underlying type. Or a collection class which puts and reads the data into collection, without doing too much of laborious work. Most classes would do small and only-required tasks like calling a function-pointer/functor, performing string related stuff; and would not do intensive calculations, database or file read, sending packet to network, preparing buffer to download and other intensive work.

Two, inlining is just a request from the programmer to the compiler and compiler would inline/not-inline the code on its own discretion. It all depends on code complexity, how often it (method) gets called, other possible optimizations around it etc. Linker and profile guided optimization (PGO) also play important role in code optimization, inlining etc. Therefore, putting entire code within class-definition will not do any harm.

Three, not all of the code gets compiled - only the one that gets instantiated would get compiled, and this reasoning get more importance because of previous two points mentioned. So, don't worry about code inlining!

When set of class templates along with few helper function templates, the code is just like an arithmetic expression for the compiler. For example, you would use std::count_if algorithm on a vector<int>, passing a functor, which would call some comparison operator. All this, when coupled in single statement, may look complicated and seems processor-intensive. But it is not! The entire expression, even involving different class templates and function templates, is like a simple expression to the compiler - specially in a Release build.

Other model, the Separation Model, works on top of export keyword, which most compilers don't still support. Neither GCC, nor Visual C++ compiler support this keyword - both compiler would however say this is reserved keyword for future, rather than just throwing non-relevant error.


One concept that logically fits this modeling umbrella is Explicit Instantiation. I am deferring this concept and I would be elaborating it later. One important thing - Explicit Instantiation and Explicit Specialization are two different facets! Both will be discussed later.

Templates and Other Aspects of C++

Gradually, as you would gain firm understanding about the templates, the power of templates in C++ and be passionate about templates, you would get clear picture, that using templates you can craft your own language subset. You can program the C++ language so that it performs some tasks the way you like. You can use and abuse the language itself, and ask compiler to generate source- code for you!

Fortunately, or unfortunately, this section is not about how to abuse the language and make compiler do labor-work for you. This section tells how other concepts like inheritance, polymorphism, operator overloading, RTTI etc. are coupled with templates.

Class Templates, Friends

Any veteran programmer would know the real importance of friend keyword. Any newbie or by-the-books mortal may detest friend keyword, saying that it breaks encapsulation, and the third category would say "depends". Whatever your perspective may be, but I believe that friend keyword is useful, if judiciously used wherever required. A custom allocator for various classes; a class to maintain the relation between two different classes; or an inner class of a class are good candidates of being friends.

Let me first give you an example, where the friend keyword along with template is almost indispensable. If you remember the template-based assignment operator in Item<> class, you must also recollect that I had to use GetData from the other object of another type (another variant of Item<>). Here is the definition (in-class):

template<typename U>
void operator = (U other)
{
 Data = other.GetData();
}

The reason is simple: Item<T> and Item<U> would be different types, where T and U may be int and short, for example. One class cannot access private member of another class. If you implement an assignment operator for regular class, you would directly access the data of other object. What would you do to access data of other class (which is, ironically, the same class!) ?

Since the two specializations of class template belong to same class, can we make them friends? I mean, is it possible to make Item<T> and Item<U> friends of each other, where T and U are two different data-types (convertible)?

Logically, it is like:

class Item_T
{
   ...
   friend class Item_U;
};

So that Item_U can access Data (private data) of class Item_T! Remember that, in reality, Item_TandItem_U would not be just two class-types, but any set of two instantiations on top if class template Item.

Self-friendship seems logical, but how to achieve that? Following will simply not work:

template<typename t>
class Item
{
   ...
   friend class Item;
};

Since Item is a class template and not a regular class, therefore symbol Item is invalid in this context. GCC reports following:

warning: class 'Item<t />' is implicitly friends with itself [enabled by default]
error: 'int Item<int />::Data' is private

Amusingly, initially it says it is implicit friend with itself, and later it complains about private-access. Visual C++ compiler is more lenient and silently compiles, and makes them friends. Either way, the code is not compatible. We should use code that is portable and indicates Item as class template. Since target type is unknown, we cannot replace T with any particular data-type.

Following should be used:

template <class U> 
friend class Item; // No template stuff around 'Item'

It forward-declares the class, and implies that Item is class template. The compiler is now satisfied, without any warnings. And now, following code works without the penalty to call the function:

template<typename U>
void operator = (U other)
{
  Data = other.Data; // other (i.e. Item<U> has made 'me' friend.
}

Other than this self-friendship notion, the friend keyword would be useful along with templates in many other situations. Of course, it includes regular course of friendships, like connecting a model and a framework class; or a manager class being declared as friend by other worker classes. But, in case of templates, an inner class of a template-based outer class may have to make outer class a friend. Another case where template-based base class would be declared as friend by derived class.

At present, I do not have more ready and understandable examples to demonstrate friend keyword usage, that are specific to class templates.

Class Templates, Operator Overloading

Class templates would use operator-overloading idea, more often that a regular class would do. A comparator class would use one or more of relational operators, for instance. A collection class would use index-operator to facilitate get or set operations for element access by index or by key. If you remember class template Arrayfrom previous part, I used index-operator:

template<typename T, int SIZE>
class Array
{
   T Elements[SIZE];
   ...
public:
   T operator[](int nIndex) 
   {
      return Elements[nIndex];
   }
};

For another example, recollect a class template Pair discussed in previous part. So, for example, if I use this class template as:

int main()
{
   Pair<int,int> IntPair1, IntPair2;
   
   IntPair1.first = 10;
   IntPair1.second = 20;
   IntPair2.first = 10;
   IntPair2.second = 40;
 
   if(IntPair1 > IntPair2)
      cout << "Pair1 is big.";
}

That simply won't work and requires operator > to be implemented by class template Pair:

// This is in-class implementation
bool operator > (const Pair<Type1, Type2>& Other) const
{
  return first > Other.first && 
         second > Other.second;
}

</span /></span /></span />Though, same thing was already discussed in first part (for operator ==), I added this word only for relevance with the concept being elaborated.

Other than these regular overloadable operators, like relational operators, arithmetic operators etc, templates also employ other rarely used overloadable operators: Arrow Operator (- >) and the Pointer Indirection operator (*). A simple smart-pointer implementation, on top of class template, illustrates the usability.

template<typename Type>
class smart_ptr
{
   Type* ptr;
public:
   smart_ptr(Type* arg_ptr = NULL) : ptr(arg_ptr)
   {}
 
   ~smart_ptr()
   {
     // if(ptr) // Deleting a null-pointer is safe
     delete ptr;
   }
};

The class template smart_ptr would hold up a pointer of any type, and would safely delete the memory allocated, in the destructor. Usage example:

int main()
{
  int* pHeapMem = new int;
  smart_ptr<int> intptr(pHeapMem);
  //  *intptr = 10;
}

I have delegated the responsibility of memory deallocation to the smart_ptr object (intptr). When the destructor of intptr would get called, it would delete the memory allocated. Note that the first line in main function is just for better clarity. The constructor of smart_ptr may be called as:

smart_ptr<int> intptr(new int);

NOTE: This class (smart_ptr) is only for illustration purpose, and it is functionally not equivalent to any of standard smart pointer implementations (auto_ptr, shared_ptr, weak_ptr etc.).

Smart pointer would allow any type to be used for safe and sure memory-deallocation. You could also use any UDT:

smart_ptr<Currency> cur_ptr(new Currency);

After end of current block (i.e. - {}), the destructor of smart_ptr<> would get called, and would invoke delete operator on it. Since the type is known at compile time (instantiation is compile-time!), the destructor of correct type would be invoked. If you put destructor of Currency, that would be called as soon as cur_ptrceases to exist.

Coming back on track; how would you facilitate following:

smart_ptr<int> intptr(new int);
*intptr = 10;

For sure, you would implement pointer indirection (unary) operator:

Type& operator*() 
{
   return *ptr;
}

Distinctly understand that the above definition is non-const implementation, and that is the reason it returns reference of object (*ptr, not ptr) being held by class instance. Only because of the same, assignment of value 10 is allowed.

Had it been implemented as const method, it would not allow assignment to succeed. It would generally return a non-referenced object, or const-reference of the object being held:

// const Type& operator*() const
Type operator*() const
{
   return *ptr;
}

Following code snippet shows its usage:

int main()
{
   smart_ptr<int> intptr(new int);
   *intptr = 10; // Non-const
 
   show_ptr(intptr);
}
// Assume it implemented ABOVE main
void show_ptr(const smart_ptr<int>& intptr)
{
   cout << "Value is now:" << *intptr; // Const
}

You may like to return co<code>nst Type& for saving few bytes of program stack, from a const function. But, in general, class templates do return value types instead. It keeps the design simple, avoids any possible bug from creeping in if underlying type has const/non-const blunder in implementation. It also avoids any unnecessary reference creation even from small types (like int or Currency), which would turn to be more heavier than value type returns.

Quite interestingly, you can templatize the show_ptr function itself, so that it can display value of any underlying type under the smart_ptr object. There is a lot more to explicate about template functions/classes that itself take another template, but needs a separate discussion area for the same. Keeping simple and to-the-point discussion, here is modified show_ptr:

template<typename T>
void show_ptr(const smart_ptr<T>& ptr)
{
   cout << "Value is now:" << *ptr; // Const
} 

For Currency object, the function will call Currency::operator double, so that cout will work. Are you awake, or you need to refresh stuff about cout and Currency? If in confusion, please read that stuff again.

Moving on, lets see what happens when you try to do the following.

smart_ptr<Currency> cur_ptr(new Currency);
cur_ptr->Cents = 10;
show_ptr(cur_ptr);

The bold line, logically correct, but will fail. Reason is simple - cur_ptr is not a pointer, but a normal variable. Arrow operator can only be called if expression on left is a pointer to structure (or class). But, as you see, you are using smart_ptr as a pointer-wrapper around Currency type. Therefore, this should aesthetically work. Essentially, it means, you need to overload arrow operator in class smart_<code>ptr !

Type* operator->()
{
 return ptr;
}
 
const Type* operator->() const
{
 return ptr;
}

Since I do respect your comfort level with the C++ language, I don't find it necessary to explain about these two different overloads implemented. After the implementation of this operator, cur<code>_ptr->Centsassignment will work!

In general, operator -> will return a pointer only (of some struct/class). But that's not absolutely necessary - operator-> may also return reference/value of particular class type. It is not really useful, deep down concept and is rarely implement that way, I don't find it worth discussing.

Do apprehend that overloaded operator-> in smart_ptr will not cause any compile-time error for smart_ptr<int>, just because int cannot have arrow-operator applied to it. The reason is simple, you will not call call this operator on smart_ptr<int> object, and hence compiler will not (attempt to) compile smart_ptr<>::operator->() for it!

By now, you must have realized importance of operator overloading in C++ and in template arena. Under template domain, there is much more around operators, and it really helps template based development, compiler support, early binding etc.

Class Templates, Inheritance

Before discussing the usability of inheritance along with template-based classes, I would emphasize different modes of inheritance involved. No, it is not about multiple, multilevel, virtual or hybrid inheritance, or base class having virtual functions. The modes are just around single inheritance:

  1. Class template inheriting Regular class
  2. Regular class inheriting Class Template
  3. Class Template inheriting another Class Template

In template-based class designs, other than single inheritance, multiple inheritance would be more frequent than multilevel, hybrid or virtual inheritance. Let me first start off with single inheritance.

You know that class-template is a template for a class, which will be instantiated depending on the types(s) and other argument it takes. The instantiation would produce a template-class, or more distinctly, specialization of that class. The process is known as instantiation, and the outcome is known as specialization.

When inheriting, what would you inherit - a class-template (Item<T>), or the specialization (Item<int>) ?

These two different models appear same, but are entirely different. Let me give you an example.

class ItemExt : public Item<int>
{
}

Here you see that normal class ItemExt is inheriting from a specialization (Item<int>), and is not facilitating any other instantiation of Item. What does it mean? You might ask.

First consider this : The empty class ItemExt, in itself, can be classified as:

typedef Item<int> ItemExt;

Either way (typedef or inheritance), when you use ItemExt, you don't need to (or say, you cannot) specify the type:

ItemExt int_item;

int_item is nothing but a derived-class object of type Item<int>. This mean, you cannot create object of other underlying type using the derived class ItemExt. The instance of ItemExt will always be Item<int>,even if you add new methods/members to derived class. The new class may provide other features like printing the value, or comparing with other types etc, but class doesn't allow flexibility of templates. By this, I mean, you cannot do:

ItemExt<bool> bool_item;

Since ItemExt is not a class template, but a regular class.

If you are looking for this kind of inheritance, you can do so - it all depends on your requirements, and design perspective.

Another type of inheritance would be template-inheritance, where you would inherit the class template itself and pass the template-parameters to it. Example first:

template<typename T>
class SmartItem : public Item<T>
{
};

Class SmartItem is another class template which is inheriting from Item template. You would instantiate SmartItem<> with some type, and same type would be passed to class template Item. And all this would happen at compile time. If you instantiate SmartItem with char type, Item<char> and SmartItem<char>would be instantiated!

As an another example of template-inheritance, let inherit from class template Array:

template<size_t SIZE>
class IntArray : public Array<int, SIZE>
{
};
int main()
{
   IntArray<20> Arr;
   Arr[0] = 10;
}

Note that I have used int as first template argument, and SIZE as second template argument to base class Array. The argument SIZE is only argument for IntArray, and is second argument for base class Array.This is allowed, interesting feature and facilitates automatic code generation with the help of compiler. However, IntArray would always be array of ints, but the programmer may specify the size of array.

Similarly, you may inherit Array this way also:

template<typename T>
class Array64 : public Array<T, 64>
{
};
 
int main()
{
  Array64<float> Floats;
  Floats[2] = 98.4f;
}

Though, in the examples given above, the derived class itself do not do anything extra, inheritance is very-much required. If you think following template-based typedef will do the same, you are wrong!

template<typename T>
typedef Array<T, 64> Array64;

typedef<size_t SIZE>
typedef Array<int, SIZE> IntArray;

Template based typedefs are not allowed. Although I do not see any reason why compilers cannot provide such feature. On top of templates typedefs can behave different depending on the context (i.e. based on template parameters). But template-based typedefs at global level are not allowed.

Though, not specific to the template-inheritance discussion, you may achieve typedef without using inheritance also. But in that case too, you need to define a new class.

template<size_t SIZE>
struct IntArrayWrapper
{
   typedef Array<int, SIZE> IntArray;
};

Usage is slightly different:

IntArrayWrapper<40>::IntArray Arr;
Arr[0] = 10;

The choice entirely depends on the requirement, flexibility, readability, some coding-standards and by personal choice. The second version is quite cumbersome, in my opinion.

But, if inheritance is desired, and you are going to provide extra features on top of base-template, and/or there is "is-a"relationship between base and derived classes, you should use template-inheritance model.

Note that, in almost all cases, template based classes wouldn't have virtual functions; therefore, there is no added penalty on using inheritance. Inheritance is just a data-type modeling, and in simple cases, derived class would also be POD (Plain Old Data). Virtual functions, with templates, will be described later.

By now, I have elaborate two models of inheritance:

  • Regular class inheriting class template
  • Class template inheriting another class template

I also explicated the difference between template-inheritance (where you pass the template argument(s) to base), and instantiation-inheritance where you inherit from very specific type of template instantiation (called specialization). Note that both IntArray and Array64 would be classified as template-inheritance, since at least one template-argument is keeping the specialization to happen, and would happen only when derived type is instantiated with specific arguments.

Note that only ItemExt is an example of 'Regular class inheriting class template'. All other examples given are 'class -template inheriting class-template'.

Now the third type. Can a class-template inherit a regular class?
Who said No? Why not!

I don't find or craft any example where base class would be (absolutely) basic, without any stain of template. It is actually unreasonable and would be a bad design to represent "is-a" relationship from a non-template base class. Initially, I thought of giving an example where base class would be singly-linked list, and derived class, based on template would be somewhat smarter (say doubly) linked list.

The bad example:

class SinglyLinkedList
{
 // Assume this class implements singly linked list
 // But uses void* mechanism, where sizeof data is 
 // specified in constructor.
};

template<class T>
class SmartLinkedList : public SinglyLinkedList
{
};

Now, you can say a SmartLinkedList<> object is-a SinglyLinkedList, which defeats the whole purpose of templates. A template-based class must not depend on non-template class. Templates are abstraction around some data-type for some algorithm, programming-model, a data-structure.

In fact, templates do avoid inheritance feature of OOP, altogether. It represents most of the abstractions by a single class. By this I do not mean templates would not use inheritance. In fact, many features around templates rely on inheritance feature of C++ - but it would not use inheritance-of-features, as in classical sense of Object-Oriented-Programming.

Class Templates would use inheritance of rules, inheritance of modeling, inheritance of designs and so on. One example would be make a base class, having copy-constructor and assignment-operator private, without having any data-members in it. Now, you can inherit this Rule class, and make all desired classes non-copyable!

Let me finish all major aspects of C++ along with templates, then I would show you some truly intriguing techniques using templates!

Function Pointers and Callbacks

As you do know function-pointer is one of the mechanism in C/C++ language to achieve dynamic-polymorphism. Not necessarily, but generally coupled with callback-feature - you set particular user-defined function as a callback, that would be called later. The actual function to be called is determined at runtime, and hence late-binding of specific callable function occurs.

To understand, let us consider simple code-snippet.

typedef void (*DisplayFuncPtr)(int);

void RenderValues(int nStart, int nEnd, DisplayFuncPtr func)
{
   for(;nStart<=nEnd; ++nStart)
       func(nStart); // Display using the desired display-function
}
 
void DisplayCout(int nNumber)
{
   cout << nNumber << " ";
}

void DisplayPrintf(int nNumber)
{
   printf("%d ", nNumber);
}

int main()
{
   RenderValues(1,40, DisplayCout); // Address-Of is optional
   RenderValues(1,20, &DisplayPrintf);

   return 0;
}

In this code, DisplayFuncPtr gives the prototype of the desired function, and is only for better readability. Function RenderValues will display the numbers using the given function. I called this function with different callbacks (DisplayCout and DisplayPrintf) from main function. Late-binding occurs at the following statement.

func(nStart);

Here func may point to either of the two Display-functions (or any other UDF). This type of dynamic-binding has several issues:

  • The prototype of callback function must exactly match. If you change void DisplayCout(int) to void DisplayCout(float), the compiler will get upset:
    error C2664: 'RenderValues' : cannot convert parameter 3 from 'void (__cdecl *)(double)' to 'DisplayFuncPtr'
  • Even though the return value of func is not used by RenderValues, compiler will not allow any callback function returning non-void.
  • And this one troubles me a lot! The calling convention must also match. If function specifies cdecl as callback function, a function implemented as stdcall (__stdcall), will not be allowed.

Since function-pointers and callbacks comes from the C language itself, compilers have to impose these restrictions. Compiler just cannot allow incorrect function to avoid call-stack to get corrupted.

And here is template based solution to overcome all the mentioned issues.

template<typename TDisplayFunc>
void ShowValues(int nStart, int nEnd, TDisplayFunc func)
{
   for(;nStart<=nEnd; ++nStart)
       func(nStart); // Display using the desired display-function
}

You can happily supply any of the functions to ShowValues templated-based function:

void DisplayWithFloat(float);
int DisplayWithNonVoid(int);
void __stdcall DisplayWithStd(int);

...

ShowValues(1,20, DisplayWithFloat);
ShowValues(1,40, DisplayWithNonVoid);
ShowValues(1,50, DisplayWithStd);

Yes, you would get float to int conversion warning for first function. But return type and calling convention would not matter. In fact, any function that can be called with int argument would be allowed in this case. You can modify the third function taking double, returning a pointer:

int* __stdcall DisplayWithStd(double);

The reason is simple. Actual type of TDisplayFunc is determined at compile time, depending on the type-of argument passed. In case of function-pointer implementation, there is exactly one implementation. But in case of function templates, there would be different instantiations of ShowValues, depending on unique function-prototypes you instantiate it with.

Along with the concerns mentioned above for normal C-style function-pointer/callback approach, following are also not allowed as display-function argument:

  • Functors, i.e. Function objects - A class may implement operator() with required signature. For example:
struct DisplayHelper
{
   void operator()(int nValue)
   {
   }
};

The following code is illegal.

DisplayHelper dhFunctor;
RenderValues(1,20,dhFunctor); // Cannot convert...

But when you pass the dhFunction (a functor, aka function-object) to function template ShowValues, the compiler will make no complains. As I said earlier TDisplayFunc may be any type that can be called with intargument.

ShowValues(1,20, dhFunctor);
  • Lambdas - Locally defined functions (C++11 feature). Lambdas will also be not allowed as function-pointer argument to C-style function. Following is erroneous.
RenderValues(1,20, [](int nValue)
{
   cout << nValue;
} );

But it is perfectly valid for ShowValues function template.

ShowValues(1,20, [](int nValue)
  {
   cout << nValue;
 });

Of-course, using lambda requires C++11 complaint compiler (VC10 and above, GCC 4.5 and above).

Interestingly, the function template may be crafted in different way - where you need not to pass a functor as function parameter. Instead, you can pass it as template argument itself.

template<typename TDisplayFunc>
void ShowValuesNew(int nStart, int nEnd)
{
   TDisplayFunc functor; // Create functor here

   for(;nStart<=nEnd; ++nStart)
      functor(nStart);
}

...
 
ShowValuesNew<DisplayHelper>(1,20); // 1 template, 2 function arguments

In this case, I have passed struct DisplayHelper as template type argument. The function itself now takes only two arguments. The creation of functor is now done by template function itself. The only disadvantage is that you can now only pass struct or classes, having operator() defined in it. You cannot pass a normal function to ShowValuesNew. You can however pass a lambda's type using decltype keyword.

auto DispLambda = [](int nValue)
{
   printf("%d ", nValue);
};

ShowValuesNew<decltype(DispLambda)>(1,20);

Since the type of any lambda is around std::function, which is a class type, and hence object creation (TDisplayFunc functor;) is allowed.

By now, you have realized that function-pointer approach is very restrictive. The only advantage is code-size reduction and possibility of putting a function into some library, and later call that function passing different callbacks. The callable callback is truly late-bound. Since the core function is defined at one place, compiler does not have much of liberty to optimize the code based on functions (callbacks) passed, especially of core-function resides in other library (DLL/SO). Of course, if the core function is large, and restrictive nature is desired/acceptable, you would use function-pointer approach.

Template based approach, on the other side, do advocate for early-binding. Early binding is the core and heart of template based programming. As mentioned before, template code would generally not be intensive and big, like huge data-processing, a gaming engine, batch image processing, security subsystem - but a helper for all these systems. Therefore, early-bound nature actually helps in optimizing the code, since everything is under the compiler-territory.

Templates and Virtual Functions

Virtual functions and templates don't go together - they are into different leagues. Reason is simple - One employs late-binder and other one employs early-binder. Do very well remember that templates are compile-time, unlike generics in other managed languages (like C#). The type of a generic is determined at runtime, depending on how it is instantiated. But in case of templates, type is determined at compile time only. There are many other differences, pros and cons of templates and generics, but I am deferring that explanation.

Just for a logical understanding of this separation, consider following code.

class Sample
{
public:
   template<class T>
   virtual void Processor()  // ERROR!
   {  
   }
};

It asks Sample::Processor<T> method template to be virtual, which does not make any sense. How the sample class it to be used and inherited. So, for example, if you make new class SampleEx and inherit it from Sample, and attempt implement this virtual function. Which specialization would you override? A Processor<int> or Processor<string>, for example?

Since there are possibilities of infinite specializations of Processor method that can be overridden, depending on how method-template Processor is being called (via base of any of derived classes) - the virtualkeyword loses its meaning. Compiler cannot create virtual-function-table for such design. Also, compiler cannot enforce derived class to implement all those infinite implementations, if base class declares the given method-template as pure-virtual!

ATL library, from Microsoft, uses template based design on top of inheritance principle - but without virtual functions. For performance reasons, it uses templates, and not virtual functions - it means ATL uses more of static-binding, rather than dynamic-binding.

How would you utilize template based classes, having inheritance, but without virtual-functions? And yet facilitate that base class would know and call methods of derived class?

Before I explicate that feature, do remember that such classes will not be complete without derived classes. By this I don't mean abstract classes or pure-virtuals. Class templates, as you know, gets compiled only when instantiated with particular type - and this rule holds true for all of the methods in class. Similar way, base class will not be complete without its partner in crime -derived class. It also means that such classes cannot be exported from a library, whereas normal classes (even abstract) may be exported from a library.

Let me start it with normal inheritance model - a base class, having pure virtual function, and a derived class implementing it.

class WorkerCore
{
public:
   void ProcessNumbers(int nStart, int nEnd)
   {
      for (;nStart<=nEnd; ++nStart)
      {
         ProcessOne(nStart);
      }
   }

   virtual void ProcessOne(int nNumber) = 0;
};

class ActualWorker : public WorkerCore
{
   void ProcessOne(int nNumber) 
   {
      cout << nNumber * nNumber;
   }
};

...
 
WorkerCore* pWorker =new ActualWorker;
pWorker->ProcessNumbers(1,200);

You know that WorkerCore class is abstract, and a pointer of this type may point to derived class. ProcessOne is the function that would be doing actual work. The binding with actual function (in ProcessNumbers) depends where this pointer is actually pointing. This is very-much utilizing late-binding feature of the language.

For this trivial task, you don't want heavy runtime penalty - you would prefer early-binding. And there the nifty feature, templates, come to rescue! Carefully understand the following code.

template<class TDerived>
class WorkerCoreT 
{
public:
   void ProcessNumbers(int nStart, int nEnd)
   {
     for (;nStart<=nEnd; ++nStart)
     {
        TDerived& tDerivedObj = (TDerived&)*this;
    
        tDerivedObj.ProcessOne(nStart);
     }
   }
};
 
class ActualWorkerT : public WorkerCoreT<ActualWorkerT>
{
public:
   void ProcessOne(int nNumber)
   {
      cout << nNumber * nNumber;
   }
};

First understand the bold ones:

  • TDerived in base class: Specifies the actual type of derived class. The derived class, when inheriting, must specify it.
  • Typecasting in ProcessNumbers: Since we only WorkerCoreT is actually a TDerived object, we can safely typecast this to TDerived. And then call ProcessOne method using the object-reference.
  • <ActualWorkerT> specification: The derived class itself tells the base that "Here I am". This line is important, otherwise type of TDerived would be wrong, and so the typecasting.

Important thing to know that ProcessOne is not a virtual function, not even a regular member in base class. The base class just assumes it exists in derived class, and makes a call to it. If ProcessOne doesn't exist in derived class, the compiler will simply raise an error:

  • 'ProcessOne' : is not a member of 'ActualWorkerT'

Even though there is typecasting involved, there is no runtime penalty involved, no runtime polymorphism, function-pointer drama etc. The said function exists in derived class, is accessible from base class, and is not restricted to be void (int). It could be, as mentioned in function-pointers section, int (float), or anything else that can be called with int parameter.

The only thing is that a pointer of type WorkerCoreT cannot simply point to derived class, and make a successful call to ProcessOne. And you can justify that such thing doesn't make sense - either take early binding, or late, not both.

STL - An Introduction

STL stands for Standard Template Library, which is a part of C++ Standard Library. From programmer's point of view, even though it is (optional) part of C++ Library, most of other features (classes, functions) are dependent on STL itself. As the "template" word suggests, STL is mainly on top of C++ templates - there are class templates and function templates.

STL contains set of collection classes for representing arrays, linked lists, trees, sets, maps etc. It also contains helper functions to act on container classes (like finding maximum, sum or a particular element), and other auxiliary functions. Iterators are important classes that allow iteration over collection classes. First let me give simple example.

vector<int> IntVector;

Here vector is a class template, which is functionally equivalent to arrays. It takes one (mandatory) argument - the type. The above statement declares IntVector to be a vector<> of type int. Few points:

  • vector, along with other elements of STL, comes under std namespace.
  • To use vector, you need to include vector header (and not vector.h)
  • vector stores its elements in contiguous memory- meaning that any element can be directly accessed. Yes, very much same as array.

Stepped up example:

#include <vector>
int main()
{
   std::vector<int> IntVector;
 
   IntVector.push_back(44);
   IntVector.push_back(60);
   IntVector.push_back(79);

   cout << "Elements in vector: " << IntVector.size();
}

About the bold-marked content:

  • The header that must be included to use vector class.
  • Namespace specification: std.
  • vector::push_back method is used to add elements to vector. Initially there are no elements in vector, you insert using push_back. Other techniques also exist, but push_back is paramount.
  • To determine current size (not capacity) of vector, we use vector::size method. Thus the program will display 3.

If you were to implement vector, you would implement it like:

template<typename Type>
class Vector
{ 
  Type* pElements; // Allocate dynamically, depending on demands.
  int ElementCount; // Number of elements in vector

public:
  Vector() : pElements(NULL), ElementCount(0)
  {}

  size_t size() const { return ElementCount; };

  void push_back(const Type& element); // Add element, allocate more if required.
};

No rocket science here, you know all of it. Implementation of push_back would be to allocate additional memory, if required, and set/add the element to given location. This rises obvious question: How much memory to allocate on each new-element insertion? And here comes the Capacity subject.

vector also has, not frequently used method: capacity. Capacity of vector is currently allocated memory (in element count), and can be retrieved using this function. Initial capacity, or the additional memory allocated on each push_back depends on implementation (how VC or GCC or other compiler vendors implement it). Method capacity will always return more or equal value, than the size method would return.

I request you to implement push_back and capacity methods. Add any more data members or methods you may want to add.

One major advantage of vector is that it can be used like standard array; except that the size of array (i.e. element count of vector) is not constant. It may vary. You may think of it as dynamically allocated array, where you allocate desired memory (re-allocate if needed), keep track of size of array, check for memory allocation failure and need to free memory at the end. std::vector handles it all, yet for all data-types that meet the "Requirements of this class template".

Having said that vector is functionally equivalent to array, following is valid code. (Yes, there has to be at least 3 elements in vector for this code to work).

IntVector[0] = 59; // Modify First element

cout << IntVector[1]; // Display Second element
 
int *pElement = &IntVector[2]; // Hold Third element
cout << *pElement; // Third

It clearly means that vector has operator[] overloaded, which is like:

Type& operator[](size_t nIndex)
{
    return pElements[nIndex]; 
}

const Type& operator[](size_t nIndex)
{
    return pElements[nIndex];
}

I have not shown the basic validations here. Important to note the two overloads based on const. Class std::vector also has the two overloads - one which returns the reference of actual element, other one returns const-reference. Former will allow modification of actual element stored(See "Modify First element" comment above), and latter will not allow modification.

What about showing all the elements of a vector? Well, the following code will work for vector<int> :

for (int nIndex = 0 ; nIndex < IntVector.size(); nIndex++)
{
   cout << "Element [" << nIndex << 
                    "] is " << IntVector[nIndex] << "\n";
}

Nothing important to explain here, until I explicate the flaws with this type of collection-iteration code. Anyway, we can utilize function template feature to write-up a function that can display any type of vector. Here it is:

template<typename VUType>  // Vector's Underlying type!
void DisplayVector(const std::vector<VUType>& lcVector)
{
   for (int nIndex = 0 ; nIndex < lcVector.size(); nIndex++)
   {
      cout << "Element [" << nIndex << "] is " << lcVector[nIndex] << "\n";
   }
}

Now this templated vector-iteration code can display any vector -vector<float> , vector<string> or vector<Currency>, as long as cout can display the type, or the underlying type can make it cout-able. Please understand the bold-marked content yourself!

Following code is added only for better grip and understanding.

...
IntVector.push_back(44);
IntVector.push_back(60);
IntVector.push_back(79);
 
DisplayVector(IntVector);  // DisplayVector<int>(const vector<int>&);

Would the implementation of DisplayVector work for all type containers, like sets and maps? It won't! I will cover up it soon.

Another container in STL is set. A set<> would store only unique elements of type T. You need to include <set> header to use it. An example:

std::set<int> IntSet;

IntSet.insert(16);
IntSet.insert(32);
IntSet.insert(16);
IntSet.insert(64);

cout << IntSet.size();

Usage is very similar to vector, except that you need to use insert method. Reason is simple and justified: New element may be placed anywhere in set, not just at the end - and you can't force an element to be inserted at end.

The output of this code snipped will be 3, and not 4. Value 16 is being inserted twice, and set will ignore the second insertion request. Only 16, 32 and 64 would exist in IntSet.

Well, this article is not about STL, but about templates. I briefed about set class also for a reason I am going to explain. You may find relevant documentation, articles, sample code etc on the net for STL. Use following keywords to search your favorites: vector, map, set, multimap, unordred_map, count_if, make_pair, tuple, for_each etc.

Let me bring attention to the subject I have to elaborate.

How would you iterate through all elements of a set ? Following code is not going to work for set.

for (int nIndex = 0; nIndex < IntSet.size(); nIndex)
{
   cout << IntSet[nIndex];  // ERROR!
}

Unlike vector class, set does not define operator[]. You cannot access any element based on its index - the index doesn't exist for set. The order of elements in set are ascending: from smaller to larger. There exist weak-strict ordering, comparer class etc, but lets consider his (ascending) as default behavior for the subject in hand.

So, at some point, if elements of set<int> are (40,60,80), and later you insert 70, the sequence of elements would become (40, 60, 70, 80). Therefore, logically, index is inappropriate for set.

And here comes another important facet of STL: Iterators. All container classes do have support for iterators, so that elements of collection can be iterated through. Different kind of iterators are represented by various classes. First let me present you a sample code to iterator a standard-array.

int IntArray[10] = {1,4,8,9,12,12,55,8,9};
 
for ( int* pDummyIterator = &IntArray[0]; // BEGIN 
      pDummyIterator <= &IntArray[9];   // Till LAST element
      pDummyIterator++)
{
    cout << *pDummyIterator << " ";
}

Using simple pointer arithmetic, the code is displaying values of all elements of array. Similarly, iterators can be used to iterate a vector:

vector<int>::iterator lcIter;

for (lcIter = IntVector.begin(); 
     lcIter != IntVector.end(); 
     ++lcIter)
{
    cout << (*lcIter);
}

Carefully understand about the bold-marked content:

  • iterator is a class. Specifically a typedef inside vector<int>. Thus, a variable of type vector<int>::iterator may only iterate a vector<int>, and not vector<float> or set<int>. How exactly iterator is typedef'd, shouldn't matter you or any STL programmer.
  • begin and end are methods that return iterator of same type. An instance of vector<Currency>would return vector<Currency>::iterator, when you invoke begin or end on it.
    • begin returns an iterator that points to the first element of container. Think it of as &IntArray[0].
    • Method end returns an iterator that points to the next-to-last element of container. Think it of as &IntArray[SIZE], where IntArray is of SIZE size. You know that, for size 10 array, &IntArray[10] would be (logically) pointing to next element of &IntArray[9].
    • The expression ++lcIter calls operator++ on iterator object, which moves the iterator to point to the next element of collection. It is very much same as ++ptr pointer arithmetic,.
    • The loop starts with iterator pointing to begin, and goes till it points to end.
  • The expression *lcIter calls the unary operator* on iterator, which returns the reference/const-reference of element currently pointed. For example above, it simply returns int.

You may not be able to grasp this complex iterator concept, so easily, so soon. You should regularly play with iterators - let the compiler bring you some weird errors, let your program crash down or disturb your debugger and cause assertions. More you bring these errors and assertions, the more you learn!

Exactly the same way, you may iterate a set:

set<int>::iterator lcIter;
for (lcIter = IntSet.begin(); 
     lcIter != IntSet.end(); 
     ++lcIter)
{
   cout << (*lcIter);
}

If I ask you to write up iteration-loop for:

  • vector<float>
  • set<Currency>
  • vector<Pair>

Soon you would realize you need to change only container-type and/or the underlying type and rest of the code remain same! You may tempt to write a function template, which would take the container and the underlying-type as its template type arguments. Something like:

template<typename Container, typename Type>
void DisplayCollection(const Container<Type>& lcContainer)
{
   Container<Type>::iterator lcIter;
   for (lcIter = lcContainer.begin();  lcIter != lcContainer.end();  ++lcIter)
   {
     cout << (*lcIter);
   } 
}

Seems logically correct, but that's not going to compile. Similar to DisplayVector function, this function attempts to take lcContainer argument, having Container as collection-class, and its underlying type as Type. It won't be easy to understand why it won't work, but it's not that much odd to understand why it won't work.

The syntax of DisplayVector was:

template<typename VUType>
void DisplayVector(const std::vector<VUType>& lcVector)

Where the actual type being passed to function is complete expression: vector<VUType>&. The type being passed was not just: vector&

The syntax of DisplayCollection is something like:

template<typename Container, typename Type>
void DisplayCollection(const Container<Type>& lcContainer)

Here the type being passed to function (not template) is complete: Container<Type>&. Suppose if could call it as:

vector<float> FloatVector;

DisplayCollection<vector, float>(FloatVector);

The (first) type being passed to template is just: vector, which is not a complete type. Some specialization of vector (like vector<float>) would make it qualified for being a complete type. Since first template (type) argument cannot be classified as type, we cannot use it that way. Though there exist techniques to pass class-template itself (like just vector), and make it complete type based on other arguments/aspects. Anyway, here is modified DisplayCollection prototype:

template<typename Container>
void DisplayCollection(const Container& lcContainer);

Yes, just that simple! But the implementation now demands some changes. So, lets implement it gradually.

template<typename Container>
void DisplayCollection(const Container& lcContainer)
{
   cout << "Items in collection: " << lcContainer.size() << "\n";
}

All STL containers do have size method implemented, and they do return size_t. So, irrespective which container is being passed (set, map, deque etc) - the method size will work.

The iteration of collection:

Container::const_iterator lcIter;

for (lcIter = lcContainer.begin(); 
     lcIter != lcContainer.end(); 
     ++lcIter)
{
    cout << (*lcIter);
}

Few things to learn:

Since the argument ( lcContainer) is being passed with const qualifier, it is rendered as non-mutable object in this function. It means you cannot insert, delete or (re)assign anything to the container. If a vector is being passed, lcContainer.push_back would be an error, since object is const. Further it means that you cannot iterate it using mutable iterator.

  • Using iterator class, you can change the contents. It is thus referred as mutable-iterator.
  • Use const_iterator when you don't need to change, or you cannot use mutable-iterator. When object/container itself is const (non-mutable), you must use const_iterator.
  • Important! An object of const_iterator is not same as constant object of iterator.
    That means: const_iterator != const iterator - note the space!

How the compiler would return iterator or const_iterator, when I am calling same methods: begin and end?

Valid question, and simple answer:

class SomeContainerClass
{
   iterator begin();
   iterator end();
 
   const_iterator begin() const;
   const_iterator end() const;
};

When object is const, the const version of method is called - simple C++ rule!

Now, one more important point to consider. The code given above, will not compile on all compilers (specifically the following line):

Container::const_iterator lcIter

Visual C++ 2008 compiles it fine, but GCC reports following error:

error: need 'typename' before 'Container::const_iterator' because 'Container' is a dependent scope

To understand the reasoning, consider following class:

class TypeNameTest
{
public:
   static int IteratorCount;
   typedef long IteratorCounter;
};

int main()
{
   TypeNameTest::IteratorCounter = 10; // ERROR
   TypeNameTest::IteratorCount var;    //ERROR
}

Using ClassName::Symbol notation, we can access both typedef symbol and and statically defined symbol. For simple classes, the compiler is able to distinguish, the programmer is able to resolve and there is no potential ambiguity.

But in case of template functions, where the the underlying type itself is dependent on template type argument (like const_iterator being based on Container), the compiler must be told that specified symbol is actually a type, and not a static symbol of given class. And there we use typename keyword to classify the symbol as a type.

Therefore we should (must) use typename keyword:

typename Container::const_iterator lcIter; // const_iterator is a type, not static symbol in Container

Why can't we use class keyword instead, and why does VC++ compile it fine?
Ah! Compiler vendors and their tendency to follow some non-standards. VC++ doesn't need typename, GCC needs it (and tells you!). GCC will even accept class in-place of typename, VC will not accept class. But thankfully, both adhere to the standards and do accept typename keyword!

The new C++ standard (C++11) brings up some respite, specially while working with STL, templates and complicated iterator definitions. The auto keyword. The iteration loop can be modified as:

for (auto lcIter = lcContainer.begin();  lcIter != lcContainer.end();  ++lcIter)
{
   cout << (*lcIter);
}

The actual type of lcIter will be determined automatically, at compile-time. You may read about autokeyword on the Internet, your favorite author's book or may refer my article.

Templates and Library Development

As you know that the template code doesn't directly go to object file, it gets compiled (second phase compilation) only when instantiated with appropriate template parameters. Since the actual code generation (i.e. specialization) happens only by instantiating a function/class template with appropriate template-arguments, the function/class template cannot be exported through a library.

When you attempt to export a function template, such as DisplayCollection, through a library (.LIB,.DLL or .SO), the compiler and linker may depict that it exported the given function. Linker may throw a warning or error that some symbols (e.g. DisplayCollection) was not exported or not found. Since there were no call to function template(s) in the library itself, no actual code was generated, and thus nothing actually was exported.

When you later use that library in other project, you would get set of linker errors that some symbols were not found. To recollect this problem, read this section again.

It is therefore not possible to export template code from a library, without disclosing the source code, and delivering it (generally via header-files). Though, it is very much possible to expose source-code for only the templated-stuff, and not the core library stuff, which may be playing with void-pointers, and sizeof keyword. The core library may actually be made private by exporting it from the library, since core stuff may not be template-based.

External Templates, a feature still pending to be part of C++ standard, is not supported by major compilers.

Some libraries may export entire class for particular template arguments using Explicit Instantiation feature.

Explicit Instantiation

This feature is particularly important if you are exposing your template-based library, either by header-only implementation, or through wrapper-mode implementation (hiding core, but exposing features via templates). With Explicit Instantiation, you can instruct the compiler (and thus, the linker) to generate the code for specific template arguments. It means you are demanding a specialization of template, without actually instantiating it in your code. Consider a simple example.

template class Pair<int, int>;

This statement simply asks the compiler to instantiate Pair with <int,int> arguments for all method of Pairclass. That means, compiler will generate code for:

  • Data-members - first and second.
  • All three constructors (as mentioned in this and previous article).
  • Operators > and == (as mentioned).

To verify this, you may look at the generated binary (executable, DLL/SO), using appropriate tool On Windows, you may use Dependency Walker to see if code was generated or not. A simpler method exist to assert if compiler/linker are actually performing explicit instantiation - Let the compiler break on failure. For example:

template struct Pair<Currency, int>;

Would make the type of first as Currency. Compiler will attempt to generate code for all methods, and would fail on operator ==, saying that it (Currency) doesn't have operator defined:

bool operator == (const Pair<Type1, Type2>& Other) const
{
 return first == Other.first &&  // ERROR Currency.operator== isn't available.
        second == Other.second;
}

It fails on this method, just because it comes before any other method that fails (before operator< in this case).

This was just the example to check if compiler is actually generating code for all methods or not. But the main question is: Why would you want to utilize this feature?

For example, you expose a string class (like std::string, or CString). And that class is on top of a template argument - the character type - ANSI or Unicode. A very simple definition of String class template:

template<typename CharType>
class String
{
    CharType m_str[1024];
public:
    CharType operator[](size_t nIndex)
    {
       return m_str[nIndex];
    }
 
    size_t GetLength() const
    {
        size_t nIndex = 0;
        while(m_str[nIndex++]);

        return nIndex;
    }
};

And a pretty simple usage example:

String<char> str;
str.GetLength();

And you know that it will produce only following:

  • String<char>::m_str
  • String<char>::String - default compiler provided constructor.
  • String<char>::GetLength method

If you were to put String into some library, you may put entire String class into header, and ship the header. Here is question is not at all about private-stuff, encapsulation etc, it about unnecessary increase in size of different executables produced.

There would be thousands of binaries (DLLs, SOs, executables), and almost all of them would be using Stringclass. Wouldn't it be better if you could pack them into one library? Yes, I mean the non-templated traditional approach?

To do so, you just ask explicit instantiation for all types you are supposed to export through library.

template class String<char>;
template class String<wchar_t>;

For programmer's convinience, you may typedef different String types. The std::string type is, in fact, typedef'd and exported in this manner:

typedef basic_string<char, ... > string;
typedef basic_string<wchar_t, ... > wstring;

// Explicit Instantiation
template class /*ATTR*/ basic_string<char, ...>;
template class /*ATTR*/ basic_string<wchar_t, ... >;

The base class is basic_string, which is class template. Few arguments not shown here only for simplicity, and vendors may have different signature of reset of template arguments (for basic_string). Second group shows explicit instantiations for these types. The commented part, /*ATTR*/ - would depend on compiler vendor. It may be an expression that these instantiations do actually go in library being compiled, or are only acting as header-only. In VC++ implementation, these two instantiations are actually in a DLL.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

An Idiot's Guide to C++ Templates - Part 1

 

Prolusion

Most C++ programmers stay away from C++ templates due to their perplexed nature. The excuses against templates:

  • Hard to learn and adapt.
  • Compiler errors are vague, and very long.
  • Not worth the effort.

Admitted that templates are slightly hard to learn, understand, and adapt. Nevertheless, the advantages we gain from using templates would outweigh the negatives. There is a lot more than generic functions or classes that can be wrapped around templates. I would explicate them.

While C++ templates and STL (Standard Template Library) are siblings, technically. In this article, I would only cover templates at the core level. Next parts of this series would cover more advanced and interesting stuff around templates, and some know-how about STL.

Table of Contents

The Syntax Drama

As you probably know, template largely uses the angle brackets: The less than ( < ) and the greater than ( > ) operators. For templates, they are always used together in this form:

< Content >

Where Content can be:

  1. class T / typename T
  2. A data type, which maps to T
  3. An integral specification
  4. An integral constant/pointer/reference which maps to specification mentioned above.

For point 1 and 2, the symbol T is nothing but some data-type, which can be any data-type - a basic datatype (int, double etc), or a UDT.

Let's jump to an example. Suppose you write a function that prints double (twice) of a number:

void PrintTwice(int data)
{
    cout << "Twice is: " << data * 2 << endl;         
}

Which can be called passing an int:

PrintTwice(120); // 240

Now, if you want to print double of a double, you would overload this function as:

void PrintTwice(double data)
{
    cout << "Twice is: " << data * 2 << endl;         
}

Interestingly, class type ostream (the type of cout object) has multiple overloads for operator << - for all basic data-types. Therefore, same/similar code works for both int and double, and no change is required for our PrintTwice overloads - yes, we just copy-pasted it. Had we used one of printf-functions, the two overloads would look like:

void PrintTwice(int data)
{
    printf("Twice is: %d", data * 2 );
}

void PrintTwice(double data)
{
    printf("Twice is: %lf", data * 2 );
}

Here the point is not about cout or print to display on console, but about the code - which is absolutely same. This is one of the many situations where we can utilize the groovy feature provided by the C++ language: Templates!

Templates are of two types:

  • Function Templates
  • Class Templates

C++ templates is a programming model that allows plugging-in of any data-type to the code (templated code). Without template, you would need to replicate same code all over again and again, for all required data-types. And obviously, as said before, it requires code maintenance.

Anyway, here is the simplified PrintTwice, utilizing templates:

void PrintTwice(TYPE data)
{
    cout<<"Twice: " << data * 2 << endl;
}

Here, actual type of TYPE would be deduced (determined) by the compiler depending on argument passed to the function. If PrintTwice is called as PrintTwice(144); it would be an int, if you pass 3.14 to this function, TYPE would be deduced as double type.

You might be confused what TYPE is, how the compiler is going to determine that this is a function template. Is TYPE type defined using typedef keyword somewhere?

No, my boy! Here we use the keyword template to let compiler know that we are defining a function template.

Function Templates

Here is the templated function PrintTwice:

template<class TYPE>
void PrintTwice(TYPE data)
{
    cout<<"Twice: " << data * 2 << endl;
}

The first line of code:

template<class TYPE>

tells the compiler that this is a function-template. The actual meaning of TYPE would be deduced by compiler depending on the argument passed to this function. Here, the name, TYPE is known as template type parameter.

For instance, if we call the function as:

PrintTwice(124);

TYPE would be replaced by compiler as int, and compiler would instantiate this template-function as:

void PrintTwice(int data)
{
    cout<<"Twice: " << data * 2 << endl;
}

And, if we call this function as:

PrintTwice(4.5547);

It would instantiate another function as:

void PrintTwice(double data)
{
    cout<<"Twice: " << data * 2 << endl;
}

It means, in your program, if you call PrintTwice function with int and double parameter types, two instances of this function would be generated by compiler:

void PrintTwice(int data) { ... }
void PrintTwice(double data) { ... }

Yes, the code is duplicated. But these two overloads are instantiated by the compiler and not by the programmer. The true benefit is that you need not to do copy-pasting the same code, or to manually maintain the code for different data-types, or to write up a new overload for new data-type that arrives later. You would just provide a template of a function, and rest would be managed by compiler.

It is also true that code size would increase, since there are now two function definitions. The code-size (at binary/assembly level) would almost be same. Effectively, for N number of data-types, N instances of same function (i.e. overloaded functions) would be created. There are advanced compiler/linker level optimizations which can somewhat reduce the code size, if instantiated functions are same, or some part of function body is same. I wouldn't discuss it now.

But, on a positive side, when you manually define N different overloads (say N=10), those N different overloads would be anyway be compiled,linked and packed in binary (the executable). However, with templates, only the required instantiations of function would get into final executable. With templates, the overloaded copies of function might be less than N, and it can be more than N - but exactly the number of required copies - no more no less!

Also, for non-templated implementations, the compiler has to compile all those N copies - since they are in your source-code! When you attach template with a generic function, compiler would compile only for required set of data-types. It basically means the compilation would be faster if number of different data-types is less than N!

It would be a perfectly valid argument, that the compiler/linker would likely do all possible optimizations to remove unused non-template function' implementations from the final image. But, again, do understand that compiler has to compile all those overloads (for syntax checking etc). With templates, the compilation would happen only for required data-types - you can call it as "On demand compilation".

Enough of text-only content for now! You can come back and re-read it again. Let's move ahead.

Now, let's write another function template that would return the twice of given number:

template<typename TYPE>
TYPE Twice(TYPE data)
{
   return data * 2;
}

You should have noticed that I used typename, instead of class. No, it is not required to use typenamekeyword if a function returning something. For template programming, these two keywords are very much the same. There is a historical reason for having two keywords for same purpose, and I hate history.

However, there are instances where you can only use the newer keyword - typename. (When a particular type is defined in another type, and is dependent on some template parameter - Let this discussion be deferred to another part).

Moving ahead. When we call this function as:

cout << Twice(10);
cout << Twice(3.14);
cout << Twice( Twice(55) );

Following set of functions would be generated:

int     Twice(int data) {..}
double  Twice(double data) {..}

Two things:

  • In third line of code snipped above, Twice is called twice - the return value/type of first call would be the argument/type of second call. Hence, both calls are of int type (Since argument type TYPE, and return type are same).
  • If a template function is instantiated for a particular data-type, compiler would re-use the same function' instance - if the function is invoked again for same data-type. It means, irrespective of where, in your code, you invoke the function template with same type - in same function, in different function, or anywhere in another source file (of same project/build).

Let's write up a function template that would return the addition of two numbers:

template<class T>
T Add(T n1, T n2)
{
    return n1 + n2;
}

Firstly, I just replaced template-type parameter's name - TYPE with symbol T. In template programming, you would generally use T - but that's a personal choice. You should better use a name that reflects the meaning of type-parameter, and that improves code readability. This symbol can be any name which follows variable naming rules in the C++ language.

Secondly, I re-used the template parameter T - for both of arguments (n1 and n2).

Let's slightly modify Add function, which would store the addition in local variable and then return the calculated value.

template<class T>
T Add(T n1, T n2)
{
    T result;
    result = n1 + n2;
    
    return result;
}

Quite explanatory, I used the type parameter T within the function's body. You might ask (you should): "How the compiler would know what is type of result, when it tries to compile/parse the function Add?"

Well, while looking at the body of function template (Add), compiler would not see if T (template type parameter) is correct or not. It would simply check for basic syntax (such as semi-colons, proper usage of keywords, matching braces etc), and would report errors for those basic checks. Again, it depends on compiler to compiler how it handles the template code - but it would not report any errors resulting due to template type parameters.

Just for completeness, I would reiterate, compiler will not check if (currently relevant only for function Add):

  • T is having a default constructor (so that T result; is valid)
  • T supports the usage of operator + (so that <code>n1+n2 is valid)
  • T has an accessible copy/move-constructor (so that return statement succeeds)

Essentially, the compiler would have to compile the template code in two phases: Once for basic syntax checks; and later for each instantiation of function template - where it would perform actual code compilation against the template data-types.

It is perfectly okay if you did not completely understood this two phase compilation process. You would get firm understanding as you read through this tutorial, and then you would come back to read these theory-sessions later!

Pointers, References and Arrays with Templates

First a code sample (No worries - it is simple code snippet!):

template<class T>
double GetAverage(T tArray[], int nElements)
{
    T tSum = T(); // tSum = 0

    for (int nIndex = 0; nIndex < nElements; ++nIndex)
    {
        tSum += tArray[nIndex];
    }

    // Whatever type of T is, convert to double
    return double(tSum) / nElements;
}
  

int main()
{
    int  IntArray[5] = {100, 200, 400, 500, 1000};
    float FloatArray[3] = { 1.55f, 5.44f, 12.36f};

    cout << GetAverage(IntArray, 5);
    cout << GetAverage(FloatArray, 3);
}

For the first call of GetAverage, where IntArray is passed, compiler would instantiate this function as:

double GetAverage(int tArray[], int nElements);

And similarly for float. The return type is kept as double since average of numbers would logically fit in double data-type. Note that this is just for this example - the actual data-type that comes under T may be a class, which may not be converted to double.

You should notice that a function template may have template type arguments, along with non-template type arguments. It it not required to have all arguments of a function template to arrive from template types. int nElements is such function argument.

Clearly note and understand that template type-parameter is just T, and not T* or T[] - compilers are smart enough to deduce the type int from an int[] (or int*). In the example given above, I have used T tArray[] as an argument to function template, and actual data-type of T would intelligently be determined from this.

Most often, you would come across and would also require to use initialization like:

T tSum = T();

First thing first, this is not template specific code - this comes under C++ language itself. It essentially means: Call the default constructor for this datatype. For int, it would be:

int tSum = int();

Which effectively initializes the variable with 0. Similarly, for float, it would set this variable to 0.0f. Though not covered yet, if a user defined class type comes from T, it would call the default constructor of that class (If callable, otherwise relevant error). As you can understand that T might be any data-type, we cannot initialize tSum simply with an integer zero (0). In real case, it may be a some string class, which initializes it with empty string ("").

Since the template type T may be any type, it must also have += operator available. As we know, it is available for all basic data types (int, float, char etc.). If actual type (for T), doesn't have += operator available (or any possibility), compiler would raise an error that actual type doesn't have this operator, or any possible conversion.

Similarly, the type T must be able to convert itself to double (see the return statement). I will cover up these nitty-gritties, later. Just for better understanding, I am re-listing the required support from type T (now, only applicable for GetAverage function template):

  • Must have an accessible default constructor.
  • Must have += operator callable.
  • Must be able to convert itself to double (or equivalent).

For GetAverage function template prototype, you may use T* instead of T[], and would mean the same:

template<class T>
GetAverage(T* tArray, int nElements){}

Since the caller would be passing an array (allocated on stack or heap), or an address of a variable of type T. But, as you should be aware, these rules comes under rule-book of C++, and not specifically from template programming!

Moving ahead. Let's ask the actor 'reference' to come into template programming flick. Quite self-explanatory now, you just use T& as a function template argument for the underlying type T:

template<class T>
void TwiceIt(T& tData)
{
    tData *= 2;    
    // tData = tData + tData;
}

Which calculates the twice value of argument, and puts into same argument' value. You would call it simply as:

int x = 40;
TwiceIt(x); // Result comes as 80

Note that I used operator *= to get twice of argument tData. You may also use operator + to gain the same effect. For basic data-types, both operators are available. For class type, not both operators would be available, and you might ask the class' to implement required operator.

In my opinion, it is logical to ask operator + be defined by class. The reason is simple - doing T+T is more appropriate for most UDTs (User Defined Type), than having *= operator . Ask yourself: What does it mean if some class String or Date implements, or is asked, to implement following operator:

void operator *= (int); // void return type is for simplicity only.

At this point, you now clearly understand that template parameter type T may be inferred from T&, T* or T[].Therefore, it is also possible and very reasonable to add const attribute to the parameter which is arriving to function template, and that parameter would not be changed by function template. Take it easy, it is as simple as this:

template<class TYPE>
void PrintTwice(const TYPE& data)
{
    cout<<"Twice: " << data * 2 << endl;
}

Observe that I modified the template parameter TYPE to TYPE&, and also added const to it. Few or most of the readers would have realized the importance of this change. For those who didn't:

  • The TYPE type may be large in size, and would demand more space on stack (call-stack). It includes double which requires 8 bytes*, some structure or a class, which would demand more bytes to be kept on stack. It essentially means - a new object of given type would be created, copy constructor called, and be put into call stack, followed by destructor call at then end of function.
    Addition of reference (&) avoids all this - reference of same object is passed.
  • Function would not change the argument passed, and therefore addition of const to it. It ensures, to the caller of function, that this function (here PrintTwice), is not going to change the parameter's value. It also ensures a compiler error if, by mistake, the function itself tries to modify content of (constant) argument.
* On 32-bit platform, function arguments would require 4-bytes minimum, and in multiple of 4-bytes. This means a char or short would require 4 bytes in call-stack. An 11-byte object, for example would require 12-bytes in stack.
Similarly, for 64-bit platform, 8-bytes would be needed. An 11-byte object would require 16-bytes. Argument of type double would need 8-bytes.
All pointers/references takes 4-bytes/8-bytes respectively on 32-bit/64-bit platform, and therefore passing double or double& would mean the same for 64-bit platform.

And similarly, we should change other function templates as:

template<class TYPE>
TYPE Twice(const TYPE& data) // No change for return type
{
   return data * 2;
}

template<class T>
T Add(const T& n1, const T& n2) // No return type change
{
    return n1 + n2;
}

template<class T>
GetAverage(const T tArray[], int nElements)
// GetAverage(const T* tArray, int nElements)
{}

Note that it is not possible to have reference and const added to return type, unless we intend to return reference (or pointer) of original object that was passed to the function template. The following code exemplifies it:

template<class T>
T& GetMax(T& t1, T& t2)
{
    if (t1 > t2)
    {
        return t2;
    }
    // else 
    return t2;
}

It is how we utilize return reference:

int x = 50;
int y = 64;

// Set the max value to zero (0)
GetMax(x,y) = 0;

Note that this is just for illustration, and you would rarely see or write such code. You may, however see such code and may need to write, if the returned object is reference of some UDT. In that case member access operator dot (.) or arrow (->) would follow the function call. Anyway, this function template returns the reference of object which wins the greater-than race. This, definitely, requires operator > be defined by type T.

You should have noticed, I have not added const to any of two parameters passed. This is required; since function returns non-const reference of type T. Had it been like:

T& GetMax(const T& t1, const T& t2)

At the return statements, compiler would complain that t1 or t2 cannot be converted to non-const. If we add const to return type also ( const T& GetMax(...) ), following line at call site would fail to compile:

GetMax(x,y) = 0;

Since const object cannot be modified! You can definitely do forceful const/non-const typecasting, either in function or at call site. But that's a different aspect, a bad design and a non-recommended approach.

Multiple Types with Function Templates

Till now I have covered only one type as template type parameters. With templates, you may have more than one template-type parameters. It goes like:

template<class T1, class T2, ... >

Where T1 and T2 are type-names to the function template. You may use any other specific name, rather than T1, T2. Note that the usage of '...' above does not mean that this template specification can take any number of arguments. It is just illustration that template may have any number of arguments.
(As with C++11 standard, templates would allow variable number of arguments - but that thing is out of topic, for now.)

Let's have a simple example taking two template parameters:

template<class T1, class T2>
void PrintNumbers(const T1& t1Data, const T2& t2Data)
{
     cout << "First value:" << t1Data;
     cout << "Second value:" << t2Data;
}

And we can simply call it as:

PrintNumbers(10, 100);    // int, int
PrintNumbers(14, 14.5);   // int, double
PrintNumbers(59.66, 150); // double, int

Where each call demands separate template instantiation for the first and second types being passed (or say inferred). Therefore, following three function template instances would be populated by compiler:

// const and reference removed for simplicity
void PrintNumbers(int t1Data, int t2Data);
void PrintNumbers(int t1Data, double t2Data);
void PrintNumbers(double t1Data, int t2Data);

Realize that second and third instantiations are not same, as T1 and T2 would infer different data-types (int, double and double,int). Compiler will not perform any automatic conversion, as it might do for normal function call - A normal function taking int, for example, may be passed short, or vice-versa. But with templates, if you pass short - it is absolutely short, not (upgraded to) int. So, if you pass (short, int), (short, short), (long, int) - this would result in three different instantiations for PrintNumbers!

In similar fashion, function templates may have 3 or more type parameters, and each of them would map to the argument types specified in function call. As an example, the following function template is legal:

template<class T1, class T2, class T3>
T2 DoSomething(const T1 tArray[], T2 tDefaultValue, T3& tResult)
{
   ... 
}

Where T1 specifies the type of array that would be passed by caller. If array (or pointer) is not passed, compiler will render appropriate error. The type T2 is used as return type as well as second argument, which is passed by value. Type T3 is passed as reference (a non-const reference). This function template example given above is just haphazardly chosen, but is a valid function template specification.


By now, I have stepped-up and elaborated multiple template parameters. But for a reason, I am stepping-down to one parameter function. There is a reason for this, and you will understand it in no time.Assume that there is a function (non templated), which takes an int argument:

void Show(int nData);

And you call it as:

Show( 120 );    // 1
Show( 'X' );    // 2
Show( 55.64 );  // 3
  • Call 1 is perfectly valid since function takes int argument, and we are passing 120.
  • Call 2 is valid call since we are passing char, which will be promoted by compiler to int.
  • Call 3 would demand demotion of value - compiler has to convert double to int, and hence 55 would be passed instead of 55.64. Yes, this will trigger appropriate compiler warning.

One solution is to modify the function such that it takes double, where all three types can be passed. But that wouldn't support all types and may not fit in, or convertible to, double. Therefore, you may write set of overloaded functions, taking appropriate types. Armed with knowledge, now, you would appreciate the importance of templates, and would ask to write it a function template instead:

template<class Type>
void Show(Type tData) {}

Of course, assuming all existing overloads of Show were doing the same thing.

Alright, you know this drill. So, what's new in this which caused me to step-down?Well, what if you wanted to pass int to function template Show, but wish the compiler instantiates as if doublewas passed?

// This will produce (instantiate) 'Show(int)'
Show ( 1234 );

// But you want it to produce 'Show(double)'

May seem illogical to demand this thing, as of now. But there is valid reason to demand such instantiation, which you will understand and appreciate soon!

Anyway, first see how to demand such absurd thing:

Show<double> ( 1234 );

Which instantiates the following template function (as you know):

void Show(double);

With this special syntax (Show<>()), you are demanding compiler to instantiate Show function for the type being explicitly passed, and asking the compiler not to deduce type by function argument.

Function Template - Template Function

Important! There is a difference between function template and template function.

A function template is body of a function that is bracketed around template keyword, which is not an actual function, and will not be fully compiled by compiler, and is not accountable by the linker. At least one call, for particular data-type(s) is needed to instantiate it, and be put into accountability of compiler and linker. Therefore, the instance of function template Show is instantiated as Show(int) or Show(double).

A template function? Simply put, an "instance of a function template", which is produced when you call it, or cause it to get instantiated for particular data type. The instance of function-template is actually a valid function.

An instance of a function template (aka template-function) is not a normal function, under the umbrella of name-decoration system of compiler and linker. That means, an instance of function-template:

template<class T> 
void Show(T data) 
{ }

for template argument double, it is not:

void Show(double data){}

but actually:

void Show<double>(double x){}

For long, I did not uncover this, only for simplicity, and now you know it! Use your compiler/debugger to find out the actual instantiation of a function template, and see the full prototype of a function in call-stack or generated code.

And hence, now you know the mapping between these two:

Show<double>(1234);
...
void Show<double>(double data); // Note that data=1234.00, in this case!

Explicit Template Argument Specification

Stepping back (up) to the multiple template argument discussion.

We have following function template:

template<class T1, class T2>
void PrintNumbers(const T1& t1Data, const T2& t2Data)
{}

And have following function calls, causing 3 different instances of this function template:

PrintNumbers(10, 100);    // int, int
PrintNumbers(14, 14.5);   // int, double
PrintNumbers(59.66, 150); // double, int

And what if you just needed only one instance - both arguments taking double? Yes, you are willing to pass ints and let them be promoted doubles. Coupled with the understanding you just gained, you would call this function-template as:

PrintNumbers<double, double>(10, 100);    // int, int
PrintNumbers<double, double>(14, 14.5);   // int, double
PrintNumbers<double, double>(59.66, 150); // double, int

Which would produce only the following template function:

void PrintNumbers<double, double>(const double& t1Data, const T2& t2Data)
{}

And the concept of passing template type parameters this way, from the call-site, is known as Explicit Template Argument Specification.

Why would you need explicit type specification? Well, there are multiple reasons:

  • You want only specific type to be passed, and not let the compiler intelligently deduce one or more template argument types solely by the actual arguments (function parameters).

For example, there is one function template, max, taking two arguments (via only one template type parameter):

template<class T>
T max(T t1, T t2)
{
   if (t1 > t2)
      return t1;
   return t2;
}

And you attempt to call it as:

max(120, 14.55);

It would cause a compiler error, mentioning there is an ambiguity with template-type T. You are asking the compiler to deduce one type, from two types! One solution is to change max template so that it takes two template parameters - but you aren't author of that function template.

There you use explicit argument specification:

max<double>(120, 14.55); // Instantiates max<double>(double,double);

Undoubtedly notice and understand that I have passed explicit specification only for first template parameter, the second type is deduced from second argument of function call.

  • When function-template takes template-type, but not from its function arguments.

A simple example:

template<class T>
void PrintSize()
{
   cout << "Size of this type:" << sizeof(T);
}

You cannot call such function template simply as:

PrintSize();

Since this function template would require template type argument specification, and it cannot be deduced automatically by compiler. The correct call would be:

PrintSize<float>();

which would instantiate PrintSize with float template argument.

  • When function template has a return type, which cannot be deduced from arguments, or when function template doesn't have any argument.

An example:

template<class T>
T SumOfNumbers(int a, int b)
{
   T t = T(); // Call default CTOR for T

   t = T(a)+b;

 
   return t;
}

Which takes two ints, and sums them up. Though, summing them in int itself is appropriate, this function template gives opportunity to calculate the sum (using operator+) in any type as required by caller. For example, the get the result in double, you would call it as:

double nSum;
nSum = SumOfNumbers<double>(120,200);

The last two are just simplified examples for completeness, just to give you the hint where Explicit Template Argument Specification would fit in. There are more concrete scenarios where this explicitness would be needed, and would be covered in next part.

Default Arguments with Function Templates

For readers, who do know about default template type specification in template' arena - this is not about default template-type arguments. Default template-types, is anyway, not allowed with function-templates. For readers, who do not know about it, do not worry - this paragraph is not about default template type specification.

As you know, a C++ function may have default arguments. The default-ness may only go from right to left, meaning, if nth argument is required to be default, (n+1)th must also be default, and so on till last argument of function.

A simple example to explicate this:

template<class T>
void PrintNumbers(T array[], int array_size, T filter = T())
{
   for(int nIndex = 0; nIndex < array_size; ++nIndex)
   {
       if ( array[nIndex] != filter) // Print if not filtered
           cout << array[nIndex];
   }
}

This function template would print, as you can guess, all numbers except which are filtered out by third argument: filter. The last, optional function argument, is defaulted to default-value of type T, which, for all basic type would mean zero. Therefore, when you call it as:

int Array[10] = {1,2,0,3,4,2,5,6,0,7};
PrintNumbers(Array, 10);

It would be instantiated as:

void PrintNumbers(int array[], int array_size, int filter = int())
{}

The filter argument would be rendered as: int filter = 0.

As obvious, when you call it as:

PrintNumbers(Array, 10, 2);

The third argument gets value 2, and not the default value 0.

It should be clearly understood that:

  • Type T must have default constructor available. And of course, all operators as may be required by function body, for type T.
  • The default argument must be deducible from the other non-default types the template takes. In PrintNumbers example, type of array would facilitate deduction for filter.
    If not, you must use explicit template argument specification to specify type of default argument.

For sure, the default argument may not necessarily be default value for type T (pardon the pun). It means, the default-argument may not always need to depend on default-constructor of type T:

template<class T>
void PrintNumbers(T array[], int array_size, T filter = T(60))

Here, the default function argument doesn't use default-value for type T. Instead, it uses value 60. This, for sure, requires the type T to have copy-constructor which accepts an int (for 60).

Finally, here comes an end to 'Function Templates' for this part of article. I assume you enjoyed reading and grasping these basics of function templates. Next part would cover more intriguing aspects of Template programming.

Class Templates

More often, you would design and use class templates, than function templates. In general, you use a class template to define an abstract type whose behavior is generic and is reusable, adaptable. While some text would start by giving example about data-structures like linked-lists, stacks, queues and similar containers. I would start with very basic examples, that are easy to understand.

Let's take a simple class, which sets, gets and prints the value stored:

class Item
{
    int Data;
public:
    Item() : Data(0)
    {}

    void SetData(int nValue)
    { 
        Data = nValue;
    }

    int GetData() const
    {
        return Data;
    }

    void PrintData()
    {
        cout << Data;
    }
};

One constructor which initializes Data to 0, Set and Get methods, and a method to print current value. Usage is also quite simple:

Item item1;
item1.SetData(120);
item1.PrintData(); // Shows 120

Nothing new for you, for sure! But when you need similar abstraction for other data-type, you need to duplicate code of entire class (or at least the required methods). It incurs code maintenance issues, increases code size at source code as well as at binary level.

Yes, I can sense your intelligence that I am going to mention C++ templates! The templated version of the same class in form of class template is as below:

template<class T>
class Item
{
    T Data;
public:
    Item() : Data( T() )
    {}

    void SetData(T nValue)
    {
        Data = nValue;
    }

    T GetData() const
    {
        return Data;
    }

    void PrintData()
    {
        cout << Data;
    }
};

The class template declaration starts with same syntax as function templates:

template<class T>
class Item

Note that the keyword class is used two times - firstly to specify template type specification (T), and secondly to specify that this is a C++ class declaration.

To completely turn Item into a class template, I replaced all instances of int with T. I also used T() syntax to call default constructor of T, instead of hard-coded 0 (zero), in the constructor's initializer' list. If you've read function templates section completely, you know the reason!

And usage as also quite simple:

Item<int> item1;
item1.SetData(120);
item1.PrintData();

Unlike function template instantiation, where arguments of function itself helps the compiler to deduce template type arguments, with class templates you must explicitly pass template type (in angle brackets).

The code snippet shown above causes class template Item to instantiate as Item<int>. When you create another object with different type using Item class template as:

Item<float> item2;
float n = item2.GetData();

It would cause Item<float> to get instantiated. It is important to know that there is absolutely no relation between two instantiations of class template - Item<int> and Item<float>. For the compiler and linker, these two are different entities - or say, different classes.

First instantiation with type int produces following methods:

  • Item<int>::Item() constructor
  • SetData and PrintData methods for type int

Similarly, second instantiation with type float would produce:

  • Item<float>::Item() constructor
  • GetData method for float type

As you know Item<int> and Item<float> are two different classes/types; and therefore, following code will not work:

item1 = item2; // ERROR : Item<float> to Item<int>

Since both types are different, the compiler will not call possible default assignment operator. Had item1 and item2 were of same types (say both of Item<int>), the compiler would happily call assignment operator. Though, for the compiler, conversion between int and float is possible, it is not possible for different UDT conversions, even if underlying data members are same - this is simple C++ rule.

At this point, clearly understand that only following set of methods would get instantiated:

  • Item<int>::Item() - constructor
  • void Item<int>::SetData(int) method
  • void Item<int>::PrintData() const method
  • Item<float>::Item() - constructor
  • float Item<float>::GetData() const method

The following methods will not get second phase compilation:

  • int Item<int>::GetData() const
  • void Item<float>::SetData(float)
  • void Item<float>::PrintData() const

Now, what is a second phase compilation? Well, as I already elaborated that template-code would be compiled for basic syntax checks, irrespective of it being called/instantiated or not. This is known as first-phase compilation.

When you actually call, or somehow trigger it to be called, the function/method for the particular type(s) - then only it gets special treatment of second-phase compilation. Only through the second phase compilation, the code actually gets fully-compiled, against the type for which it is being instantiated.

Though, I could have elaborated this earlier, but this place is appropriate. How do you find out if function is getting first-phase and/or second-phase compilation?

Let's do something weird:

T GetData() const
 { 
  for())

  return Data;
 }

There is an extra parenthesis at the end of for - which is incorrect. When you compile it, you would get host of errors, irrespective of it being called or not. I have checked it using Visual C++ and GCC compilers, and both complain. This validates first-phase compilation.

Let's slightly change this to:

T GetData() const
{ 
  T temp = Data[0]; // Index access ?
  return Data;
}

Now compile it without calling GetData method for any type - and there won't be any quench from the compiler. This means, at this point, this function doesn't get phase-two compilation treatment!

As soon as you call:

Item<double> item3;
item2.GetData();

you would get error from compiler that Data is not an array or pointer, which could have operartor []attached to it. It proves that only selected functions would get special privilege of phase-two compilation. And this phase-two compilation would happen separately for all unique types you instantiate class/function template for.

One interesting thing, you can do is:

T GetData() const
{ 
  return Data % 10;
}

Which would get successfully compiled for Item<int>, but would fail for Item<float>:

item1.GetData(); // item1 is Item<int>

// ERROR
item2.GetData(); // item2 is Item<float>

Since operator % is not applicable for float type. Isn't it interesting?

Multiple Types with Class Templates

Our first class-template Item had only one template type. Now let's construct a class that would have two template-type arguments. Again, there could have been somewhat complex class template example, I would like to keep it simple.

At times, you do require some native structure to keep few data members. Crafting a unique struct for the same appears somewhat needless and unnecessary-work. You would soon come out of names for different structures having few members in it. Also, it increases code length. Whatever your perspective may be for this, I am using it as an example, and deriving a class template having two members in it.

STL programmers would find this as equivalent to std::pair class template.

Assume you have a structure Point,

struct Point
{
    int x;
    int y;
};

which is having two data-members. Further, you may also have another structure Money:

struct Money
{
    int Dollars;
    int Cents;
};

Both of these structures have almost similar data-members in it. Instead of re-writing different structures, wouldn't it be better to have it at one place, which would also facilitate:

  • Constructor having one or two arguments of given types, and a copy-constructor.
  • Methods to compare two objects of same type.
  • Swapping between two types
  • And more.

You might say you can use inheritance model, where you'd define all required methods and let derive class customize it. Does it fit in? What about the data-types you chosen? It might be int, string, or float, some-class as types. In short, inheritance will only complicate the design, and will not allow plug-in feature facilitated by C++ templates.

There we use class templates! Just define a class template for two types, having all required methods. Let's start!

template<class Type1, class Type2>
struct Pair
{
    // In public area, since we want the client to use them directly.
    Type1 first;
    Type2 second;
};

Now, we can use Pair class template to derive any type having two members. An example:

// Assume as Point struct
Pair<int,int> point1;

// Logically same as X and Y members
point1.first = 10;
point1.second = 20;

Understand that type of first and second are now int and int, respectively. This is because we instantiated Pair with these types.

When we instantiate it like:

Pair<int, double> SqRoot;

SqRoot.first = 90;
SqRoot.second = 9.4868329;

first would be of int type, and second would be of double type. Clearly understand that first and second are data-members, and not functions, and therefore there is no runtime penalty of assumed function call.

Note: In this part of article, all definitions are within the class declaration body only. In next part, I would explain how to implement methods in separate implementation file, and issues related with that. Therefore, all method definitions shown should be assumed within class ClassName{...}; only.

The following given default constructor would initialize both members to their default values, as per data type of Type1 and Type2:

Pair() : first(Type1()), second(Type2())
{}

Following is a parameterized constructor taking Type1 and Type2 to initialize values of first and second:

Pair(const Type1& t1, const Type2& t2) : 
  first(t1), second(t2)
  {}

Following is a copy-constructor which would copy one Pair object from another Pair object, of exactly same type:

Pair(const Pair<Type1, Type2>& OtherPair) : 
  first(OtherPair.first),
  second(OtherPair.second)
 {}

Please note that it is very much required to specify template type arguments of Pair<>, for the argument of this copy-constructor. The following specification wouldn't make sense, since Pair is not a non-template type:

Pair(const Pair& OtherPair) // ERROR: Pair requires template-types

And here is an example using parametrized constructor and copy-constructor:

Pair<int,int> point1(12,40);
Pair<int,int> point2(point1);

It is important to note that if you change any of the template type parameters of either of the objects, point2 or point1, you wouldn't be able to copy-construct it using point1 object. Following would be an error:

Pair<int,float> point2(point1);  // ERROR: Different types, no conversion possible.

Though, there is a possible conversion between float to int, but there is no possible conversion between Pair<int,float> to Pair<int,int>. The copy constructor cannot take other type as copyable object. There is a solution to this, but I would discuss it in next part.

In similar fashion, you can implement comparison operators to compare two objects of same Pair type. Following is an implementation of equal-to operator:

bool operator == (const Pair<Type1, Type2>& Other) const
{
  return first == Other.first && 
         second == Other.second;
}

Note that I used const attribute, for argument and for the method itself. Please fully understand the first line of above' method definition!

Just like copy-constructor call, you must pass exactly the same type to this comparison operator - compiler will not attempt to convert different Pair types. An example:

if (point1 == point2) // Both objects must be of same type.
 
   ...

For a solid understanding for the concepts covered till here, please implement following methods by on your own:

  • All remaining 5 relational operators
  • Assignment operator
  • Swap method
  • Modify both constructors (except copy-constructor), and combine them into one so that they take both parameters as default. This means, implement only one constructor that can take 0,1 or 2 arguments.

Pair class is an example for two types, and it can be used instead of defining multiple structures having just two data-members. The drawback is just with remembering what first and second would mean (X or Y?). But when you well-define a template instantiation, you would always know and use first and second members appropriately.

Ignoring this one disadvantage, you would achieve all the features in the instantiated type: constructors, copy-constructor, comparison operators, swap method etc. And, you'd get all this without re-writing the required code for various two-member structures you would need. Furthermore, as you know, only the set of required methods would get compiled and linked. A bug fix in class template would automatically be reflected to all instantiations. Yes, a slight modification to class template may also raise bunch of errors, of other types, if the modification fails to comply with existing usage.

Likewise, you can have a class template tuple which allows three (or more) data-members. Please try to implement class tuple with three members (first, second, third) by yourself:

template<class T1, class T2, class T3>
class tuple

Non-type Template Arguments

Alright, we have seen that class templates, just like function templates, can take multiple type arguments. But class templates also allow few non-type template arguments. In this part, I will elaborate only one non-type: integer.

Yes, a class template may take a integer as template argument. First a sample:

template<class T, int SIZE>
class Array{};

In this class template declaration, int SIZE is a non-type argument, which is an integer.

  • Only integral data-types can be non-type integer argument, it includes int, char, long, long long, unsigned variants and enums. Types such as float and double are not allowed.
  • When being instantiated, only compile time constant integer can be passed. This means 100, 100+99, 1<<3 etc are allowed, since they are compiled time constant expressions. Arguments, that involve function call, like abs(-120), are not allowed.
    As a template argument, floats/doubles etc may be allowed, if they can be converted to integer.

Fine. We can instantiate class template Array as:

Array<int, 10> my_array;

So what? What's the purpose of SIZE argument?

Well, within the class template you can use this non-type integer argument, wherever you could have used an integer. It includes:

  • Assigning static const data-member of a class.
template<class T, int SIZE>
class Array
{
 static const int Elements_2x = SIZE * 2; 
};

[First two lines of class declaration will not be shown further, assume everything is within class' body.]

Since it is allowed to initialize a static-constant-integer within class declaration, we can use non-type integer argument.

  • To specify default value for a method.
    (Though, C++ also allows any non-constant to be a default parameter of a function, I have pointed this one for just for illustration.)
void DoSomething(int arg = SIZE); 
// Non-const can also appear as default-argument...
  • To define the size of an array.

This one is important, and non-type integer argument is often used for this purpose. So, let's implement the class template Array utilizing SIZE argument.

private:
   T TheArray[SIZE];

T is the type of array, SIZE is the size (integer) - as simple as that. Since the array is in private area of class, we need define several methods/operators.

// Initialize with default (i.e. 0 for int)
void Initialize()
{ 
    for(int nIndex = 0; nIndex < SIZE; ++nIndex)
        TheArray[nIndex] = T();
}

For sure, the type T must have a default constructor and an assignment operator. I will cover these things (requirements) for function template and class templates in next part.

We also need to implement array-element access operators. One of the overloaded index-access operator sets, and the other one gets the value (of type T):

T operator[](int nIndex) const
{
  if (nIndex>0 && nIndex<SIZE)
  {
    return TheArray[nIndex];
  }
  return T();
}

T& operator[](int nIndex)
{
   return TheArray[nIndex];
}

Note that first overload (which is declared const) is get/read method, and has a check to see if index is valid or not, otherwise returns default value for type T.

The second overload returns the reference of an element, which can be modified by caller. There is no index validity check, since it has to return a reference, and therefore local-object (T()) cannot be returned. You may, however, check the index argument, return default value, use assertion and/or throw an exception.

Let's define another method, which would logically sum all the elements of Array:

T Accumulate() const
{
  T sum = T();
  for(int nIndex = 0; nIndex < SIZE; ++nIndex)
  {
     sum += TheArray[nIndex];
  }
  return sum;
}

As you can interpret, it requires operator += to be available for target type T. Also note that return type is Titself, which is appropriate. So, when instantiate Array with some string class, it will call += on each iteration and would return the combined string. If target type doesn't have this += operator defined, and you call this method, there would be an error. In that case, you either - don't call it; or implement the required operator overload in the target class.

Template Class as Argument to Class Template

While it is a vague statement to understand, and invites some ambiguities, I would attempt my best to remove the fogginess.

First, recollect the difference between template-function and a function-template. If the neurons have helped to transfer the correct information to the cache of your brain-box, you now callback that template-function is an instance of function-template. If search-subsystem of your brain is not responding, please reload the information again!

An instance of class template is template class. Therefore, for following class template:

template<class T1, class T2>
class Pair{};

The instantiation of this template is a template-class:

Pair<int,int> IntPair;

Clearly understand that IntPair is not a template-class, is not instantiation for class template. It is an object of a particular instantiation/class-template. The template-class/instantiation is Pair<int,int>, which produces another class type (compiler, our friend does this, you know!). Essentially this is what template-class would be produced by compiler for this case:

class Pair<int,int>{};

There is more precise definition for a template-class, select this single line of code for easy understanding. Detailed explication would come in next installment of this series.

Now, let's come to the point. What if you pass a template-class to some class-template? I mean, what does it mean by following statement?

Pair<int, Pair<int,int> > PairOfPair;

Is it valid - if so, what does it mean?
Firstly, it is perfectly valid. Secondly, it instantiates two template classes:

  • Pair<int,int> - A
  • Pair<int, Pair<int,int> > -- B

Both A and B types would be instantiated by compiler, and if there is any error, arising due to any type of these two template classes, compiler would report. To simplify this complex instantiation, you may do:

typedef Pair<int,int> IntIntPair;
...
Pair<int, IntIntPair> PairOfPair;

You can assign first and second members of PairOfPair object like this:

PairOfPair.first = 10;
PairOfPair.second.first = 10;
PairOfPair.second.second= 30;

Note that second member in last two lines is of type Pair<int,int>, and therefore it has same set of members to be accessed further. That's the reason first and second members can be used, in cascaded manner.

Now you (hopefully) understand that class template (Pair) is taking template-class (Pair<int,int>) as argument and inducing the final instantiation!

An interesting instantiation, in this discussion, would be of Array with Pair! You know that Pair takes two template type arguments, and Array takes one type argument, and a size (integer) argument.

Array< Pair<int, double>, 40> ArrayOfPair;

Here int and double are type-arguments for Pair. Hence, the first template-type of Array (marked bold) is Pair<int,double>. The second argument is constant 40. Can you answer this: Would the constructor of Pair<int,double> be called? When it will be called? Before you answer that, I just reverse the instantiation as:

Pair<int, Array<double, 50>> PairOfArray;

Wohoo! What does it mean?
Well, it means: PairOfArray is an instantiation of Pair, which is taking first type as int (for firstmember), and second type (second) is an Array. Where Array (the second type of Pair) is 50 elements of type double!

Don't kill me for this! Slowly and clearly understand these basic concepts of templates. Once you get the crystal-clear understanding, you would love templates!

Here again, I used a template-class (Array<double,50>) as an argument to an instance of other type (Pair<int,...>).

Okay, but what right-shift operator (>>) is doing above? Well, that's not an operator, but just ending of Array's type specification, followed by ending of Pair type specification. Some old compilers required us to put a space in between two greater-than symbols, so to avoid error or confusion.

Pair<int, Array<double, 50>  > PairOfArray;

At present, almost all modern C++ compilers are smart enough to understand that this is used to end template type specification, and therefore you need not to worry. Therefore, you may freely use two or more > symbols to end template specification(s).

Kindly note that passing a template class (instantiation) is nothing very specific in C++ terms - it is just a type that a class template would take.

Finally, Here I put usage examples both objects. First the constructors.

Array< Pair<int, double>, 40> ArrayOfPair;

This will cause the constructor of Pair to be called 40 times, since there is declaration of constant-size array in Array class template:

T TheArray[SIZE];

Which would mean:

Pair<int,double> TheArray[40];

And hence the required number of calls to constructor of Pair.

For the following object construction:

Pair<int, Array<double, 50>> PairOfArray;

The constructor of Pair would initialize first argument with 0 (using int() notation), and would call constructor of Array with Array() notation, as shown below:

Pair() : first(int()), second(Array())
 {}

Since the default constructor of Array class template is provided by compiler, it would be called. If you don't understand the stuff written here, please sharpen your C++ skills.

Assigning one element of ArrayOfPair:

ArrayOfPair[0] = Pair<int,double>(40, 3.14159);

Here, you are calling non-const version of Array::operator[], which would return the reference of first element of Array (from TheArray). The element, as you know, is of type Pair<int,double>. The expression on right side of assignment operator is just calling the constructor for Pair<int,double> and passing required two arguments. The assignment is done!

Default Template Arguments with Class Templates

First let me eliminate any ambiguity with 'Default Argument' phrase. The same phrase was used in Function Template section. In that sub-section, the default-argument referred to arguments of function parameters itself, not the type-arguments of function template. Function templates, anyway, do not support default arguments for template-arguments. As a side note, please know that methods of a class template can take default arguments, as any ordinary function/method would take.

Class templates, on other hand, do support default-argument for the type/non-type arguments for the template parameters. Throwing you an example:

template<class T, int SIZE=100>
class Array
{
private:
   T TheArray[SIZE];
   ...
};

I have just modified around SIZE in first line for class template Array. The second template parameter, an integer constant specification, is now set to 100. It means, when you use it in following manner:

Array<int> IntArray;

It would essentially mean:

Array<int, 100> IntArray;

Which would be automatically placed by compiler during the instantiation of this class template. Of course, you can specify custom array size by explicitly passing the second template argument:

Array<int, 200> IntArray;

Do remember that when you explicitly pass the default argument's parameter, with the same argument specified in class template declaration, it would instantiate it only once. By this, I mean, the following two objects created would instantiate only one class: Array<int,100>

Array<int> Array1;
Array<int,100> Array2;

Of course, if you change the default' template parameter in class template definition, with value other than 100, it would cause two template instantiations, since they would be different types.

You can customize the default argument by using const or #define:

const int _size = 120;
// #define _size 150 
template<class T, int SIZE=_size>
class Array

For sure, using _size symbol instead of hard-coded constant value mean the same. But using a symbol would ease the default' specification. Irrespective of how you specify the default template parameter for integer (which is a non-type template argument), it must be a compile time constant expression.

You would generally not use default specification for non-type integer parameter, unless you are utilizing templates for advanced stuff, like meta-programming, static-asserts, SFINAE etc, which definitely demands a separate part. More often you would see and implement default parameters for class templates, which are data-types. An example:

template<class T = int>
class Array100
{
    T TheArray[100];
};

It defines an array of type T of size 100. Here, the type argument is defaulted to int. That means, if you don't specify the type while instantiating Array100, it would map to int. Following is an example on how to use it:

Array100<float> FloatArray;
Array100<> IntArray;

In the first instantiation, I passed float as template type, and in second call I kept it default (to int), by using <> notation. While there are more uses of this notation in template programming, which I would cover up in later parts, it is very much required for this case also. If you try to use the class template as:

Array100 IntArray;

It would result in compiler errors, saying Array100 requires template parameters. Therefore, you must use empty-set of angle-brackets (<>) to instantiate a class template, if all template arguments are default, and you wish to use defaults.

Important thing to remember is that a non-template class of name Array100 will not be allowed also. Definition of a non-template class like as given below, along with template class (above or below each other), will upset the compiler:

class Array100{}; // Array100 demands template arguments!

Now, let's mix both type and non-type argument in our class Array:

template<class T = int, int SIZE=100>
class Array
{
    T TheArray[SIZE];
    ...
};

Finally, both type and the size arguments are marked default with int and 100 respectively. Clearly understand that first int is for default specification of T, and second int is for non-template constant specification. For simplicity and better readability, you should keep them in different lines:

template<class T = int, 
         int SIZE=100>
class Array{};

Now, use use your intelligence to parse the meaning of following instantiations:

Array<>            IntArray1;
Array<int>         IntArray2;
Array<float, 40>   FlaotArray3;

Just like explicit specification in function templates, specifying only the trailing template arguments is not allowed. Following is an error:

Array<, 400> IntArrayOf500; // ERROR

As a final note, do remember that following two object creations would instantiate only one class template, since essentially they are exactly the same:

Array<>          IntArray1;
Array<int>       IntArray2
Array<int, 100>  IntArray3;

Defaulting a template type on another type

It is also possible to default a type/non-type parameter on a previously arrived template parameter. For instance, we can modify the Pair class so that second type would be same as first type, if second type is not explicitly specified.

template<class Type1, class Type2 = Type1>
class Pair
{
    Type1 first;
    Type2 second;
};

In this modified class template Pair, Type2 now defaults to Type1 type. An instantiation example:

Pair<int> IntPair;

Which, as you can guess, is same as:

Pair<int,int> IntPair;

But saves you from typing the second parameter. It is also possible to let the first argument of Pair be default also:

template<class Type1=int, class Type2 = Type1>
class Pair
{
    Type1 first;
    Type2 second;
};

Which means, if you don't pass any template argument, Type1 would be int, and hence Type2 would also be int !

The following usage:

Pair<> IntPair;

Instantiates following class:

class Pair<int,int>{};

For sure, it is also possible to default non-type arguments on another non-type argument. An example:

template<class T, int ROWS = 8, int COLUMNS = ROWS>
class Matrix
{
    T TheMatrix[ROWS][COLUMNS];
};

But, the dependent template parameter must be on the right of which it is dependent upon. Following would cause errors:

template<class Type1=Type2, class Type2 = int>
class Pair{};

template<class T, int ROWS = COLUMNS, int COLUMNS = 8>
class Matrix

Class' Methods as Function Templates

Though, this one isn't for absolute beginners, but since I have covered both function templates and class templates - the elaboration of this concept is logical for this first part of this series itself.

Consider a simple example:

class IntArray
{
    int TheArray[10];
public:
    template<typename T>
    void Copy(T target_array[10])
    {
       for(int nIndex = 0; nIndex<10; ++nIndex)
       {
          target_array[nIndex] = TheArray[nIndex];
          // Better approach: 
            //target_array[nIndex] = static_cast<T>(TheArray[nIndex]);
       }
    }
};

The class IntArray is simple, non-template class, having an integer array of 10 elements. But the method Copy is a designed as a function template (method template?). It takes one template type parameter, which would be deduced by compiler automatically. Here is how we can use it:

IntArray int_array;
float float_array[10];

int_array.Copy(float_array);

As you can guess, IntArray::Copy would be instantiated with type float, since we are passing float-array to it. To avoid confusion and understand it better, just think of int_array.Copy as Copy only, and IntArray::Copy<float>(..) as Copy<float>(..) only. The method template of a class is nothing but an ordinary function template embedded in a class.

Do notice that I used 10 as array size everywhere. Interestingly, we can also modify the class as:

template<int ARRAY_SIZE>
class IntArray
{
    int TheArray[ARRAY_SIZE];
public:
    template<typename T>
    void Copy(T target_array[ARRAY_SIZE])
    {
       for(int nIndex = 0; nIndex<ARRAY_SIZE; ++nIndex)
       {
            target_array[nIndex] = static_cast<T>(TheArray[nIndex]);
       }
    }
};

Which makes the class IntArray and the method Copy, better candidates to be in the realm of template programming!

As you would have intelligently guessed, Copy method is nothing but an array-conversion routine, which converts from int to any type, wherever conversion from int to given type is possible. This is one of the valid case where class method can be written as function templates, taking template arguments by themselves. Please modify this class template so that it can work for any type of array, not just int.

For sure, 'explicit template argument specification' with method template is also possible. Consider another example:

template<class T>
class Convert
{   
   T data;
public: 
   Convert(const T& tData = T()) : data(tData)
   { }

   template<class C>   
   bool IsEqualTo( const C& other ) const      
   {        
       return data == other;   
   }
};

Which can be utilized as:

Convert<int> Data;
float Data2 = 1 ;

bool b = Data.IsEqualTo(Data2);

It instantiates Convert::IsEqualTo with float parameter. Explicit specification, as given below, would instantiate it with double:

bool b = Data.IsEqualTo<double>(Data2);

One of the astounding thing, with the help of templates, you can do it by defining conversion operator on top of template!

template<class T>
operator T() const
{
    return data;
}

It would make possible to convert the Convert' class template instance into any type, whenever possible. Consider following usage example:

Convert<int> IntData(40);
float FloatData;
double DoubleData;

 
FloatData = IntData;
DoubleData = IntData;

Which would instantiate following two methods (fully qualified names):

Convert<int>::operator<float> float();
Convert<int>::operator<double> double();

On one hand it provides good flexibility, since without writing extra code, Convert can convert itself (the specific instantiation) to any data-type - whenever conversion is possible at compilation level. If conversion is not possible, like from double to string-type, it would raise an error.

But on the other hand, it also invites trouble by possibility of inadvertently inserting bugs. You may not want to have conversion operator called, and it is called (compiler code generated) without you knowing about it.

At The End

You have just seen slight glimpse of power and flexibility provided by templates. Next part will cover more of advanced and intriguing concepts. My humble and aspiring request to all readers is to play more and more with templates. Try to gain firm understanding on one aspect first (like function template only), rather than just hastily jumping to other concept. Initially do it with your test projects/code-base, and not with any existing/working/production code.

Following is a summary of what we have covered:

  • To avoid unnecessary code duplication and code maintenance issues, specially when code is exactly same, we can use templates. Templates are far better approach than using C/C++ macros or functions/classes running on top of void-pointers.
  • Templates are not only type-safe, but also reduces unnecessary code-bloat which would not be referred (not generated by compiler).
  • Function templates are used to put a code that is not a part of class, and is same/almost-same for different data-types. At most of the places, compiler would automatically determine the type. Otherwise you have to specify the type, or you may also specify explicit type yourself.
  • Class templates makes it possible to wrap any data type around specific implementation. It may be an array, string, queue, linked-list, thread-safe atomic implementation etc. Class templates do facilitate default template type specification, which function template don't support.

Hope you have enjoyed the article, and cleared out the mental-block that templates are complicated, unnecessarily, bizarre. Second part would be arriving soon!

History

  • First release: October 8, 2011 - Covered basics of templates in C++, function templates.
  • First amendment: Oct 9, 2011 - Class Templates and multiple types in class templates, non-template type
  • Third amendment: Oct 11, 2011 - Template class as argument to class template
  • Fourth amendment: Oct 12, 2011 - Default Template Arguments with Class Templates
  • Fifth amendment: Oct 13, 2011 - Methods as Function Templates, finishing lines.
  • Sixth amendment: Mar 8, 2012 - Basic corrections.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Ajay Vijayvargiya

Software Developer (Senior)
India India
Started programming with GwBasic back in 1996 (Those lovely days!). Found the hidden talent!

Touched COBOL and Quick Basic for a while.

Finally learned C and C++ entirely on my own, and fell in love with C++, still in love! Began with Turbo C 2.0/3.0, then to VC6 for 4 years! Finally on VC2008/2010.

I enjoy programming, mostly the system programming, but the UI is always on top of MFC! Quite experienced on other environments and platforms, but I prefer Visual C++. Zeal to learn, and to share!

The Complete Idiot's Guide to Writing Shell Extensions - Part I

A step-by-step tutorial on writing shell extensions

Contents

README.TXT

This is the stuff I want you to read first, before proceeding on or posting messages to this article's discussion board.

This series of articles was originally written for VC 6 users. Now that VC 8 is out, I felt it was about time to update the articles to cover VC 7.1. 😉 (Also, the automatic 6-to-7 conversion done by VC 7.1 doesn't always go smoothly so VC 7.1 users could get stuck when trying to use the demo source code.) So as I go through and update this series, the articles will be updated to reflect new VC 7.1 features, and I'll have VC 7.1 projects in the source downloads.

Important note for VC 2005 users: The Express edition of VC 2005 does not come with ATL or MFC. Since the articles in this series use ATL, and some use MFC, you won't be able to use the Express editions with the articles' sample code.

If you are using VC 6, you should get an updated Platform SDK. You can use the web install version, or download the CAB files or an ISO image and run the setup locally. Be sure you use the utility to add the SDK include and lib directories to the VC search path. You can find this in the Visual Studio Registration folder in the Platform SDK program group. It's a good idea to get the latest Platform SDK even if you're using VC 7 or 8 so you have the latest headers and libs.

Important note for VC 7 users: You must make a change to the default include path if you haven't updated your Platform SDK. Make sure that $(VCInstallDir)PlatformSDK\include is first in the list, above ($VCInstallDir)include, as shown here:

Since I haven't used VC 8 yet, I don't know if the sample code will compile on 8. Hopefully the 7-to-8 upgrade process will work better than the 6-to-7 process did. Please post on this article's forum if you have any trouble with VC 8.

Introduction to the Series

A shell extension is a COM object that adds some kind of functionality to the Windows shell (Explorer). There are all kinds of extensions out there, but very little easy-to-follow documentation about what they are. (Although I bet the situation has improved during the six years since I originally wrote that!) I highly recommend Dino Esposito's great book Visual C++ Windows Shell Programming (ISBN 1861001843) if you want an in-depth look into lots of aspects of the shell, but for folks who don't have the book, or only care about shell extensions, I've written up this tutorial that will astound and amaze you, or failing that, get you well on your way to understanding how to write your own extensions. This guide assumes you are familiar with the basics of COM and ATL. If you need a refresher on COM basics, check out my Intro to COM article.

Part I contains a general introduction to shell extensions, and a simple context menu extension to whet your appetite for the following parts.

There are two parts in the term "shell extension." Shell refers to Explorer, and extension refers to code you write that gets run by Explorer when a predetermined event happens (e.g., a right-click on a .DOC file). So a shell extension is a COM object that adds features to Explorer.

A shell extension is an in-process server that implements some interfaces that handle the communication with Explorer. ATL is the easiest way to get an extension up and running quickly, since without it you'd be stuck writing QueryInterface() and AddRef() code over and over. It is also much easier to debug extensions on Windows NT-based OSes, as I will explain later.

There are many types of shell extensions, each type being invoked when different events happen. Here are a few of the more common types, and the situations in which they are invoked:

Type When it's invoked What it does
Context menu handler User right-clicks on a file or folder. In shell versions 4.71+, also invoked on a right-click in the background of a directory window. Adds items to the context menu.
Property sheet handler Properties dialog displayed for a file. Adds pages to the property sheet.
Drag and drop handler User right-drags items and drops them on a directory window or the desktop. Adds items to the context menu.
Drop handler User drags items and drops them on a file. Any desired action.
QueryInfo handler (shell version 4.71+) User hovers the mouse over a file or other shell object like My Computer. Returns a string that Explorer displays in a tooltip.

Introduction to Part I

By now you many be wondering what an extension looks like in Explorer. One example is WinZip - it contains many types of extensions, one of them being a context menu handler. Here some commands that WinZip adds to the context menu for compressed files:

 [WinZip menu items - 4K]

WinZip contains the code that adds the menu items, provides fly-by help (text that appears in Explorer's status bar), and carries out the appropriate actions when the user chooses one of the WinZip commands.

WinZip also contains a drag and drop handler. This type is very similar to a context menu extension, but it is invoked when the user drags a file using the right mouse button. Here is what WinZip's drag and drop handler adds to the context menu:

 [WinZip menu items - 2K]

There are many other types (and Microsoft keeps adding more in each version of Windows!). For now, we'll just look at context menu extensions, since they are pretty simple to write and the results are easy to see (instant gratification!).

Before we begin coding, there are some tips that will make the job easier. When you cause a shell extension to be loaded by Explorer, it will stay in memory for a while, making it impossible to rebuild the DLL. To have Explorer unload extensions more often, create this registry key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\AlwaysUnloadDLL

and set the default value to "1". On 9x, that's the best you can do. On NT, go to this key:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer

and create a DWORD called DesktopProcess with a value of 1. This makes the desktop and Taskbar run in one process, and subsequent Explorer windows each run in its own process. This means that you can do your debugging with a single Explorer window, and when you close it, your DLL is automatically unloaded, avoiding any problems with the file being in use. You will need to log off and back on for these changes to take effect.

I will explain how to debug on 9x a little later.

Using AppWizard to Get Started

Let's start simple, and make an extension that just pops up a message box to show that it's working. We'll hook the extension up to .TXT files, so our extension will be called when the user right-clicks a text file.

OK, it's time to get started! What's that? I haven't told you how to use the mysterious shell extension interfaces yet? Don't worry, I'll be explaining as I go along. I find that it's easier to follow along with examples if the concepts are explained, and followed immediately by sample code. I could explain everything first, then get to the code, but I find that harder to absorb. Anyway, fire up VC and we'll get started.

Run the AppWizard and make a new ATL COM program. We'll call it "SimpleExt". Keep all the default settings in the AppWizard, and click Finish. We now have an empty ATL project that will build a DLL, but we need to add our shell extension's COM object. In the ClassView tree, right-click the SimpleExt classes item, and pick New ATL Object. (In VC 7, right-click the item and pick Add|Add Class.)

In the ATL Object Wizard, the first panel already has Simple Object selected, so just click Next. On the second panel, enter "SimpleShlExt" in the Short Name edit box (the other edit boxes on the panel will be filled in automatically):

By default, the wizard creates a COM object that can be used from C and script-based clients through OLE Automation. Our extension will only be used by Explorer, so we can change some settings to remove the Automation features. Go to the Attributes page, and change the Interface type to Custom, and change the Aggregation setting to No:

When you click OK, the wizard creates a class called CSimpleShlExt that contains the basic code for implementing a COM object, and adds this class to the project. We will add our code to this class.

The Initialization Interface

When our shell extension is loaded, Explorer calls our QueryInterface() function to get a pointer to an IShellExtInit interface. This interface has only one method, Initialize(), whose prototype is:

HRESULT IShellExtInit::Initialize (
  LPCITEMIDLIST pidlFolder,
  LPDATAOBJECT pDataObj,
  HKEY hProgID )

Explorer uses this method to give us various information. pidlFolder is the PIDL of the folder containing the files being acted upon. (A PIDL [pointer to an ID list] is a data structure that uniquely identifies any object in the shell, whether it's a file system object or not.) pDataObj is an IDataObject interface pointer through which we retrieve the names of the files being acted upon. hProgID is an open HKEY which we can use to access the registry key containing our DLL's registration data. For this simple extension, we'll only need to use the pDataObj parameter.

To add this to our COM object, open the SimpleShlExt.h file, and add the lines listed below in bold. Some of the COM-related code generated by the wizard isn't needed, since we're not implementing our own interface, so I've indicated the code that can be removed with strikeout type:

#include <shlobj.h>
#include <comdef.h>
 
class ATL_NO_VTABLE CSimpleShlExt :
  public CComObjectRootEx<CComSingleThreadModel>,
  public CComCoClass<CSimpleShlExt, &CLSID_SimpleShlExt>,
  public ISimpleShlExt,
  public IShellExtInit
{
  BEGIN_COM_MAP(CSimpleShlExt)
    COM_INTERFACE_ENTRY(ISimpleShlExt)
    COM_INTERFACE_ENTRY(IShellExtInit)
  END_COM_MAP()

The COM_MAP is how ATL implements QueryInterface(). It tells ATL what interfaces other programs can retrieve from our COM objects.

Inside the class declaration, add the prototype for Initialize(). We'll also need a buffer to hold a filename:

protected:
  TCHAR m_szFile[MAX_PATH];
 
public:
  // IShellExtInit
  STDMETHODIMP Initialize(LPCITEMIDLIST, LPDATAOBJECT, HKEY);

Then, in the SimpleShlExt.cpp file, add the definition of the function:

STDMETHODIMP CSimpleShlExt::Initialize ( 
  LPCITEMIDLIST pidlFolder,
  LPDATAOBJECT pDataObj,
  HKEY hProgID )

What we'll do is get the name of the file that was right-clicked, and show that name in a message box. If there is more than one selected file, you could access them all through the pDataObj interface pointer, but since we're keeping this simple, we'll only look at the first filename.

The filename is stored in the same format as the one used when you drag and drop files on a window with theWS_EX_ACCEPTFILES style. That means we get the filenames using the same API: DragQueryFile(). We'll begin the function by getting a handle to the data contained in the IDataObject:

HRESULT CSimpleShlExt::Initialize(...)
{
FORMATETC fmt = { CF_HDROP, NULL, DVASPECT_CONTENT,
                  -1, TYMED_HGLOBAL };
STGMEDIUM stg = { TYMED_HGLOBAL };
HDROP     hDrop;
 
  // Look for CF_HDROP data in the data object. If there
  // is no such data, return an error back to Explorer.
  if ( FAILED( pDataObj->GetData ( &fmt, &stg ) ))
    return E_INVALIDARG;
 
  // Get a pointer to the actual data.
  hDrop = (HDROP) GlobalLock ( stg.hGlobal );
 
  // Make sure it worked.
  if ( NULL == hDrop )
    return E_INVALIDARG;

Note that it's vitally important to error-check everything, especially pointers. Since our extension runs in Explorer's process space, if our code crashes, we take down Explorer too. On 9x, such a crash might necessitate rebooting the computer.

Now that we have an HDROP handle, we can get the filename we need.

  // Sanity check – make sure there is at least one filename.
UINT uNumFiles = DragQueryFile ( hDrop, 0xFFFFFFFF, NULL, 0 );
HRESULT hr = S_OK;
  
  if ( 0 == uNumFiles )
    {
    GlobalUnlock ( stg.hGlobal );
    ReleaseStgMedium ( &stg );
    return E_INVALIDARG;
    }
 
  // Get the name of the first file and store it in our
  // member variable m_szFile.
  if ( 0 == DragQueryFile ( hDrop, 0, m_szFile, MAX_PATH ) )
    hr = E_INVALIDARG;
 
  GlobalUnlock ( stg.hGlobal );
  ReleaseStgMedium ( &stg );
 
  return hr;
}

If we return E_INVALIDARG, Explorer will not call our extension for this right-click event again. If we return S_OK, then Explorer will call QueryInterface() again and get a pointer to another interface: IContextMenu.

The Interface for Interacting with the Context Menu

Once Explorer has initialized our extension, it will call the IContextMenu methods to let us add menu items, provide fly-by help, and carry out the user's selection.

Adding IContextMenu to our shell extension is similar to adding IShellExtInit. Open up SimpleShlExt.h and add the lines listed here in bold:

class ATL_NO_VTABLE CSimpleShlExt : 
    public CComObjectRootEx<CComSingleThreadModel>,
    public CComCoClass<CSimpleShlExt, &CLSID_SimpleShlExt>,
    public IShellExtInit,
    public IContextMenu
{
  BEGIN_COM_MAP(CSimpleShlExt)
    COM_INTERFACE_ENTRY(IShellExtInit)
    COM_INTERFACE_ENTRY(IContextMenu)
  END_COM_MAP()

And then add the prototypes for the IContextMenu methods:

public:
  // IContextMenu
  STDMETHODIMP GetCommandString(UINT, UINT, UINT*, LPSTR, UINT);
  STDMETHODIMP InvokeCommand(LPCMINVOKECOMMANDINFO);
  STDMETHODIMP QueryContextMenu(HMENU, UINT, UINT, UINT, UINT);

Modifying the context menu

IContextMenu has three methods. The first one, QueryContextMenu(), lets us modify the menu. The prototype of QueryContextMenu() is:

HRESULT IContextMenu::QueryContextMenu (
  HMENU hmenu, UINT uMenuIndex, UINT uidFirstCmd,
  UINT uidLastCmd, UINT uFlags );

hmenu is a handle to the context menu. uMenuIndex is the position in which we should start adding our items. uidFirstCmd and uidLastCmd are the range of command ID values we can use for our menu items. uFlags indicates why Explorer is calling QueryContextMenu(), and I'll get to this later.

The return value is documented differently depending on who you ask. Dino Esposito's book says it's the number of menu items added by QueryContextMenu(). The MSDN docs from VC 6 says it's the command ID of the last menu item we add, plus 1. The online MSDN says this:

If successful, returns an HRESULT value that has its severity value set to SEVERITY_SUCCESS and its code value set to the offset of the largest command identifier that was assigned, plus one. For example, assume that idCmdFirst is set to 5 and you add three items to the menu with command identifiers of 5, 7, and 8. The return value should be MAKE_HRESULT(SEVERITY_SUCCESS, 0, 8 - 5 + 1). Otherwise, it returns an OLE error value.

I've been following Dino's explanation so far in the code I've written, and it's worked fine. Actually, his method of making the return value is equivalent to the online MSDN method, as long as you start numbering your menu items with uidFirstCmd and increment it by 1 for each item.

Our simple extension will have just one item, so the QueryContextMenu() function is quite simple:

HRESULT CSimpleShlExt::QueryContextMenu (
  HMENU hmenu, UINT uMenuIndex, UINT uidFirstCmd,
  UINT uidLastCmd, UINT uFlags )
{
  // If the flags include CMF_DEFAULTONLY then we shouldn't do anything.
  if ( uFlags & CMF_DEFAULTONLY )
    return MAKE_HRESULT ( SEVERITY_SUCCESS, FACILITY_NULL, 0 );
 
  InsertMenu ( hmenu, uMenuIndex, MF_BYPOSITION,
               uidFirstCmd, _T("SimpleShlExt Test Item") );
 
  return MAKE_HRESULT ( SEVERITY_SUCCESS, FACILITY_NULL, 1 );
}

The first thing we do is check uFlags. You can look up the full list of flags in MSDN, but for context menu extensions, only one value is important: CMF_DEFAULTONLY. This flag tells namespace extensions to add only the default menu item. Shell extensions should not add any items when this flag is present. That's why we return 0 immediately if the CMF_DEFAULTONLY flag is present. If that flag isn't present, we modify the menu (using the hmenu handle), and return 1 to tell the shell that we added 1 menu item.

Showing fly-by help in the status bar

The next IContextMenu that can be called is GetCommandString(). If the user right-clicks a text file in an Explorer window, or selects a text file and then clicks the File menu, the status bar will show fly-by help when our menu item is highlighted. Our GetCommandString() function will return the string that we want Explorer to show.

The prototype for GetCommandString() is:

HRESULT IContextMenu::GetCommandString (
  UINT idCmd, UINT uFlags, UINT* pwReserved,
  LPSTR pszName, UINT cchMax );

idCmd is a zero-based counter that indicates which menu item is selected. Since we have just one menu item, idCmd will always be zero. But if we had added, say, 3 menu items, idCmd could be 0, 1, or 2. uFlags is another group of flags that I'll describe later. We can ignore pwReserved. pszName is a pointer to a buffer owned by the shell where we will store the help string to be displayed. cchMax is the size of the buffer. The return value is one of the usual HRESULT constants, such as S_OK or E_FAIL.

GetCommandString() can also be called to retrieve a "verb" for a menu item. A verb is a language-independent string that identifies an action that can be taken on a file. The docs for ShellExecute() have more to say, and the subject of verbs is best suited for another article, but the short version is that verbs can be either listed in the registry (such as "open" and "print"), or created dynamically by context menu extensions. This lets an action implemented in a shell extension be invoked by a call to ShellExecute().

Anyway, the reason I mentioned all that is we have to determine why GetCommandString() is being called. If Explorer wants a fly-by help string, we provide it. If Explorer is asking for a verb, we'll just ignore the request. This is where the uFlags parameter comes into play. If uFlags has the GCS_HELPTEXT bit set, then Explorer is asking for fly-by help. Additionally, if the GCS_UNICODE bit is set, we must return a Unicode string.

The code for our GetCommandString() looks like this:

#include <atlconv.h>  // for ATL string conversion macros
 
HRESULT CSimpleShlExt::GetCommandString (
  UINT idCmd, UINT uFlags, UINT* pwReserved,
  LPSTR pszName, UINT cchMax )
{
USES_CONVERSION;
 
  // Check idCmd, it must be 0 since we have only one menu item.
  if ( 0 != idCmd )
    return E_INVALIDARG;
 
  // If Explorer is asking for a help string, copy our string into the
  // supplied buffer.
  if ( uFlags & GCS_HELPTEXT )
    {
    LPCTSTR szText = _T("This is the simple shell extension's help");
 
    if ( uFlags & GCS_UNICODE )
      {
      // We need to cast pszName to a Unicode string, and then use the
      // Unicode string copy API.
      lstrcpynW ( (LPWSTR) pszName, T2CW(szText), cchMax );
      }
    else
      {
      // Use the ANSI string copy API to return the help string.
      lstrcpynA ( pszName, T2CA(szText), cchMax );
      }
 
    return S_OK;
    }
 
  return E_INVALIDARG;
}

Nothing fancy here; I just have the string hard-coded and convert it to the appropriate character set. If you have never used the ATL conversion macros, check out Nish and my article on string wrapper classes; they make life a lot easier when having to pass Unicode strings to COM methods and OLE functions.

One important thing to note is that the lstrcpyn() API guarantees that the destination string will be null-terminated. This is different from the CRT function strncpy(), which does not add a terminating null if the source string's length is greater than or equal to cchMax. I suggest always using lstrcpyn(), so you don't have to insert checks after every strncpy() call to make sure the strings end up null-terminated.

Carrying out the user's selection

The last method in IContextMenu is InvokeCommand(). This method is called if the user clicks on the menu item we added. The prototype for InvokeCommand() is:

HRESULT IContextMenu::InvokeCommand (
  LPCMINVOKECOMMANDINFO pCmdInfo );

The CMINVOKECOMMANDINFO struct has a ton of info in it, but for our purposes, we only care about lpVerband hwnd. lpVerb performs double duty - it can be either the name of the verb that was invoked, or it can be an index telling us which of our menu items was clicked on. hwnd is the handle of the Explorer window where the user invoked our extension; we can use this window as the parent window for any UI that we show.

Since we have just one menu item, we'll check lpVerb, and if it's zero, we know our menu item was clicked. The simplest thing I could think to do is pop up a message box, so that's just what this code does. The message box shows the filename of the selected file, to prove that it's really working.

HRESULT CSimpleShlExt::InvokeCommand (
  LPCMINVOKECOMMANDINFO pCmdInfo )
{
  // If lpVerb really points to a string, ignore this function call and bail out.
  if ( 0 != HIWORD( pCmdInfo->lpVerb ) )
    return E_INVALIDARG;
 
  // Get the command index - the only valid one is 0.
  switch ( LOWORD( pCmdInfo->lpVerb ) )
    {
    case 0:
      {
      TCHAR szMsg[MAX_PATH + 32];
 
      wsprintf ( szMsg, _T("The selected file was:\n\n%s"), m_szFile );
 
      MessageBox ( pCmdInfo->hwnd, szMsg, _T("SimpleShlExt"),
                   MB_ICONINFORMATION );
 
      return S_OK;
      }
    break;
 
    default:
      return E_INVALIDARG;
    break;
    }
}

Other code details

There are a couple more tweaks we can make to the wizard-generated code to remove OLE Automation features that we don't need. First, we can remove some registry entries from the SimpleShlExt.rgs file (the purpose of this file is explained in the next section):

HKCR
{
  SimpleExt.SimpleShlExt.1 = s 'SimpleShlExt Class'
  {
    CLSID = s '{5E2121EE-0300-11D4-8D3B-444553540000}'
  }
  SimpleExt.SimpleShlExt = s 'SimpleShlExt Class'
  {
    CLSID = s '{5E2121EE-0300-11D4-8D3B-444553540000}'
    CurVer = s 'SimpleExt.SimpleShlExt.1'
  }
  NoRemove CLSID
  {
    ForceRemove {5E2121EE-0300-11D4-8D3B-444553540000} = s 'SimpleShlExt Class'
    {
      ProgID = s 'SimpleExt.SimpleShlExt.1'
      VersionIndependentProgID = s 'SimpleExt.SimpleShlExt'
      InprocServer32 = s '%MODULE%'
      {
        val ThreadingModel = s 'Apartment'
      }
      'TypeLib' = s '{73738B1C-A43E-47F9-98F0-A07032F2C558}'
    }
  }
}

We can also remove the type library from the DLL's resources. Click View|Resource Includes. In the Compile-time directives box, you'll see a line that includes the type library:

Remove that line, then click OK when VC warns about modifying the includes.

In VC 7, the command is in a different location. On the Resource View tab, right-click the SimpleExt.rc folder, and pick Resource Includes on the menu.

Now that we've removed the type library, we need to change two lines of code and tell ATL that it shouldn't do anything with the type library. Open SimpleExt.cpp, go to the DllRegisterServer() function, and change the RegisterServer() parameter to FALSE:

STDAPI DllRegisterServer()
{
//...
  return _Module.RegisterServer(TRUE FALSE);
}

DllUnregisterServer() needs a similar change:

STDAPI DllUnregisterServer()
{
//...
  return _Module.UnregisterServer(TRUE FALSE);
}

Registering the Shell Extension

So now we have all of the COM interfaces implemented. But... how do we get Explorer to use our extension? ATL automatically generates code that registers our DLL as a COM server, but that just lets other apps use our DLL. In order to tell Explorer our extension exists, we need to register it under the key that holds info about text files:

HKEY_CLASSES_ROOT\txtfile

Under that key, a key called ShellEx holds a list of shell extensions that will be invoked on text files. Under ShellEx, the ContextMenuHandlers key holds a list of context menu extensions. Each extension creates a subkey under ContextMenuHandlers and sets the default value of that key to its GUID. So, for our simple extension, we'll create this key:

HKEY_CLASSES_ROOT\txtfile\ShellEx\ContextMenuHandlers\SimpleShlExt

and set the default value to our GUID: "{5E2121EE-0300-11D4-8D3B-444553540000}".

You don't have to write any code to do this, however. If you look at the list of files on the FileView tab, you'll see SimpleShlExt.rgs. This is a text file that is parsed by ATL, and tells ATL what registry entries to add when the server is registered, and which ones to delete when the server is unregistered. Here's how we specify the registry entries to add so Explorer knows about our extension:

HKCR
{
  NoRemove txtfile
  {
    NoRemove ShellEx
    {
      NoRemove ContextMenuHandlers
      {
        ForceRemove SimpleShlExt = s '{5E2121EE-0300-11D4-8D3B-444553540000}'
      }
    }
  }
}

Each line is a registry key name, with "HKCR" being an abbreviation for HKEY_CLASSES_ROOT. The NoRemove keyword means that the key should not be deleted when the server is unregistered. The last line has another keyword, ForceRemove, which means that if the key exists, it will be deleted before the new key is written. The rest of the line specifies a string (that's what the "s" means) that will be stored in the default value of the SimpleShlExt key.

I need to editorialize a bit here. The key we register our extension under is HKEY_CLASSES_ROOT\txtfile. However, the name "txtfile" isn't a permanent or pre-determined name. If you look in HKEY_CLASSES_ROOT\.txt, the default value of that key is where the name is stored. This has a couple of side effects:

  • We can't reliably use an RGS script since "txtfile" may not be the correct key name.
  • Some other text editor may be installed that associates itself with .TXT files. If it changes the default value of the HKEY_CLASSES_ROOT\.txt key, all existing shell extensions will stop working.

This sure seems like a design flaw to me. I think Microsoft thinks the same way, since recently-created extensions, like the QueryInfo extension, are registered under the key that's named after the file extension.

OK, end of editorial. There's one final registration detail. On NT, it's advisable to put our extension in a list of "approved" extensions. There is a system policy that can be set to prevent extensions from being loaded if they are not on the approved list. The list is stored in:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Shell Extensions\Approved

In this key, we create a string value whose name is our GUID. The contents of the string can be anything. The code to do this goes in our DllRegisterServer() and DllUnregisterServer() functions. I won't show the code here, since it's just simple registry access, but you can find the code in the article's sample project.

Debugging the shell extension

Eventually, you'll be writing an extension that isn't quite so simple, and you'll have to debug it. Open up your project settings, and on the Debug tab, enter the full path to Explorer in the "Executable for debug session" edit box, for example "C:\windows\explorer.exe". If you're using NT, and you've set the DesktopProcess registry entry described earlier, a new Explorer window will open when you press F5 to start debugging. As long as you do all your work in that window, you'll have no problem rebuilding the DLL later, since when you close that window, your extension will be unloaded.

On Windows 9x, you will have to shut down the shell before running the debugger. Click Start, and then Shut Down. Hold down Ctrl+Alt+Shift and click Cancel. That will shut down Explorer, and you'll see the Taskbar disappear. You can then go back to MSVC and press F5 to start debugging. To stop the debugging session, press Shift+F5 to shut down Explorer. When you're done debugging, you can run Explorer from a command prompt to restart the shell normally.

What Does It All Look Like?

Here's what the context menu looks like after we add our item:

 [SimpleShlExt menu item - 3K]

Our menu item is there!

Here's what Explorer's status bar looks like with our fly-by help displayed:

 [SimpleShlExt help text - 3K]

And here's what the message box looks like, showing the name of the file that was selected:

 [SimpleShlExt msg box - 9K]

Copyright and license

This article is copyrighted material, ©2003-2006 by Michael Dunn. I realize this isn't going to stop people from copying it all around the 'net, but I have to say it anyway. If you are interested in doing a translation of this article, please email me to let me know. I don't foresee denying anyone permission to do a translation, I would just like to be aware of the translation so I can post a link to it here.

The demo code that accompanies this article is released to the public domain. I release it this way so that the code can benefit everyone. (I don't make the article itself public domain because having the article available only on CodeProject helps both my own visibility and the CodeProject site.) If you use the demo code in your own application, an email letting me know would be appreciated (just to satisfy my curiosity about whether folks are benefitting from my code) but is not required. Attribution in your own source code is also appreciated but not required.

Revision History

March 27, 2000: Article first published.
March 14, 2006: Updated to cover changes in VC 7.1. Removed OLE Automation-related code from the project to simplify things a bit.

Series Navigation: » Part II

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Michael Dunn

Software Developer (Senior) VMware
United States United States
Michael lives in sunny Mountain View, California. He started programming with an Apple //e in 4th grade, graduated from UCLA with a math degree in 1994, and immediately landed a job as a QA engineer at Symantec, working on the Norton AntiVirus team. He pretty much taught himself Windows and MFC programming, and in 1999 he designed and coded a new interface for Norton AntiVirus 2000.
Mike has been a a developer at Napster and at his own lil' startup, Zabersoft, a development company he co-founded with offices in Los Angeles and Odense, Denmark. Mike is now a senior engineer at VMware.

He also enjoys his hobbies of playing pinball, bike riding, photography, and Domion on Friday nights (current favorite combo: Village + double Pirate Ship). He would get his own snooker table too if they weren't so darn big! He is also sad that he's forgotten the languages he's studied: French, Mandarin Chinese, and Japanese.

Mike was a VC MVP from 2005 to 2009.

You may also be interested in...

Trump Reverses Obama's Mortgage Fee Cuts on First Day

https://www.bloomberg.com/politics/articles/2017-01-20/trump-administration-overturns-obama-s-fha-mortgage-fee-cut

The FHA came under severe stress after the financial crisis. In 2013, it needed $1.7 billion from the U.S. Treasury, its first bailout in 79 years, due to a wave of defaults. To replenish the FHA’s coffers, the Obama administration several times increased the fees the agency charges. The law requires the FHA’s capital reserve ratio to stay above 2 percent, and the agency hit that level in 2015 for the first time since the bailout.

“It is important to ensure that the FHA fund remains strong to support homeownership in the future while minimizing taxpayer risk,” Teresa Bryce Bazemore, president of Radian Group, said in a statement.

 

Soon after Donald Trump was sworn in as president, his administration undid one of Barack Obama’s last-minute economic-policy actions: a mortgage-fee cut under a government program that’s popular with first-time home buyers and low-income borrowers.

The new administration on Friday said it’s canceling a reduction in the Federal Housing Administration’s annual fee for most borrowers. The cut would have reduced the annual premium for someone borrowing $200,000 by $500 in the first year.

The reversal comes after Trump’s team criticized the Obama administration for adopting new policies as it prepared to leave office. In the waning days of the administration, the White House announced new Russia sanctions, a ban on drilling in parts of the Arctic and many other regulations.

No Consultation

Last week, Obama’s Housing and Urban Development secretary, Julian Castro, said the FHA would cut its fees. The administration didn’t consult Trump’s team before the announcement.

Republicans have argued in the past that reductions put taxpayers at risk by lowering the funds the FHA has to deal with mortgage defaults.

Shares of private mortgage insurance companies, including MGIC Investment Corp. and Radian Group Inc., erased earlier losses, trading up about one percent as of mid-afternoon. They closed little changed from the day before. Private insurers, which back loans guaranteed by mortgage-finance companies Fannie Mae and Freddie Mac, compete with the FHA for market share and have been critics of fee cuts in the past.

A letter Friday from HUD to lenders and others in the real-estate industry said, “more analysis and research are deemed necessary to assess future adjustments while also considering potential market conditions in an ever-changing global economy that could impact our efforts.”

Senate Democratic leader Chuck Schumer of New York took to the chamber’s floor to denounce the reversal.

“It took only an hour after his positive words on the inaugural platform for his actions to ring hollow,” Schumer said. “One hour after talking about helping working people and ending the cabal in Washington that hurts people, he signs a regulation that makes it more expensive for new homeowners to buy mortgages.”

Mark Calabria, director of financial regulation studies for the libertarian Cato Institute, said it was appropriate for the administration to examine last-minute decisions by its predecessor, “especially when those decisions appear to be purely motivated by politics."

Ben Carson, Trump’s nominee to lead HUD, FHA’s parent agency, said at his confirmation hearing last week that he was disappointed the cut was announced in Obama’s final days in office.

FHA Role

The FHA sells insurance to protect against defaults and doesn’t issue mortgages. It is a popular program among first-time home buyers because it allows borrowers to make a down payment of as low as 3.5 percent with a credit score of 580, on a scale of 300 to 850.

The Obama administration announced last week it would cut the insurance premium by a quarter of a percentage point to 0.60 percent, effective on Jan. 27.

Some housing industry groups lauded the change, saying it could increase home buying by offsetting recent rises in mortgage rates. Supporters of the reduction were disappointed that the Trump administration reversed course.

‘Average People’

“This action is completely out of alignment with President Trump’s words about having the government work for the people,” said John Taylor, president of the National Community Reinvestment Coalition, through a spokesman. “Exactly how does raising the cost of buying a home help average people?”

Sarah Edelman, director of housing policy for the left-leaning Center for American Progress, in an e-mail wrote, “On Day 1, the president has turned his back on middle-class families -- this decision effectively takes $500 out of the pocketbooks of families that were planning to buy a home in 2017. This is not the way to build a strong economy.”

The FHA came under severe stress after the financial crisis. In 2013, it needed $1.7 billion from the U.S. Treasury, its first bailout in 79 years, due to a wave of defaults. To replenish the FHA’s coffers, the Obama administration several times increased the fees the agency charges. The law requires the FHA’s capital reserve ratio to stay above 2 percent, and the agency hit that level in 2015 for the first time since the bailout.

“It is important to ensure that the FHA fund remains strong to support homeownership in the future while minimizing taxpayer risk,” Teresa Bryce Bazemore, president of Radian Group, said in a statement.

 

 

 

An Idiots Guide to Really Really Bad Programming

https://www.codeproject.com/kb/scrapbook/bad_programming.aspx

An Idiots Guide to Really Really Bad Programming

A satirical look at bad programming techniques

 

Introduction

I have noticed that certain individuals persist in writing good clear code that is well documented and easy to understand. So I thought it was about time to draw the main techniques of really really bad programming together into a single reference document. Remember the most important two things when writing truly awful software are to include as many subtle bugs as possible and to make the code really confusing so that it is hard to track those devious little bugs down. That way you can annoy both users and programmers to the maximum extent.

If you follow these simple rules, I promise you that you will never be far from users’ and other programmers’ thoughts.

1. Turn Off All Compiler Warnings

If you want to write really awful code, this is a great place to start. After all, you do not want that pesky compiler nagging you all day. All real programmers turn the warnings off completely.

2. Variables

Beginners should ensure that all variables are global. It is much easier to keep them all together in one place. As you become more advanced, add local variables with the same names as the globals, this will allow you to create subtle bugs with ease.

3. Functions

Avoid using functions wherever possible. It is much easier to write one single enormous long passage of code than going through the tiresome process of trying to divide it into separate functions and having the tedious job of passing parameters to them.

4. Cut and Paste is your Friend

Avoid using loops at all costs, it might take longer to cut and paste a section of code over and over, but it is well worth it when you consider the hours of fun for anyone checking each and every repeat of that code for subtle variations.

5. Variable Names for Beginners

You can have so much fun with variable names, try using the most meaningless name that you can think of, Fred is one of my personal favourites. But always include a smattering of semi meaningful names to keep other programmers guessing.

6. Advanced Variable Name Techniques

Ideally try to think of names that are very similar for as many entirely different variables as you can. Another great idea is to create two variables with the same name, but one ending in ‘1’ and the other in ‘l’, these are very easy to confuse at a glance and should get other programmers really guessing. Example: Slopel and Slope1. However, you can achieve the best results by occasionally using a label to mean the exact opposite of what people would assume, a classic example is to use the label Horizontal to mean Vertical and vice versa, that will really please any reader of your code, causing hours of amusement and pleasure as they try to unravel it.

7. Initialising

Play programmers bingo by allowing all of your variables to start with whatever value the memory had in from the last application. Great fun!

8. Bounds Checking

Allow the user to enter whatever values they want, if they enter a stupid value the program may crash. So? What do they expect?

9. Comments

It is best for beginners to avoid comments at all times, they waste valuable space and take ages to type. As you become more advanced, you can begin by adding comments which are entirely useless, because they state the obvious such as:

i++; // increment i

However, the most advanced comments will be as cryptic as possible, such as:

i++; 	//check inside the chicken string

10. Layout

It takes real dedication to make your software layout truly bad. The most important thing is NEVER EVER be consistent. Whatever layout philosophy you choose, be sure to change it regularly. Advanced programmers should remember to use the same layout for long enough for any other programmer to become accustomed to it before changing to something entirely different.

11. Hungarian Notation

This is entirely optional, but for best results use a smattering of Hungarian Notation, thus annoying everyone (those who like it AND those who don't) and of course include a few incorrect uses just to make things more interesting.

12. Lastly Never Ever Test

Don't bother to test your code, just wait for people to complain, that way you will find out which are the most commonly occurring bugs first! Brilliant!

I have tried to cover most aspects of Really Really Bad Programming, but I am sure many of you can think of a few more. And don't be down hearted if after a while you start to slip into good habits, it doesn't take much effort to get back to those bad bad ways.

Disclaimer

I am not in any way suggesting that I never write bad code, nor am I suggesting that anyone else at CodeProject does. Just having a little fun!

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Swarm Robotics ! The Nueroscience

 

Swarm Robotics | the Neuroscience and Robotics Laboratory We're developing a framework for self-organizing robot systems based on decentralized estimation and control. The goal is to "compile" desired group behaviors into local communication, estimation, and control laws running on individual robots. For properly designed control laws, the interactions of individual robots result in the desired group behavior, without any centralized control.