fm_tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Factorization Machines with tensorflow tutorial - Python3\n",
    "\n",
    "In this tutial we demonstrate how to create a FMs model with tensorflow step-by-step.\n",
    "\n",
    "### References:\n",
    "\n",
    "Blog post by Gabriele Modena: [Factorization Machines with Tensorflow](http://nowave.it/factorization-machines-with-tensorflow.html)\n",
    "\n",
    "Factorization Machines paper: [Factorization Machines with LibFm (pdf)](http://www.csie.ntu.edu.tw/~b97053/paper/Factorization%20Machines%20with%20libFM.pdf)\n",
    "\n",
    "\n",
    "### Relevant repository:\n",
    "\n",
    "[tffm library](https://github.com/geffy/tffm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Utility function to convert list to sparse matrix\n",
    "\n",
    "Here we created a utility function to create a sparse matrix (that is needed by factorization machines) from a list of user/item ids.\n",
    "\n",
    "Check [this gist](https://gist.github.com/babakx/7a3fc9739b7778f6673a458605e18963) for more details about this utitly function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import count \n",
    "from collections import defaultdict\n",
    "from scipy.sparse import csr    \n",
    "from __future__ import print_function \n",
    "\n",
    "def vectorize_dic(dic, ix=None, p=None):\n",
    "    \"\"\" \n",
    "    Creates a scipy csr matrix from a list of lists (each inner list is a set of values corresponding to a feature) \n",
    "    \n",
    "    parameters:\n",
    "    -----------\n",
    "    dic -- dictionary of feature lists. Keys are the name of features\n",
    "    ix -- index generator (default None)\n",
    "    p -- dimension of featrure space (number of columns in the sparse matrix) (default None)\n",
    "    \"\"\"\n",
    "    if (ix == None):\n",
    "        d = count(0)\n",
    "        ix = defaultdict(lambda: next(d)) \n",
    "        \n",
    "    n = len(list(dic.values())[0]) # num samples\n",
    "    g = len(list(dic.keys())) # num groups\n",
    "    nz = n * g # number of non-zeros\n",
    "\n",
    "    col_ix = np.empty(nz, dtype=int)     \n",
    "    \n",
    "    i = 0\n",
    "    for k, lis in dic.items():     \n",
    "        # append index el with k in order to prevet mapping different columns with same id to same index\n",
    "        col_ix[i::g] = [ix[str(el) + str(k)] for el in lis]\n",
    "        i += 1\n",
    "        \n",
    "    row_ix = np.repeat(np.arange(0, n), g)      \n",
    "    data = np.ones(nz)\n",
    "    \n",
    "    if (p == None):\n",
    "        p = len(ix)\n",
    "        \n",
    "    ixx = np.where(col_ix < p)\n",
    "\n",
    "    return csr.csr_matrix((data[ixx],(row_ix[ixx], col_ix[ixx])), shape=(n, p)), ix          "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data\n",
    "\n",
    "In this tutorial we use the [MovieLens100k Dataset](https://grouplens.org/datasets/movielens/100k/). Here we convert data to scipy csr (sparse) matrix format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "\n",
    "# laod data with pandas\n",
    "cols = ['user', 'item', 'rating', 'timestamp']\n",
    "train = pd.read_csv('data/ua.base', delimiter='\\t', names=cols)\n",
    "test = pd.read_csv('data/ua.test', delimiter='\\t', names=cols)\n",
    "\n",
    "# vectorize data and convert them to csr matrix\n",
    "X_train, ix = vectorize_dic({'users': train.user.values, 'items': train.item.values})\n",
    "X_test, ix = vectorize_dic({'users': test.user.values, 'items': test.item.values}, ix, X_train.shape[1])\n",
    "y_train = train.rating.values\n",
    "y_test= test.rating.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Densifying the input matrices\n",
    "Here we convert the two matrices of `X_train` and `X_test` to dense format to be able to feed them to the tf model. For large datasets this trick is not recommended. You can use `tf.SparseTensor` for large sparse datasets. Check [this file from tffm library](https://github.com/geffy/tffm/blob/a98c786917f5ca74a249748ddef8b694b7f823c9/tffm/core.py#L127) to see how a sparse tensor can be defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(90570, 2623)\n",
      "(9430, 2623)\n"
     ]
    }
   ],
   "source": [
    "X_train = X_train.todense()\n",
    "X_test = X_test.todense()\n",
    "\n",
    "# print shape of data\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define FM Model with tensorflow\n",
    "\n",
    "We first initialize the parameters of the model as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "n, p = X_train.shape\n",
    "\n",
    "# number of latent factors\n",
    "k = 10\n",
    "\n",
    "# design matrix\n",
    "X = tf.placeholder('float', shape=[None, p])\n",
    "# target vector\n",
    "y = tf.placeholder('float', shape=[None, 1])\n",
    "\n",
    "# bias and weights\n",
    "w0 = tf.Variable(tf.zeros([1]))\n",
    "W = tf.Variable(tf.zeros([p]))\n",
    "\n",
    "# interaction factors, randomly initialized \n",
    "V = tf.Variable(tf.random_normal([k, p], stddev=0.01))\n",
    "\n",
    "# estimate of y, initialized to 0.\n",
    "y_hat = tf.Variable(tf.zeros([n, 1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we define how the output values y should be calculated\n",
    "Using the trick in Rendle's paper, the output of a give feature vector `x` can be calculated using the following equation. The next cell implements that with tensorflow operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$\\hat{y}(\\mathbf{x}) = w_0 + \\sum_{j=1}^{p}w_jx_j + \\frac{1}{2} \\sum_{f=1}^{k} ((\\sum_{j=1}^{p}v_{j,f}x_j)^2-\\sum_{j=1}^{p}v_{j,f}^2 x_j^2)$$"
      ],
      "text/plain": [
       "<IPython.core.display.Math object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Math, Latex\n",
    "display(Math(r'\\hat{y}(\\mathbf{x}) = w_0 + \\sum_{j=1}^{p}w_jx_j + \\frac{1}{2} \\sum_{f=1}^{k} ((\\sum_{j=1}^{p}v_{j,f}x_j)^2-\\sum_{j=1}^{p}v_{j,f}^2 x_j^2)'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate output with FM equation\n",
    "linear_terms = tf.add(w0, tf.reduce_sum(tf.multiply(W, X), 1, keep_dims=True))\n",
    "pair_interactions = (tf.multiply(0.5,\n",
    "                    tf.reduce_sum(\n",
    "                        tf.subtract(\n",
    "                            tf.pow( tf.matmul(X, tf.transpose(V)), 2),\n",
    "                            tf.matmul(tf.pow(X, 2), tf.transpose(tf.pow(V, 2)))),\n",
    "                        1, keep_dims=True)))\n",
    "y_hat = tf.add(linear_terms, pair_interactions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss function\n",
    "\n",
    "Here we implement FM point-wise loss function with tensorflow operations. The loss is defined as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$L = \\sum_{i=1}^{n} (y_i - \\hat{y}_i)^2 + \\lambda_w ||W||^2 + \\lambda_v ||V||^2$$"
      ],
      "text/plain": [
       "<IPython.core.display.Math object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Math(r'L = \\sum_{i=1}^{n} (y_i - \\hat{y}_i)^2 + \\lambda_w ||W||^2 + \\lambda_v ||V||^2'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# L2 regularized sum of squares loss function over W and V\n",
    "lambda_w = tf.constant(0.001, name='lambda_w')\n",
    "lambda_v = tf.constant(0.001, name='lambda_v')\n",
    "\n",
    "l2_norm = (tf.reduce_sum(\n",
    "            tf.add(\n",
    "                tf.multiply(lambda_w, tf.pow(W, 2)),\n",
    "                tf.multiply(lambda_v, tf.pow(V, 2)))))\n",
    "\n",
    "error = tf.reduce_mean(tf.square(tf.subtract(y, y_hat)))\n",
    "loss = tf.add(error, l2_norm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization\n",
    "\n",
    "Given a loss function, tensorflow can automatically calculate the derivatives of the loss function and find the optimal values for the 'variables' of the loss function. Under the hood, gradient descent optimizer update model parameters iteratively with the following update rule:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$$\\Theta_{i+1} = \\Theta_{i} - \\eta \\frac{\\delta L}{\\delta \\Theta}$$"
      ],
      "text/plain": [
       "<IPython.core.display.Math object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Math(r'\\Theta_{i+1} = \\Theta_{i} - \\eta \\frac{\\delta L}{\\delta \\Theta}'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Batches\n",
    "\n",
    "With SGD or Adam optimization methods, you can update model parameters in mini-batches. The larges the size of mini-batches are, the faster is the optimization method but it can become harder to find the optimized parameters. Size of mini-batches is a trade-off between accuracy and complexity. Here we implement a method to generate mini-batches from the input data. Check this great weblog for an [overview of gradient descent optimization methods](http://ruder.io/optimizing-gradient-descent/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batcher(X_, y_=None, batch_size=-1):\n",
    "    n_samples = X_.shape[0]\n",
    "\n",
    "    if batch_size == -1:\n",
    "        batch_size = n_samples\n",
    "    if batch_size < 1:\n",
    "       raise ValueError('Parameter batch_size={} is unsupported'.format(batch_size))\n",
    "\n",
    "    for i in range(0, n_samples, batch_size):\n",
    "        upper_bound = min(i + batch_size, n_samples)\n",
    "        ret_x = X_[i:upper_bound]\n",
    "        ret_y = None\n",
    "        if y_ is not None:\n",
    "            ret_y = y_[i:i + batch_size]\n",
    "            yield (ret_x, ret_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lanching tensorflow graph and training the model\n",
    "Finally we can strat the tensorflow session to initialize the variabeles and optimize the model prameters. Training consists of running the `optimizer` operation and feeding the mini-batches to the optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "983ce6eb5e604ee5aee7c85314180d77",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=10), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm_notebook as tqdm\n",
    "\n",
    "epochs = 10\n",
    "batch_size = 1000\n",
    "\n",
    "# Launch the graph\n",
    "init = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "\n",
    "sess.run(init)\n",
    "\n",
    "for epoch in tqdm(range(epochs), unit='epoch'):\n",
    "    perm = np.random.permutation(X_train.shape[0])\n",
    "    # iterate over batches\n",
    "    for bX, bY in batcher(X_train[perm], y_train[perm], batch_size):\n",
    "        sess.run(optimizer, feed_dict={X: bX.reshape(-1, p), y: bY.reshape(-1, 1)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the model\n",
    "We can now evaluate the trained model on out test set. We use RMSE to measure the error of predictions. Note that here we need to run the `error` operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2851532\n"
     ]
    }
   ],
   "source": [
    "errors = []\n",
    "for bX, bY in batcher(X_test, y_test):\n",
    "    errors.append(sess.run(error, feed_dict={X: bX.reshape(-1, p), y: bY.reshape(-1, 1)}))\n",
    "\n",
    "RMSE = np.sqrt(np.array(errors).mean())\n",
    "print(RMSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closing tensorflow session\n",
    "After finishing your experiment make sure you close the tf session that you crated to free the memory it uses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}