You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This repository contains examples and best practices for building recommendation systems, provided as Jupyter notebooks. The examples detail our learnings on five key tasks:
4
-
-[Prepare Data](notebooks/01_prepare_data/README.md): Preparing and loading data for each recommender algorithm
5
-
-[Model](notebooks/02_model/README.md): Building models using various classical and deep learning recommender algorithms such as Alternating Least Squares ([ALS](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/recommendation.html#ALS)) or eXtreme Deep Factorization Machines ([xDeepFM](https://arxiv.org/abs/1803.05170)).
6
-
-[Evaluate](notebooks/03_evaluate/README.md): Evaluating algorithms with offline metrics
6
+
-[Prepare Data](notebooks/01_prepare_data): Preparing and loading data for each recommender algorithm
7
+
-[Model](notebooks/02_model): Building models using various classical and deep learning recommender algorithms such as Alternating Least Squares ([ALS](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/recommendation.html#ALS)) or eXtreme Deep Factorization Machines ([xDeepFM](https://arxiv.org/abs/1803.05170)).
8
+
-[Evaluate](notebooks/03_evaluate): Evaluating algorithms with offline metrics
7
9
-[Model Select and Optimize](notebooks/04_model_select_and_optimize): Tuning and optimizing hyperparameters for recommender models
8
-
-[Operationalize](notebooks/05_operationalize/README.md): Operationalizing models in a production environment on Azure
10
+
-[Operationalize](notebooks/05_operationalize): Operationalizing models in a production environment on Azure
9
11
10
12
Several utilities are provided in [reco_utils](reco_utils) to support common tasks such as loading datasets in the format expected by different algorithms, evaluating model outputs, and splitting training/test data. Implementations of several state-of-the-art algorithms are included for self-study and customization in your own applications. See the [reco_utils documentation](https://readthedocs.org/projects/microsoft-recommenders/).
11
13
12
14
13
15
For a more detailed overview of the repository, please see the documents at the [wiki page](https://github.com/microsoft/recommenders/wiki/Documents-and-Presentations).
14
16
15
17
## Getting Started
16
-
Please see the [setup guide](SETUP.md) for more details on setting up your machine locally, on Spark, or on [Azure Databricks](SETUP.md#setup-guide-for-azure-databricks).
18
+
Please see the [setup guide](SETUP.md) for more details on setting up your machine locally, on a [data science virtual machine (DSVM)](https://azure.microsoft.com/en-gb/services/virtual-machines/data-science-virtual-machines/) or on [Azure Databricks](SETUP.md#setup-guide-for-azure-databricks).
17
19
18
20
To setup on your local machine:
19
21
1. Install Anaconda with Python >= 3.6. [Miniconda](https://conda.io/miniconda.html) is a quick way to get started.
@@ -35,27 +37,11 @@ To setup on your local machine:
35
37
```
36
38
5. Start the Jupyter notebook server
37
39
```
38
-
cd notebooks
39
40
jupyter notebook
40
41
```
41
-
6. Run the [SAR Python CPU MovieLens](notebooks/00_quick_start/sar_movielens.ipynb) notebook under the 00_quick_start folder. Make sure to change the kernel to "Python (reco)".
42
-
43
-
**NOTE** - The [Alternating Least Squares (ALS)](notebooks/00_quick_start/als_movielens.ipynb) notebooks require a PySpark environment to run. Please follow the steps in the [setup guide](SETUP.md#dependencies-setup) to run these notebooks in a PySpark environment.
44
-
45
-
## Install this repository via PIP
46
-
A [setup.py](reco_utils/setup.py) file is provided in order to simplify the installation of this utilities in this repo from the main directory.
47
-
This still requires the conda environment to be installed as described above. Once the necessary dependencies are installed you can use the following command to install reco_utils as it's own python package.
48
-
49
-
pip install -e reco_utils
50
-
51
-
It is also possible to install directly from Github. Or from a specific branch as well.
**NOTE** - The pip installation does not install any of the necessary package dependencies, it is expected that conda will be used as shown above to setup the environment for the utilities being used.
42
+
6. Run the [SAR Python CPU MovieLens](notebooks/00_quick_start/sar_movielens.ipynb) notebook under the `00_quick_start` folder. Make sure to change the kernel to "Python (reco)".
58
43
44
+
**NOTE** - The [Alternating Least Squares (ALS)](notebooks/00_quick_start/als_movielens.ipynb) notebooks require a PySpark environment to run. Please follow the steps in the [setup guide](SETUP.md#dependencies-setup) to run these notebooks in a PySpark environment. For the deep learning algorithms, it is recommended to use a GPU machine.
59
45
60
46
## Algorithms
61
47
@@ -90,31 +76,32 @@ We provide a [benchmark notebook](benchmarks/movielens.ipynb) to illustrate how
This project welcomes contributions and suggestions. Before contributing, please see our [contribution guidelines](CONTRIBUTING.md).
96
80
81
+
This project welcomes contributions and suggestions. Before contributing, please see our [contribution guidelines](CONTRIBUTING.md).
97
82
98
83
## Build Status
99
84
100
-
| Build Type | Branch | Status | | Branch | Status |
101
-
| --- | --- | --- | --- | --- | --- |
85
+
These tests are the nightly builds, which compute the smoke and integration tests. `master` is our main branch and `staging` is our development branch. We use `pytest` for testing python utilities in [reco_utils](reco_utils) and `papermill` for the [notebooks](notebooks). For more information about the testing pipelines, please see the [test documentation](tests/README.md).
86
+
87
+
### DSVM Build Status
88
+
89
+
The following tests run on a Windows and Linux DSVM daily. These machines run 24/7.
90
+
91
+
| Build Type | Branch | Status | | Branch | Status |
These DevOps pipelines run the existing tests on AzureML.
102
+
The following tests run on an AzureML [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-compute-target). AzureML allows to programmatically start a virtual machine, execute the tests, gather the results in [Azure DevOps](https://azure.microsoft.com/en-gb/services/devops/) and shut down the machine.
112
103
113
104
| Build Type | Branch | Status | | Branch | Status |
**NOTE** - these tests are the nightly builds, which compute the smoke and integration tests. Master is our main branch and staging is our development branch. We use `pytest` for testing python utilities in [reco_utils](reco_utils) and `papermill` for the [notebooks](notebooks). For more information about the testing pipelines, please see the [test documentation](tests/README.md).
*[Troubleshooting Installation on Azure Databricks](#Troubleshooting-Installation-on-Azure-Databricks)
22
-
*[Prepare Azure Databricks for Operationalization](#prepare-azure-databricks-for-operationalization)
22
+
*[Prepare Azure Databricks for Operationalization](#prepare-azure-databricks-for-operationalization)
23
+
*[Install the utilities via PIP](#install-the-utilities-via-pip)
23
24
*[Setup guide for Docker](#setup-guide-for-docker)
24
25
25
26
## Compute environments
@@ -270,7 +271,7 @@ import reco_utils
270
271
271
272
* For the [reco_utils](reco_utils) import to work on Databricks, it is important to zip the content correctly. The zip has to be performed inside the Recommenders folder, if you zip directly above the Recommenders folder, it won't work.
272
273
273
-
## Prepare Azure Databricks for Operationalization
274
+
###Prepare Azure Databricks for Operationalization
274
275
275
276
This repository includes an end-to-end example notebook that uses Azure Databricks to estimate a recommendation model using matrix factorization with Alternating Least Squares, writes pre-computed recommendations to Azure Cosmos DB, and then creates a real-time scoring service that retrieves the recommendations from Cosmos DB. In order to execute that [notebook](notebooks/05_operationalize/als_movie_o16n.ipynb), you must install the Recommenders repository as a library (as described above), **AND** you must also install some additional dependencies. With the *Quick install* method, you just need to pass an additional option to the [installation script](scripts/databricks_install.py).
276
277
@@ -313,6 +314,21 @@ Additionally, you must install the [spark-cosmosdb connector](https://docs.datab
313
314
314
315
</details>
315
316
317
+
## Install the utilities via PIP
318
+
319
+
A [setup.py](reco_utils/setup.py) file is provided in order to simplify the installation of the utilities in this repo from the main directory.
320
+
321
+
This still requires the conda environment to be installed as described above. Once the necessary dependencies are installed, you can use the following command to install `reco_utils` as a python package.
322
+
323
+
pip install -e reco_utils
324
+
325
+
It is also possible to install directly from Github. Or from a specific branch as well.
**NOTE** - The pip installation does not install any of the necessary package dependencies, it is expected that conda will be used as shown above to setup the environment for the utilities being used.
331
+
316
332
## Setup guide for Docker
317
333
318
334
A [Dockerfile](docker/Dockerfile) is provided to build images of the repository to simplify setup for different environments. You will need [Docker Engine](https://docs.docker.com/install/) installed on your system.
Copy file name to clipboardExpand all lines: notebooks/01_prepare_data/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ data preparation tasks witnessed in recommendation system development.
8
8
| --- | --- |
9
9
|[data_split](data_split.ipynb)| Details on splitting data (randomly, chronologically, etc). |
10
10
|[data_transform](data_transform.ipynb)| Guidance on how to transform (implicit / explicit) data for building collaborative filtering typed recommender. |
11
-
|[wikidata knowledge graph](wikidata_KG.ipynb)| Details on how to create a knowledge graph using Wikidata |
11
+
|[wikidata knowledge graph](wikidata_knowledge_graph.ipynb)| Details on how to create a knowledge graph using Wikidata |
Copy file name to clipboardExpand all lines: notebooks/01_prepare_data/wikidata_knowledge_graph.ipynb
+11-15Lines changed: 11 additions & 15 deletions
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@
5
5
"metadata": {},
6
6
"source": [
7
7
"## Wikidata Knowledge Graph Extraction\n",
8
-
"Many recommendation algorithms (DKN, RippleNet, KGCN) use Knowledge Graphs as an external source of information. We found that one of the bottlenecks to benchmark current algorithms like DKN, RippleNet or KGCN is that they used Microsoft Satori. As Satori is not open source, it's not possible to replicate the results found in the papers. The solution is using other open source KGs.\n",
8
+
"Many recommendation algorithms (DKN, RippleNet, KGCN) use Knowledge Graphs (KGs) as an external source of information. We found that one of the bottlenecks to benchmark current algorithms like DKN, RippleNet or KGCN is that they used Microsoft Satori. As Satori is not open source, it's not possible to replicate the results found in the papers. The solution is using other open source KGs.\n",
9
9
"\n",
10
10
"The goal of this notebook is to provide examples of how to interact with Wikipedia queries and Wikidata to extract a Knowledge Graph that can be used with the mentioned algorithms.\n",
11
11
"\n",
@@ -24,7 +24,8 @@
24
24
"name": "stdout",
25
25
"output_type": "stream",
26
26
"text": [
27
-
"System version: 3.6.8 |Anaconda, Inc.| (default, Feb 21 2019, 18:30:04) [MSC v.1916 64 bit (AMD64)]\n"
0 commit comments