The Ultimate Guide to Random Forest Regression

2. Machine learning approaches to random forest

Random forest is both a supervised learning algorithm and an ensemble algorithm. 

It is supervised in the sense that during training, it learns the mappings between inputs and outputs. For example, an input feature (or independent variable) in the training dataset would specify that an apartment has “3 bedrooms” (feature: number of bedrooms) and this maps to the output feature (or target) that the apartment will be sold for “$200,000” (target: price sold). 

Ensemble algorithms combine multiple other machine learning algorithms, in order to make more accurate predictions than any underlying algorithm could on its own. In the case of random forest, it ensembles multiple decision trees into its final decision.

Random forest can be used on both regression tasks (predict continuous outputs, such as price) or classification tasks (predict categorical or discrete outputs). Here, we will take a deeper look at using random forest for regression predictions.

2.1 The random forest regression model

The random forest algorithm follows a two-step process:

1. Builds n decision tree regressors (estimators). The number of estimators n defaults to 100 in Scikit Learn (the machine learning Python library), where it is called n_estimators. The trees are built following the specified hyperparameters (e.g. minimum number of samples at the leaf nodes, maximum depth that a tree can grow, etc.).
2. Average prediction across estimators. Each decision tree regression predicts a number as an output for a given input. Random forest regression takes the average of those predictions as its ‘final’ output.

‍

Let’s delve deeper into how random forest regression builds regression trees. 

2.1.1 Decision tree regression

Regression using decision trees follows the same pattern as any decision tree algorithm:

1. Attribute selection. The decision tree regression algorithm looks at all attributes and their values to determine which attribute value would lead to the ‘best split’. For regression problems, the algorithm looks at MSE (mean squared error) as its objective or cost function, which needs to be minimized. This is equal to variance reduction as a feature selection criterion. Note: Scikit learn also has an MAE (mean absolute error) implementation.

2. Once it finds the best split point candidate, it splits the dataset at that value (called the root node) and repeats the process of attribute selection for the other ranges.

3. The algorithm continues iteratively until either:

a) We have grown terminal or leaf nodes so that they reach each sample (there are no stopping criteria).

b) We reached some stopping criteria. For example, we might have set a maximum depth, which only allows a certain number of splits from the root node to the terminal nodes. Or we might have set a minimum number of samples in each terminal node to prevent them from splitting beyond a certain point.

So, why is a single tree not enough? Why do we need a forest of trees?

2.1.2 Why are decision trees not enough?

Decision trees have a couple of problems:

1. The main problem is that they tend to overfit very easily. This causes high variance, which can be seen as high test errors on the test dataset, despite high accuracy on the training dataset. In other words, decision trees do not generalize well to novel data.
2. Decision trees are easily swayed by data that splits the attributes well. Imagine that there’s a single feature, whose values almost deterministically split the dataset. Let's say that when X1 = 0, then Y will always be below 10, while when X1 = 1, then Y will always be equal or greater to 10. Almost every decision tree will use this feature in its split criteria, making the trees overly correlated with each other.
   

The ensemble of decision trees introduces randomness, which mitigates the issues above. So how does random forest impose randomness? And how does this help make better predictions?

2.1.3 What is random about random forest regression?

The ensemble of decision trees has high accuracy because it uses randomness on two levels:

1. The algorithm randomly selects a subset of features, which can be used as candidates at each split. This prevents the multitude of decision trees from relying on the same set of features, which automatically solves Problem 2 above and decorrelates individual trees.
2. Each tree draws a random sample of data from the training dataset when generating its splits. This introduces a further element of randomness, which prevents the individual trees from overfitting the data. Since they cannot see all of the data, they cannot overfit it.
   

Ensembling decision trees allows us to compensate for the weaknesses of each individual tree.

2.3 Beyond random forest: how to improve the model

The base model can be improved in a couple of ways by tuning the parameters of the random forest regressor:

1. Specify the maximum depth of the trees. By default, trees are expanded until all leaves are either pure or contain less than the minimum samples for the split. This can still cause the trees to overfit or underfit. Play with the hyperparameter to find an optimal number for max_depth.
2. Increase or decrease the number of estimators. How does changing the number of trees affect performance? More trees usually means higher accuracy at the cost of slower learning. If you wish to speed up your random forest, lower the number of estimators. If you want to increase the accuracy of your model, increase the number of trees.
3. Specify the maximum number of features to be included at each node split. This depends very heavily on your dataset. If your independent variables are highly correlated, you’ll want to decrease the maximum number of features. If your input attributes are not correlated and your model is suffering from low accuracy, increase the number of features to be included.

3. Random forest regression in practice

The way in which you use random forest regression in practice depends on how much you know about the entire data science process.

We recommend that beginners start by modeling data on datasets that have already been collected and cleaned, while experienced data scientists can scale their operations by choosing the right software for the task at hand.

3.1 Beginner projects to try out random forest regression

There are over 84 datasets to try out random forest regression in practice. Among the best ones are:

1. Predict the salary based on years of work experience.
2. Build an economic prediction of US food imports based on historical data.
3. Compare how different advertising channels affect total sales.
4. Predict the number of upvotes a social media post will get.
5. Predict the price at which the house will sell on a market given the real estate description.
   

3.2 Production software for advanced data science

Data scientists spend more than 80% of their time on data collection and cleaning. If you want to speed up the entire data pipeline, use software that automates tasks to give you more time for data modeling. 

Keboola offers a platform for data scientists who want to build their own machine learning models. It comes with one-click deployed Jupyter Notebooks, through which all of the modeling can be done using Julia, R, or Python. 

Deep dive into the data science process with this Jupyter Notebook:

1. Collect the relevant data.
2. Explore and clean the data to discover patterns.
3. Train your random forest regression model.
4. Evaluate the model with a variety of metrics.
   

Want to take it a step further? Keboola can assist you with instrumentalizing your entire data operations pipeline. 

Being a data-centric platform, Keboola also allows you to build your ETL pipelines and orchestrate tasks to get your data ready for machine learning algorithms. Deploy multiple models with different algorithms to version your work and compare which ones perform best. Start building models today with our free trial. 

‍