https://github.com/Nixtla/statsforecast/blob/main/experiments/neuralprophet/README.md Skip to content Sign up * Product + Features + Mobile + Actions + Codespaces + Copilot + Packages + Security + Code review + Issues + Discussions + Integrations + GitHub Sponsors + Customer stories * Team * Enterprise * Explore + Explore GitHub + Learn and contribute + Topics + Collections + Trending + Skills + GitHub Sponsors + Open source guides + Connect with others + The ReadME Project + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing + Plans + Compare plans + Contact Sales + Education [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} Nixtla / statsforecast Public * Notifications * Fork 57 * Star 974 * Code * Issues 34 * Pull requests 3 * Discussions * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights Permalink main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags statsforecast/experiments/neuralprophet/README.md Go to file * Go to file T * Go to line L * * Copy path * Copy permalink This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. @FedericoGarza FedericoGarza feat: add ercot explanation [skip ci] Latest commit 874b6f1 Aug 16, 2022 History 2 contributors Users who have contributed to this file @FedericoGarza @mergenthaler [ ] ETS is faster and more accurate than NeuralProphet (in most cases). Install StatsForecast Results on ERCOT, ETTm2, M3, M4, and Tourism: Background Empirical validation Notes Results Reproducibility Conclusion Misc. 78 lines (50 sloc) 5.19 KB Raw Blame Edit this file E Open in GitHub Desktop * Open with Desktop * View raw * * View blame ETS is faster and more accurate than NeuralProphet (in most cases). We benchmarked on more than 55K series and show that ETS improves MAPE and sMAPE forecast accuracy by 32% and 19%, respectively, with 104x less computational time over NeuralProphet. Install StatsForecast pip install statsforecast Results on ERCOT, ETTm2, M3, M4, and Tourism: comparison Background In this experiment, we test NeuralProphet's introduction as a "successor" to Facebook-Prophet and compare it with classic Exponential Smoothing (ETS). We show that it is not the case that ETS has "too restrictive assumptions and parametric nature limit their performance in real-world applications" as claimed by NeuralProphet's paper. Moreover, we show that NeuralProphet's predictions are outperformed both in accuracy and computation time by this statistical model. Empirical validation To compare NeuralProphet against ETS, we designed a pipeline considering the M3, M4, and Tourism datasets (standard benchmarks in the forecasting practice). In correspondence with the author he suggested to further test the model in electricity datasets and try including auto regresor and covariates. Therefore we also included the ERCOT dataset (used in the NeuralProphet's documentation), and ETTm2. NeuralProphet fits the time series globally using autoregressive terms and produces forecasts using a multistep approach. Notes * We used the out-of-the-box configuration of the NeuralProphet model in its global-multistep version, using autoregressive terms based on the seasonality of the data and the forecast horizon. This experiment concludes that hyperparameter optimization could be highly costly, particularly for big datasets. * For the ERCOT dataset, we used the NeuralProphet configuration used in its documentation. * Additionally, we test the performance of NeuralProphet using different learning rates (1e-5, 1e-4, 1e-3, 1e-2, 1e-1). The performance is similar. * During the execution of the experiment, we found issues with the NeuralProphet implementation related to Monthly, Quarterly, and Yearly frequencies. We fixed the issue and opened a Pull Request to solve the problem. * According to the paper and a discussion on GitHub, the NeuralProphet implementation is not available in GPU. There is a work-in-progress Pull Request, though. * We also performed experiments for the M4-Monthly data set, but NeuralProphet did not finish after three days of computation. ETS results are reported. Results The following table shows the MAPE, sMAPE, and Time (in minutes) ETS improvements over NeuralProphet for each dataset. table Reproducibility 1. Create a conda environment exp_neuralprophet using the environment.yml file. conda env create -f environment.yml 3. Activate the conda environment using conda activate exp_neuralprophet 4. Run the experiments for each dataset and each model using python -m src.[model] --dataset [dataset] --group [group] The variable model can be statsforecast (ETS model) or neuralprophet. For M4, the groups are Yearly, Quarterly, Weekly, Daily, and Hourly. For M3, the groups are Yearly, Monthly, Quarterly, and Other. For Tourism, the groups are Yearly, Monthly, and Quarterly. To run ETTm2 use LongHorizon as dataset and ETTm2 as group. To run ERCOT use ERCOT as dataset and Other as group. 5. Evaluate the results using python -m src.evaluation Conclusion * Always use strong baselines when forecasting. * Quick and easy results are sometimes misleading. * Simpler models are sometimes better. * Both Prophet and NeuralProphet are definitely not models for Forecasting at Scale. * We find the claim that "NeuralProphet bridges the gap between traditional time-series models and deep learning methods." simply to be false, given that this model does not outperform classical statistical methods neither in accuracy nor speed. Misc. * StatsForecast also includes a variety of lightning fast baseline models. * If you really need to do forecast at scale, here we show how to forecast 1 million time series under 30 minutes using Ray. * If you are interested in SOTA Deep Learning models, check NeuralForecast [ ] Go Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.