https://github.com/sogou/workflow Skip to content Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Project management - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} sogou / workflow * Notifications * Star 4.2k * Fork 813 C++ Parallel Computing and Asynchronous Networking Engine Apache-2.0 License 4.2k stars 813 forks Star Notifications * Code * Issues 9 * Pull requests 1 * Actions * Projects 0 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 3 branches 6 tags Code Clone HTTPS GitHub CLI [https://github.com/s] Use Git or checkout with SVN using the web URL. [gh repo clone sogou/] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @Barenboim Barenboim Merge pull request #384 from Barenboim/master ... 11cd77a May 23, 2021 Merge pull request #384 from Barenboim/master Make sure WFServer's new_connection() returns a WFConnection. 11cd77a Git stats * 728 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time benchmark docs: move images to another repository Oct 20, 2020 docs Update tutorial-12-mysql_cli.md May 19, 2021 src Make sure WFServer's new_connection() returns a WFConnection, and pro... May 22, 2021 test update testcase Mar 26, 2021 tutorial Merge pull request #373 from holmes1412/master May 19, 2021 .editorconfig initialization Jul 29, 2020 .gitignore Initial commit Jul 28, 2020 .travis.yml Update .travis.yml Feb 9, 2021 CMakeLists.txt fix addrinfo with unix domain socket (#337) Apr 14, 2021 CMakeLists_Headers.txt add MySQLSSLRequest for SSL MySQL client. May 17, 2021 CODE_OF_CONDUCT.md add CODE_OF_CONDUCT Aug 14, 2020 GNUmakefile add kafka client (#96) Nov 13, 2020 LICENSE Initial commit Jul 28, 2020 README.md Update README.md May 1, 2021 README_cn.md Update README_cn.md May 1, 2021 configure initialization Jul 29, 2020 workflow-config.cmake.in fix targets in workflow-config.cmake.in Mar 31, 2021 View code Sogou C++ Workflow You can use it: Compiling and running environment Get started (Linux, macOS): Tutorials System design features More design documents README.md Zhong Wen Ban Ru Kou Sogou C++ Workflow License Language Platform Build Status As Sogou`s C++ server engine, Sogou C++ Workflow supports almost all back-end C++ online services of Sogou, including all search services, cloud input method,online advertisements, etc., handling more than 10 billion requests every day. This is an enterprise-level programming engine in light and elegant design which can satisfy most C++ back-end development requirements. You can use it: * To quickly build an HTTP server: #include #include "workflow/WFHttpServer.h" int main() { WFHttpServer server([](WFHttpTask *task) { task->get_resp()->append_output_body("Hello World!"); }); if (server.start(8888) == 0) { // start server on port 8888 getchar(); // press "Enter" to end. server.stop(); } return 0; } * As a multifunctional asynchronous client, it currently supports HTTP, Redis, MySQL and Kafka protocols. * To implement client/server on user-defined protocol and build your own RPC system. + srpc is based on it and it is an independent open source project, which supports srpc, brpc, trpc and thrift protocols. * To build asynchronous workflow; support common series and parallel structures, and also support any DAG structures. * As a parallel computing tool. In addition to networking tasks, Sogou C++ Workflow also includes the scheduling of computing tasks. All types of tasks can be put into the same flow. * As a asynchronous file IO tool in Linux system, with high performance exceeding any system call. Disk file IO is also a task. * To realize any high-performance and high-concurrency back-end service with a very complex relationship between computing and networking. * To build a micro service system. + This project has built-in service governance and load balancing features. Compiling and running environment * This project supports Linux, macOS, Windows, Android and other operating systems. + Windows version is currently released as an independent branch, using iocp to implement asynchronous networking. All user interfaces are consistent with the Linux version. * Supports all CPU platforms, including 32 or 64-bit x86 processors, big-endian or little-endian arm processors. * Relies on OpenSSL; OpenSSL 1.1 and above is recommended. If you don't like SSL, you may checkout the nossl branch. But still need to link crypto for md5 and sha1. * Uses the C++11 standard and therefore, it should be compiled with a compiler which supports C++11. Does not rely on boost or asio. * No other dependencies. However, if you need Kafka protocol, some compression libraries should be installed, including lz4, zstd and snappy. Get started (Linux, macOS): $ git clone https://github.com/sogou/workflow $ make $ cd tutorial $ make Tutorials * Client + Creating your first task:wget + Implementing Redis set and get:redis_cli + More features about series:wget_to_redis * Server + First server:http_echo_server + Asynchronous server:http_proxy * Parallel tasks and Series + A simple parallel wget:parallel_wget * Important topics + About error + About timeout + About global configuration + About DNS + About exit * Computing tasks + Using the build-in algorithm factory:sort_task + User-defined computing task:matrix_multiply + Use computing task in a simple way: go task * Asynchronous File IO tasks + Http server with file IO:http_file_server * User-defined protocol + A simple user-defined portocol: client/server * Timing tasks and counting tasks + About timer + About counter * Service governance + About service governance + More documents about upstream * Connection context + About connection context * Built-in protocols + Asynchronous MySQL client:mysql_cli + Asynchronous Kafka client: kafka_cli System design features We believe that a typical back-end program= protocol+algorithm+workflow and should be developed completely independently. * Protocol + In most cases, users use built-in common network protocols, such as HTTP, Redis or various rpc. + Users can also easily customize user-defined network protocol. In the customization, they only need to provide serialization and deserialization functions to define their own client/server. * Algorithm + In our design, the algorithm is a concept symmetrical to the protocol. o If protocol call is rpc, then algorithm call is an apc (Async Procedure Call). + We have provided some general algorithms, such as sort, merge, psort, reduce, which can be used directly. + Compared with a user-defined protocol, a user-defined algorithm is much more common. Any complicated computation with clear boundaries should be packaged into an algorithm. * Workflow + Workflow is the actual bussiness logic, which is to put the protocols and algorithms into the flow graph for use. + The typical workflow is a closed series-parallel graph. Complex business logic may be a non-closed DAG. + The workflow graph can be constructed directly or dynamically generated based on the results of each step. All tasks are executed asynchronously. Basic task, task factory and complex task * Our system contains six basic tasks: networking, file IO, CPU, GPU, timer, and counter. * All tasks are generated by the task factory and automatically recycled after callback. + Server task is one kind of special networking task, generated by the framework which calls the task factory, and handed over to the user through the process function. * In most cases, the task generated by the user through the task factory is a complex task, which is transparent to the user. + For example, an HTTP request may include many asynchronous processes (DNS, redirection), but for user, it is just a networking task. + File sorting seems to be an algorithm, but it actually includes many complex interaction processes between file IO and CPU computation. + If you think of business logic as building circuits with well-designed electronic components, then each electronic component may be a complex circuit. Asynchrony and encapsulation based on C++11 std::function * Not based on user mode coroutines. Users need to know that they are writing asynchronous programs. * All calls are executed asynchronously, and there are almost no operation that occupys a thread. + Although we also provide some facilities with semi-synchronous interfaces, they are not core features. * We try to avoid user's derivations, and encapsulate user behavior with std::function instead, including: + The callback of any task. + Any server's process. This conforms to the FaaS (Function as a Service) idea. + The realization of an algorithm is simply a std::function. But the algorithm can also be implemented by derivation. Memory reclamation mechanism * Every task will be automatically reclaimed after the callback. If a task is created but a user does not want to run it, the user needs to release it through the dismiss method. * Any data in the task, such as the response of the network request, will also be recycled with the task. At this time, the user can use std::move() to move the required data. * SeriesWork and ParallelWork are two kinds of framework objects, which are also recycled after their callback. + When a series is a branch of a parallel, it will be recycled after the callback of the parallel that it belongs to. * This project doesn't use std::shared_ptr to manage memory. More design documents To be continued... About C++ Parallel Computing and Asynchronous Networking Engine Topics mysql redis http kafka tasking asychronous io-cpu Resources Readme License Apache-2.0 License Releases 6 workflow v0.9.5 Latest Apr 12, 2021 + 5 releases Packages 0 No packages published Contributors 13 * @Barenboim * @holmes1412 * @sonullx * @kedixa * @alpc62 * @qicosmos * @wzl12356 * @westfly * @dengjunplusplus * @MaybeShewill-CV * @shopee-jin + 2 contributors Languages * C++ 80.0% * C 18.3% * CMake 1.3% * Makefile 0.4% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.