![]() hangfire is cool and all, but it is extremely limited and suffers a lot of performance issues when working with large ammounts of jobs.modilar solution that would allow me to use hangfire, quartz or your scheduling solution internally, this way I would be able to adopt your solution in steps on my own software, or maybe even reuse my current custom Hangfire solution internally that contains distributed tracing, etc.headless solution that will allow me to build or manage my pipelines from a UI, but also allow me to directly access your solution through c# code if I want to build or manage my own pipelines programatically or using my own custom front end solution.When I think about it feature wise, I would love to see the following: It could even use Hangfire or Quartz internally. Would love to see a solution for that built with dotnet. I worked as a data engineer for many many years and I feel like HangFire and Quartz solve issues that are way more generic than Airflow, Prefect, and many other ETL or pipeline tools. Built-in notification system for email, Slack, Microsoft Teams, and webhooks.SQL Server/Azure SQL Database persistent storage.SignalR and Pinia for real-time stat updates in the dashboard UI.Ability to visualize TPL pipelines and TPL networks as diagrams/graphs.DevExtreme components for powerful datagrids and filtering.Include retries, zombie task killers, etc.Workflows built from the Task Parallel Library.NET Standard backend (I'm thinking Web API). Focus on the power of the criminally-underrated Task Parallel Library. Make it friendly for on-premises servers and cloud/scale. Purpose: provide a fully-featured, modern, managed orchestration engine for data pipelines and task execution. NET, and I thought it would be cool if we had a comprehensive data pipeline orchestration engine. I've fallen in love with Data Engineering and. Fast forward 'til now, and I still don't think we have anything close to these frameworks. NET ecosystem needs its own equivalent of Apache Airflow or Prefect. A few years ago, I opened a GitHub issue with Microsoft telling them that I think the. ![]()
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |