FastAPI with async Celery tasks
Like all developers with too many side projects and an immense imposter syndrome, I was working on the next great feature for my side-project. Import via CSV!
First I implemented a working solution with FastAPI's BackoundTasks feature. But this didn't feel right. In the documentation Celery recommended, so I go with that. No problem. I've already worked with Celery, so that should not be a big problem!
It was a big problem
While BackgroundTasks are for running tasks in the background after the request-response cycle, it falls short in scenarios requiring long-running or resource-intensive operations. You know, such as processing large CSV files for imports.
This is out of the way, what exactly is the problem with Celery? Async! Because of the nature of pecuny, I'm using async database sessions. Which was no problem with BackgrounTasks, but Celery is not supporting async tasks yet. What a bummer. Which means I need to find a way that suits my needs.
The Solution
Reading a ton of Celery tutorials (like 3!), which none covered async tasks. Even FastAPI + Celery = ♥ which promised me it would equal love, but didn't deliver 😔.
Luckily I was not the only one facing this issue. There are a couple of working solutions, but only one of them I found perfect for me. AsyncCelery! A big thanks to Cyril for providing this class.
Why do I love it? Because it solves my problem, without the need to load in another library or do some weird shenanigans. It is working, clean and simple enough that I understand what is going on.
Minimal Example
I know that it would be a lot of struggle to go through pecuny's codebase to find out how exactly I implemented it, so I've created a dedicated repository for this. Because I love you!
Conclusion
In the world of software development, finding the right tool for the job can often be as challenging as squaring the circle. My journey with pecuny's import feature epitomizes this, starting with FastAPI's BackgroundTasks and transitioning to Celery in hopes of a more robust solution. Yet, the asynchronous nature of Pecuny's database sessions threw a wrench in the works, underscoring Celery's current limitations with async tasks.
This experience reminds us that no tool is a one-size-fits-all solution, prompting a continuous search for alternatives that align with our unique project needs.