Across multiple runs of a regular asyncio event loop, I would get as high as 3s for the same 4000 requests with uvloop, it never broke 2.1s.įor a drop in replacement it seems pretty great - I don’t think it’ll help much at this stage, though, because most of the timing is due to the actual network call at this point. It doesn’t seem to have impacted the performance all that much - it did have lower variance, though.
Python http client how to#
If you know how to set up HTTPX or P圜url in a way that’s faster let me know! UVLoopĪddendum: 8/27/21 - I received an email from Steve telling me about uvloop, a faster, drop in replacement for asyncio’s event loop.
This was my first time writing a pycurl implementation, though, based on this template - introducing native threading might be able to speed it up, but I still haven’t seen anything faster than the 393 microseconds approach. The results were impressive, but the aiohttp library was still faster.
Python http client code#
Writing the code felt more like dispatching actions and opening sockets than dealing with a nice http library. P圜url is different in that it feels like a pretty raw wrapper to curl. Someone on that same lobste.er’s thread suggested pycurl. I also tried a native gather, with punting the concurrency down to the library - this did not help either. I used their async library with the same sempahore restricting the number of processes ran, but it was still slower. Unfortunately, in my testing, it was strictly slower than aiohttp. HTTPX is a modern implementation of a python web client. Know of a faster implementation of an HTTP library that is stateful and works locally? Let me know! HTTPXĪ poster on lobste.rs said that I should try out httpx.
I mostly do this locally at home, though, for my side projects - introducing parallelization and multiple hosts can get you numbers that are an order of magnituded better than this, but the purpose of this excercise is seeing what we can hit on a local machine with jupyter notebook. Interestingly enough the optimal semaphore value was right around 60. The optimal number will depend on your host - beefier set ups will have higher concurrency limits, and if you’re running this on a remote host on something like digital ocean you can crank this number up quite a bit. This is nearly a 3x slow down due to resource contention issues locally. If we bump our concurrent requests to 4k we see a drastic loss in performance. Urls = [ f " results" ) Optimal semaphore size? join ()Īnd the actual query code is fairly straightforward - just define a function that’ll populate a global variable using some unique ID, and have it make the request off in its own thread. add_task ( func, args ) def wait_completion ( self ): """ Wait for completion of all the tasks in the queue """ self. put (( func, args, kargs )) def map ( self, func, args_list ): """ Add a list of tasks to the queue """ for args in args_list : self. tasks ) def add_task ( self, func, * args, ** kargs ): """ Add a task to the queue """ self. tasks = Queue ( num_threads ) for _ in range ( num_threads ): Worker ( self.
task_done () class ThreadPool : """ Pool of threads consuming tasks from a queue """ def _init_ ( self, num_threads ): self. Print ( e ) finally : # Mark this task as done, whether an exception happened or not get () try : func ( * args, ** kargs ) except Exception as e : # An exception happened in this thread start () def run ( self ): while True : func, args, kargs = self. From queue import Queue from threading import Thread class Worker ( Thread ): """ Thread executing tasks from a given tasks queue """ def _init_ ( self, tasks ): Thread.
0 kommentar(er)
