Hitting rate limits with Modrinth API

[douglasparker]

Describe the problem

I seem to be running into rate limits when starting a server and downloading all of the mods.

The container should pause and retry after hitting rate limits.

Container definition

No response

Container logs

[mc-image-helper] 23:25:43.845 ERROR : 'modrinth' command failed. Version is 1.29.0
me.itzg.helpers.http.FailedRequestException: HTTP request of https://api.modrinth.com/v2/project/wavey-capes failed with 429 Too Many Requests: Fetching object content
        at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:186)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
        at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:122)
        at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:128)
        at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
        at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
        at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:193)
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
        at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2071)
        at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:118)
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
        at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413)
        at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:431)
        at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:485)
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:712)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Unknown Source)
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
                at reactor.core.publisher.Mono.block(Mono.java:1710)
                at me.itzg.helpers.modrinth.ModrinthCommand.getProject(ModrinthCommand.java:230)
                at me.itzg.helpers.modrinth.ModrinthCommand.processProject(ModrinthCommand.java:262)
                at java.base/java.util.stream.ReferencePipeline$7$1.accept(Unknown Source)
                at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
                at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
                at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
                at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
                at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
                at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
                at me.itzg.helpers.modrinth.ModrinthCommand.processProjects(ModrinthCommand.java:100)
                at me.itzg.helpers.modrinth.ModrinthCommand.call(ModrinthCommand.java:83)
                at me.itzg.helpers.modrinth.ModrinthCommand.call(ModrinthCommand.java:40)
                at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
                at picocli.CommandLine.access$1500(CommandLine.java:148)
                at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
                at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
                at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
                at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
                at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
                at picocli.CommandLine.execute(CommandLine.java:2170)
                at me.itzg.helpers.McImageHelper.main(McImageHelper.java:139)

[itzg]

@triphora , how would like my mc-image-helper tooling to handle to API rate limits? Retry with delay, as requested above, or some other way?

[triphora]

How exactly are you forming your requests? It shouldn't exactly be possible to get anywhere close to hitting the ratelimit if you use the bulk get routes appropriately.

[itzg]

Good question 😀 ...I'll work on getting in more use of https://docs.modrinth.com/api-spec/#tag/projects/operation/getProjects

I'm glad I asked you 😁

[itzg]

I'll initially look into delaying 429's by the duration reported by X-Ratelimit-Reset header. Reference

Longer term can switch to bulk project retrieval when this is resolved

modrinth/labrinth#185

[github-actions]

This issue is stale because it has been open 30 days with no activity. Please add a comment describing the reason to keep this issue open.

[triphora]

modrinth/labrinth#185 (comment)

[itzg]

Thanks @triphora! I'll adapt the code to use that API.