Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support creating .torrent with larger piece size #19535

Merged
merged 5 commits into from
Sep 3, 2023

Conversation

Chocobo1
Copy link
Member

@Chocobo1 Chocobo1 commented Aug 31, 2023

  • Store user data in combo box
  • Support creating .torrent with larger piece size
    Warning: users are at their own discretion to create .torrent with >= 64 MiB piece size as not every torrent client supports it.
    Larger piece sizes are only available when using libtorrent 2.x. libtorrent 1.x is not efficient with memory usage and in order to avoid user complaints it is limited to 128 MiB.
    Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now.
    Closes Support Creation of torrents with 64 MB piece size #19527.
  • Don't keep dialog in disabled state if add torrent failed
  • Control source field enabled state
  • Revise error message

@Chocobo1 Chocobo1 added the GUI GUI-related issues/changes label Aug 31, 2023
@Chocobo1 Chocobo1 added this to the 5.0 milestone Aug 31, 2023
@Chocobo1 Chocobo1 changed the title Piece size Support creating .torrent with 64 MiB piece size Aug 31, 2023
@Pentaphon
Copy link

Pentaphon commented Aug 31, 2023

  • Warning: users are at their own discretion to create .torrent with 64 MiB piece size as not every torrent client supports it.

It seems the latest Transmission, Deluge, BiglyBT, and uTorrent support 64MB along with qBitorrent so it's really just about older versions not supporting it. It's up to leechers to keep their clients up to date if they want to continue participating in swarms.

@Chocobo1
Copy link
Member Author

Chocobo1 commented Sep 1, 2023

PR updated, I've added an experimental commit that would allow creating .torrent files up to 1 GiB piece size. This is the max limit of libtorrent. I just thought that why don't we provide every options at once and let users take the responsibility (about the creation result).

@Pentaphon
Copy link

PR updated, I've added an experimental commit that would allow creating .torrent files up to 1 GiB piece size. This is the max limit of libtorrent. I just thought that why don't we provide every options at once and let users take the responsibility (about the creation result).

Totally support this and I will ping other devs to support this.

@Chocobo1 Chocobo1 changed the title Support creating .torrent with 64 MiB piece size Support creating .torrent up to 1 GiB piece size Sep 1, 2023
@Chocobo1 Chocobo1 requested a review from a team September 1, 2023 08:16
@stalkerok
Copy link
Contributor

stalkerok commented Sep 1, 2023

lt 1.2: When creating a torrent with a piece size of 256 MB or more, a memory leak occurs. The client crashed several times. In cases where the client did not crash, the log writes Torrent errored. Torrent: "test". Error: "invalid piece size". The torrent was created with a piece size of 512 MB. 256 MB creates normally.
lt 2.0: There is no leakage. But it also does not create above 256 MB. At 512 MB, the same error is Torrent errored. Torrent: "test". Error: "invalid piece size" or
2023-09-01_112122
when creating 1 GB. Torrent format v1.

@Chocobo1
Copy link
Member Author

Chocobo1 commented Sep 1, 2023

lt 1.2: When creating a torrent with a piece size of 256 MB or more, a memory leak occurs. The client crashed several times.

I didn't observe it on my machine. Wild guess, would it be possible that it exhausted all your system memory?

In cases where the client did not crash, the log writes Torrent errored. Torrent: "test". Error: "invalid piece size". The torrent was created with a piece size of 512 MB. 256 MB creates normally.

I checked the created .torrent file and it seems to be valid. It is the torrent client (or specifically, the code responsible for loading the .torrent file) that is faulty. And this can be resolved later (probably with some help from libtorrent, not sure).
IMO unless other testers are able to reproduce the memory leak/crash, I would still wish to keep the options up to 1 GiB.

Also, the error message is slightly improved.

@stalkerok
Copy link
Contributor

The client crashes after a leak, 16 GB RAM + 10 GB pagefile. There is no memory leakage on 128 MB, memory is used at a minimum.

@Chocobo1
Copy link
Member Author

Chocobo1 commented Sep 1, 2023

The client crashes after a leak, 16 GB RAM + 10 GB pagefile.

Just curious, how do you know it is a leak and not other problems?

@Pentaphon
Copy link

There is no memory leakage on 128 MB, memory is used at a minimum.

Perhaps applications like dottorrent only creates torrents up to 64MB for that very reason?

@stalkerok
Copy link
Contributor

Just curious, how do you know it is a leak and not other problems?

Actually, I do not know this, but it is very similar to a memory leak.
If it helps

Имя сбойного приложения: qbittorrent.exe, версия: 5.0.0.0, метка времени: 0x64f17d1a
Имя сбойного модуля: ucrtbase.dll, версия: 10.0.19041.789, метка времени: 0x2bd748bf
Код исключения: 0xc0000409
Смещение ошибки: 0x000000000007286e
Идентификатор сбойного процесса: 0x834
Время запуска сбойного приложения: 0x01d9dcaaffd330d8
Путь сбойного приложения: C:\Users\stalkerok\Desktop\qbittorrent p\Support creating .torrent up to 1 GiB piece size # 4430\qbittorrent.exe
Путь сбойного модуля: C:\Windows\System32\ucrtbase.dll
Идентификатор отчета: 77a00d50-a98b-4c0a-b83d-94c49b0bdaba

Нехватка виртуальной памяти в системе успешно обнаружена. Наибольший объем виртуальной памяти был выделен следующим программам: для qbittorrent.exe (2100) выделено 19390054400 байт, для qbittorrent.exe (15988) выделено 859172864 байт и для chrome.exe (7732) выделено 235847680 байт.

qbittorrent.exe.2100.zip

@thalieht
Copy link
Contributor

thalieht commented Sep 1, 2023

CI libt 1.2 build on Windows. I created a torrent with 1G 512M and 256M pieces sizes. No crash for 512M and 1G. First time with 256 it crashed (qBt closed) but second time it didn't. But because i didn't get a stacktrace, the crash may have been caused because i had virtualbox running which causes some programs to crash for me sometimes.
So because i also got a success with 256M, i'd say no crashes for me.

@glassez

This comment was marked as off-topic.

@stalkerok
Copy link
Contributor

Yes, it was necessary to put in quotation marks from the very beginning. I do not know how to explain this and why it happens only with lt 1.2 and only from 256 MB+

@Chocobo1
Copy link
Member Author

Chocobo1 commented Sep 1, 2023

Torrent errored. Torrent: "test". Error: "invalid piece size".

I found an issue within libtorrent 2.x, I'll submit a fix. This issue prevents torrents with 1 GiB piece size from being added to transfer list. Smaller piece size should already be fine and libtorrent 1.2.x isn't affected.

Perhaps applications like dottorrent only creates torrents up to 64MB for that very reason?

Not sure but I find it unlikely.

https://user-images.githubusercontent.com/63958081/265005883-c29327d4-2ad6-4101-bc78-6a68af3a2e3e.png
https://user-images.githubusercontent.com/63958081/265005922-cb26edca-bf62-4a76-b559-a88b1a87c732.png
I don't have time to copy the stacktrace.

OK, I retest it with a larger data and now I see that libtorrent 1.x is not efficient with memory usage. It sucks away all the memory. I'm going to limit the max piece size to 128 MiB when using libtorrent 1.x. Only libtorrent 2.x will be able to use 1 GiB piece size.

@Chocobo1 Chocobo1 changed the title Support creating .torrent up to 1 GiB piece size Support creating .torrent with larger piece size Sep 1, 2023
Warning: users are at their own discretion to create .torrent with >= 64 MiB piece size as not
every torrent client supports it.

Larger piece sizes are only available when using libtorrent 2.x. libtorrent 1.x is not
efficient with memory usage and in order to avoid user complaints it is limited to 128 MiB.

Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only
allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden
for now.

Closes qbittorrent#19527.
@Chocobo1
Copy link
Member Author

Chocobo1 commented Sep 1, 2023

After more investigations, it seems that libtorrent 2.0.9/1.2.19 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now.

@Pentaphon
Copy link

Pentaphon commented Sep 2, 2023

After more investigations, it seems that libtorrent 2.0.9/1.2.19 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now.

Probably for the best. This is already a huge step up from the 32MB piece size we mostly see in the wild. Even 64MB is still rare to see.

@Chocobo1 Chocobo1 merged commit 4d2015c into qbittorrent:master Sep 3, 2023
13 checks passed
@Chocobo1 Chocobo1 deleted the pieceSize branch September 3, 2023 06:31
rcarpa pushed a commit to rcarpa/qBittorrent that referenced this pull request Dec 1, 2023
Warning: users are at their own discretion to create .torrent with >= 64 MiB piece size as not
every torrent client supports it.
Larger piece sizes are only available when using libtorrent 2.x. libtorrent 1.x is not
efficient with memory usage and in order to avoid user complaints it is limited to 128 MiB.
Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only
allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden
for now.

Closes qbittorrent#19527.
PR qbittorrent#19535.
@stalkerok
Copy link
Contributor

@Chocobo1 I think in lt2.0 it also makes sense to limit the creation of torrents with a piece size of 256MB because of this issue #21011

@Chocobo1
Copy link
Member Author

Chocobo1 commented Sep 3, 2024

@Chocobo1 I think in lt2.0 it also makes sense to limit the creation of torrents with a piece size of 256MB because of this issue #21011

I don't mind. You can submit a patch.
But that won't stop other torrent creator apps from creating those large piece size torrents...

@xavier2k6
Copy link
Member

But that won't stop other torrent creator apps from creating those large piece size torrents...

Exactly, if you look at the torrents that were supplied with 256.0 MiB piece sizes - they were created with a program called py3createtorrent v1.1.0

What needs to be done is a fix for libtorrent 1.2.x branch or reject torrents of this type from being loaded via libtorrent 1.2.x based builds.

@stalkerok
Copy link
Contributor

But that won't stop other torrent creator apps from creating those large piece size torrents...

Yeah, but at least something...

@xavier2k6
Copy link
Member

Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now.

@Chocobo1 Do you mind pointing me in the direction of where this limitation is in libtorrent code & is there something similar for libtorrent 1.2.19?

@Pentaphon
Copy link

Can we compromise and just make the maximum piece size 128MB? That seems like enough considering almost all torrents in the wild are 32MB or below.

@Chocobo1
Copy link
Member Author

Chocobo1 commented Sep 9, 2024

@Chocobo1 Do you mind pointing me in the direction of where this limitation is in libtorrent code & is there something similar for libtorrent 1.2.19?

See: arvidn/libtorrent#7512

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GUI GUI-related issues/changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Creation of torrents with 64 MB piece size
7 participants