[OpenSSL, Botan]: Remove JOM, it hangs.

[jaykrell]

We have been seeing many hangs in building OpenSSL.
i.e instead of our vcpkg taking around 40 minutes overall, it times out after 8 hours.
Every case I have looked at is in the middle of building OpenSSL with jom.
(We do not build Botan.)
While the investigation is yet quite shallow (haven't reproed locally and debugged) I strongly suspect a problem in jom.
Ongoing testing in our local cloud CI is occuring with:
set VCPKG_MAX_CONCURRENCY=1
So far 1-2 runs have succeeded but I'll run 10 or so.

• Changes comply with the maintainer guide.
• SHA512s are updated for each updated download.
• The "supports" clause reflects platforms that may be fixed by this new version, or no changes were necessary.
• Any fixed CI baseline and CI feature baseline entries are removed from that file, or no entries needed to be changed.
• All patch files in the port are applied and succeed.
• The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
• Exactly one version is added in each modified versions file.

[dg0yt]

Jumping straight into a PR...

I would like to see the actual versions of vcpkg, openssl, jom.
For which versions did the problem occur first?

It doesn't occur in vcpkg CI, and I don't think I see related issues.
Maybe there should be a triplet variable to allow disabling jom in customization instead of forcing long build times for everyone.

[jaykrell]

I've done some testing overnight, a bunch of builds:
set VCPKG_MAX_CONCURRENCY=1
and the OpenSSL hangs went away. This disables jom entirely, not just runs it single threaded.
The overall build time ended up longer and builds timed out, but they got past Vcpkg (I'm not just running vcpkg here, but our much larger overall build). I also disabled vcpkg nuget caching to ensure testing. So, the slowdown normally wouldn't be often, and this was overkill, spread across all vcpkg, not just OpenSSL. We don't build Botan, that is just for completeness.

Of course this is overkill, but it is what I had available.

We are on very current of everything.
We have internal repackaging of Vcpkg. We are on: 2026.5.18.257-2b65c20fc6
Where the date is ours, and the git hash is from here, 6 weeks ago (right before our 5/18 label).

C:\s\vcpkg>git log 2b65c20
commit 2b65c20
Author: Billy O'Neal bion@microsoft.com
Date: Sat May 16 04:27:06 2026 -0700

[vtk-m] Avoid stdext::checked_array_iterator. (#51815)

Up until around April this year, we hadn't updated Vcpkg for about 2 years.
Due to other problems.
That update is when we started seeing this.

Our system also presently runs vcpkg twice concurently.
With an older vcpkg also, so we get the old zlib1.dll and not just z.dll.
I believe they all have separate inputs and outputs, so this should not matter.
This second vcpkg should go away soon. I was just afraid there might be dependency
on the older zlib1.dll.

I hope to still locally repro and debug jom, but have not yet.
It is unfortunate they release binaries without symbols, so I have to build Qt.

I can try newer Vcpkg but I doubt it will make a difference.
And to do that I need to workaround the cpprestsdk removal.

[jaykrell]

Maybe there should be a triplet variable to allow disabling jom in customization

We do have our own approximatelely 100 line triplet, so this would be adequate and welcome, for us.
But surely this is being seen by others. I realize, we should debug and fix jom.

[Osyotr]

I'd check defender exclusions first.

[jaykrell]

It does work sometimes, and when it does, in the usual 20-40 minutes. i.e. not slowed down hugely.
When it hangs, we have a dump. It is still running. But there's no symbols and I cannot see much.
We have few a processed dump -- presumably the right process trees. We have vcpkg and jom, and no compiler or linker. So I assume compile and link finished, and jom is just still waiting.
I still hope to reproduce this locally.
We run quite a variety of build tools overall. I kinda doubt we are managing which processes are allowed. And jom is run basically only once, so launching it is a rare and successful event.

[jaykrell]

C:\s>jom /version
jom version 1.1.7

[jaykrell]

Maybe it requires higher core count? I think our builds machines are 52 core, hyperthreaded to 102.
At least I got that impression years ago. Before they moved to VMs, perhaps.
On Windows, 64 is a special number, the most objects you can wait for at a time. It might be common to have bugs on systems with more than 64 CPUs?
In any case, this is a very severe problem for us. The rate of hanging in vcpkg/openssl/jom is very high. I retry, and that has often worked, but it is very expensive, and failure rate is still over 50% I think -- always the hang, no other failures.

[jaykrell]

it looks like in triplet:

if (PORT STREQUAL "openssl")
  set(VCPKG_CONCURRENCY 1)
endif()

will work.

[BillyONeal]

These are heavily used ports and they are not one of the known flaky ones so it would be better if we have some assurance that this actually fixes a real problem before proceeding.

[jaykrell]

Yes, it is a huge help. I have the triplet change and I expect it will go forward.
Otherewise we have a huge percentage of builds hang and eventually timeout.
Caching covers it up, but a huge percentage of uncached builds hang.
Switching openssl to nmake solves it.
I realize jom should be debugged and fixed.
The repro so far is only in Microsoft CloudBuild, which is very heavily used, and probably very non-debuggable.

[jaykrell]

Clarification/repitition: We do not build Botan. We do build OpenSSL. I recommend removing jom from vcpkg entirely. Maybe even leave the ports alone but make PREFE_JOM do nothing, remove the support? Until/unless debugged?

[MichalPetryka]

We've never seen OpenSSL hang for us, even with JOM.

[jaykrell]

We see it I think more than 50% of runs. It is a big problem.
Though we can disable it in the triplet. I have that PR open in our system.
I don't expect we'll be able to debug it. I hope to run it in a loop locally (after spending days/hours building Qt so I can build jom), but I cannot hope to match the CI configuration is many ways.

[dg0yt]

Again, my proposal is to add a specific triplet variable such as VCPKG_NMAKE_DISABLE_JOM so that it can be used in custom triplets incl. per-port customization.